Compare commits

..

32 Commits

Author SHA1 Message Date
overtrue 23bffc4ca0 fix(obs): trim only what the log byte budget needs
Lifting the minimum file age for an over-budget pass merged fresh files into the regular selection, so the keep_files ceiling and the per-file limit applied to them too: one byte over the budget could process every fresh file but keep_files of them. The override is now a byte trim on the files that would remain, oldest first, until the budget holds.

An unset single-file limit now rolls at a quarter of a small byte budget instead of 256 MiB, so a freshly rolled file is not larger than the whole budget. Both logging setups open the appender through one function that returns the roll signal the cleanup task listens on. The README states that a log directory must not be shared by processes whose file names match the same pattern.
2026-11-11 16:25:59 +08:00
overtrue f2655911db fix(obs): bound the log directory with default settings
With the defaults the active log file had no size limit (size rolling was off), and the cleaner counted only rolled files older than the minimum file age against the byte budget, on a 30-minute interval. A log storm could therefore fill the disk for up to two hours without the budget applying.

The active file now rolls at 256 MiB when the single-file limit is unset; an explicit 0 still turns size rolling off, and the cleaner's per-file trim keeps its previous default. The appender wakes the cleaner after every roll, and once the rolled files exceed the byte budget the cleaner processes the oldest ones regardless of age. The README no longer calls the budget a hard cap on the directory and states the actual bound.
2026-11-11 15:30:11 +08:00
Chris An 0e402b2534 fix(scanner): retain child objects beneath object prefixes (#9277) 2026-11-11 14:36:53 +08:00
Chris An db39cd5634 fix(ecstore): let missing-key deletes complete on UUID-named folders (#9280)
* fix(ecstore): let missing-key deletes complete on UUID-named folders

Deleting a key that does not exist probes the key's directory for
interrupted delete transactions. The probe was bounded at 4096 entries
per drive and charged every entry inside a UUID-named child, so a key
naming a folder of roughly 1,400 or more UUID-named child keys answered
503 SlowDown on every retry.

Inside a UUID-named child only journal files and data-directory markers
can be evidence, so every other name (a child key's own xl.meta, data
directories or children) is now skipped unread at 1/64 like plain
children of the folder. The per-call allowance of the bounded protocol
rises to 262,144 entries per drive; the resumable background step keeps
4096 and the request budget keeps its sizing, so the per-request total
is unchanged. The probe runs on the durable-mutation lane instead of
the one-worker foreground lane, and stops incomplete when its caller
goes away or after a third of the drive timeout (at most 10 s), so a
slow walk answers SlowDown instead of timing out the drive.

* test(ecstore): keep discovery fixtures meaningful under 1/64 charging

Part files inside a data directory are inert to object-scoped discovery
now, so reclaim_orphan_data_dirs_drains_empty_discovery_pages fills its
first step with the referenced data directories of many versions, which
still cost a full entry each. The charging test drives the walk directly
instead of through the wall-clock-limited disk entry point, so a loaded
machine cannot turn it into an incomplete probe.
2026-11-10 23:24:50 -05:00
Ramakrishna Chilaka 89d01aef4f perf(ecstore): reduce syscalls and directory re-opens in PUT and DELETE (#9264)
Co-authored-by: RustFS <hello@rustfs.com>
Co-authored-by: Hauser Zhi <housemecn@gmail.com>
2026-11-11 00:06:11 +00:00
dependabot[bot] 86c3395d07 chore(deps): bump the dependencies group across 1 directory with 12 updates (#9251)
Bumps the dependencies group with 12 updates in the / directory:

| Package | From | To |
| --- | --- | --- |
| [async-nats](https://github.com/nats-io/nats.rs) | `0.51.0` | `0.52.0` |
| [tar-codec](https://github.com/astral-sh/astral-codec) | `0.0.15` | `0.0.16` |
| [tar-framing](https://github.com/astral-sh/astral-codec) | `0.0.15` | `0.0.16` |
| [flate2](https://github.com/rust-lang/flate2-rs) | `1.1.10` | `1.1.11` |
| [smallvec](https://github.com/servo/rust-smallvec) | `1.16.4` | `1.16.5` |
| [starshard](https://github.com/houseme/starshard) | `2.4.0` | `2.4.1` |
| [zip](https://github.com/zip-rs/zip2) | `9.3.0` | `9.4.1` |
| [dial9](https://github.com/dial9-rs/dial9) | `0.5.7` | `0.5.8` |
| [rustfs-mimalloc](https://github.com/houseme/rustfs-mimalloc) | `0.7.3` | `0.7.4` |
| [hotpath](https://github.com/hotpath-rs/hotpath) | `0.30.3` | `0.31.1` |
| [rustfs-mimalloc-sys](https://github.com/houseme/rustfs-mimalloc) | `0.7.3` | `0.7.4` |
| [toml](https://github.com/toml-rs/toml) | `1.2.0+spec-1.1.0` | `1.2.1+spec-1.1.0` |



Updates `async-nats` from 0.51.0 to 0.52.0
- [Release notes](https://github.com/nats-io/nats.rs/releases)
- [Commits](https://github.com/nats-io/nats.rs/compare/async-nats/v0.51.0...async-nats/v0.52.0)

Updates `tar-codec` from 0.0.15 to 0.0.16
- [Release notes](https://github.com/astral-sh/astral-codec/releases)
- [Commits](https://github.com/astral-sh/astral-codec/compare/v0.0.15...v0.0.16)

Updates `tar-framing` from 0.0.15 to 0.0.16
- [Release notes](https://github.com/astral-sh/astral-codec/releases)
- [Commits](https://github.com/astral-sh/astral-codec/compare/v0.0.15...v0.0.16)

Updates `flate2` from 1.1.10 to 1.1.11
- [Release notes](https://github.com/rust-lang/flate2-rs/releases)
- [Changelog](https://github.com/rust-lang/flate2-rs/blob/main/CHANGELOG.md)
- [Commits](https://github.com/rust-lang/flate2-rs/compare/1.1.10...v1.1.11)

Updates `smallvec` from 1.16.4 to 1.16.5
- [Release notes](https://github.com/servo/rust-smallvec/releases)
- [Commits](https://github.com/servo/rust-smallvec/compare/v1.16.4...v1.16.5)

Updates `starshard` from 2.4.0 to 2.4.1
- [Changelog](https://github.com/houseme/starshard/blob/main/CHANGELOG.md)
- [Commits](https://github.com/houseme/starshard/compare/2.4.0...2.4.1)

Updates `zip` from 9.3.0 to 9.4.1
- [Release notes](https://github.com/zip-rs/zip2/releases)
- [Changelog](https://github.com/zip-rs/zip2/blob/master/CHANGELOG.md)
- [Commits](https://github.com/zip-rs/zip2/compare/v9.3.0...v9.4.1)

Updates `dial9` from 0.5.7 to 0.5.8
- [Release notes](https://github.com/dial9-rs/dial9/releases)
- [Changelog](https://github.com/dial9-rs/dial9/blob/main/CHANGELOG.md)
- [Commits](https://github.com/dial9-rs/dial9/compare/dial9-v0.5.7...dial9-v0.5.8)

Updates `rustfs-mimalloc` from 0.7.3 to 0.7.4
- [Release notes](https://github.com/houseme/rustfs-mimalloc/releases)
- [Changelog](https://github.com/houseme/rustfs-mimalloc/blob/main/CHANGELOG.md)
- [Commits](https://github.com/houseme/rustfs-mimalloc/compare/v0.7.3...v0.7.4)

Updates `hotpath` from 0.30.3 to 0.31.1
- [Release notes](https://github.com/hotpath-rs/hotpath/releases)
- [Changelog](https://github.com/hotpath-rs/hotpath/blob/main/CHANGELOG.md)
- [Commits](https://github.com/hotpath-rs/hotpath/compare/v0.30.3...v0.31.1)

Updates `rustfs-mimalloc-sys` from 0.7.3 to 0.7.4
- [Release notes](https://github.com/houseme/rustfs-mimalloc/releases)
- [Changelog](https://github.com/houseme/rustfs-mimalloc/blob/main/CHANGELOG.md)
- [Commits](https://github.com/houseme/rustfs-mimalloc/compare/v0.7.3...v0.7.4)

Updates `toml` from 1.2.0+spec-1.1.0 to 1.2.1+spec-1.1.0
- [Commits](https://github.com/toml-rs/toml/compare/toml-v1.2.0...toml-v1.2.1)

---
updated-dependencies:
- dependency-name: async-nats
  dependency-version: 0.52.0
  dependency-type: direct:production
  update-type: version-update:semver-minor
  dependency-group: dependencies
- dependency-name: tar-codec
  dependency-version: 0.0.16
  dependency-type: direct:production
  update-type: version-update:semver-patch
  dependency-group: dependencies
- dependency-name: tar-framing
  dependency-version: 0.0.16
  dependency-type: direct:production
  update-type: version-update:semver-patch
  dependency-group: dependencies
- dependency-name: flate2
  dependency-version: 1.1.11
  dependency-type: direct:production
  update-type: version-update:semver-patch
  dependency-group: dependencies
- dependency-name: smallvec
  dependency-version: 1.16.5
  dependency-type: direct:production
  update-type: version-update:semver-patch
  dependency-group: dependencies
- dependency-name: starshard
  dependency-version: 2.4.1
  dependency-type: direct:production
  update-type: version-update:semver-patch
  dependency-group: dependencies
- dependency-name: zip
  dependency-version: 9.4.1
  dependency-type: direct:production
  update-type: version-update:semver-minor
  dependency-group: dependencies
- dependency-name: dial9
  dependency-version: 0.5.8
  dependency-type: direct:production
  update-type: version-update:semver-patch
  dependency-group: dependencies
- dependency-name: rustfs-mimalloc
  dependency-version: 0.7.4
  dependency-type: direct:production
  update-type: version-update:semver-patch
  dependency-group: dependencies
- dependency-name: hotpath
  dependency-version: 0.31.1
  dependency-type: direct:production
  update-type: version-update:semver-minor
  dependency-group: dependencies
- dependency-name: rustfs-mimalloc-sys
  dependency-version: 0.7.4
  dependency-type: direct:production
  update-type: version-update:semver-patch
  dependency-group: dependencies
- dependency-name: toml
  dependency-version: 1.2.1+spec-1.1.0
  dependency-type: direct:production
  update-type: version-update:semver-patch
  dependency-group: dependencies
...

Signed-off-by: dependabot[bot] <support@github.com>
Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
Co-authored-by: Hauser Zhi <housemecn@gmail.com>
2026-11-11 00:04:47 +00:00
cxymds c9743f166e fix(s3): normalize leading-slash keys in bulk deletes (#9258)
* fix(s3): normalize leading-slash keys in bulk deletes

* test(s3): align bulk delete reserved-prefix guard expectations
2026-11-10 17:56:00 -05:00
Boyi (Elliot) Li 50707a610c test(utils,rio,checksums): add checksum SIMD equivalence and bench UTs (#9217)
Add unaligned-boundary SIMD vs scalar equivalence unit tests and on-demand (#[ignore]) utility-level throughput microbenchmarks across rustfs-utils, rustfs-rio, and rustfs-checksums:
- crates/utils/src/hash.rs: verify HighwayHasher SIMD (AVX2/SSE4.1/NEON) against highway::PortableHash scalar reference and contiguous hash_encode vs odd-chunked hash_encode_slices across unaligned tail boundaries; provide on-demand throughput microbenchmark.
- crates/rio/src/checksum.rs: verify SIMD-accelerated CRC32, CRC32C, and CRC64_NVME against bitwise scalar reference implementations across vector tail boundaries and measure utility-level throughput across all 10 BASE_CHECKSUM_TYPES.
- crates/checksums/src/lib.rs: verify streaming chunked update + finalize + headers() against one-shot digests across all 9 ChecksumAlgorithm variants for unaligned chunk and payload sizes, plus on-demand throughput microbenchmark.
2026-11-10 17:54:11 -05:00
Patrick Wicki ba3e0e1546 fix(admin): keep measured bucket usage when the overlay is unconverged (#9243)
The single-process usage overlay clears usage_snapshot_complete when it
holds counts it cannot reconcile with the persisted snapshot, for
example writes into a bucket that has no verified baseline. The admin
datausageinfo path then judged the overlaid document as structurally
incomplete and replaced it with an empty one, hiding every bucket the
scanner had measured. Because each write restarts that bucket's scanner
fence, a continuously written bucket kept the whole response empty
indefinitely.

Decide whether the persisted snapshot is usable before the overlay is
applied, and report overlay-induced incompleteness as a non-converged
snapshot instead, as is already done for buckets the scan never reached
(rustfs#5806).
2026-11-10 17:52:06 -05:00
Chris An 95e128336d ci: narrow checks for stable installation metadata (#9269)
* ci: narrow checks for stable installation metadata

* fix: reject release metadata marker collisions
2026-11-10 16:10:04 -05:00
Chris An 8b7bbdb17d chore(release): align installation references for 1.0.5 (#9268) 2026-11-10 14:48:02 -05:00
Chris An 2780bba67b ci: stage release builds and promote verified artifacts (#9265)
* ci: stage release builds and promote verified artifacts

* fix: preserve release package and channel retry contracts

* fix: align installation chart versions with sealed candidates
2026-11-10 13:15:05 -05:00
hector 3a23122104 ci: run macOS release builds on macos-latest-xlarge (#9260)
Use the larger GitHub-hosted macOS runner for the macos-aarch64
release target. The target id, triple, and other platforms stay the same.

Co-authored-by: hector <majinghe@users.noreply.github.com>
2026-11-10 18:55:19 +08:00
Ramakrishna Chilaka 0b1ebeffec perf(ecstore): reduce syscalls in multipart copy, commit, and delete (#9247) 2026-11-10 12:27:10 +08:00
Chris An 0f9258594c fix: preserve scanner backlog authority after pool replacement (#9252)
* fix(ecstore): allow bucket creation after failed decommission

* fix(scanner): preserve backlog membership across pool reindexing

* test(scanner): cover partial publication after pool reindexing
2026-11-09 20:46:20 -05:00
Chris An 33fc735ee2 chore(release): prepare 1.0.5 (#9249) 2026-11-09 17:19:35 -05:00
Chris An 509869cfc5 fix(tier): recover orphaned prepared mutations (#9246) 2026-11-09 22:18:14 +08:00
Ramakrishna Chilaka 395876ccd0 fix(ecstore): remove hardlinked backup during delete rollback (#9239)
When a fused recoverable delete is interrupted after publishing the transaction directory and before updating xl.meta, xl.meta and <transaction>/xl.meta.bkp remain hardlinks to the same inode. Under POSIX rename semantics, renaming xl.meta.bkp over xl.meta during rollback succeeds as a no-op without unlinking xl.meta.bkp, which leaves xl.meta.bkp in the transaction directory after journal removal and strands the transaction in DeleteRecoveryState::LegacyAmbiguous.

Unlink xl.meta.bkp after syncing the restored object directory and before syncing the rollback directory and removing journal markers, and extend delete_recovery_fused_prepare_checkpoint_reopens_before_head_publication to preserve the hardlink and verify rollback completion and retry.
2026-11-09 06:41:04 +08:00
Chris An 6083fbbeea fix(catalog): enumerate table owners without skipping namespaces (#9242) 2026-11-09 06:38:58 +08:00
Chris An 42d3267d88 fix(ecstore): decouple pool metadata health from the authority gate (#9241)
* fix(ecstore): share immutable pool metadata authority reads

* fix(ecstore): read pool metadata authority without the save gate

Health inferred pool metadata write authority from acquiring the local
authority gate within 100 ms. Sharing the gate among read-only
admissions still left held writers, and writers queued behind a reader
parked on the pool.bin fence, turning into cluster-write 503s.

Each writer now publishes the authority it leaves before releasing the
gate, and health reads that published state without acquiring it.
Transaction latches stay shared with the published copy; recovery
clears the latch and its failure under one lock so inspection never
observes a half-cleared state. The inspection-timeout path that can no
longer occur (check_timed_out, the check_timeout admin state, the
pool_metadata_check_timeout reason, the node timeout cache and its
metric) is removed.
2026-11-09 06:38:30 +08:00
Chris An 1d35e381b7 fix(rpc): bound replay memory and atomically admit signed requests (#9238)
* fix(rpc): index nonce expiry by signed validity

* fix(rpc): size replay cache from allocated storage layout

* fix(rpc): make replay admission atomic and release expired tables

* fix(rpc): preserve replay admission refusal at the S3 boundary
2026-11-09 06:36:55 +08:00
Chris An 0b109b4885 fix(select): bound LZ4 scratch and Arrow schema expansion (#9237)
* fix(select): admit legacy and linked LZ4 working memory

* fix(select): bound embedded Arrow schema expansion
2026-11-09 06:36:44 +08:00
cxymds 1dea00ffa7 ci: repair e2e selection and verify tool downloads via API (#9240)
* ci: align full e2e selection with retired catalog test

* ci: verify pinned tools through GitHub release asset API

* ci: fetch pinned typos through the release asset API

* ci: download pinned actionlint through the release asset API

* ci: ensure GitHub CLI exists before pinned mc download

* ci: budget full quick checks after tool installation
2026-11-08 14:09:26 +00:00
cxymds addf7a6219 fix: reduce scanner backoff and classify downstream closures (#9235) 2026-11-08 02:18:17 -05:00
Ramakrishna Chilaka 420cf32b2a perf(ecstore): optimize PUT, DELETE, and multipart commit hotpaths (#9227) 2026-11-08 02:12:40 -05:00
Chris An 907410b2f5 fix(select): bound Parquet allocations before decoding (#9231) 2026-11-08 01:36:12 -05:00
Chris An f10079f1a3 fix(rpc): size and reserve replay cache for compact entries (#9233) 2026-11-08 01:35:58 -05:00
Chris An a6c3016493 fix(cache): respect process cgroup memory limits (#9232) 2026-11-08 01:35:49 -05:00
Hauser Zhi 68fb52b752 chore(deps): update flake.lock (#9236) 2026-11-08 01:34:01 -05:00
cxymds 16f618d41b fix: correct scanner pacing and unverified MRF outcomes (#9225)
Co-authored-by: Chris An <anzhengchao@gmail.com>
2026-11-08 06:48:25 +08:00
Henry Guo 565d262ede refactor(table-catalog): retire durable-strong backing (#9130)
* refactor(table-catalog): retire durable-strong backing

* fix(table-catalog): avoid backup locks during S3 authorization

* fix(table-catalog): preserve directory and maintenance behavior

* fix(table-catalog): back up retained v1 snapshot history

* test(table-catalog): cover retained v1 backup recovery

Exercise backup inventory sealing and restore after a v1-to-v2 upgrade, including missing or corrupt references, nonzero inline snapshot sequences, v2 manifests in retained v1 history, and unchanged commit admission rules.

Clarify when direct maintenance can replace reports and discard retry budgets, and require quiescing scheduler admission and workers before invocation.

Co-Authored-By: heihutu <heihutu@gmail.com>
Co-Authored-By: zhi22915 <qiuzgang@gmail.com>

---------

Co-authored-by: Chris An <anzhengchao@gmail.com>
Co-authored-by: Hauser Zhi <housemecn@gmail.com>
Co-authored-by: heihutu <heihutu@gmail.com>
Co-authored-by: zhi22915 <qiuzgang@gmail.com>
2026-11-08 05:03:55 +08:00
Chris An 7b010ddd70 fix(ci,ecstore): restore validation and defer capacity exhaustion (#9223) 2026-11-07 15:17:25 -05:00
244 changed files with 31486 additions and 29903 deletions
+21 -26
View File
@@ -1,43 +1,38 @@
---
name: rustfs-release-publish
description: Publish a RustFS version through Console, release-branch CI, preview acceptance, human confirmation, and final publication. Use only for an explicit release request.
description: Publish a RustFS version through frozen Linux candidate acceptance, desktop completion, manifest approval, byte-preserving publication and post-release follow-up. Use only for an explicit release request.
---
# RustFS Release
Require the exact final SemVer target before version edits or publication. If unspecified, inspect current releases/tags and ask with concrete candidates; do not guess the channel or bump. A named bump type resolves to an exact version that still needs confirmation. Report the confirmed target verbatim; resolve any later mismatch before proceeding. Independent read-only preflight may continue.
Require the exact final SemVer target before version edits or publication. If unspecified, inspect releases/tags and ask with concrete candidates; do not guess the channel or bump. Reuse the user's confirmed target and authorized delivery scope. Independent read-only preflight may continue.
## Invariants
- Cargo stores `<target>`, never `-preview.N`. Before publication, installation defaults retain the previous available deliverable. Final and preview binaries embed their own tag names, so version/asset names differ despite identical source.
- Use annotated tags without `v`. Preview names are exactly `<target>-preview.<digits>`; choose the next unused iteration after fetching tags. Preview Releases are prereleases, never Latest; publish only versioned assets/exact Docker tags, never latest assets, R2, Helm, moving Docker channels, or installation/banner updates.
- `release/<target>` owns the candidate. Retain its history, backport only reviewed fixes, and never merge main wholesale or force-push it. Pin `PREVIEW_HASH` to its exact commit with complete passing CI. Preview and final tags both point there; never tag a later main/branch HEAD or insert another bump.
- Phase failures block dependents. Keep preview Releases until final publication and notes generation finish; CI removes Releases while retaining tags. Never move an existing preview or final tag.
- Cargo and every candidate executable contain the final product version. `<target>-preview.<run_number>` is an immutable candidate identifier, not a public tag or displayed binary version.
- Freeze the exact source, reviewed workflow recipe, toolchain, Console, CLI, dependency lock, previous stable version and validation plan. Newer unrelated main or Console releases do not change those inputs.
- Retain `release/<target>` history when a release branch is used; backport only reviewed fixes. Pin `CANDIDATE_SHA` to its exact CI-validated commit. Never force-push a release branch or move a public tag.
- Build Linux first. Complete field acceptance before macOS/Windows and packaging. Full certification and final acceptance are required before publication; a partial Linux PASS grants no release authority.
- Promote the accepted bytes. Final publication never recompiles binaries or rebuilds packages/images. Infrastructure retries reuse successful artifacts; changed source or inputs require a new candidate and affected acceptance.
- Phase failures block dependents. No latest/default-channel update before versioned delivery. Use the [pipeline runbook](../../../docs/operations/release-pipeline.md) for executable entry points, locks and recovery.
## Read by Phase
Read only the current phase's procedure; retain completed evidence across turns. Follow root verification, Git/PR, and authorization rules.
## Read by phase
| Phase | Procedure |
|---|---|
| 0 | Clean tree, fetch main/tags, verify GitHub access, then complete the mandatory [Console release gate](references/console-gate.md). A green Console build alone is insufficient. |
| 1–2 | [Prepare/reuse the release candidate and run exact-SHA branch CI](references/release-candidate.md). Source version changes use `rustfs-release-version-bump` with authorized delivery scope. |
| 3 | [Verify the preview build, assets, channels, and notes](references/preview-release.md). |
| 4–5 | [Run the downloaded binary, Console CRUD, and latest-rc matrix](references/preview-acceptance.md). Every check must pass. |
| Confirmation | Complete the gate below and end the turn. |
| 6 | After fresh confirmation, [publish and verify the final tag](references/final-release.md) at `PREVIEW_HASH`. |
| 7 | [Reconcile Security Advisories, milestones, installation references, and the website; reconcile main and remove the completed release branch](references/post-release-updates.md) only after complete publication. |
| 0 | Clean tree, fetch main/tags, verify access, then complete the mandatory [Console gate](references/console-gate.md). |
| 1–2 | [Prepare the exact source and run branch CI](references/release-candidate.md). Version edits use `rustfs-release-version-bump`. |
| 3 | [Build and verify the Linux candidate](references/preview-release.md). |
| 4–5 | [Run artifact, field, Console and rc acceptance](references/preview-acceptance.md), then complete desktop/package/image/chart validation. |
| Confirmation | Report the complete manifest and obtain the confirmation below. |
| 6 | [Publish the accepted manifest](references/final-release.md). |
| 7 | [Reconcile advisories, milestones, installation references, website and release branch](references/post-release-updates.md). |
## Human Confirmation
## Human confirmation
After Phases 3–5 pass, recheck that the release branch still equals `PREVIEW_HASH` and that the current Console asset tag/digest match the preview's recorded build inputs. Report the target, branch, preview tag/hash, branch CI run, preview Release URL, accepted Console asset identity/result, and rc matrix. Explicitly ask whether to publish the final tag and end the turn without creating/pushing it.
After all acceptance passes, report the target, source SHA, source CI run, Linux and completion run URLs, final manifest SHA-256, accepted Console/CLI identities, compatibility/performance results and backlog report. Explicitly ask whether to publish this exact manifest and end the turn without dispatching publication.
Only a new affirmative user reply to that exact report authorizes Phase 6. The original release request, earlier confirmation, silence, or automated follow-up does not count. Confirmation binds the target, preview tag, hash, and accepted build inputs; a changed value or repeated/failed acceptance cycle invalidates it. Revalidate and obtain fresh confirmation before continuing.
Only a new affirmative reply to that exact report authorizes Phase 6. An earlier general release request, silence or automatic continuation does not count. Reuse confirmation for retries of that same manifest; do not ask again merely because a destination failed. A changed source, input, manifest or acceptance result invalidates it and requires renewed acceptance/confirmation.
## Recovery
## Recovery and reporting
- Before the final tag exists, land fixes on main, backport to the candidate, verify target versions, and restart branch CI/acceptance. Use a new preview iteration once a preview was tagged; main may advance independently.
- If only an external build input changed, record it, create a new preview iteration even when the source SHA is unchanged, and repeat the affected build and artifact acceptance before fresh confirmation. Reuse source checks whose inputs remain valid.
- Once the final tag exists, retry publication jobs for that exact tag. A source fix requires a newly confirmed target, not retagging or a newer preview. Phase 7 retries recheck artifact availability and superseding releases.
- If abandoned after the bump merged, report the unpublished version on main and record whether to revert or let the next release overwrite it. Deleting unpublished candidate branches or preview tags requires an explicit decision. Completed release branches follow Phase 7 cleanup; keep release tags.
Report completed phase results and incomplete work with evidence: Console previous/latest tags, release need, hash and run/Release URLs; target, branch CI/SHA, previews and hash, confirmation state; preview/final assets, latest-channel state, Console/rc results and cleanup; advisory reconciliation results and pending disclosure decisions; both milestone states, installation PR checks/merge state, and website text/link/PR/deployment state. Keep artifact publication separate from follow-up delivery; do not call an open PR a live update. Report deviations and their authorization.
Resume failed infrastructure jobs without rebuilding successful units. Fix product defects on main, backport when needed, and start a new candidate. After a final tag exists, retry the same bytes or use a new patch version; never move the tag. Do not conflate implemented, CI-passed, field-accepted, approved, published and post-release-complete states. Report incomplete work and deviations with evidence; a tag or green build alone is not a completed release.
@@ -4,7 +4,7 @@ Read during Phase 0, before changing RustFS version files or tags. Follow the pa
### Console release gate
Complete this gate before changing any RustFS version file or creating any RustFS tag. RustFS `build.yml` downloads the asset returned by `repos/rustfs/console/releases/latest`, so a successful Console build alone is insufficient.
Complete this gate before changing any RustFS version file or creating any RustFS tag. Candidate creation resolves `repos/rustfs/console/releases/latest` once and freezes the exact asset for every build, so a successful Console build alone is insufficient. After freezing, later Console releases do not silently change or invalidate the accepted inputs.
1. Read the latest published Console tag and compare it with Console `main`:
@@ -1,22 +1,13 @@
# Final Release Publication
# Final release publication
Read only after fresh user confirmation of the preview acceptance report. The [parent confirmation gate](../SKILL.md#human-confirmation) applies.
Read only after fresh confirmation of the complete candidate report. The [parent confirmation gate](../SKILL.md#human-confirmation) applies.
## Phase 6 — Publish the final tag on the validated commit
Dispatch `release-publish.yml` on main with the successful completion run ID and approved `release-manifest.json` SHA-256. Do not create the tag manually. The workflow revalidates source, bytes and trusted certification, then creates the annotated final tag at `CANDIDATE_SHA`. A recipe change fails closed and requires a new candidate; unrelated main changes do not require a rebuild.
No second source version bump. Before creating the final tag, recheck the remote `release/<target>` HEAD equals `PREVIEW_HASH` and the current Console release tag/asset digest still match the accepted preview inputs. If either changed after confirmation, the approval no longer applies; validate the changed inputs with a new preview and obtain fresh confirmation. The final tag goes on the exact commit the preview validated:
Require both versioned publication and channel publication jobs to succeed. Verify the final tag's commit and manifest identity; all six platform ZIPs and compatibility aliases; DEB/RPM, checksums, SBOM, provenance and evidence bundle; both multi-architecture image variants in Docker Hub, GHCR and Quay; and the accepted Helm chart in the chart repository. Download a host-compatible final artifact and confirm its digest is exactly the accepted candidate digest and its version/source output agrees.
```bash
git fetch origin --tags
git rev-parse "<preview-tag>^{commit}" # must equal PREVIEW_HASH — abort if not
git tag -a "<target>" -m "Release <target>" "$PREVIEW_HASH"
git push origin "<target>"
```
For stable releases verify GitHub Latest, R2 download aliases and `latest.json` unless a newer stable release already owns those channels. Prereleases must remain GitHub prereleases and leave stable pointers unchanged. Verify generated release notes compare against the frozen previous stable release, excluding internal previews.
- CI rebuilds from the same source with a different tag name, so the binary self-reports `<target>`. Other inputs can change: `build.yml` reads Console `releases/latest` during each build. Compare the actual Console tag, URL, and verified digest in every final build job with the accepted preview record, and inspect available build provenance for other relevant changes. A pre-tag check cannot prevent a later input change. On a mismatch or missing evidence, repeat affected acceptance against the final artifacts and report the limitation; failed acceptance leaves release validation incomplete even if publication already succeeded. Never move the final tag to conceal a failure.
- Verify the final tag's complete publication path: all matrix and release jobs green; `gh release view "<target>"` shows the full versioned and `-latest` asset set plus checksums, SBOM, and provenance; Docker and Helm workflows succeed; `latest.json` points to `<target>`. A stable target must have `isPrerelease=false` and `isLatest=true`. An alpha/beta/rc target must have `isPrerelease=true`; GitHub does not permit prereleases to be Latest, but the project `latest.json` still advances to the final non-preview target.
- Verify the final Release body contains `## What's Changed` and a Full Changelog link. When `PREVIOUS_DELIVERABLE` exists, the link MUST be `https://github.com/rustfs/rustfs/compare/<PREVIOUS_DELIVERABLE>...<target>` and the baseline MUST equal the preview Release baseline; for example, both `1.0.0-beta.12-preview.1` and `1.0.0-beta.12` compare from `1.0.0-beta.11`.
- Verify the preview cleanup: `cleanup-preview-releases` must succeed and `gh release view "<preview-tag>"` must then report `release not found` for every preview iteration of this target. Each surviving preview tag must still resolve to its own recorded commit; the accepted preview resolves to `PREVIEW_HASH`. If the job failed, delete the leftover Releases manually with `gh release delete "<preview-tag>" --yes` and report it.
- Verify a host-compatible final artifact's checksum and `./rustfs --version`; it must report `<target>` and the expected source SHA. This identity check does not replace affected acceptance when build inputs changed.
On infrastructure failure retry the same workflow/seal; matching destinations are reused. Never clobber an asset, force a tag, rebuild the image or silently accept a changed digest. A source fix after the final tag requires a new patch version. Cross-registry writes are not atomic: report the unfinished destination rather than claiming publication complete.
Never pass `--cleanup-tag` when removing preview Releases. Remove them manually only if `cleanup-preview-releases` failed, after final publication and notes generation. Record deleted Releases and surviving tags.
New candidates create no preview Releases to clean up. For a legacy preview release, preserve its tag and remove its Release only after final delivery and notes are verified. Do not pass `--cleanup-tag`. Complete the [post-release follow-up](post-release-updates.md) separately.
@@ -24,7 +24,7 @@ Reconcile advisories after every completed final release, including Console-only
1. Re-read the published deliverable state before edits or a retry. If a newer release has superseded this target, do not downgrade installation references or the website announcement. Report the superseding release and leave those defaults intact.
2. Invoke `rustfs-release-version-bump` with stage `post-release`, the exact target, and the authorized delivery scope. Prepare the follow-up on current main; do not edit or move either validated release tag. Cargo versions may already be preparing the next target and must remain untouched.
3. Verify the exact Docker image manifest, Release download links, RPM source archive, and Helm repository entry before publishing the installation PR. The chart workflow overrides chart/app versions from the final tag, pins the source checkout to that release, and waits for its default Docker image before packaging.
3. Verify the exact Docker image manifest, Release download links, RPM source archive, and Helm repository entry before publishing the installation PR. The chart is packaged and tested before sealing; publication uploads those same bytes only after versioned Docker images are available.
4. Report the installation PR and its merge state separately from the completed release. Reuse existing merge authorization and repository checks; do not infer new merge authority from this reference.
## rustfs.com top banner
@@ -54,4 +54,4 @@ Use the existing announcement configuration from [website PR #112](https://githu
1. Compare the exact published tag with current main. Audit every release-side commit since their merge base, including inherited backports. An ahead count is a history difference, not proof of missing changes. Use ancestry and patch equivalence, then inspect current code and regression evidence for squashed, adapted, superseded, or reverted changes. Record each commit as present, superseded, release-only, or missing; a positive `git cherry` result alone does not prove a missing fix.
2. Bring missing shared changes to main through a reviewed PR and the required checks before closing the follow-up. Preserve main's later changes; decide version bumps and pinned Console configuration separately. If a history merge is needed, use a merge commit and verify the release commit is an ancestor of main; squash does not establish that ancestry. Do not merge main into the published release branch or move its tags. Report unresolved changes instead of claiming reconciliation is complete.
3. After publication and authorized follow-ups are complete, delete the remote `release/<target>` branch. Immediately before deletion, verify its HEAD still equals the published tag's commit and no open PR targets or originates from it. Preserve a branch with newer work or open PRs and report the blocker. Use a deletion with an expected-old-SHA check to avoid removing a concurrently advanced branch.
4. Verify the remote branch is absent and the final and preview tags still resolve to their recorded commits. Remove a local release branch only if it is task-owned, unchecked out, and has no unpreserved work. Record cleanup separately from artifact publication; an explicitly authorized deletion before reconciliation does not make unresolved main changes complete.
4. Verify the remote branch is absent and the final tag and any legacy preview tags still resolve to their recorded commits. Remove a local release branch only if it is task-owned, unchecked out, and has no unpreserved work. Record cleanup separately from artifact publication; an explicitly authorized deletion before reconciliation does not make unresolved main changes complete.
@@ -1,10 +1,10 @@
# Preview Artifact Acceptance
# Candidate Artifact Acceptance
Read after Phase 3 succeeds. Complete every check below before the parent skill's manual confirmation gate. These checks cover the downloaded artifact, embedded Console, and latest rc client.
Read after Phase 3 succeeds. Complete every check below before the parent skill's manual confirmation gate. These checks cover the downloaded artifact, embedded Console, and the rc release frozen in the candidate inputs.
## Phase 4 — Run the artifact locally, verify the console
Use a unique task-owned scratch directory. Select the versioned archive matching the host OS, architecture, and runtime, download its checksum file, and verify the archive before extraction. Do not substitute a different build when a matching artifact is unavailable; report that acceptance gap. The shell example below assumes the verified executable is at `RUSTFS_BINARY` (`rustfs.exe` on Windows), `SCRATCH` is the task directory, and `S3_PORT` and `CONSOLE_PORT` are distinct available ports. Use equivalent native commands when the shell differs.
Use a unique task-owned scratch directory. Select the versioned archive matching the acceptance host, and verify it against the sealed manifest before extraction. Before desktop completion, use the Linux acceptance environment rather than rebuilding for the local desktop. Do not substitute a different build when a matching artifact is unavailable; report that acceptance gap. The shell example below assumes the verified executable is at `RUSTFS_BINARY` (`rustfs.exe` on Windows), `SCRATCH` is the task directory, and `S3_PORT` and `CONSOLE_PORT` are distinct available ports. Use equivalent native commands when the shell differs.
Bind both listeners to loopback. Choose task-only credentials, set loopback proxy exclusions for the server/client processes, and record the server PID. A port probe does not reserve a port; retry startup with fresh ports on a bind conflict rather than connecting to an existing service.
@@ -24,16 +24,16 @@ PREVIEW_SERVER_PID=$!
Checks (all must pass):
- `"$RUSTFS_BINARY" --version` reports the preview tag name and the short SHA of `PREVIEW_HASH`. Reporting `<target>` without the `-preview.N` suffix means the build did not embed the tag — treat as FAIL and investigate before proceeding.
- `"$RUSTFS_BINARY" --version` reports the exact final `<target>` and full `CANDIDATE_SHA`. A preview suffix, another version or another source is a failure.
- Wait with a bounded readiness deadline while checking the recorded server process remains alive; `curl --noproxy '*' -fsS "$S3_ENDPOINT/health/ready"` must return ready from this process's listener.
- Startup log shows the embedded console being served (this was the regression that `fix(release): require embedded console assets` guards).
- Open `CONSOLE_ENDPOINT` in the browser and log in with the task credentials. The dashboard must render without JS console errors; create a bucket, upload a file, download it back (byte-identical), and delete the object and bucket. Keep the server running for Phase 5.
## Phase 5 — Validate with the latest rc client
## Phase 5 — Validate with the pinned rc client
`rc` is the RustFS CLI client from <https://github.com/rustfs/cli>.
- Resolve the latest release with `gh api repos/rustfs/cli/releases/latest --jq .tag_name`; verify the local client's version or download and verify a host-compatible client into the task directory. Set `RC_BIN` to that executable and record its version/digest. Avoid changing the user's installed client for this check.
- Read the CLI release and asset digests from `inputs.json`; verify the local client's version or download a host-compatible asset from that exact release into the task directory. If the host is a desktop, resolve its asset within the same pinned CLI release and record the digest; do not independently follow `latest`. Set `RC_BIN` to that executable and record its version/digest. Avoid changing the user's installed client for this check.
- Inspect the client's current help for an isolated configuration option and use it when supported. Otherwise choose a unique `PREVIEW_ALIAS` and confirm it is absent before creating it; never overwrite an existing alias. Use task-owned bucket/user names and input/output files. The names below are safe only inside the newly started isolated server.
- Point the client at `S3_ENDPOINT` with loopback proxy exclusions and run the command matrix, recording PASS/FAIL per command:
@@ -60,3 +60,9 @@ cmp <local-file> "$SCRATCH/out"
```
- Any FAIL blocks acceptance. On success or failure, stop only the recorded task process and remove only task-created aliases/data after retaining the needed failure evidence. Do not erase an existing client configuration or unrelated server data.
## Field acceptance and desktop completion
Before accepting the Linux manifest, record Scanner/Heal, multi-pool, memory, major features, high-concurrency 503, performance and previous-version integrity/upgrade results in backlog. Preserve the release-specific agreed limits and baseline; a PASS label without artifact identities or test results does not satisfy acceptance.
Dispatch `release-complete.yml` only after that report passes, using the exact Linux manifest digest and backlog URL. Require native macOS/Windows checks, package extraction, image startup/scans, chart tests and complete certification to pass. Complete any desktop Console/client checks applicable to the release against these same archives. Record the completion run and final manifest SHA-256 before entering the parent confirmation gate. Partial Linux acceptance does not authorize final publication.
@@ -1,25 +1,9 @@
# Preview Release Verification
# Linux candidate verification
Read for Phase 3 after the candidate passed exact-SHA CI. The [parent release invariants](../SKILL.md) apply.
Read for Phase 3 after the source passed exact-SHA CI. The [parent invariants](../SKILL.md) apply; commands and recovery live in the [pipeline runbook](../../../../docs/operations/release-pipeline.md).
## Preview channels
Dispatch `release-candidate.yml` on main with the confirmed final version and `CANDIDATE_SHA`. Record the run ID, frozen `inputs.json`, every Linux archive digest and `linux-manifest.json` SHA-256. Require the four Linux builds, native startup/S3 checks, previous-release upgrade matrix and Linux seal to succeed. Missing, skipped, cancelled or failed required work is incomplete acceptance.
- Use `<target>-preview.N` for every target, e.g. `1.0.0-beta.10-preview.3` or `1.1.0-preview.1`.
- The canonical suffix is exactly `-preview.<digits>`. `build.yml` recognizes it before alpha/beta/rc classification and routes it to the preview-only path; any other tag containing `-preview` fails closed instead of being treated as a release.
- A preview Release MUST be published with `isPrerelease=true` and `isLatest=false`. Any `*-latest` preview asset or preview-triggered `latest.json`, R2, Helm, or moving Docker channel tag is a pipeline failure. Docker images may use the exact preview version tag, including its existing variant suffixes.
- Preview Releases are cleaned up by the `cleanup-preview-releases` job after `publish-release` succeeds for the deliverable tag. It deletes every Release whose tag is exactly `<target>-preview.<digits>` and never passes `--cleanup-tag`, so the tags survive.
Verify the frozen Console identity equals the one accepted in Phase 0. Every matrix job consumes that same digest and exact toolchain. The binary must report the final product version and source commit; its candidate ID is separate. No public tag, Release, registry tag, R2 release path, Helm repository or latest pointer should change during this stage.
## Phase 3 — CI and preview Release verification
- Find and watch the tag build: `gh run list --workflow build.yml --branch "<preview-tag>" --limit 1` then `gh run watch <run-id>`. Every build matrix target must succeed (linux x86_64/aarch64 × musl/gnu, macos-aarch64, windows-x86_64).
- Confirm the preview tag resolves to the Phase 2 CI-validated `PREVIEW_HASH`. If the release branch advances during acceptance, repeat Phases 2–5 with a new preview tag.
- Confirm the Release publication jobs (`create-release`, `upload-release-assets`, and `publish-release`) succeed while `update-latest-version` is skipped.
- Record the inputs actually consumed by each build job: source SHA and the embedded Console release tag, asset URL, and verified SHA256 from the download logs. The workflow reads Console `releases/latest` separately for each build; confirm the matrix used the Console asset accepted in Phase 0. A missing identity or differing asset blocks acceptance until resolved. Retain available toolchain and build provenance with the artifact checksums; the same source SHA alone does not establish equivalent builds.
- Verify `gh release view "<preview-tag>" --json isPrerelease,assets,url`: `isPrerelease` must be `true`, and the Release must contain all 6 versioned platform zips, checksums, SBOM, and provenance with no `-latest` assets. Confirm `gh api repos/{owner}/{repo}/releases/latest --jq .tag_name` does not return `<preview-tag>`.
- Record `PREVIOUS_DELIVERABLE`, selected from published Releases by `publishedAt` after excluding the current tag and every `-preview.N` tag. Verify `gh release view "<preview-tag>" --json body --jq .body` contains `## What's Changed` and, when `PREVIOUS_DELIVERABLE` exists, `**Full Changelog**: https://github.com/rustfs/rustfs/compare/<PREVIOUS_DELIVERABLE>...<preview-tag>`. For a repository with no previous deliverable, verify a Full Changelog link exists and record the GitHub baseline fallback.
- Confirm Helm is skipped. If preview Docker images are published, confirm they use only exact preview tags (and variant suffixes), with no `latest`, `alpha`, `beta`, or `rc` channel updates. Preview validation covers the built RustFS binaries, embedded console, and rc compatibility; it does not authorize advancing any installation default.
## Release notes
- When a previous deliverable exists, GitHub Release notes for the preview and final tags MUST use it as their shared comparison baseline: the most recently published non-preview Release before the target. Internal `-preview.N` Releases are explicitly excluded from that selection, even when they point at the same commit as the final tag — cleanup runs after the notes are generated, so the preview Release is still present and would otherwise be picked as the baseline. If no previous deliverable exists, omit `previous_tag_name` and record that GitHub's default baseline fallback was used.
- Generated Release notes carry a workflow-management marker so retries can repair them. Before manually curating a generated body, remove that marker; unmarked non-placeholder notes are preserved by later workflow runs.
A green Linux run is ready for field acceptance, not publication. The partial certification must have `publish_allowed: false`. Retain the exact Linux manifest for the next stage; a new source/build-input candidate must not inherit the old approval. A newer unrelated main commit does not invalidate this evidence.
@@ -16,22 +16,13 @@ Read for Phases 1–2. The [parent release invariants](../SKILL.md) and Console
- Only when neither the release branch, an unfinished preview tag, nor current main already has `<target>`: if main is at a later version, select and verify a historical commit containing `<target>` rather than downgrading main; block if none exists. Otherwise invoke the `rustfs-release-version-bump` skill with stage `prepare`, the final `<target>` (NOT a preview version), and full GitHub flow (commit/push/PR).
- If a bump PR was needed, get it merged into main and record its merge commit. For a new release branch when main already has `<target>`, identify the main commit that introduced those versions. Phase 2 chooses a release candidate containing that commit; an existing release branch keeps its own history and a newer main HEAD is not required.
Do not wait for the moving latest main HEAD to become green. The release-branch CI in Phase 2 validates the chosen commit before a preview tag is created.
Do not wait for the moving latest main HEAD to become green. The release-branch CI in Phase 2 validates the chosen commit before a candidate build is dispatched.
## Phase 2 — Pin and validate the release candidate, then publish the preview tag
## Phase 2 — Pin and validate the release source
- Use `release/<target>`. On the first attempt, create it from a selected main commit containing the version bump and intended changes; it need not be the latest main HEAD. On a restart, retain the existing branch and backport only reviewed fixes from main. Verify the branch's Cargo versions still equal `<target>`. If adopting an existing preview, run this branch CI before continuing its acceptance; reuse the tag only when it points to the validated SHA.
- The current `ci.yml` runs automatically on main pushes, not release-branch pushes. Dispatch it explicitly with `gh workflow run ci.yml --ref "release/<target>"`. Locate that dispatch run, verify its `head_sha` equals the release branch's remote HEAD, and require the full expected CI job set to pass, including `Quick Checks`, `Test and Lint`, and `End-to-End Tests (full merge gate)`. Treat a missing or skipped required lane as incomplete. A successful PR check or CI run for a different SHA does not qualify. If the branch advances or a run is cancelled, validate the new SHA again. Do not tag a candidate with failing or incomplete CI.
- Use `release/<target>`. On the first attempt, create it from a selected main commit containing the version bump and intended changes; it need not be the latest main HEAD. On a restart, retain the existing branch and backport only reviewed fixes from main. Verify the branch's Cargo versions still equal `<target>`. A legacy preview can anchor source selection, but its tag and old artifacts do not constitute a candidate in the new protocol; freeze and verify a new candidate after the release recipe is aligned.
- The current `ci.yml` runs automatically on main pushes, not release-branch pushes. Dispatch it explicitly with `gh workflow run ci.yml --ref "release/<target>"`. Locate that dispatch run, verify its `head_sha` equals the release branch's remote HEAD, and require the full expected CI job set to pass, including `Quick Checks`, `Test and Lint`, and `End-to-End Tests (full merge gate)`. Treat a missing or skipped required lane as incomplete. A successful PR check or CI run for a different SHA does not qualify. If the branch advances or a run is cancelled, validate the new SHA again. Do not dispatch a candidate with failing or incomplete CI.
- Prefer protecting `release/*` against force pushes and deletion and requiring reviewed backport PRs. Check live rulesets rather than assuming the main ruleset covers release branches; report a missing release-branch rule as a gap.
- Set `PREVIEW_HASH` to the CI-validated release-branch commit and recheck the remote branch HEAD immediately before tagging. Report the branch, commit, and CI run URL; do not derive `PREVIEW_HASH` from a later `origin/main` fetch. When adopting an existing preview tag at this SHA, skip tag creation and continue Phase 3.
```bash
git tag -a "<preview-tag>" -m "Release <preview-tag>" "$PREVIEW_HASH"
git push origin "<preview-tag>"
```
Pushing the tag triggers `.github/workflows/build.yml` ("Build and Release"); `docker.yml` chains off it via `workflow_run`.
The preview run builds versioned artifacts and publishes them in a GitHub prerelease. Docker may publish exact preview image tags. Latest-channel, R2, and Helm publication must be skipped; installation references and the website announcement stay unchanged.
On a restart after a backport, re-run release-branch CI before setting the new `PREVIEW_HASH`. Use the next unused preview iteration when the validated commit differs from an existing preview tag; never move an existing tag.
- Set `CANDIDATE_SHA` to the CI-validated release-branch commit. Record the branch, exact commit and CI run URL. The new candidate protocol starts from a commit whose release recipe matches the reviewed main dispatcher; backport the recipe before freezing an older release branch if needed.
- Dispatch the Linux candidate as described in [candidate verification](preview-release.md). Do not create a preview or final tag here. The final executable already contains `<target>`; publication later promotes those same bytes.
- After a source or build-input change, run the affected source checks and create a new candidate. Reuse the frozen candidate after an infrastructure failure. Do not restart acceptance for unrelated main advances.
@@ -4,18 +4,18 @@ description: Prepare an exact RustFS Cargo version or align installation referen
---
# Release Version Files
Require an exact target; ask if missing or ambiguous. Reject targets containing `-preview`: those are tag-only. Infer `post-release` only from an explicit request or parent workflow; otherwise use `prepare`. Delivery defaults to local edit/verify; commit, push, PR, and merge follow existing authorization and [Git rules](../../references/pull-requests.md).
Require an exact target; ask if missing or ambiguous. Reject targets containing `-preview`: those identify candidates, not product versions or new public tags. Infer `post-release` only from an explicit request or parent workflow; otherwise use `prepare`. Delivery defaults to local edit/verify; commit, push, PR, and merge follow existing authorization and [Git rules](../../references/pull-requests.md).
| Stage | Files and constraints |
|---|---|
| `prepare` | Only `Cargo.toml` and `Cargo.lock`: workspace package, internal dependency, and workspace member versions become `<target>`. External dependencies stay unchanged. |
| `post-release` | Only `README.md`, `README_ZH.md`, `flake.nix`, `helm/rustfs/Chart.yaml`, and `rustfs.spec`. Cargo may already target the next release; leave it untouched. |
Before installation edits, verify the exact non-preview Release, downloadable assets/source archive, pullable `rustfs/rustfs:<target>` image manifest, and published Helm chart. Block on missing artifacts. Do not downgrade defaults if a newer deliverable superseded the target. Preview preparation retains the previous published installation references; follow-up commits never replace either validated tag.
Before installation edits, verify the exact non-preview Release, downloadable assets/source archive, pullable `rustfs/rustfs:<target>` image manifest, and published Helm chart. Block on missing artifacts. Do not downgrade defaults if a newer deliverable superseded the target. Candidate preparation retains the previous published installation references; follow-up commits never replace validated tags or sealed artifacts.
Packaging policy:
- Docker image tags have no `v` prefix. Derive chart/app versions with `scripts/helm_chart_version.sh <target>` (`beta.N -> 0.N.0`; other versions unchanged). Helm CI derives versions from the final tag, so no early chart bump is required.
- Docker image tags have no `v` prefix. Derive chart/app versions with `scripts/helm_chart_version.sh <target>`; both equal the final product SemVer, including prerelease suffixes. Helm CI uses the frozen candidate version before sealing, so no early installation chart bump is required. Previously published chart versions remain unchanged.
- RPM `Version` is numeric; `Release` is the prerelease suffix or `1` for stable. Expanded `Source0` and the unpack directory must resolve to the exact target including its suffix. Add the changelog entry using current git identity, date, and target; never copy an example identity/date.
- Changing these mappings requires explicit confirmation.
+2 -2
View File
@@ -1,2 +1,2 @@
sha256-darwin=219ac11e720b3f54dfee4d5d5e95171785ba159b2fe01db3037a8f5061c1ad32
sha256-linux=2d34698cc8ec52d5b6c37ac612919a1d21f36cd56a1af539a7522a4eea518e26
sha256-darwin=9901a4950099b94d3d2618524d540f1ed4bf6023044903886f98b6c4bf5f30fb
sha256-linux=09fd5f48f5556c9fc576357222dccd7ebb0b90b790798f1834db2ec5d67b4952
+10
View File
@@ -22,6 +22,14 @@ TEST_THREADS ?= 1
# differ from CI and .config/nextest.toml test-groups will NOT apply.
.PHONY: script-tests
.PHONY: release-package-tests
release-package-tests: ## Run release package integration on Linux with fpm, dpkg and rpm installed
$(RUSTFS_PYTHON_BIN) ./scripts/test_release_packages.py
.PHONY: release-image-tests
release-image-tests: ## Run OCI integration with Docker, Go and skopeo; set RELEASE_IMAGE_VARIANT and RELEASE_IMAGE_OUTPUT
$(RUSTFS_PYTHON_BIN) ./scripts/test_release_images.py
script-tests: ## Run shell script tests
@echo "Running script tests..."
trap 'status=$$?; printf "ERROR: script-tests failed (exit %s): %s\n" "$$status" "$$BASH_COMMAND" >&2; exit "$$status"' ERR
@@ -54,8 +62,10 @@ script-tests: ## Run shell script tests
./scripts/check_embedded_secrets.sh --self-test
$(RUSTFS_PYTHON_BIN) ./scripts/check_test_wiring.py --self-test
$(RUSTFS_PYTHON_BIN) ./scripts/test_e2e_binary.py
$(RUSTFS_PYTHON_BIN) ./scripts/test_download_release_asset.py
$(RUSTFS_PYTHON_BIN) ./scripts/test_inspect_btf_prototypes.py
$(RUSTFS_PYTHON_BIN) ./scripts/test_release_build_workflow.py
$(RUSTFS_PYTHON_BIN) -m unittest discover -s scripts/release -p "test_*.py"
$(RUSTFS_PYTHON_BIN) ./scripts/test_release_glibc.py
$(RUSTFS_PYTHON_BIN) ./scripts/test_upgrade_workflow.py
$(RUSTFS_PYTHON_BIN) ./scripts/test_migration_gate_evidence.py
+2 -2
View File
@@ -110,7 +110,7 @@ threads-required = "num-test-threads"
# Reserve capacity so sibling disk-heavy tests cannot consume its 2s
# registration budget; retain the queued backup and deadlock assertions.
[[profile.default.overrides]]
filter = 'package(rustfs) & test(=admin::handlers::table_catalog::tests::cold_strong_registration_completes_while_other_bucket_backup_waits)'
filter = 'package(rustfs) & test(=admin::handlers::table_catalog::tests::cold_object_registration_completes_while_other_bucket_backup_waits)'
threads-required = "num-test-threads"
[[profile.default.overrides]]
@@ -410,7 +410,7 @@ threads-required = "num-test-threads"
# Reserve capacity so sibling disk-heavy tests cannot consume its 2s
# registration budget; retain the queued backup and deadlock assertions.
[[profile.ci.overrides]]
filter = 'package(rustfs) & test(=admin::handlers::table_catalog::tests::cold_strong_registration_completes_while_other_bucket_backup_waits)'
filter = 'package(rustfs) & test(=admin::handlers::table_catalog::tests::cold_object_registration_completes_while_other_bucket_backup_waits)'
threads-required = "num-test-threads"
[[profile.ci.overrides]]
+82 -24
View File
@@ -1,44 +1,93 @@
{
"schema_version": 1,
"plan_id": "rustfs-release-validation",
"description": "Versioned certification plan for every public RustFS release exit (rustfs/backlog#2662, R01). preview, prerelease and stable all require this explicit plan. Scripts/release/release_gate.py fail-closes publication on any missing, duplicate, stale, substituted or non-PASS required item.",
"release_types": ["preview", "prerelease", "stable"],
"description": "Certification for immutable candidates. All shipped platforms require native identity and startup; stable and prerelease require the pinned upgrade matrix. Linux-only acceptance never authorizes publication.",
"release_types": [
"preview",
"prerelease",
"stable"
],
"targets": [
{"os": "linux", "arch": "x86_64", "abi": "musl"},
{"os": "linux", "arch": "aarch64", "abi": "musl"},
{"os": "linux", "arch": "x86_64", "abi": "gnu"},
{"os": "linux", "arch": "aarch64", "abi": "gnu"},
{"os": "macos", "arch": "aarch64", "abi": "none"},
{"os": "windows", "arch": "x86_64", "abi": "msvc"}
{
"os": "linux",
"arch": "x86_64",
"abi": "musl"
},
{
"os": "linux",
"arch": "aarch64",
"abi": "musl"
},
{
"os": "linux",
"arch": "x86_64",
"abi": "gnu"
},
{
"os": "linux",
"arch": "aarch64",
"abi": "gnu"
},
{
"os": "macos",
"arch": "aarch64",
"abi": "none"
},
{
"os": "windows",
"arch": "x86_64",
"abi": "msvc"
}
],
"targets_basis": "Platform matrix of .github/workflows/build.yml (prepare-platform-matrix). gnu and macOS targets build with the pyroscope feature; musl and Windows targets build with no extra features.",
"features": ["pyroscope"],
"trusted_workflows": [
{"repository": "rustfs/s3chaos", "workflow_id": "release-gate.yml"}
"features": [
"pyroscope"
],
"trusted_workflows_basis": "The shared evidence envelope producer lives in rustfs/s3chaos (G01, fault/release_gate.rs) and its plan/live split workflow is .github/workflows/release-gate.yml there. Only results from these identities may enter a release certification; PR-uploaded JSON never gains release authority. R02 must extend this list when additional verifier workflows land, never widen it to untrusted sources.",
"trusted_workflows": [
{
"repository": "rustfs/rustfs",
"workflow_id": "release-candidate.yml"
},
{
"repository": "rustfs/rustfs",
"workflow_id": "release-complete.yml"
}
],
"trusted_workflows_basis": "Receipts are produced by the reviewed main workflows and fetched by exact run ID with GitHub artifact digest verification. Candidate inputs freeze the harness commit separately from product source.",
"lanes": [
{
"lane": "identity",
"harness_repository": "rustfs/s3chaos",
"harness_sha": "9a48f55c1d6d58ce63a8fb1ea7317e024f32f11d",
"harness_basis": "Fixed G01 evidence-envelope producer revision (rustfs/s3chaos PR #102 head recorded in rustfs/backlog#2658). R02 re-pins this when the live verifier ships.",
"harness_repository": "rustfs/rustfs",
"harness_sha": "candidate.workflow_sha",
"harness_basis": "The exact workflow commit is frozen in candidate inputs; no mutable branch is accepted as a harness identity.",
"timeout_seconds": 1800,
"environment_capabilities": ["process-exec"],
"cases": ["artifact-identity", "startup"],
"environment_capabilities": [
"process-exec"
],
"cases": [
"artifact-identity",
"startup"
],
"applies_to": {
"release_types": ["preview", "prerelease", "stable"],
"release_types": [
"preview",
"prerelease",
"stable"
],
"platforms": "all"
},
"applies_basis": "Artifact identity and startup checks are mandatory for every release type, preview included, on every shipped target (rustfs/backlog#2662 frozen design item 5)."
},
{
"lane": "upgrade",
"harness_repository": "rustfs/s3chaos",
"harness_sha": "9a48f55c1d6d58ce63a8fb1ea7317e024f32f11d",
"harness_basis": "Same fixed envelope producer as the identity lane; the workflow SHA and the product build SHA are bound separately and need not be equal.",
"harness_repository": "rustfs/rustfs",
"harness_sha": "candidate.workflow_sha",
"harness_basis": "The exact workflow commit is frozen in candidate inputs; no mutable branch is accepted as a harness identity.",
"timeout_seconds": 5400,
"environment_capabilities": ["process-exec", "previous-release-artifact"],
"environment_capabilities": [
"process-exec",
"previous-release-artifact"
],
"cases": [
"direct-upgrade",
"mixed-version-upgrade",
@@ -49,8 +98,17 @@
],
"cases_basis": "Case IDs of the pinned previous-release matrix in .github/workflows/e2e-upgrade.yml (source.version/asset/sha256 and the case list mapped there).",
"applies_to": {
"release_types": ["prerelease", "stable"],
"platforms": [{"os": "linux", "arch": "x86_64", "abi": "gnu"}]
"release_types": [
"prerelease",
"stable"
],
"platforms": [
{
"os": "linux",
"arch": "x86_64",
"abi": "gnu"
}
]
},
"applies_basis": "preview releases use the pre-declared smaller set (identity lane only). The upgrade lane is only exercised on linux-x86_64-gnu, the platform of the pinned source assets."
}
+4 -3
View File
@@ -28,11 +28,12 @@ runs:
- name: Install actionlint
shell: bash
env:
GH_TOKEN: ${{ github.token }}
ASSET_DOWNLOADER: ${{ github.action_path }}/../setup/download_release_asset.py
run: |
actionlint_dir="$(mktemp -d "${RUNNER_TEMP}/actionlint.XXXXXX")"
curl --fail --location --silent --show-error \
--output "$actionlint_dir/actionlint.tar.gz" \
https://github.com/rhysd/actionlint/releases/download/v1.7.12/actionlint_1.7.12_linux_amd64.tar.gz
python3 "$ASSET_DOWNLOADER" --repository rhysd/actionlint --tag v1.7.12 --asset actionlint_1.7.12_linux_amd64.tar.gz --output "$actionlint_dir/actionlint.tar.gz" --sha256 8aca8db96f1b94770f1b0d72b6dddcb1ebb8123cb3712530b08cc387b349a3d8
echo "8aca8db96f1b94770f1b0d72b6dddcb1ebb8123cb3712530b08cc387b349a3d8 $actionlint_dir/actionlint.tar.gz" | sha256sum --check --status
tar -xzf "$actionlint_dir/actionlint.tar.gz" -C "$actionlint_dir" actionlint
rm "$actionlint_dir/actionlint.tar.gz"
+23 -5
View File
@@ -66,10 +66,7 @@ inputs:
runs:
using: "composite"
steps:
# protobuf-compiler is deliberately absent: the setup-protoc step below
# installs 35.1 into the tool cache and prepends it to PATH, so the apt
# build (older, and never version-matched) was shadowed on every run and
# simply never used.
# Use the pinned code generators rather than Ubuntu's older packages.
- name: Install system dependencies (Ubuntu)
if: runner.os == 'Linux'
shell: bash
@@ -79,7 +76,9 @@ runs:
build-essential \
pkg-config \
libssl-dev \
ripgrep
ripgrep \
python3 \
gh
# musl-gcc is needed by the native musl release leg, and zip/unzip by the
# release packaging steps. No CI test lane touches any of them.
@@ -88,13 +87,32 @@ runs:
shell: bash
run: sudo apt-get install -y musl-tools zip unzip
- name: Install pinned code generators through GitHub API (Linux x64)
if: runner.os == 'Linux' && runner.arch == 'X64'
shell: bash
env:
GH_TOKEN: ${{ github.token }}
ASSET_DOWNLOADER: ${{ github.action_path }}/download_release_asset.py
run: |
python3 "$ASSET_DOWNLOADER" --repository protocolbuffers/protobuf --tag v35.1 --asset protoc-35.1-linux-x86_64.zip --output "$RUNNER_TEMP/protoc.zip"
python3 "$ASSET_DOWNLOADER" --repository google/flatbuffers --tag v25.12.19 --asset 'Linux.flatc.binary.g++-13.zip' --output "$RUNNER_TEMP/flatc.zip"
python3 -m zipfile -e "$RUNNER_TEMP/protoc.zip" "$RUNNER_TEMP/protoc"
python3 -m zipfile -e "$RUNNER_TEMP/flatc.zip" "$RUNNER_TEMP/flatc"
chmod +x "$RUNNER_TEMP/protoc/bin/protoc" "$RUNNER_TEMP/flatc/flatc"
"$RUNNER_TEMP/protoc/bin/protoc" --version
"$RUNNER_TEMP/flatc/flatc" --version
echo "$RUNNER_TEMP/protoc/bin" >> "$GITHUB_PATH"
echo "$RUNNER_TEMP/flatc" >> "$GITHUB_PATH"
- name: Install protoc
if: runner.os != 'Linux' || runner.arch != 'X64'
uses: rustfs/setup-protoc@a3705324d8f9bf5b6c3573fb6cf8ae421db55dd6 # v3.0.1
with:
version: "35.1"
repo-token: ${{ github.token }}
- name: Install flatc
if: runner.os != 'Linux' || runner.arch != 'X64'
uses: Nugine/setup-flatc@698800de72a96bfb22cf60431dc21a2ff9a7e07b # v1
with:
version: "25.12.19"
@@ -0,0 +1,84 @@
#!/usr/bin/env python3
"""Download a named GitHub release asset through its API and verify its digest."""
import argparse
import hashlib
import json
from pathlib import Path
import re
import subprocess
import sys
import tempfile
from urllib.parse import quote
def download(repository, tag, name, output, expected_sha256=None):
if not re.fullmatch(r"[A-Za-z0-9][A-Za-z0-9_.-]*/[A-Za-z0-9][A-Za-z0-9_.-]*", repository):
raise ValueError("invalid release repository")
if not tag or not name or "/" in name or "\\" in name:
raise ValueError("invalid release tag or asset name")
if expected_sha256 is not None and not re.fullmatch(r"[0-9a-f]{64}", expected_sha256):
raise ValueError("invalid pinned asset digest")
result = subprocess.run(
["gh", "api", "--hostname", "github.com", f"repos/{repository}/releases/tags/{quote(tag, safe='')}"],
check=True, capture_output=True, text=True, timeout=60,
)
release = json.loads(result.stdout)
if not isinstance(release, dict) or release.get("tag_name") != tag:
raise ValueError("release metadata does not match the requested tag")
assets = release.get("assets")
if not isinstance(assets, list) or any(not isinstance(asset, dict) for asset in assets):
raise ValueError("invalid release asset inventory")
matches = [asset for asset in assets if asset.get("name") == name]
if len(matches) != 1:
raise ValueError("release must contain exactly one named asset")
asset = matches[0]
asset_id, size, digest = asset.get("id"), asset.get("size"), asset.get("digest")
if type(asset_id) is not int or asset_id <= 0 or type(size) is not int or not 0 < size <= 512 * 1024 * 1024:
raise ValueError("invalid release asset identity or size")
if not isinstance(digest, str) or not re.fullmatch(r"sha256:[0-9a-f]{64}", digest):
raise ValueError("release asset has no valid SHA-256 digest")
sha256 = digest.removeprefix("sha256:")
if expected_sha256 is not None and expected_sha256 != sha256:
raise ValueError("release digest differs from the pinned digest")
output = Path(output)
output.parent.mkdir(parents=True, exist_ok=True)
temporary = None
try:
with tempfile.NamedTemporaryFile(dir=output.parent, prefix=".release-asset-", delete=False) as stream:
temporary = Path(stream.name)
# gh authenticates to the API; its HTTP client strips credentials on
# cross-host redirects. No browser URL or token is supplied in argv.
subprocess.run(
["gh", "api", "--hostname", "github.com", "--header", "Accept: application/octet-stream", f"repos/{repository}/releases/assets/{asset_id}"],
check=True, stdout=stream, timeout=300,
)
with temporary.open("rb") as stream:
actual = hashlib.file_digest(stream, "sha256").hexdigest()
if temporary.stat().st_size != size or actual != sha256:
raise ValueError("downloaded release asset failed size or SHA-256 verification")
temporary.replace(output)
print(f"Verified release asset: {repository} {tag} {name} sha256={actual}")
finally:
if temporary is not None:
temporary.unlink(missing_ok=True)
def main():
parser = argparse.ArgumentParser(description=__doc__)
parser.add_argument("--repository", required=True)
parser.add_argument("--tag", required=True)
parser.add_argument("--asset", required=True)
parser.add_argument("--output", required=True, type=Path)
parser.add_argument("--sha256")
args = parser.parse_args()
try:
download(args.repository, args.tag, args.asset, args.output, args.sha256)
except (ValueError, KeyError, TypeError, json.JSONDecodeError, subprocess.SubprocessError, OSError) as error:
print(f"Release asset download failed: {error}", file=sys.stderr)
return 1
return 0
if __name__ == "__main__":
raise SystemExit(main())
+81 -21
View File
@@ -350,10 +350,10 @@
"roles": [
"periodic",
"manual",
"release"
"reusable"
],
"events": [
"push",
"workflow_call",
"schedule",
"workflow_dispatch"
],
@@ -367,7 +367,7 @@
"monitor": "workflow",
"max_age_hours": 36,
"result_owner": "build.yml",
"disposition": "retained independent entry",
"disposition": "Builds artifacts only; public release writes belong to release-publish.yml.",
"failure_owner": "workflow",
"origin": "baseline",
"alert_contract": "scheduled-validation"
@@ -745,14 +745,13 @@
},
{
"workflow": ".github/workflows/docker.yml",
"name": "Docker Images",
"name": "Build candidate images",
"roles": [
"manual",
"release"
"release",
"reusable"
],
"events": [
"workflow_run",
"workflow_dispatch"
"workflow_call"
],
"cadence": "event",
"timezone": "UTC",
@@ -761,7 +760,7 @@
"policy": "active",
"monitor": "none",
"result_owner": "docker.yml",
"disposition": "retained independent entry",
"disposition": "Builds artifacts only; public release writes belong to release-publish.yml.",
"monitor_reason": "Event-driven or operator-requested work has no periodic freshness obligation.",
"origin": "baseline"
},
@@ -954,14 +953,13 @@
},
{
"workflow": ".github/workflows/helm-package.yml",
"name": "Publish helm chart to artifacthub",
"name": "Build candidate Helm chart",
"roles": [
"manual",
"release"
"release",
"reusable"
],
"events": [
"workflow_run",
"workflow_dispatch"
"workflow_call"
],
"cadence": "event",
"timezone": "UTC",
@@ -970,7 +968,7 @@
"policy": "active",
"monitor": "none",
"result_owner": "helm-package.yml",
"disposition": "retained independent entry",
"disposition": "Builds artifacts only; public release writes belong to release-publish.yml.",
"monitor_reason": "Event-driven or operator-requested work has no periodic freshness obligation.",
"origin": "baseline"
},
@@ -1182,14 +1180,13 @@
},
{
"workflow": ".github/workflows/package.yml",
"name": "Package DEB/RPM",
"name": "Build candidate packages",
"roles": [
"manual",
"release"
"release",
"reusable"
],
"events": [
"workflow_run",
"workflow_dispatch"
"workflow_call"
],
"cadence": "event",
"timezone": "UTC",
@@ -1198,7 +1195,7 @@
"policy": "active",
"monitor": "none",
"result_owner": "package.yml",
"disposition": "retained independent entry",
"disposition": "Builds artifacts only; public release writes belong to release-publish.yml.",
"monitor_reason": "Event-driven or operator-requested work has no periodic freshness obligation.",
"origin": "baseline"
},
@@ -1727,6 +1724,69 @@
"disposition": "retained independent entry",
"monitor_reason": "Event-driven or operator-requested work has no periodic freshness obligation.",
"origin": "baseline"
},
{
"workflow": ".github/workflows/release-candidate.yml",
"name": "Release candidate (Linux)",
"roles": [
"manual",
"release"
],
"events": [
"workflow_dispatch"
],
"cadence": "event",
"timezone": "UTC",
"schedule": [],
"owner": "@overtrue",
"policy": "active",
"monitor": "none",
"result_owner": "release-candidate.yml",
"disposition": "Explicit immutable candidate stage",
"monitor_reason": "Operator-requested release stage has no periodic freshness obligation.",
"origin": "release-candidate-migration"
},
{
"workflow": ".github/workflows/release-complete.yml",
"name": "Release candidate (complete)",
"roles": [
"manual",
"release"
],
"events": [
"workflow_dispatch"
],
"cadence": "event",
"timezone": "UTC",
"schedule": [],
"owner": "@overtrue",
"policy": "active",
"monitor": "none",
"result_owner": "release-complete.yml",
"disposition": "Explicit immutable candidate stage",
"monitor_reason": "Operator-requested release stage has no periodic freshness obligation.",
"origin": "release-candidate-migration"
},
{
"workflow": ".github/workflows/release-publish.yml",
"name": "Publish accepted release",
"roles": [
"manual",
"release"
],
"events": [
"workflow_dispatch"
],
"cadence": "event",
"timezone": "UTC",
"schedule": [],
"owner": "@overtrue",
"policy": "active",
"monitor": "none",
"result_owner": "release-publish.yml",
"disposition": "Explicit immutable candidate stage",
"monitor_reason": "Operator-requested release stage has no periodic freshness obligation.",
"origin": "release-candidate-migration"
}
]
}
+11 -4
View File
@@ -5,7 +5,7 @@
## Billing facts
This repository is public. For a public repository, GitHub's standard hosted runners (`ubuntu-latest`, `windows-latest`, `macos-latest`) cost nothing and have no minute cap, whatever plan the organisation is on. GitHub bills only two things: larger hosted runners, which have custom labels and are not used here, and standard runners in private repositories.
This repository is public. For a public repository, GitHub's standard hosted runners (`ubuntu-latest`, `windows-latest`, `macos-latest`) cost nothing and have no minute cap, whatever plan the organisation is on. GitHub bills only two things: larger hosted runners, including the `macos-latest-xlarge` release lane used here, and standard runners in private repositories.
Every other label in this repository is capacity RustFS pays for. ARC pods and lab machines accrue cost while the machine exists. A Ubicloud label is metered for the minutes a job holds its VM:
@@ -13,6 +13,7 @@ Every other label in this repository is capacity RustFS pays for. ARC pods and l
|---|---|---|---|
| `ubuntu-latest` | nobody | GitHub-hosted, Ubuntu 24.04, 4 vCPU, 16 GB RAM, about 14 GB of free disk. Ships a C toolchain, `docker`, `gh`, `curl`, `wget`, `jq`, Python. Can `mount` tmpfs and loop devices; io_uring works | the default for every job |
| `windows-latest`, `macos-latest` | nobody | GitHub-hosted standard runners | Windows and macOS targets |
| `macos-latest-xlarge` | RustFS | GitHub-hosted larger Apple Silicon runner | macOS release builds, deferred until Linux acceptance for candidates |
| `sm-standard-2` | RustFS | ARC pod, Ubuntu 24.04, minimal image: no `gh`, no C toolchain; Docker only on the `dind-` label | only a job that a hosted runner cannot run (see the placement rule) |
| `sm-standard-4` | RustFS | ARC pod, Ubuntu 24.04, 14 CPUs and 28 GB (the label is not the core count), large disk | workspace-scale Cargo builds |
| `ubicloud-standard-16-arm` | RustFS | Ubicloud VM, alias of `ubicloud-standard-16-arm-ubuntu-2404`: Ubuntu 24.04 arm64, 16 vCPU, 48 GB, 300 GB disk. Reached through the `build.yml` matrix, so `actionlint.yaml` does not list it | native aarch64 release builds (`linux-aarch64-musl`, `linux-aarch64-gnu`) |
@@ -32,7 +33,7 @@ A job runs on `ubuntu-latest` unless it has one of these reasons, and the reason
Jobs that do only the following have no reason to be self-hosted: shell or Python scripts, `cargo fmt`, `cargo deny`, typos, actionlint, API calls (`gh`, dependency review, issue and stale bots, CLA checks), downloading an artifact and running a prebuilt binary, or a `cargo build -p rustfs` with its own cache key. `e2e-distributed.yml` and `e2e-upgrade.yml` build the server and the `e2e_test` harness on `ubuntu-latest` inside their budgets, so "it compiles something" alone is not a reason.
The functional chain's `prepare` and `complete-chain` jobs, the performance runner availability check, and runner hygiene use `ubuntu-latest`: they inspect APIs or artifacts, not the lab hosts. The actual functional and performance suites remain on their lab runners. Functional-chain health collection uses `ubuntu-latest` so a lab outage cannot also stop its independent monitor. Docker image packaging also uses `ubuntu-latest` for both musl and glibc variants; it downloads release binaries rather than compiling the workspace, and still sets up QEMU and Buildx explicitly.
The functional chain's `prepare` and `complete-chain` jobs, the performance runner availability check, and runner hygiene use `ubuntu-latest`: they inspect APIs or artifacts, not the lab hosts. The actual functional and performance suites remain on their lab runners. Functional-chain health collection uses `ubuntu-latest` so a lab outage cannot also stop its independent monitor. Docker image packaging also uses `ubuntu-latest` for both musl and glibc variants; it wraps verified candidate binaries rather than compiling the workspace, and still sets up QEMU and Buildx explicitly.
Moving these jobs off AKS reduces capacity demand, not necessarily the Azure bill immediately. VM costs fall only when nodes scale down or smaller capacity replaces them.
@@ -52,7 +53,7 @@ A hosted runner is also the only choice for a few things the pods cannot do: `mo
These are the patterns that have cost real money here without validating anything more:
- An `if: always()` summary or alert job on a self-hosted label inside a `workflow_run`-triggered workflow. The upstream workflow completes many times a day, the real jobs skip, and the summary still starts a pod each time. Gate it on the upstream job having run, and keep it on `ubuntu-latest`.
- Per-push workflows on `main`. A merge triggers the full `ci.yml` matrix, `cache-warm.yml` and the `build.yml` development build; with tens of merges a day that is as much self-hosted time as all pull requests together. Anything that does not verify the merged tree itself belongs on a schedule or behind a `paths` filter.
- Per-push workflows on `main`. A merge triggers the applicable `ci.yml` matrix and cache warming; `build.yml` development builds are scheduled or manual; with tens of merges a day that is as much self-hosted time as all pull requests together. Anything that does not verify the merged tree itself belongs on a schedule or behind a `paths` filter.
- `apt-get install` of tools the hosted image already ships. It is only needed on the minimal pod images.
- A throttled `CARGO_BUILD_JOBS` on a 14-CPU pod. The limit exists for link-storm reasons documented in `ci.yml`; measure before raising it, and measure before lowering it further.
- Tiny jobs on ARC pods. Each one pays roughly a minute of pod start-up before its first step; a compile-free job that takes ten seconds costs more to schedule than to run.
@@ -66,7 +67,7 @@ gh api "repos/rustfs/rustfs/actions/runs/<run_id>/jobs?per_page=100" \
--jq '.jobs[] | "\(.conclusion)\t\(.labels | join(","))\t\(.started_at)\t\(.completed_at)\t\(.name)"'
```
Only rows whose label starts with `sm-`, `dind-`, `ubicloud-`, `smoke-` or `pf-` cost anything. At the October 2026 review, a full pull-request run of `ci.yml` spent about 150 self-hosted minutes, a third of it in jobs that never compiled; treat that as the scale of what a placement mistake costs, not as a current figure.
Rows using `macos-latest-xlarge` or a label starting with `sm-`, `dind-`, `ubicloud-`, `smoke-` or `pf-` use paid capacity. At the October 2026 review, a full pull-request run of `ci.yml` spent about 150 self-hosted minutes, a third of it in jobs that never compiled; treat that as the scale of what a placement mistake costs, not as a current figure.
## Checklist for a new or moved job
@@ -76,3 +77,9 @@ Only rows whose label starts with `sm-`, `dind-`, `ubicloud-`, `smoke-` or `pf-`
- A self-hosted `runs-on` carries a comment naming its reason from the placement rule.
- Every `./.github/actions/setup` call states `cache-save-if`, and a key read from one pool is not written from the other.
- Moving a job does not change its check name, so the `main` ruleset and `docs/testing/ci-gates.md` stay as they are; renaming it does, and then that document's change checklist applies.
## Release stages
The [candidate release pipeline](../../docs/operations/release-pipeline.md) builds Linux first and defers paid desktop capacity until acceptance. Daily development builds also select Linux; use manual platform selection for desktop development artifacts. All six platforms remain required before publication. The pipeline records and promotes the exact tested bytes. Builder workflows have no public-release credentials. Keep periodic development builds, PR verification and field-test chains separate when their contracts differ; do not re-enable disabled workflows as part of a source-file migration.
Larger runners remain billed for public repositories; see [GitHub runner billing](https://docs.github.com/en/actions/concepts/runners/larger-runners#billing). Actual spending comes from organization billing, not runner labels or an estimated build duration.
+149 -8
View File
@@ -23,9 +23,16 @@ on:
- 'deny.toml'
- '.github/actions/**'
- '.github/workflows/**'
- 'scripts/release/create_or_update_release.sh'
- 'scripts/release/package_service_scripts.sh'
- 'scripts/release/package_versions.sh'
- 'scripts/release/**'
- 'scripts/security/generate_release_supply_chain_assets.py'
- '.config/release-validation-plan.json'
- '.github/scheduled-validations.json'
- 'scripts/test_helm_release_workflow.py'
- 'scripts/test_release_packages.py'
- 'scripts/test_release_images.py'
- 'Dockerfile*'
- 'entrypoint.sh'
- 'bootstrap/**'
- 'scripts/test_package_service_scripts.sh'
- 'scripts/test_package_versions.sh'
- 'scripts/test_docker_workflow.py'
@@ -41,9 +48,16 @@ on:
- 'deny.toml'
- '.github/actions/**'
- '.github/workflows/**'
- 'scripts/release/create_or_update_release.sh'
- 'scripts/release/package_service_scripts.sh'
- 'scripts/release/package_versions.sh'
- 'scripts/release/**'
- 'scripts/security/generate_release_supply_chain_assets.py'
- '.config/release-validation-plan.json'
- '.github/scheduled-validations.json'
- 'scripts/test_helm_release_workflow.py'
- 'scripts/test_release_packages.py'
- 'scripts/test_release_images.py'
- 'Dockerfile*'
- 'entrypoint.sh'
- 'bootstrap/**'
- 'scripts/test_package_service_scripts.sh'
- 'scripts/test_package_versions.sh'
- 'scripts/test_docker_workflow.py'
@@ -132,12 +146,38 @@ jobs:
if: github.repository == 'rustfs/rustfs' && (github.event_name != 'pull_request' || github.event.action != 'closed')
runs-on: ubuntu-latest
timeout-minutes: 5
permissions:
contents: read
actions: read
outputs:
release_changed: ${{ steps.scope.outputs.release_changed }}
steps:
- name: Checkout repository
uses: actions/checkout@f548e57e544e1ff5a4c46bf1e1b8685f8e4a348a # v7
with:
fetch-depth: 0
persist-credentials: false
- name: Scope native packaging checks to release changes
id: scope
env:
BASE_SHA: ${{ github.event.pull_request.base.sha || github.event.before }}
run: |
python3 - <<'PY'
import os, re, subprocess
event, base = os.environ['GITHUB_EVENT_NAME'], os.environ.get('BASE_SHA', '')
changed = event == 'workflow_dispatch'
if event in ('push', 'pull_request'):
if not re.fullmatch(r'[0-9a-f]{40}', base) or base == '0' * 40:
changed = True
else:
paths = subprocess.check_output(['git', 'diff', '--name-only', base, os.environ['GITHUB_SHA']], text=True).splitlines()
pattern = r'^(scripts/release/|scripts/test_release_|Dockerfile|entrypoint.sh$|bootstrap/|\.dockerignore$|\.config/release-validation-plan.json$|\.github/workflows/(audit|build|docker|package|helm-package|release-[^.]+)\.yml$)'
changed = any(re.match(pattern, path) for path in paths)
with open(os.environ['GITHUB_OUTPUT'], 'a') as output:
output.write(f'release_changed={str(changed).lower()}\n')
PY
- name: Report unpinned GitHub Actions
run: ./scripts/security/check_workflow_pins.sh --enforce
@@ -150,8 +190,35 @@ jobs:
- name: Check checkouts clear their credentials
run: ./scripts/security/check_persist_credentials.sh
- name: Check preview release workflow policy
run: ./scripts/security/check_preview_release_workflow.sh
- name: Check release candidate and promotion contracts
run: python3 -m unittest discover -s scripts/release -p 'test_*.py'
- name: Prepare artifact transport rehearsal
run: |
mkdir -p "$RUNNER_TEMP/release-transport"
printf '{"purpose":"transport contract, not release evidence"}\n' > "$RUNNER_TEMP/release-transport/transport.json"
- uses: actions/upload-artifact@043fb46d1a93c77aae656e7c1c64a875d1fc6a0a # v7.0.1
with:
name: release-transport-contract-${{ github.run_attempt }}
path: ${{ runner.temp }}/release-transport/transport.json
if-no-files-found: error
retention-days: 1
- name: Verify actual GitHub artifact metadata and downloaded bytes
env:
GH_TOKEN: ${{ github.token }}
run: |
python3 - <<'PY'
import os, sys
from pathlib import Path
sys.path.insert(0, 'scripts/release')
import candidate as c
run = c.github(f"repos/{c.REPOSITORY}/actions/runs/{os.environ['GITHUB_RUN_ID']}")
destination = Path(os.environ['RUNNER_TEMP']) / 'release-transport-downloaded'
c.download_artifact(run, f"release-transport-contract-{os.environ['GITHUB_RUN_ATTEMPT']}", destination)
c.require(c.read_json(destination / 'transport.json') == {'purpose': 'transport contract, not release evidence'}, 'artifact round trip changed')
PY
- name: Check tier evidence workflow isolation
run: ./scripts/security/check_tier_artifact_workflow.sh
@@ -162,6 +229,80 @@ jobs:
- name: Check package service lifecycle contract
run: ./scripts/test_package_service_scripts.sh
release-package-contract:
name: Release package contract
needs: workflow-pin-report
if: needs.workflow-pin-report.outputs.release_changed == 'true'
runs-on: ubuntu-latest
timeout-minutes: 15
steps:
- uses: actions/checkout@f548e57e544e1ff5a4c46bf1e1b8685f8e4a348a # v7
with:
persist-credentials: false
- name: Install real package managers
run: |
sudo apt-get update
sudo apt-get install -y ruby ruby-dev build-essential rpm fakeroot cpio
sudo gem install fpm --version 1.17.0 --no-document
- name: Exercise accepted and substituted payloads through both recipes
run: python3 scripts/test_release_packages.py
release-image-contract:
name: Release image contract (${{ matrix.variant }})
needs: workflow-pin-report
if: needs.workflow-pin-report.outputs.release_changed == 'true'
runs-on: ubuntu-latest
timeout-minutes: 35
strategy:
fail-fast: false
matrix:
variant: [musl, gnu]
env:
RELEASE_IMAGE_VARIANT: ${{ matrix.variant }}
NO_PROXY: 127.0.0.1,localhost,::1
steps:
- uses: actions/checkout@f548e57e544e1ff5a4c46bf1e1b8685f8e4a348a # v7
with:
persist-credentials: false
- name: Install OCI tools
run: |
echo "RELEASE_IMAGE_OUTPUT=$RUNNER_TEMP/image-contract" >> "$GITHUB_ENV"
echo 'no_proxy=127.0.0.1,localhost,::1' >> "$GITHUB_ENV"
sudo apt-get update
sudo apt-get install -y skopeo
go version
- uses: docker/setup-qemu-action@c7c53464625b32c7a7e944ae62b3e17d2b600130 # v3
- uses: docker/setup-buildx-action@8d2750c68a42422c14e847fe6c8ac0403b4cbd6f # v3
- name: Exercise real OCI packaging, startup, registry retry and conflicts
run: python3 scripts/test_release_images.py
- name: Exercise amd64 OCI archive scanning
uses: aquasecurity/trivy-action@ed142fd0673e97e23eac54620cfb913e5ce36c25
env:
TRIVY_PLATFORM: linux/amd64
with:
input: ${{ runner.temp }}/image-contract-scan/image-${{ matrix.variant }}-amd64.tar
format: sarif
output: ${{ runner.temp }}/image-contract/scan-${{ matrix.variant }}-amd64.sarif
exit-code: "0"
- name: Exercise arm64 OCI archive scanning
uses: aquasecurity/trivy-action@ed142fd0673e97e23eac54620cfb913e5ce36c25
env:
TRIVY_PLATFORM: linux/arm64
with:
input: ${{ runner.temp }}/image-contract-scan/image-${{ matrix.variant }}-arm64.tar
format: sarif
output: ${{ runner.temp }}/image-contract/scan-${{ matrix.variant }}-arm64.sarif
exit-code: "0"
- name: Verify scan receipts before recording the archive
run: |
python3 - <<'PY'
import os, sys
from pathlib import Path
sys.path.insert(0, 'scripts/release')
import deliver, test_candidate
deliver.image_record(test_candidate.inputs(), os.environ['RELEASE_IMAGE_VARIANT'], Path(os.environ['RELEASE_IMAGE_OUTPUT']))
PY
dependency-review:
name: Dependency Review
runs-on: ubuntu-latest
File diff suppressed because it is too large Load Diff
+16 -3
View File
@@ -142,15 +142,23 @@ jobs:
- uses: actions/checkout@f548e57e544e1ff5a4c46bf1e1b8685f8e4a348a # v7
with:
persist-credentials: false
- name: Install pinned typos through GitHub API
env:
GH_TOKEN: ${{ github.token }}
TYPOS_VERSION: 1.50.2
run: |
python3 .github/actions/setup/download_release_asset.py --repository crate-ci/typos --tag "v${TYPOS_VERSION}" --asset "typos-v${TYPOS_VERSION}-x86_64-unknown-linux-musl.tar.gz" --output "$RUNNER_TEMP/typos.tar.gz"
mkdir -p "$RUNNER_TEMP/typos"
tar -xzf "$RUNNER_TEMP/typos.tar.gz" -C "$RUNNER_TEMP/typos"
- name: Typos check with custom config file
uses: crate-ci/typos@512fc24f32f44ab01972217aaaf3dc86ec234d53 # v1.50.2
run: '"$RUNNER_TEMP/typos/typos" .'
# Fail early with compile-free checks for every pull request.
quick-checks:
name: Quick Checks
if: github.event_name != 'pull_request' || github.event.action != 'closed'
runs-on: ubuntu-latest
timeout-minutes: 10
timeout-minutes: 20
steps:
- name: Checkout repository
uses: actions/checkout@f548e57e544e1ff5a4c46bf1e1b8685f8e4a348a # v7
@@ -1137,11 +1145,16 @@ jobs:
- name: Install mc
env:
GH_TOKEN: ${{ github.token }}
MC_VERSION: RELEASE.2025-08-13T08-35-41Z
MC_SHA256: 01f866e9c5f9b87c2b09116fa5d7c06695b106242d829a8bb32990c00312e891
run: |
if ! command -v gh >/dev/null 2>&1; then
sudo apt-get update
sudo apt-get install -y gh
fi
MC_BINARY="mc.linux-amd64.${MC_VERSION}"
curl -fsSLo "$RUNNER_TEMP/mc" "https://github.com/minio/mc/releases/download/${MC_VERSION}/${MC_BINARY}"
python3 .github/actions/setup/download_release_asset.py --repository minio/mc --tag "$MC_VERSION" --asset "$MC_BINARY" --output "$RUNNER_TEMP/mc" --sha256 "$MC_SHA256"
echo "${MC_SHA256} $RUNNER_TEMP/mc" | sha256sum --check --status
chmod +x "$RUNNER_TEMP/mc"
echo "$RUNNER_TEMP" >> "$GITHUB_PATH"
+71 -751
View File
@@ -1,779 +1,99 @@
# Copyright 2024 RustFS Team
#
# Licensed under the Apache License, Version 2.0 (the "License");
# you may not use this file except in compliance with the License.
# You may obtain a copy of the License at
#
# http://www.apache.org/licenses/LICENSE-2.0
#
# Unless required by applicable law or agreed to in writing, software
# distributed under the License is distributed on an "AS IS" BASIS,
# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
# See the License for the specific language governing permissions and
# limitations under the License.
# Docker Images Workflow
#
# This workflow builds Docker images using pre-built binaries from the build workflow.
#
# Trigger Types:
# 1. workflow_run: Automatically triggered when "Build and Release" workflow completes
# 2. workflow_dispatch: Manual trigger for standalone Docker builds
#
# Key Features:
# - Only triggers when Linux builds (x86_64 + aarch64) are successful
# - Independent of macOS/Windows build status
# - Uses workflow_run event for precise control
# - Only builds Docker images for releases and prereleases (development builds are skipped)
name: Docker Images
# Permissions needed for workflow_run event and Docker registry access
# Copyright 2026 RustFS Team
# SPDX-License-Identifier: Apache-2.0
name: Build candidate images
on:
workflow_call:
inputs:
candidate_manifest:
type: string
required: true
permissions:
contents: read
packages: write
on:
# Automatically triggered when build workflow completes
workflow_run:
workflows: [ "Build and Release" ]
types: [ completed ]
# Manual trigger with same parameters for consistency
workflow_dispatch:
inputs:
push_images:
description: "Push images to registries"
required: false
default: true
type: boolean
version:
description: "Version to build (latest, v1.0.0, v1.0.0-alpha1, or v1.0.1-preview.1)"
required: false
default: "latest"
type: string
force_rebuild:
description: "Force rebuild even if binary exists (useful for testing)"
required: false
default: false
type: boolean
env:
CONCLUSION: ${{ github.event.workflow_run.conclusion }}
HEAD_BRANCH: ${{ github.event.workflow_run.head_branch }}
HEAD_SHA: ${{ github.event.workflow_run.head_sha }}
TRIGGERING_EVENT: ${{ github.event.workflow_run.event }}
DOCKERHUB_USERNAME: rustfs
CARGO_TERM_COLOR: always
REGISTRY_DOCKERHUB: rustfs/rustfs
REGISTRY_GHCR: ghcr.io/${{ github.repository }}
REGISTRY_QUAY: quay.io/rustfs/rustfs
DOCKER_PLATFORMS: linux/amd64,linux/arm64
actions: read
jobs:
# Check if we should build Docker images.
# Short-circuit at job level so per-merge development builds (workflow_run
# from a push to main) skip the whole run without spawning a checkout job:
# images are only published for tag builds — build.yml push triggers cover
# only main and tags, so a non-main head_branch is a tag name — and for
# manual dispatch.
build-check:
name: Docker Build Check
if: >-
github.event_name == 'workflow_dispatch' ||
(github.event.workflow_run.conclusion == 'success' &&
github.event.workflow_run.event == 'push' &&
github.event.workflow_run.head_branch != 'main')
runs-on: ubuntu-latest
timeout-minutes: 30
outputs:
should_build: ${{ steps.check.outputs.should_build }}
should_push: ${{ steps.check.outputs.should_push }}
build_type: ${{ steps.check.outputs.build_type }}
version: ${{ steps.check.outputs.version }}
short_sha: ${{ steps.check.outputs.short_sha }}
is_prerelease: ${{ steps.check.outputs.is_prerelease }}
create_latest: ${{ steps.check.outputs.create_latest }}
source_ref: ${{ steps.check.outputs.source_ref }}
steps:
- name: Checkout repository
uses: actions/checkout@f548e57e544e1ff5a4c46bf1e1b8685f8e4a348a # v7
with:
persist-credentials: false
# For workflow_run events, checkout the specific commit that triggered the workflow
ref: ${{ github.event.workflow_run.head_sha || github.sha }}
- name: Check build conditions
id: check
env:
# dispatch inputs via env, not `${{ }}` interpolation: they are
# free-form strings and would otherwise be evaluated by bash.
INPUT_VERSION: ${{ github.event.inputs.version }}
INPUT_PUSH_IMAGES: ${{ github.event.inputs.push_images }}
INPUT_FORCE_REBUILD: ${{ github.event.inputs.force_rebuild }}
run: |
should_build=false
should_push=false
build_type="none"
version=""
short_sha=""
is_prerelease=false
create_latest=false
source_ref="$GITHUB_SHA"
# Pre-GA policy: until the first stable (vX.Y.Z) tag exists, every
# prerelease (alpha/beta/rc) also moves `latest`, so users pulling
# `latest` get the newest test build. Once a stable tag is published
# this returns false and `latest` follows stable releases only.
prerelease_moves_latest() {
local stable_tags
stable_tags=$(git ls-remote --tags --refs origin 2>/dev/null \
| awk '{print $2}' \
| grep -E '^refs/tags/v?[0-9]+\.[0-9]+\.[0-9]+$' || true)
if [[ -z "$stable_tags" ]]; then
return 0
fi
echo "ℹ️ Stable release tag(s) already exist; prereleases no longer update latest"
return 1
}
if [[ "${{ github.event_name }}" == "workflow_run" ]]; then
# Triggered by build workflow completion
echo "🔗 Triggered by build workflow completion"
# Check if the triggering workflow was successful
# If the workflow succeeded, it means ALL builds (including Linux x86_64 and aarch64) succeeded
if [[ "$CONCLUSION" == "success" ]]; then
echo "✅ Build workflow succeeded, all builds including Linux are successful"
should_build=true
should_push=true
else
echo "❌ Build workflow failed (conclusion: $CONCLUSION), skipping Docker build"
should_build=false
fi
# Extract version info from commit message or use commit SHA
# Use Git to generate consistent short SHA (ensures uniqueness like build.yml)
short_sha=$(git rev-parse --short "$HEAD_SHA")
source_ref="$HEAD_SHA"
# Determine build type based on triggering workflow event and ref
triggering_event="$TRIGGERING_EVENT"
head_branch="$HEAD_BRANCH"
echo "🔍 Analyzing triggering workflow:"
echo " 📋 Event: $triggering_event"
echo " 🌿 Head branch: $head_branch"
echo " 📎 Head SHA: $HEAD_SHA"
# Check if this was triggered by a tag push
if [[ "$triggering_event" == "push" ]]; then
# For tag pushes, head_branch will be like "refs/tags/v1.0.0" or just "v1.0.0"
if [[ "$head_branch" == refs/tags/* ]]; then
# Extract tag name from refs/tags/TAG_NAME
tag_name="${head_branch#refs/tags/}"
version="$tag_name"
elif [[ "$head_branch" =~ ^v?[0-9]+\.[0-9]+\.[0-9]+ ]]; then
# Direct tag name like "v1.0.0" or "1.0.0-alpha.1"
version="$head_branch"
elif [[ "$head_branch" == "main" ]]; then
# Regular branch push to main
build_type="development"
version="dev-${short_sha}"
should_build=false
echo "⏭️ Skipping Docker build for development version (main branch push)"
else
# Other branch push
build_type="development"
version="dev-${short_sha}"
should_build=false
echo "⏭️ Skipping Docker build for development version (branch: $head_branch)"
fi
# If we extracted a version (tag), determine release type
if [[ -n "$version" ]] && [[ "$version" != "dev-${short_sha}" ]]; then
# Remove 'v' prefix if present for consistent version format
if [[ "$version" == v* ]]; then
version="${version#v}"
fi
if [[ "$version" =~ -preview\.[0-9]+$ ]]; then
build_type="preview"
is_prerelease=true
echo "🔍 Building Docker image for preview: $version (version tags only)"
elif [[ "$version" == *"-preview"* ]]; then
echo "❌ Invalid preview tag: $version (expected suffix: -preview.<number>)" >&2
exit 1
elif [[ "$version" == *"alpha"* ]] || [[ "$version" == *"beta"* ]] || [[ "$version" == *"rc"* ]]; then
build_type="prerelease"
is_prerelease=true
# Pre-GA policy: prereleases update latest until the first stable tag exists.
if prerelease_moves_latest; then
create_latest=true
echo "🧪 Building Docker image for prerelease: $version (creating latest tag)"
else
echo "🧪 Building Docker image for prerelease: $version"
fi
else
build_type="release"
create_latest=true
echo "🚀 Building Docker image for release: $version"
fi
fi
else
# Non-push events
build_type="development"
version="dev-${short_sha}"
should_build=false
echo "⏭️ Skipping Docker build for development version (event: $triggering_event)"
fi
echo "🔄 Build triggered by workflow_run:"
echo " 📋 Conclusion: $CONCLUSION"
echo " 🌿 Branch: $HEAD_BRANCH"
echo " 📎 SHA: $HEAD_SHA"
echo " 🎯 Event: $TRIGGERING_EVENT"
elif [[ "${{ github.event_name }}" == "workflow_dispatch" ]]; then
# Manual trigger
input_version="$INPUT_VERSION"
version="${input_version}"
should_push="$INPUT_PUSH_IMAGES"
should_build=true
# Get short SHA
short_sha=$(git rev-parse --short HEAD)
echo "🎯 Manual Docker build triggered:"
echo " 📋 Requested version: $input_version"
echo " 🔧 Force rebuild: $INPUT_FORCE_REBUILD"
echo " 🚀 Push images: $should_push"
case "$input_version" in
"latest")
build_type="release"
create_latest=true
echo "🚀 Building with latest stable release version"
;;
*-preview*)
if [[ ! "$input_version" =~ ^v?[0-9]+\.[0-9]+\.[0-9]+(-[0-9A-Za-z.-]+)?-preview\.[0-9]+$ ]]; then
echo "❌ Invalid preview version: $input_version" >&2
exit 1
fi
build_type="preview"
is_prerelease=true
version="${input_version#v}"
echo "🔍 Building with preview version: $version (version tags only)"
;;
# Prerelease versions (must match first, more specific)
v*alpha*|v*beta*|v*rc*|*alpha*|*beta*|*rc*)
build_type="prerelease"
is_prerelease=true
# Pre-GA policy: prereleases update latest until the first stable tag exists.
if prerelease_moves_latest; then
create_latest=true
echo "🧪 Building with prerelease version: $input_version (creating latest tag)"
else
echo "🧪 Building with prerelease version: $input_version"
fi
;;
# Release versions (match after prereleases, more general)
v[0-9]*|[0-9]*.*.*)
build_type="release"
create_latest=true
echo "📦 Building with specific release version: $input_version"
;;
*)
# Invalid version for Docker build
should_build=false
echo "❌ Invalid version for Docker build: $input_version"
echo "⚠️ Only release versions (latest, v1.0.0, 1.0.0) and prereleases (v1.0.0-alpha1, 1.0.0-beta2) are supported"
;;
esac
if [[ "$should_build" == true && "$input_version" != "latest" ]]; then
tag_ref="refs/tags/$input_version"
if ! git ls-remote --exit-code origin "$tag_ref" >/dev/null 2>&1; then
if [[ "$input_version" == v* ]]; then
tag_ref="refs/tags/${input_version#v}"
else
tag_ref="refs/tags/v$input_version"
fi
fi
if ! git ls-remote --exit-code origin "$tag_ref" >/dev/null 2>&1; then
echo "❌ Release tag not found for Docker build: $input_version"
exit 1
fi
source_ref="$tag_ref"
fi
fi
{
echo "should_build=$should_build"
echo "should_push=$should_push"
echo "build_type=$build_type"
echo "version=$version"
echo "short_sha=$short_sha"
echo "is_prerelease=$is_prerelease"
echo "create_latest=$create_latest"
echo "source_ref=$source_ref"
} >> "$GITHUB_OUTPUT"
echo "🐳 Docker Build Summary:"
echo " - Should build: $should_build"
echo " - Should push: $should_push"
echo " - Build type: $build_type"
echo " - Version: $version"
echo " - Short SHA: $short_sha"
echo " - Is prerelease: $is_prerelease"
echo " - Create latest: $create_latest"
echo " - Source ref: $source_ref"
# Consume the trusted release certification for the exact version this
# workflow wraps and verify the certified input digests. A manual dispatch
# without that certification fails here and pushes no official tag; only
# results from the trusted verification sources carry release authority.
release-certification:
name: Release Certification
needs: build-check
if: needs.build-check.outputs.should_build == 'true'
runs-on: ubuntu-latest
timeout-minutes: 30
permissions:
contents: read
actions: read
steps:
- name: Checkout repository
uses: actions/checkout@f548e57e544e1ff5a4c46bf1e1b8685f8e4a348a # v7
with:
persist-credentials: false
ref: ${{ needs.build-check.outputs.source_ref }}
- name: Download and verify release certification
env:
GH_TOKEN: ${{ secrets.RELEASE_VALIDATION_TOKEN || github.token }}
TRIGGERING_RUN_ID: ${{ github.event.workflow_run.id }}
VERSION: ${{ needs.build-check.outputs.version }}
shell: bash
run: |
set -euo pipefail
mkdir -p certification
run_id="$TRIGGERING_RUN_ID"
if [[ -z "$run_id" ]]; then
run_id="$(gh api --method GET \
"repos/${GITHUB_REPOSITORY}/actions/workflows/build.yml/runs" \
-f branch="$VERSION" -f status=success -F per_page=1 \
--jq '.workflow_runs[0].id // empty' || true)"
fi
if [[ -z "$run_id" ]]; then
echo "No certified Build and Release run found for ${VERSION}; failing closed" >&2
exit 1
fi
if ! gh run download "$run_id" -n release-certification -D certification; then
echo "Build run ${run_id} published no release-certification artifact; failing closed" >&2
exit 1
fi
python3 scripts/release/release_gate.py verify-promotion \
--plan .config/release-validation-plan.json \
--candidate certification/candidate.json \
--summary certification/release-gate-output \
--artifact-sha256 "$(jq -r '.deliverables[0].sha256' certification/candidate.json)" \
--artifact-name "$(jq -r '.deliverables[0].name' certification/candidate.json)"
# Input digest binding: every zip this image wraps must be the
# certified byte-for-byte content, never a mutable latest pointer.
while IFS=$'\t' read -r name location expected; do
echo "Verifying certified input ${name}"
actual="$(curl -fsSL "$location" | sha256sum | awk '{print $1}')"
if [[ "$actual" != "$expected" ]]; then
echo "Certified input ${name} does not match its recorded digest" >&2
exit 1
fi
done < <(jq -r '.deliverables[] | select(.name | contains("linux")) | [.name, .location, .sha256] | @tsv' certification/candidate.json)
# Build multi-arch Docker images
# Strategy: Build images using pre-built binaries from dl.rustfs.com
# Supports both release and dev channel binaries based on build context
# Only runs when should_build is true (which includes workflow success check)
build-docker:
name: Build Docker Images
needs: [ build-check, release-certification ]
if: needs.build-check.outputs.should_build == 'true'
runs-on: ubuntu-latest
timeout-minutes: 60
timeout-minutes: 90
strategy:
fail-fast: false
matrix:
include:
- variant: musl
file: Dockerfile
suffix: ""
- variant: glibc
- variant: gnu
file: Dockerfile.glibc
suffix: "-glibc"
env:
CANDIDATE_MANIFEST: ${{ inputs.candidate_manifest }}
VARIANT: ${{ matrix.variant }}
steps:
- name: Checkout repository
uses: actions/checkout@f548e57e544e1ff5a4c46bf1e1b8685f8e4a348a # v7
- uses: actions/checkout@f548e57e544e1ff5a4c46bf1e1b8685f8e4a348a # v7
with:
ref: ${{ fromJson(inputs.candidate_manifest).source_sha }}
persist-credentials: false
ref: ${{ needs.build-check.outputs.source_ref }}
- name: Login to Docker Hub
uses: docker/login-action@c94ce9fb468520275223c153574b00df6fe4bcc9 # v3
with:
username: ${{ env.DOCKERHUB_USERNAME }}
password: ${{ secrets.DOCKERHUB_TOKEN }}
- name: Login to GitHub Container Registry
uses: docker/login-action@c94ce9fb468520275223c153574b00df6fe4bcc9 # v3
with:
registry: ghcr.io
username: ${{ secrets.GHCR_USERNAME }}
password: ${{ secrets.GHCR_PASSWORD }}
- name: Login to Quay.io
uses: docker/login-action@c94ce9fb468520275223c153574b00df6fe4bcc9 # v3
with:
registry: quay.io
username: ${{ secrets.QUAY_USERNAME }}
password: ${{ secrets.QUAY_PASSWORD }}
- name: Set up QEMU
uses: docker/setup-qemu-action@c7c53464625b32c7a7e944ae62b3e17d2b600130 # v3
- name: Set up Docker Buildx
uses: docker/setup-buildx-action@8d2750c68a42422c14e847fe6c8ac0403b4cbd6f # v3
- name: Extract metadata and generate tags
id: meta
- name: Reuse completed image
id: reuse
env:
GH_TOKEN: ${{ github.token }}
run: python3 scripts/release/deliver.py reuse --kind image --target "$VARIANT"
- name: Install OCI tools
if: steps.reuse.outputs.reused != 'true'
run: |
BUILD_TYPE="${{ needs.build-check.outputs.build_type }}"
VERSION="${{ needs.build-check.outputs.version }}"
CREATE_LATEST="${{ needs.build-check.outputs.create_latest }}"
VARIANT_SUFFIX="${{ matrix.suffix }}"
# Convert version format for Dockerfile compatibility. The former
# DOCKER_CHANNEL was "release" down every branch and was passed as a
# build-arg no Dockerfile declares, so it is gone.
case "$VERSION" in
"latest")
DOCKER_RELEASE="latest"
;;
v*)
# For versioned releases (v1.0.0), remove 'v' prefix for Dockerfile
DOCKER_RELEASE="${VERSION#v}"
;;
*)
# For other versions, pass as-is
DOCKER_RELEASE="${VERSION}"
;;
esac
echo "docker_release=$DOCKER_RELEASE" >> "$GITHUB_OUTPUT"
echo "🐳 Docker build parameters:"
echo " - Original version: $VERSION"
echo " - Docker RELEASE: $DOCKER_RELEASE"
# Generate tags based on build type
# Only support release and prerelease builds (no development builds)
TAG_BASE="${VERSION}${VARIANT_SUFFIX}"
TAGS="${{ env.REGISTRY_DOCKERHUB }}:$TAG_BASE,${{ env.REGISTRY_GHCR }}:$TAG_BASE,${{ env.REGISTRY_QUAY }}:$TAG_BASE"
# Add latest when requested (stable releases, and prereleases before GA)
if [[ "$CREATE_LATEST" == "true" ]]; then
TAGS="$TAGS,${{ env.REGISTRY_DOCKERHUB }}:latest${VARIANT_SUFFIX},${{ env.REGISTRY_GHCR }}:latest${VARIANT_SUFFIX},${{ env.REGISTRY_QUAY }}:latest${VARIANT_SUFFIX}"
fi
# Always add the channel tag for prereleases, independent of latest
if [[ "$BUILD_TYPE" == "prerelease" ]]; then
# Prerelease channel tags (alpha, beta, rc)
if [[ "$VERSION" == *"alpha"* ]]; then
CHANNEL="alpha"
elif [[ "$VERSION" == *"beta"* ]]; then
CHANNEL="beta"
elif [[ "$VERSION" == *"rc"* ]]; then
CHANNEL="rc"
fi
if [[ -n "$CHANNEL" ]]; then
TAGS="$TAGS,${{ env.REGISTRY_DOCKERHUB }}:${CHANNEL}${VARIANT_SUFFIX},${{ env.REGISTRY_GHCR }}:${CHANNEL}${VARIANT_SUFFIX},${{ env.REGISTRY_QUAY }}:${CHANNEL}${VARIANT_SUFFIX}"
fi
fi
# Output tags
echo "tags=$TAGS" >> "$GITHUB_OUTPUT"
# Staging identity for this run's promotion: official tags are only
# created after the staged image verifies against the certification,
# and they point at this same digest (no rebuild-after-verify).
STAGING_TAG="staging-${GITHUB_RUN_ID}-${GITHUB_RUN_ATTEMPT}${VARIANT_SUFFIX}"
STAGING_TAGS="${{ env.REGISTRY_DOCKERHUB }}:${STAGING_TAG},${{ env.REGISTRY_GHCR }}:${STAGING_TAG},${{ env.REGISTRY_QUAY }}:${STAGING_TAG}"
echo "staging_tags=$STAGING_TAGS" >> "$GITHUB_OUTPUT"
echo "staging_tag=$STAGING_TAG" >> "$GITHUB_OUTPUT"
# Generate labels
LABELS="org.opencontainers.image.title=RustFS"
LABELS="$LABELS,org.opencontainers.image.description=RustFS distributed object storage system"
LABELS="$LABELS,org.opencontainers.image.version=$VERSION"
SOURCE_REVISION="$(git rev-parse HEAD)"
LABELS="$LABELS,org.opencontainers.image.revision=$SOURCE_REVISION"
LABELS="$LABELS,org.opencontainers.image.source=${{ github.server_url }}/${{ github.repository }}"
LABELS="$LABELS,org.opencontainers.image.created=$(date -u +'%Y-%m-%dT%H:%M:%SZ')"
LABELS="$LABELS,org.opencontainers.image.build-type=$BUILD_TYPE"
echo "labels=$LABELS" >> "$GITHUB_OUTPUT"
echo "🐳 Generated Docker tags:"
echo "$TAGS" | tr ',' '\n' | sed 's/^/ - /'
echo "📋 Build type: $BUILD_TYPE"
echo "🔖 Version: $VERSION"
- name: Build and push Docker image
sudo apt-get update
sudo apt-get install -y skopeo
- uses: docker/setup-qemu-action@c7c53464625b32c7a7e944ae62b3e17d2b600130 # v3
if: steps.reuse.outputs.reused != 'true'
- uses: docker/setup-buildx-action@8d2750c68a42422c14e847fe6c8ac0403b4cbd6f # v3
if: steps.reuse.outputs.reused != 'true'
- name: Supply accepted binaries and pinned CLI
if: steps.reuse.outputs.reused != 'true'
env:
GH_TOKEN: ${{ github.token }}
run: python3 scripts/release/deliver.py image-prepare --target "$VARIANT"
- name: Build OCI archive without registry writes
if: steps.reuse.outputs.reused != 'true'
uses: docker/build-push-action@10e90e3645eae34f1e60eeb005ba3a3d33f178e8 # v6
with:
context: .
file: ${{ matrix.file }}
platforms: ${{ env.DOCKER_PLATFORMS }}
push: ${{ needs.build-check.outputs.should_push == 'true' }}
# Push to staging tags only: official tags are created by the
# certification-gated promotion step below from this same digest.
tags: ${{ steps.meta.outputs.staging_tags }}
labels: ${{ steps.meta.outputs.labels }}
# No layer cache. This build compiles nothing — it downloads a
# release zip and runs apk/apt — so the cache could only save the
# minute or two those take, while creating a correctness problem: with
# RELEASE=latest the binary URL is resolved by curl *inside* a RUN
# layer, and the layer key does not include what that resolved to. A
# rebuild at the same RELEASE value (dispatch with version=latest, or
# a re-run of the same version) would hit the old layer and ship the
# previous release's binary. mode=max also consumed the same 10GB
# Actions cache quota the Rust lanes are fighting over.
#
# Only RELEASE is passed: it is the sole build-arg the Dockerfiles
# declare besides TARGETARCH. BUILDTIME, VERSION, BUILD_TYPE, REVISION
# and CHANNEL were never read by any stage (and BUILDTIME's $(date ...)
# was a literal here, not a shell substitution). BUILD_DATE and VCS_REF
# are declared by the Dockerfiles but deliberately left unset —
# supplying them would change the published image labels.
platforms: linux/amd64,linux/arm64
push: false
provenance: false
outputs: type=oci,dest=delivery/image-${{ matrix.variant }}.tar
build-args: |
RELEASE=${{ steps.meta.outputs.docker_release }}
provenance: true
sbom: true
# Add retry mechanism by splitting the build process
no-cache: false
pull: true
# A later-built wrapper: the staged image must complete this platform's
# own binary identity verification before any official tag is written.
# It may not inherit an unverified package's conclusion.
- name: Verify staged image against certification
if: needs.build-check.outputs.should_push == 'true'
env:
GH_TOKEN: ${{ secrets.RELEASE_VALIDATION_TOKEN || github.token }}
TRIGGERING_RUN_ID: ${{ github.event.workflow_run.id }}
VERSION: ${{ needs.build-check.outputs.version }}
shell: bash
run: |
set -euo pipefail
mkdir -p certification
run_id="$TRIGGERING_RUN_ID"
if [[ -z "$run_id" ]]; then
run_id="$(gh api --method GET \
"repos/${GITHUB_REPOSITORY}/actions/workflows/build.yml/runs" \
-f branch="$VERSION" -f status=success -F per_page=1 \
--jq '.workflow_runs[0].id // empty' || true)"
fi
gh run download "$run_id" -n release-certification -D certification
STAGING_REF="${{ env.REGISTRY_DOCKERHUB }}:staging-${GITHUB_RUN_ID}-${GITHUB_RUN_ATTEMPT}${{ matrix.suffix }}"
case "${{ matrix.variant }}" in
musl) libc="musl" ;;
glibc) libc="gnu" ;;
esac
for platform in linux/amd64 linux/arm64; do
case "$platform" in
linux/amd64) arch_sub="x86_64" ;;
linux/arm64) arch_sub="aarch64" ;;
esac
docker pull --platform "$platform" "$STAGING_REF"
container="$(docker create --platform "$platform" "$STAGING_REF")"
docker cp "$container:/usr/bin/rustfs" "$RUNNER_TEMP/staged-rustfs"
docker rm "$container" >/dev/null
actual="$(sha256sum "$RUNNER_TEMP/staged-rustfs" | awk '{print $1}')"
expected="$(jq -r --arg needle "linux-${arch_sub}-${libc}" \
'[.deliverables[] | select(.name | contains($needle)) | .binary_sha256][0] // empty' \
certification/candidate.json)"
if [[ -z "$expected" || "$actual" != "$expected" ]]; then
echo "Staged image binary for ${platform} does not match the certified binary digest" >&2
exit 1
fi
python3 scripts/release/release_gate.py verify-promotion \
--plan .config/release-validation-plan.json \
--candidate certification/candidate.json \
--summary certification/release-gate-output \
--artifact "$RUNNER_TEMP/staged-rustfs" \
--artifact-role binary
done
- name: Promote staged image to official tags
if: needs.build-check.outputs.should_push == 'true'
shell: bash
run: |
set -euo pipefail
STAGING_REF="${{ env.REGISTRY_DOCKERHUB }}:staging-${GITHUB_RUN_ID}-${GITHUB_RUN_ATTEMPT}${{ matrix.suffix }}"
IFS=',' read -r -a official_tags <<< "${{ steps.meta.outputs.tags }}"
for tag in "${official_tags[@]}"; do
docker buildx imagetools create -t "$tag" "$STAGING_REF"
echo "Promoted $tag -> $STAGING_REF"
done
# Downstream consumers (Helm) verify this record instead of inheriting
# an unverified wrapper's conclusion.
- name: Record image promotion
if: needs.build-check.outputs.should_push == 'true'
shell: bash
env:
GH_TOKEN: ${{ secrets.RELEASE_VALIDATION_TOKEN || github.token }}
TRIGGERING_RUN_ID: ${{ github.event.workflow_run.id }}
VERSION: ${{ needs.build-check.outputs.version }}
run: |
set -euo pipefail
mkdir -p certification
run_id="$TRIGGERING_RUN_ID"
if [[ -z "$run_id" ]]; then
run_id="$(gh api --method GET \
"repos/${GITHUB_REPOSITORY}/actions/workflows/build.yml/runs" \
-f branch="$VERSION" -f status=success -F per_page=1 \
--jq '.workflow_runs[0].id // empty' || true)"
fi
gh run download "$run_id" -n release-certification -D certification
STAGING_REF="${{ env.REGISTRY_DOCKERHUB }}:staging-${GITHUB_RUN_ID}-${GITHUB_RUN_ATTEMPT}${{ matrix.suffix }}"
digest="$(docker buildx imagetools inspect "$STAGING_REF" --format '{{json .Manifest}}' | jq -r '.digest')"
candidate_sha="$(python3 -c "import json,hashlib,sys; sys.path.insert(0,'scripts/release'); import release_gate; print(release_gate.canonical_sha256(json.load(open('certification/candidate.json'))))")"
jq -n \
--arg version "$VERSION" \
--arg digest "$digest" \
--arg staging_ref "$STAGING_REF" \
--arg candidate_sha256 "$candidate_sha" \
--arg tags "${{ steps.meta.outputs.tags }}" \
--slurpfile candidate certification/candidate.json \
'{schema_version: 1, version: $version, image_digest: $digest, staging_ref: $staging_ref,
promoted_tags: ($tags | split(",")), candidate_sha256: $candidate_sha256,
binary_sha256s: ($candidate[0].deliverables | map(.binary_sha256) | unique)}' \
> image-promotion-record.json
- name: Upload image promotion record
if: needs.build-check.outputs.should_push == 'true'
uses: actions/upload-artifact@043fb46d1a93c77aae656e7c1c64a875d1fc6a0a # v7.0.1
with:
name: image-promotion-record${{ matrix.suffix }}
path: image-promotion-record.json
if-no-files-found: error
retention-days: 30
# Note: Manifest creation is no longer needed as we only build one variant
# Multi-arch manifests are automatically created by docker/build-push-action
# Report-only container image vulnerability scan
scan-docker-image:
name: Scan Docker Images
needs: [ build-check, build-docker ]
if: needs.build-check.outputs.should_build == 'true' && needs.build-check.outputs.should_push == 'true'
runs-on: ubuntu-latest
timeout-minutes: 30
permissions:
contents: read
security-events: write
strategy:
fail-fast: false
matrix:
include:
- variant: musl
suffix: ""
- variant: glibc
suffix: "-glibc"
steps:
- name: Login to GitHub Container Registry
# docker/login-action v3
uses: docker/login-action@c94ce9fb468520275223c153574b00df6fe4bcc9
with:
registry: ghcr.io
username: ${{ secrets.GHCR_USERNAME }}
password: ${{ secrets.GHCR_PASSWORD }}
- name: Scan image with Trivy
# aquasecurity/trivy-action v0.36.0
BINARY_SOURCE=supplied
RELEASE=${{ fromJson(inputs.candidate_manifest).version }}
VCS_REF=${{ fromJson(inputs.candidate_manifest).source_sha }}
- name: Verify wrapped binaries and startup on both architectures
if: steps.reuse.outputs.reused != 'true'
run: python3 scripts/release/deliver.py image-smoke --target "$VARIANT"
- name: Scan amd64 image before sealing
if: steps.reuse.outputs.reused != 'true'
uses: aquasecurity/trivy-action@ed142fd0673e97e23eac54620cfb913e5ce36c25
env:
TRIVY_PLATFORM: linux/amd64
with:
image-ref: ${{ env.REGISTRY_GHCR }}:${{ needs.build-check.outputs.version }}${{ matrix.suffix }}
input: delivery-scan/image-${{ matrix.variant }}-amd64.tar
format: sarif
output: trivy-${{ matrix.variant }}.sarif
ignore-unfixed: true
vuln-type: os,library
severity: CRITICAL,HIGH
output: delivery/scan-${{ matrix.variant }}-amd64.sarif
exit-code: "0" # Keep the repository's report-only vulnerability policy.
- name: Scan arm64 image before sealing
if: steps.reuse.outputs.reused != 'true'
uses: aquasecurity/trivy-action@ed142fd0673e97e23eac54620cfb913e5ce36c25
env:
TRIVY_PLATFORM: linux/arm64
with:
input: delivery-scan/image-${{ matrix.variant }}-arm64.tar
format: sarif
output: delivery/scan-${{ matrix.variant }}-arm64.sarif
exit-code: "0"
# Surface findings in the Security tab; an artifact alone is write-only
# reporting nobody reviews.
- name: Upload scan results to code scanning
uses: github/codeql-action/upload-sarif@99df26d4f13ea111d4ec1a7dddef6063f76b97e9 # v4.37.0
- name: Record verified OCI identity
if: steps.reuse.outputs.reused != 'true'
run: python3 scripts/release/deliver.py image-record --target "$VARIANT"
- uses: actions/upload-artifact@043fb46d1a93c77aae656e7c1c64a875d1fc6a0a # v7.0.1
if: steps.reuse.outputs.reused != 'true'
with:
sarif_file: trivy-${{ matrix.variant }}.sarif
category: container-image-${{ matrix.variant }}
- name: Upload container scan report
uses: actions/upload-artifact@043fb46d1a93c77aae656e7c1c64a875d1fc6a0a # v7.0.1
with:
name: container-image-scan-${{ matrix.variant }}
path: trivy-${{ matrix.variant }}.sarif
name: image-${{ matrix.variant }}
path: delivery/*
if-no-files-found: error
retention-days: 30
# Docker build summary
docker-summary:
name: Docker Build Summary
needs: [ build-check, build-docker ]
if: always() && needs.build-check.outputs.should_build == 'true'
runs-on: ubuntu-latest
timeout-minutes: 10
steps:
- name: Docker build completion summary
run: |
BUILD_TYPE="${{ needs.build-check.outputs.build_type }}"
VERSION="${{ needs.build-check.outputs.version }}"
CREATE_LATEST="${{ needs.build-check.outputs.create_latest }}"
echo "🐳 Docker build completed successfully!"
echo "📦 Build type: $BUILD_TYPE"
echo "🔢 Version: $VERSION"
echo "🚀 Strategy: Images using pre-built binaries (release channel only)"
echo ""
case "$BUILD_TYPE" in
"preview")
echo "🔍 Preview Docker image has been built with ${VERSION} tags"
echo "⏭️ Preview images do not update latest or prerelease channels"
;;
"release")
echo "🚀 Release Docker image has been built with ${VERSION} tags"
echo "✅ This image is ready for production use"
if [[ "$CREATE_LATEST" == "true" ]]; then
echo "🏷️ Latest tag has been created for stable release"
fi
;;
"prerelease")
echo "🧪 Prerelease Docker image has been built with ${VERSION} tags"
echo "⚠️ This is a prerelease image - use with caution"
# Prereleases move latest until the first stable tag exists (pre-GA policy).
if [[ "$CREATE_LATEST" == "true" ]]; then
echo "🏷️ Latest tag has been created for prerelease: $VERSION"
else
echo "🚫 Latest tag NOT created for prerelease"
fi
;;
*)
echo "❌ Unexpected build type: $BUILD_TYPE"
;;
esac
compression-level: 0
+51 -277
View File
@@ -1,229 +1,26 @@
# Copyright 2024 RustFS Team
#
# Licensed under the Apache License, Version 2.0 (the "License");
# you may not use this file except in compliance with the License.
# You may obtain a copy of the License at
#
# http://www.apache.org/licenses/LICENSE-2.0
#
# Unless required by applicable law or agreed to in writing, software
# distributed under the License is distributed on an "AS IS" BASIS,
# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
# See the License for the specific language governing permissions and
# limitations under the License.
name: Publish helm chart to artifacthub
# Copyright 2026 RustFS Team
# SPDX-License-Identifier: Apache-2.0
name: Build candidate Helm chart
on:
workflow_run:
workflows: [ "Build and Release" ]
types: [ completed ]
workflow_dispatch:
workflow_call:
inputs:
version:
description: "Release version to publish, e.g. 1.0.0-beta.1 or v1.0.0-beta.1"
required: true
default: "1.0.0-beta.1"
candidate_manifest:
type: string
required: true
permissions:
contents: read
actions: read
jobs:
# Consume the same trusted release certification as every other publication
# exit and verify the image promotion record of the Docker workflow. The
# chart wraps no binary of its own, so it may only reference an image whose
# platform verification already passed; it must not inherit an unverified
# package's conclusion. A run without that evidence fails closed here and
# never reaches the chart repository.
release-certification:
package:
runs-on: ubuntu-latest
timeout-minutes: 30
permissions:
contents: read
actions: read
outputs:
version: ${{ steps.version.outputs.raw_tag }}
timeout-minutes: 15
steps:
- name: Checkout repository
uses: actions/checkout@f548e57e544e1ff5a4c46bf1e1b8685f8e4a348a # v7
- uses: actions/checkout@f548e57e544e1ff5a4c46bf1e1b8685f8e4a348a # v7
with:
ref: ${{ fromJson(inputs.candidate_manifest).source_sha }}
persist-credentials: false
- name: Normalize release version
id: version
env:
RAW_INPUT: ${{ github.event.inputs.version }}
RAW_BRANCH: ${{ github.event.workflow_run.head_branch }}
run: |
set -euo pipefail
if [ "${{ github.event_name }}" = "workflow_dispatch" ]; then
RAW="$RAW_INPUT"
else
RAW="$RAW_BRANCH"
fi
case "$RAW" in
refs/tags/*) RAW_TAG="${RAW#refs/tags/}" ;;
*) RAW_TAG="$RAW" ;;
esac
echo "raw_tag=$RAW_TAG" >> "$GITHUB_OUTPUT"
- name: Download and verify release certification and image record
env:
GH_TOKEN: ${{ secrets.RELEASE_VALIDATION_TOKEN || github.token }}
TRIGGERING_RUN_ID: ${{ github.event.workflow_run.id }}
RAW_TAG: ${{ steps.version.outputs.raw_tag }}
shell: bash
run: |
set -euo pipefail
mkdir -p certification image-record
build_run="$TRIGGERING_RUN_ID"
if [[ -z "$build_run" ]]; then
build_run="$(gh api --method GET \
"repos/${GITHUB_REPOSITORY}/actions/workflows/build.yml/runs" \
-f branch="$RAW_TAG" -f status=success -F per_page=1 \
--jq '.workflow_runs[0].id // empty' || true)"
fi
if [[ -z "$build_run" ]]; then
echo "No certified Build and Release run found for ${RAW_TAG}; failing closed" >&2
exit 1
fi
if ! gh run download "$build_run" -n release-certification -D certification; then
echo "Build run ${build_run} published no release-certification artifact; failing closed" >&2
exit 1
fi
python3 scripts/release/release_gate.py verify-promotion \
--plan .config/release-validation-plan.json \
--candidate certification/candidate.json \
--summary certification/release-gate-output \
--artifact-sha256 "$(jq -r '.deliverables[0].sha256' certification/candidate.json)" \
--artifact-name "$(jq -r '.deliverables[0].name' certification/candidate.json)"
docker_run="$(gh api --method GET \
"repos/${GITHUB_REPOSITORY}/actions/workflows/docker.yml/runs" \
-f branch="$RAW_TAG" -f status=success -F per_page=1 \
--jq '.workflow_runs[0].id // empty' || true)"
if [[ -z "$docker_run" ]] || ! gh run download "$docker_run" -n image-promotion-record -D image-record; then
echo "No image promotion record for ${RAW_TAG}; a chart may not wrap an unverified image" >&2
exit 1
fi
candidate_sha="$(python3 -c "import json,hashlib,sys; sys.path.insert(0,'scripts/release'); import release_gate; print(release_gate.canonical_sha256(json.load(open('certification/candidate.json'))))")"
record_sha="$(jq -r '.candidate_sha256 // empty' image-record/image-promotion-record.json)"
if [[ "$record_sha" != "$candidate_sha" ]]; then
echo "Image promotion record does not belong to this certification" >&2
exit 1
fi
overlap="$(jq -r --slurpfile candidate certification/candidate.json \
'[.binary_sha256s[]] - ($candidate[0].deliverables | map(.binary_sha256)) | length' \
image-record/image-promotion-record.json)"
if [[ "$overlap" -ne 0 ]]; then
echo "Image promotion record carries binary digests outside the certification" >&2
exit 1
fi
build-helm-package:
runs-on: ubuntu-latest
timeout-minutes: 30
needs: [ release-certification ]
if: |
(github.event_name == 'workflow_dispatch' && !contains(github.event.inputs.version, '-preview')) ||
(
github.event.workflow_run.conclusion == 'success' &&
github.event.workflow_run.event == 'push' &&
contains(github.event.workflow_run.head_branch, '.') &&
!contains(github.event.workflow_run.head_branch, '-preview')
)
outputs:
raw_tag: ${{ steps.version.outputs.raw_tag }}
app_version: ${{ steps.version.outputs.app_version }}
chart_version: ${{ steps.version.outputs.chart_version }}
steps:
- name: Checkout release source
uses: actions/checkout@f548e57e544e1ff5a4c46bf1e1b8685f8e4a348a # v7
with:
ref: ${{ github.event.workflow_run.head_sha || (startsWith(inputs.version, 'refs/tags/') && inputs.version || format('refs/tags/{0}', inputs.version)) }}
persist-credentials: false
# Both inputs reach the shell through env rather than `${{ }}`
# interpolation. A git ref name may contain `$(...)` — anything without a
# space is a legal tag — and interpolation pastes it into the script
# verbatim, where bash would run it. Reading "$RAW_INPUT" instead makes it
# data.
- name: Normalize release version
id: version
env:
RAW_INPUT: ${{ github.event.inputs.version }}
RAW_BRANCH: ${{ github.event.workflow_run.head_branch }}
run: |
set -eux
if [ "${{ github.event_name }}" = "workflow_dispatch" ]; then
RAW="$RAW_INPUT"
else
RAW="$RAW_BRANCH"
fi
case "$RAW" in
refs/tags/*)
RAW_TAG="${RAW#refs/tags/}"
;;
*)
RAW_TAG="$RAW"
;;
esac
./scripts/helm_chart_version.sh "$RAW_TAG"
# Docker and Helm share the binary-build trigger. Do not publish a chart
# until the image selected by its default appVersion is pullable.
- name: Wait for release image
timeout-minutes: 20
env:
APP_VERSION: ${{ steps.version.outputs.app_version }}
run: |
set -euo pipefail
if [[ ! "$APP_VERSION" =~ ^[0-9]+\.[0-9]+\.[0-9]+(-[0-9A-Za-z.-]+)?$ ]] ||
[[ "$APP_VERSION" == *-preview* ]]; then
echo "Invalid deliverable image version: $APP_VERSION" >&2
exit 1
fi
for attempt in {1..60}; do
if token=$(curl --fail --silent --show-error --connect-timeout 10 --max-time 20 \
'https://auth.docker.io/token?service=registry.docker.io&scope=repository:rustfs/rustfs:pull' \
| jq -er '.token // empty') &&
curl --fail --silent --show-error --head --connect-timeout 10 --max-time 20 \
--header "Authorization: Bearer $token" \
--header 'Accept: application/vnd.oci.image.index.v1+json, application/vnd.docker.distribution.manifest.list.v2+json' \
"https://registry-1.docker.io/v2/rustfs/rustfs/manifests/$APP_VERSION" > /dev/null; then
echo "Release image is available: rustfs/rustfs:$APP_VERSION"
exit 0
fi
echo "Waiting for rustfs/rustfs:$APP_VERSION ($attempt/60)"
if (( attempt < 60 )); then
sleep 15
fi
done
echo "Release image is unavailable; refusing to publish Helm chart" >&2
exit 1
- name: Replace chart version and app version
env:
CHART_VERSION: ${{ steps.version.outputs.chart_version }}
APP_VERSION: ${{ steps.version.outputs.app_version }}
run: |
set -eux
sed -i -E "s/^version:.*/version: \"${CHART_VERSION}\"/" helm/rustfs/Chart.yaml
sed -i -E "s/^appVersion:.*/appVersion: \"${APP_VERSION}\"/" helm/rustfs/Chart.yaml
- name: Set up Helm
uses: azure/setup-helm@b9e51907a09c216f16ebe8536097933489208112 # v4.3.0
- name: Install yq
- uses: azure/setup-helm@b9e51907a09c216f16ebe8536097933489208112 # v4.3.0
- name: Install pinned yq
run: |
set -euo pipefail
yq_version="4.54.1"
@@ -238,66 +35,43 @@ jobs:
echo "$tools_dir" >> "$GITHUB_PATH"
"$tools_dir/yq" --version
- name: Test Helm Chart Templates
run: ./scripts/test_helm_templates.sh
- name: Package Helm Chart
run: |
set -eux
cp helm/README.md helm/rustfs/
helm package ./helm/rustfs \
--destination helm/rustfs/ \
--version "${{ steps.version.outputs.chart_version }}"
- name: Upload helm package as artifact
uses: actions/upload-artifact@043fb46d1a93c77aae656e7c1c64a875d1fc6a0a # v7.0.1
with:
name: helm-package
path: |
helm/rustfs/*.tgz
helm/rustfs/README.md
retention-days: 1
publish-helm-package:
runs-on: ubuntu-latest
timeout-minutes: 30
needs: [ build-helm-package ]
if: needs.build-helm-package.result == 'success'
steps:
- name: Checkout helm package repo
uses: actions/checkout@f548e57e544e1ff5a4c46bf1e1b8685f8e4a348a # v7
with:
# persist-credentials-exempt: this checkout's token IS the push credential —
# the job git-pushes to rustfs/helm below. Clearing it breaks chart publishing.
repository: rustfs/helm
token: ${{ secrets.RUSTFS_HELM_PACKAGE }}
- name: Download helm package
uses: actions/download-artifact@3e5f45b2cfb9172054b4087a40e8e0b5a5461e7c # v8.0.1
with:
name: helm-package
path: ./
- name: Set up helm
uses: azure/setup-helm@b9e51907a09c216f16ebe8536097933489208112 # v4.3.0
- name: Generate index
run: helm repo index . --url https://charts.rustfs.com
# app_version is derived from the triggering tag name, and this job holds
# the cross-repository push token with rustfs/helm already checked out —
# the worst place in the repo to paste an attacker-influenced string into
# a shell line. Passed through env so bash treats it as data.
- name: Push helm package and index file
- name: Test and package the exact application version
id: chart
env:
GIT_USERNAME: ${{ secrets.USERNAME }}
GIT_EMAIL: ${{ secrets.EMAIL_ADDRESS }}
APP_VERSION: ${{ needs.build-helm-package.outputs.app_version }}
GH_TOKEN: ${{ github.token }}
CANDIDATE_MANIFEST: ${{ inputs.candidate_manifest }}
run: |
set -eux
git config --global user.name "${GIT_USERNAME}"
git config --global user.email "${GIT_EMAIL}"
git add .
git commit -m "Update rustfs helm package with ${APP_VERSION}." || echo "No changes to commit"
git push origin main
python3 - <<'PYCHART'
import sys, subprocess, shutil
from pathlib import Path
sys.path.insert(0, 'scripts/release')
import candidate as c, pipeline
inputs = pipeline.manifest_env()
current = pipeline.current_run('release-complete.yml')
if any(a['name'] == 'helm-package' for a in c.artifact_list(current['id'])):
c.download_artifact(current, 'helm-package', 'chart')
record = c.read_json('chart/chart.json')
c.require(record['inputs_sha256'] == c.digest(inputs), 'chart belongs to another candidate')
c.emit_output('reused', 'true')
else:
chart = Path('helm/rustfs/Chart.yaml')
import re
value = chart.read_text()
for key in ('version', 'appVersion'):
value, count = re.subn(rf'(?m)^{key}:.*$', f'{key}: "{inputs["version"]}"', value)
c.require(count == 1, 'ambiguous chart metadata')
chart.write_text(value)
subprocess.run(['bash', 'scripts/test_helm_templates.sh'], check=True)
shutil.copyfile('helm/README.md', 'helm/rustfs/README.md')
subprocess.run(['helm', 'package', 'helm/rustfs', '--destination', 'chart'], check=True)
path = Path('chart') / f"rustfs-{inputs['version']}.tgz"
c.write_json('chart/chart.json', {'version': inputs['version'], 'inputs_sha256': c.digest(inputs),
'name': path.name, 'sha256': c.file_digest(path), 'size': path.stat().st_size})
PYCHART
- uses: actions/upload-artifact@043fb46d1a93c77aae656e7c1c64a875d1fc6a0a # v7.0.1
if: steps.chart.outputs.reused != 'true'
with:
name: helm-package
path: chart/*
if-no-files-found: error
retention-days: 30
+2 -2
View File
@@ -47,8 +47,8 @@ permissions:
contents: read
concurrency:
group: oidc-keycloak-live-${{ github.ref }}
cancel-in-progress: true
group: oidc-keycloak-live-${{ github.event_name }}-${{ github.ref }}
cancel-in-progress: ${{ github.event_name != 'schedule' }}
jobs:
oidc-keycloak-live:
+38 -825
View File
@@ -1,840 +1,53 @@
# Copyright 2024 RustFS Team
#
# Licensed under the Apache License, Version 2.0 (the "License");
# you may not use this file except in compliance with the License.
# You may obtain a copy of the License at
#
# http://www.apache.org/licenses/LICENSE-2.0
#
# Unless required by applicable law or agreed to in writing, software
# distributed under the License is distributed on an "AS IS" BASIS,
# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
# See the License for the specific language governing permissions and
# limitations under the License.
# Package Workflow - Build DEB/RPM packages
#
# This workflow builds DEB and RPM packages from pre-built Linux binaries
# and uploads them to Cloudflare R2 and the GitHub release.
#
# Trigger:
# - workflow_run: automatically package after "Build and Release" completes
# for a release tag (the mac/windows/linux binaries are already uploaded
# to the GitHub release before packaging starts)
# - workflow_dispatch: manual fallback with a release tag and/or exact build run ID
#
# Flow:
# 1. Resolve and validate the selected Build workflow run and source identity
# 2. Download Linux binaries (x86_64-gnu, aarch64-gnu, x86_64-musl,
# aarch64-musl) from build artifacts
# 3. Build DEB packages for amd64 and arm64, gnu and musl variants
# 4. Build RPM packages for x86_64 and aarch64, gnu and musl variants
# 5. Upload all packages to Cloudflare R2 and the GitHub release
name: Package DEB/RPM
permissions:
# contents: write is required to upload packages to the GitHub release
contents: write
actions: read
# Copyright 2026 RustFS Team
# SPDX-License-Identifier: Apache-2.0
name: Build candidate packages
on:
# Follows the same pattern as docker.yml: run after the release build
# workflow completes, so packaging is triggered only by release tags
# (e.g. 1.0.0-rc.2, 1.0.0-rc.3), never by development builds.
workflow_run:
workflows: [ "Build and Release" ]
types: [ completed ]
workflow_dispatch:
workflow_call:
inputs:
tag:
description: "Release tag to package (e.g. 1.0.0-beta.12). Leave empty for latest main build."
required: false
candidate_manifest:
type: string
build_run_id:
description: "Build workflow run ID (when combined with tag, both must identify the same release commit)"
required: false
type: string
concurrency:
group: ${{ github.workflow }}-${{ github.event.workflow_run.head_branch || github.event.inputs.tag || github.run_id }}
cancel-in-progress: true
env:
HEAD_BRANCH: ${{ github.event.workflow_run.head_branch }}
WORKFLOW_RUN_ID: ${{ github.event.workflow_run.id }}
required: true
permissions:
contents: read
actions: read
jobs:
# Resolve which build run to use and extract version info
resolve:
name: Resolve Build
# Auto-trigger only from successful tag builds of "Build and Release".
# Tag pushes arrive as event == push with head_branch != main (a
# non-main push head_branch is the release tag name). Manual dispatch
# stays available as a fallback for backfills and re-runs.
if: >-
github.event_name == 'workflow_dispatch' ||
(github.event.workflow_run.conclusion == 'success' &&
github.event.workflow_run.event == 'push' &&
github.event.workflow_run.head_branch != 'main')
runs-on: ubuntu-latest # gh CLI required; the sm-standard-2 fleet does not ship it
timeout-minutes: 10
outputs:
version: ${{ steps.resolve.outputs.version }}
build_type: ${{ steps.resolve.outputs.build_type }}
build_run_id: ${{ steps.resolve.outputs.build_run_id }}
build_run_number: ${{ steps.resolve.outputs.build_run_number }}
head_sha: ${{ steps.resolve.outputs.head_sha }}
dev_sequence: ${{ steps.resolve.outputs.dev_sequence }}
tag: ${{ steps.resolve.outputs.tag }}
steps:
- name: Resolve build run
id: resolve
shell: bash
env:
GH_TOKEN: ${{ github.token }}
EVENT_NAME: ${{ github.event_name }}
REPOSITORY: ${{ github.repository }}
INPUT_TAG: ${{ github.event.inputs.tag }}
INPUT_RUN_ID: ${{ github.event.inputs.build_run_id }}
run: |
set -euo pipefail
fail() {
echo "❌ $1" >&2
exit 1
}
TAG=""
BUILD_RUN_ID=""
case "$EVENT_NAME" in
workflow_run)
TAG="$HEAD_BRANCH"
BUILD_RUN_ID="$WORKFLOW_RUN_ID"
;;
workflow_dispatch)
TAG="$INPUT_TAG"
BUILD_RUN_ID="$INPUT_RUN_ID"
;;
*) fail "unsupported event: $EVENT_NAME" ;;
esac
# Validate and classify tags before using them in API paths or logs.
semver_core='(0|[1-9][0-9]*)\.(0|[1-9][0-9]*)\.(0|[1-9][0-9]*)'
prerelease_id='(alpha|beta|rc)\.(0|[1-9][0-9]*)'
if [[ -n "$TAG" ]]; then
if [[ "$TAG" =~ ^${semver_core}(-${prerelease_id})?-preview\.(0|[1-9][0-9]*)$ ]]; then
BUILD_TYPE=preview
elif [[ "$TAG" =~ ^${semver_core}-${prerelease_id}$ ]]; then
BUILD_TYPE=prerelease
elif [[ "$TAG" =~ ^${semver_core}$ ]]; then
BUILD_TYPE=release
else
fail "tag is not a supported strict package version"
fi
else
BUILD_TYPE=development
fi
if [[ -n "$BUILD_RUN_ID" ]]; then
[[ "$BUILD_RUN_ID" =~ ^[1-9][0-9]*$ ]] || fail "build run ID must be a positive decimal integer"
echo "Using selected build run: $BUILD_RUN_ID"
elif [[ -n "$TAG" ]]; then
echo "Looking for build run for tag: $TAG"
BUILD_RUN_ID=$(gh api --method GET \
"repos/${REPOSITORY}/actions/workflows/build.yml/runs" \
-f branch="$TAG" -f status=success -F per_page=1 \
--jq '.workflow_runs[0].id // empty' 2>/dev/null || true)
if [[ -z "$BUILD_RUN_ID" ]]; then
BUILD_RUN_ID=$(gh api --method GET \
"repos/${REPOSITORY}/actions/workflows/build.yml/runs" \
-f event=push -f status=success -F per_page=100 2>/dev/null |
jq -r --arg tag "$TAG" \
'[.workflow_runs[] | select(.head_branch == $tag)][0].id // empty' || true)
fi
[[ "$BUILD_RUN_ID" =~ ^[1-9][0-9]*$ ]] || fail "no successful build run found for tag"
echo "Found build run: $BUILD_RUN_ID"
else
echo "No tag specified, looking for latest main build"
BUILD_RUN_ID=$(gh api --method GET \
"repos/${REPOSITORY}/actions/workflows/build.yml/runs" \
-f branch=main -f status=success -F per_page=1 \
--jq '.workflow_runs[0].id // empty' 2>/dev/null || true)
[[ "$BUILD_RUN_ID" =~ ^[1-9][0-9]*$ ]] || fail "no successful main build found"
echo "Latest main build: $BUILD_RUN_ID"
fi
# Fetch once and use the same immutable run metadata for identity,
# ordering, workflow provenance, and release-channel validation.
RUN_JSON=$(gh api "repos/${REPOSITORY}/actions/runs/${BUILD_RUN_ID}") ||
fail "cannot read selected build run"
RUN_ID=$(jq -r '.id // empty' <<<"$RUN_JSON")
RUN_NUMBER=$(jq -r '.run_number // empty' <<<"$RUN_JSON")
RUN_STATUS=$(jq -r '.status // empty' <<<"$RUN_JSON")
RUN_CONCLUSION=$(jq -r '.conclusion // empty' <<<"$RUN_JSON")
RUN_PATH=$(jq -r '.path // empty' <<<"$RUN_JSON")
HEAD_SHA=$(jq -r '.head_sha // empty' <<<"$RUN_JSON")
RUN_HEAD_BRANCH=$(jq -r '.head_branch // empty' <<<"$RUN_JSON")
[[ "$RUN_ID" == "$BUILD_RUN_ID" ]] || fail "run metadata ID mismatch"
[[ "$RUN_NUMBER" =~ ^[1-9][0-9]*$ ]] || fail "build run number must be a positive decimal integer"
[[ "$RUN_STATUS" == completed && "$RUN_CONCLUSION" == success ]] || fail "selected build run is not successful"
[[ "$RUN_PATH" == .github/workflows/build.yml ]] || fail "selected run is not Build and Release"
[[ "$HEAD_SHA" =~ ^[0-9a-f]{40}$ ]] || fail "selected build run has an invalid head SHA"
[[ "$RUN_HEAD_BRANCH" != *$'\n'* && -n "$RUN_HEAD_BRANCH" ]] || fail "selected build run has an invalid head branch"
if [[ -n "$TAG" ]]; then
[[ "$RUN_HEAD_BRANCH" == "$TAG" ]] || fail "tag and build run head branch do not match"
TAG_REF_JSON=$(gh api "repos/${REPOSITORY}/git/ref/tags/${TAG}") ||
fail "cannot resolve release tag ref"
TAG_OBJECT_TYPE=$(jq -r '.object.type // empty' <<<"$TAG_REF_JSON")
TAG_OBJECT_SHA=$(jq -r '.object.sha // empty' <<<"$TAG_REF_JSON")
depth=0
while [[ "$TAG_OBJECT_TYPE" == tag && $depth -lt 5 ]]; do
TAG_OBJECT_JSON=$(gh api "repos/${REPOSITORY}/git/tags/${TAG_OBJECT_SHA}") ||
fail "cannot peel annotated release tag"
TAG_OBJECT_TYPE=$(jq -r '.object.type // empty' <<<"$TAG_OBJECT_JSON")
TAG_OBJECT_SHA=$(jq -r '.object.sha // empty' <<<"$TAG_OBJECT_JSON")
depth=$((depth + 1))
done
[[ "$TAG_OBJECT_TYPE" == commit && "$TAG_OBJECT_SHA" =~ ^[0-9a-f]{40}$ ]] ||
fail "release tag does not resolve to a commit"
[[ "$TAG_OBJECT_SHA" == "$HEAD_SHA" ]] || fail "release tag commit and build run head SHA do not match"
VERSION="$TAG"
DEV_SEQUENCE=""
else
VERSION="dev-${HEAD_SHA}"
DEV_SEQUENCE="$RUN_NUMBER"
fi
{
echo "version=$VERSION"
echo "build_type=$BUILD_TYPE"
echo "build_run_id=$BUILD_RUN_ID"
echo "build_run_number=$RUN_NUMBER"
echo "head_sha=$HEAD_SHA"
echo "dev_sequence=$DEV_SEQUENCE"
echo "tag=${TAG}"
} >> "$GITHUB_OUTPUT"
echo "📊 Resolved:"
echo " Version: $VERSION"
echo " Build type: $BUILD_TYPE"
echo " Build run ID: $BUILD_RUN_ID"
echo " Build run number: $RUN_NUMBER"
# Consume the trusted release certification of the resolved build run and
# verify the certified input digests before any package is wrapped. A run
# without that certification fails closed here; packages built later must
# complete their own verification and cannot inherit an unverified
# conclusion.
release-certification:
name: Release Certification
needs: resolve
if: needs.resolve.result == 'success'
runs-on: ubuntu-latest # gh CLI required; the sm-standard-2 fleet does not ship it
timeout-minutes: 30
permissions:
contents: read
actions: read
steps:
- name: Checkout repository
uses: actions/checkout@f548e57e544e1ff5a4c46bf1e1b8685f8e4a348a # v7
with:
persist-credentials: false
- name: Download and verify release certification
env:
GH_TOKEN: ${{ secrets.RELEASE_VALIDATION_TOKEN || github.token }}
BUILD_RUN_ID: ${{ needs.resolve.outputs.build_run_id }}
BUILD_TYPE: ${{ needs.resolve.outputs.build_type }}
shell: bash
run: |
set -euo pipefail
mkdir -p certification
if [[ "$BUILD_TYPE" == "development" ]]; then
# Development artifacts are explicitly uncertified and never touch
# official or latest channels; only the staging/dev paths remain.
jq -n '{schema_version: 1, certified: false, reason: "development artifacts are explicitly uncertified"}' \
> certification/development-uncertified.json
exit 0
fi
if ! gh run download "$BUILD_RUN_ID" -n release-certification -D certification; then
echo "Build run ${BUILD_RUN_ID} published no release-certification artifact; failing closed" >&2
exit 1
fi
python3 scripts/release/release_gate.py verify-promotion \
--plan .config/release-validation-plan.json \
--candidate certification/candidate.json \
--summary certification/release-gate-output \
--artifact-sha256 "$(jq -r '.deliverables[0].sha256' certification/candidate.json)" \
--artifact-name "$(jq -r '.deliverables[0].name' certification/candidate.json)"
# Build DEB and RPM packages for each architecture and libc variant
package:
name: Package (${{ matrix.arch }}-${{ matrix.libc }})
needs: [ resolve, release-certification ]
runs-on: ubuntu-latest # gh CLI required; the sm-standard-2 fleet does not ship it
timeout-minutes: 30
runs-on: ubuntu-latest
timeout-minutes: 45
strategy:
fail-fast: false
max-parallel: 1
matrix:
include:
- arch: x86_64
libc: gnu
deb_arch: amd64
rpm_arch: x86_64
artifact_name: "rustfs-linux-x86_64-gnu"
# gnu binaries dynamically link glibc; musl binaries are static
# and run without a libc dependency.
deb_depends: "libc6 (>= 2.31)"
rpm_depends: "glibc >= 2.31"
- arch: aarch64
libc: gnu
deb_arch: arm64
rpm_arch: aarch64
artifact_name: "rustfs-linux-aarch64-gnu"
deb_depends: "libc6 (>= 2.31)"
rpm_depends: "glibc >= 2.31"
- arch: x86_64
libc: musl
deb_arch: amd64
rpm_arch: x86_64
artifact_name: "rustfs-linux-x86_64-musl"
deb_depends: ""
rpm_depends: ""
- arch: aarch64
libc: musl
deb_arch: arm64
rpm_arch: aarch64
artifact_name: "rustfs-linux-aarch64-musl"
deb_depends: ""
rpm_depends: ""
target: [linux-x86_64-musl, linux-aarch64-musl, linux-x86_64-gnu, linux-aarch64-gnu]
steps:
- name: Checkout repository
uses: actions/checkout@f548e57e544e1ff5a4c46bf1e1b8685f8e4a348a # v7
- uses: actions/checkout@f548e57e544e1ff5a4c46bf1e1b8685f8e4a348a # v7
with:
ref: ${{ fromJson(inputs.candidate_manifest).source_sha }}
persist-credentials: false
- name: Normalize package metadata
id: versions
shell: bash
env:
BUILD_TYPE: ${{ needs.resolve.outputs.build_type }}
SOURCE_VERSION: ${{ needs.resolve.outputs.version }}
DEV_SEQUENCE: ${{ needs.resolve.outputs.dev_sequence }}
ARCH: ${{ matrix.rpm_arch }}
LIBC: ${{ matrix.libc }}
run: |
set -euo pipefail
normalized=$(./scripts/release/package_versions.sh \
"$BUILD_TYPE" "$SOURCE_VERSION" "$DEV_SEQUENCE" "$ARCH" "$LIBC")
printf '%s\n' "$normalized" >> "$GITHUB_OUTPUT"
- name: Download binary artifact from build run
uses: actions/download-artifact@3e5f45b2cfb9172054b4087a40e8e0b5a5461e7c # v8.0.1
with:
pattern: ${{ matrix.artifact_name }}*
path: ./binary-artifact
run-id: ${{ needs.resolve.outputs.build_run_id }}
github-token: ${{ github.token }}
merge-multiple: true
- name: Extract binary
id: binary
shell: bash
run: |
set -euo pipefail
ZIP_FILE=$(find ./binary-artifact -name "*.zip" -type f | head -1)
if [[ -z "$ZIP_FILE" ]]; then
echo "❌ No binary artifact found"
find ./binary-artifact -mindepth 1 -maxdepth 1 -print 2>/dev/null || true
exit 1
fi
echo "Found artifact: $ZIP_FILE"
mkdir -p ./bin
unzip -o "$ZIP_FILE" -d ./bin
if [[ ! -f ./bin/rustfs ]]; then
echo "❌ rustfs binary not found in archive"
exit 1
fi
chmod +x ./bin/rustfs
stat --printf='%n %s bytes\n' ./bin/rustfs
echo "✅ Binary extracted"
- name: Build DEB package
id: deb
shell: bash
env:
DEB_VERSION: ${{ steps.versions.outputs.deb_version }}
DEB_ARCH: ${{ matrix.deb_arch }}
DEB_FILE: ${{ steps.versions.outputs.deb_file }}
DEB_DEPENDS: ${{ matrix.deb_depends }}
LIBC: ${{ matrix.libc }}
run: |
set -euo pipefail
PKG_DIR="${DEB_FILE%.deb}"
echo "Building DEB: ${DEB_FILE}"
mkdir -p "${PKG_DIR}/DEBIAN"
mkdir -p "${PKG_DIR}/usr/bin"
mkdir -p "${PKG_DIR}/etc/default"
mkdir -p "${PKG_DIR}/lib/systemd/system"
mkdir -p "${PKG_DIR}/usr/share/doc/rustfs"
cp ./bin/rustfs "${PKG_DIR}/usr/bin/"
chmod 755 "${PKG_DIR}/usr/bin/rustfs"
cp deploy/build/rustfs.service "${PKG_DIR}/lib/systemd/system/"
cat > "${PKG_DIR}/etc/default/rustfs" << 'ENVEOF'
# RustFS Environment Configuration
# See https://docs.rustfs.com for more information
# RUSTFS_ACCESS_KEY=<changeme>
# RUSTFS_SECRET_KEY=<changeme>
# RUSTFS_VOLUMES=""
# RUSTFS_ADDRESS=":9000"
# RUSTFS_CONSOLE_ADDRESS=":9001"
# RUSTFS_CONSOLE_ENABLE=true
# RUSTFS_OBS_LOGGER_LEVEL=error
# RUSTFS_OBS_LOG_DIRECTORY="/var/log/rustfs/"
ENVEOF
cat > "${PKG_DIR}/DEBIAN/control" << EOF
Package: rustfs
Version: ${DEB_VERSION}
Section: utils
Priority: optional
Architecture: ${DEB_ARCH}
EOF
# gnu binaries dynamically link glibc and must declare it; musl
# binaries are statically linked and need no libc dependency.
if [[ -n "$DEB_DEPENDS" ]]; then
echo "Depends: ${DEB_DEPENDS}" >> "${PKG_DIR}/DEBIAN/control"
fi
cat >> "${PKG_DIR}/DEBIAN/control" << EOF
Maintainer: RustFS Team <support@rustfs.com>
Description: High-performance distributed object storage (${LIBC} build)
RustFS is a high-performance distributed object storage software
built using Rust. It is compatible with MinIO and S3 API.
Homepage: https://rustfs.com
EOF
# Declare /etc/default/rustfs as a conffile so dpkg preserves user
# modifications on upgrade instead of silently overwriting them.
cat > "${PKG_DIR}/DEBIAN/conffiles" << 'CONFFILES'
/etc/default/rustfs
CONFFILES
./scripts/release/package_service_scripts.sh after-install > "${PKG_DIR}/DEBIAN/postinst"
chmod 755 "${PKG_DIR}/DEBIAN/postinst"
./scripts/release/package_service_scripts.sh before-remove > "${PKG_DIR}/DEBIAN/prerm"
chmod 755 "${PKG_DIR}/DEBIAN/prerm"
./scripts/release/package_service_scripts.sh after-remove > "${PKG_DIR}/DEBIAN/postrm"
chmod 755 "${PKG_DIR}/DEBIAN/postrm"
cp LICENSE "${PKG_DIR}/usr/share/doc/rustfs/"
cp README.md "${PKG_DIR}/usr/share/doc/rustfs/"
fakeroot dpkg-deb --build "${PKG_DIR}" "$DEB_FILE"
[[ $(dpkg-deb -f "$DEB_FILE" Package) == rustfs ]]
[[ $(dpkg-deb -f "$DEB_FILE" Version) == "$DEB_VERSION" ]]
[[ $(dpkg-deb -f "$DEB_FILE" Architecture) == "$DEB_ARCH" ]]
dpkg-deb --fsys-tarfile "$DEB_FILE" | tar -tf - | grep -Fx './usr/bin/rustfs' >/dev/null
stat --printf='%n %s bytes\n' "$DEB_FILE"
echo "deb_file=$DEB_FILE" >> "$GITHUB_OUTPUT"
echo "✅ DEB built: $DEB_FILE"
- name: Build RPM package
id: rpm
shell: bash
env:
RPM_VERSION: ${{ steps.versions.outputs.rpm_version }}
RPM_RELEASE: ${{ steps.versions.outputs.rpm_release }}
RPM_ARCH: ${{ matrix.rpm_arch }}
RPM_FILE: ${{ steps.versions.outputs.rpm_file }}
RPM_DEPENDS: ${{ matrix.rpm_depends }}
LIBC: ${{ matrix.libc }}
run: |
set -euo pipefail
echo "Building RPM for ${RPM_ARCH} (${LIBC})"
sudo apt-get update && sudo apt-get install -y ruby ruby-dev build-essential rpm
sudo gem install fpm
./scripts/test_package_versions.sh --require-package-managers
# gnu binaries dynamically link glibc and must declare it; musl
# binaries are statically linked and need no libc dependency.
depends_args=()
if [[ -n "$RPM_DEPENDS" ]]; then
depends_args+=(--depends "$RPM_DEPENDS")
fi
# Create config file for fpm (DEB build creates it in its package dir structure,
# but fpm needs the file to exist before packaging)
mkdir -p ./tmp-pkg/etc/default
cat > ./tmp-pkg/etc/default/rustfs << 'ENVEOF'
# RustFS Environment Configuration
# See https://docs.rustfs.com for more information
# RUSTFS_ACCESS_KEY=<changeme>
# RUSTFS_SECRET_KEY=<changeme>
# RUSTFS_VOLUMES=""
# RUSTFS_ADDRESS=":9000"
# RUSTFS_CONSOLE_ADDRESS=":9001"
# RUSTFS_CONSOLE_ENABLE=true
# RUSTFS_OBS_LOGGER_LEVEL=error
# RUSTFS_OBS_LOG_DIRECTORY="/var/log/rustfs/"
ENVEOF
fpm -s dir -t rpm \
--name rustfs \
--version "$RPM_VERSION" \
--iteration "$RPM_RELEASE" \
--architecture "$RPM_ARCH" \
--package "$RPM_FILE" \
"${depends_args[@]}" \
--description "High-performance distributed object storage (${LIBC} build)" \
--maintainer "RustFS Team <support@rustfs.com>" \
--url "https://rustfs.com" \
--license "Apache-2.0" \
--before-install <(./scripts/release/package_service_scripts.sh rpm-before-install) \
--before-upgrade <(./scripts/release/package_service_scripts.sh rpm-before-upgrade) \
--after-install <(./scripts/release/package_service_scripts.sh after-install) \
--after-upgrade <(./scripts/release/package_service_scripts.sh after-install) \
--before-remove <(./scripts/release/package_service_scripts.sh before-remove) \
--after-remove <(./scripts/release/package_service_scripts.sh after-remove) \
--rpm-posttrans <(./scripts/release/package_service_scripts.sh rpm-posttrans) \
--config-files /etc/default/rustfs \
./bin/rustfs=/usr/bin/rustfs \
./tmp-pkg/etc/default/rustfs=/etc/default/rustfs \
deploy/build/rustfs.service=/lib/systemd/system/rustfs.service \
LICENSE=/usr/share/doc/rustfs/LICENSE \
README.md=/usr/share/doc/rustfs/README.md
if [[ ! -f "$RPM_FILE" ]]; then
echo "❌ RPM build failed"
exit 1
fi
RPM_METADATA=$(rpm -qp --qf '%{NAME}\n%{VERSION}\n%{RELEASE}\n%{ARCH}\n' "$RPM_FILE")
EXPECTED_METADATA=$(printf 'rustfs\n%s\n%s\n%s' "$RPM_VERSION" "$RPM_RELEASE" "$RPM_ARCH")
[[ "$RPM_METADATA" == "$EXPECTED_METADATA" ]]
rpm -qpl "$RPM_FILE" | grep -Fx '/usr/bin/rustfs' >/dev/null
stat --printf='%n %s bytes\n' "$RPM_FILE"
echo "rpm_file=$RPM_FILE" >> "$GITHUB_OUTPUT"
echo "✅ RPM built: $RPM_FILE"
- name: Download certification evidence
if: needs.resolve.outputs.build_type != 'development'
uses: actions/download-artifact@3e5f45b2cfb9172054b4087a40e8e0b5a5461e7c # v8.0.1
with:
name: release-certification
path: ./certification
# Later-built wrapper verification: the binary inside each package must
# match the certified binary digest for this platform before anything is
# uploaded, and the package itself stays on staging until promotion.
- name: Verify wrapped binary against certification
if: needs.resolve.outputs.build_type != 'development'
shell: bash
env:
DEB_FILE: ${{ steps.deb.outputs.deb_file }}
RPM_FILE: ${{ steps.rpm.outputs.rpm_file }}
ARTIFACT_NAME: ${{ matrix.artifact_name }}
run: |
set -euo pipefail
expected="$(jq -r --arg needle "$ARTIFACT_NAME" \
'[.deliverables[] | select(.name | contains($needle)) | .binary_sha256][0] // empty' \
certification/candidate.json)"
if [[ -z "$expected" ]]; then
echo "No certified binary digest recorded for ${ARTIFACT_NAME}" >&2
exit 1
fi
if ! command -v rpm2cpio >/dev/null 2>&1; then
sudo apt-get update && sudo apt-get install -y rpm2cpio
fi
if ! command -v cpio >/dev/null 2>&1; then
sudo apt-get update && sudo apt-get install -y cpio
fi
for package_file in "$DEB_FILE" "$RPM_FILE"; do
case "$package_file" in
*.deb)
dpkg-deb --fsys-tarfile "$package_file" | tar -xOf - ./usr/bin/rustfs > "$RUNNER_TEMP/wrapped-rustfs"
;;
*.rpm)
rpm2cpio "$package_file" | cpio -i --to-stdout ./usr/bin/rustfs > "$RUNNER_TEMP/wrapped-rustfs"
;;
esac
python3 scripts/release/release_gate.py verify-promotion \
--plan .config/release-validation-plan.json \
--candidate certification/candidate.json \
--summary certification/release-gate-output \
--artifact "$RUNNER_TEMP/wrapped-rustfs" \
--artifact-role binary
done
- name: Upload packages to artifacts
uses: actions/upload-artifact@043fb46d1a93c77aae656e7c1c64a875d1fc6a0a # v7.0.1
with:
name: packages-${{ matrix.arch }}-${{ matrix.libc }}
path: |
*.deb
*.rpm
retention-days: 30
- name: Upload packages to Cloudflare R2
if: needs.resolve.outputs.build_type != 'preview' && env.R2_ACCESS_KEY_ID != ''
env:
R2_ACCESS_KEY_ID: ${{ secrets.R2_ACCESS_KEY_ID }}
R2_SECRET_ACCESS_KEY: ${{ secrets.R2_SECRET_ACCESS_KEY }}
R2_ENDPOINT: ${{ secrets.R2_ENDPOINT }}
R2_BUCKET: ${{ secrets.R2_BUCKET }}
AWS_EC2_METADATA_DISABLED: true
BUILD_TYPE: ${{ needs.resolve.outputs.build_type }}
DEB_FILE: ${{ steps.deb.outputs.deb_file }}
RPM_FILE: ${{ steps.rpm.outputs.rpm_file }}
shell: bash
run: |
set -euo pipefail
if [[ -z "$R2_ACCESS_KEY_ID" || -z "$R2_SECRET_ACCESS_KEY" || -z "$R2_ENDPOINT" || -z "$R2_BUCKET" ]]; then
echo "⚠️ R2 credentials missing, skipping upload"
exit 0
fi
if ! command -v aws >/dev/null 2>&1; then
sudo apt-get update && sudo apt-get install -y awscli
fi
export AWS_ACCESS_KEY_ID="$R2_ACCESS_KEY_ID"
export AWS_SECRET_ACCESS_KEY="$R2_SECRET_ACCESS_KEY"
export AWS_DEFAULT_REGION="auto"
# Build-phase writes stay on staging for release types: the official
# packages/release and packages/latest paths are written only by the
# certification-gated promote-packages job, from these same staged
# objects. Development packages keep the uncertified dev channel.
if [[ "$BUILD_TYPE" == "development" ]]; then
R2_PREFIX="artifacts/rustfs/packages/dev"
else
R2_PREFIX="artifacts/rustfs/packages/staging/${GITHUB_RUN_ID}/${GITHUB_RUN_ATTEMPT}"
fi
R2_PATH="s3://${R2_BUCKET}/${R2_PREFIX}/"
echo "📤 Uploading to $R2_PATH"
for f in "$DEB_FILE" "$RPM_FILE"; do
if [[ -n "$f" && -f "$f" ]]; then
echo "Uploading: $f"
aws s3 cp "$f" "$R2_PATH" --endpoint-url "$R2_ENDPOINT" --only-show-errors
fi
done
echo "✅ Upload complete"
- name: Upload packages to GitHub Release
if: needs.resolve.outputs.tag != ''
- name: Reuse completed package set
id: reuse
env:
GH_TOKEN: ${{ github.token }}
TAG: ${{ needs.resolve.outputs.tag }}
DEB_FILE: ${{ steps.deb.outputs.deb_file }}
RPM_FILE: ${{ steps.rpm.outputs.rpm_file }}
shell: bash
CANDIDATE_MANIFEST: ${{ inputs.candidate_manifest }}
TARGET: ${{ matrix.target }}
run: python3 scripts/release/deliver.py reuse --kind packages --target "$TARGET"
- name: Install packaging tools
if: steps.reuse.outputs.reused != 'true'
run: |
set -euo pipefail
# Upload the packages, then refresh the release checksums so the new
# assets are covered, matching the binary release flow.
for f in "$DEB_FILE" "$RPM_FILE"; do
if [[ -n "$f" && -f "$f" ]]; then
echo "📤 Uploading $(basename "$f") to GitHub release ${TAG}..."
gh release upload "$TAG" "$f" --clobber
fi
done
CHECKSUM_DIR="$(mktemp -d)"
gh release download "$TAG" -p 'SHA256SUMS' -p 'SHA512SUMS' \
-D "$CHECKSUM_DIR" --clobber
for spec in "SHA256SUMS:sha256sum" "SHA512SUMS:sha512sum"; do
asset="${spec%%:*}"
checksum_cmd="${spec##*:}"
checksum_file="${CHECKSUM_DIR}/${asset}"
for f in "$DEB_FILE" "$RPM_FILE"; do
if [[ -n "$f" && -f "$f" ]]; then
base="$(basename "$f")"
# GitHub stores release asset names with '~' normalized to '.'.
# Current package file names embed the raw tag and contain no
# '~', but older releases may still ship such assets, so the
# checksum entries must reference the name as stored on the
# release.
github_base="${base//\~/.}"
# Remove any stale entry (both naming variants), then append
grep -Fv -- "$base" "$checksum_file" > "${checksum_file}.tmp" || true
grep -Fv -- "$github_base" "${checksum_file}.tmp" > "${checksum_file}.tmp2" || true
mv "${checksum_file}.tmp2" "$checksum_file"
digest=$("$checksum_cmd" -- "$f" | awk '{print $1}')
printf '%s %s\n' "$digest" "$github_base" >> "$checksum_file"
fi
done
echo "📤 Updating ${asset} for release ${TAG}..."
gh release upload "$TAG" "$checksum_file" --clobber
done
echo "✅ GitHub release assets updated"
# The only writer of the official packages/release and packages/latest R2
# paths. Promotion re-checks each package's wrapped binary against the
# passed certification list and copies the staged objects byte-for-byte:
# no rebuild-after-verify, no mutable latest resolution downstream.
promote-packages:
name: Promote Packages
needs: [ resolve, package, release-certification ]
# Target-channel lock for the official package paths.
concurrency:
group: release-channel-r2-packages
cancel-in-progress: false
if: needs.resolve.outputs.tag != '' && (needs.resolve.outputs.build_type == 'release' || needs.resolve.outputs.build_type == 'prerelease')
runs-on: ubuntu-latest # gh CLI required; the sm-standard-2 fleet does not ship it
timeout-minutes: 30
permissions:
contents: read
actions: read
env:
R2_ACCESS_KEY_ID: ${{ secrets.R2_ACCESS_KEY_ID }}
R2_SECRET_ACCESS_KEY: ${{ secrets.R2_SECRET_ACCESS_KEY }}
R2_ENDPOINT: ${{ secrets.R2_ENDPOINT }}
R2_BUCKET: ${{ secrets.R2_BUCKET }}
AWS_EC2_METADATA_DISABLED: true
steps:
- name: Checkout repository
uses: actions/checkout@f548e57e544e1ff5a4c46bf1e1b8685f8e4a348a # v7
with:
persist-credentials: false
- name: Download packages and certification
uses: actions/download-artifact@3e5f45b2cfb9172054b4087a40e8e0b5a5461e7c # v8.0.1
with:
path: ./packages
pattern: packages-*
merge-multiple: true
- name: Download certification evidence
uses: actions/download-artifact@3e5f45b2cfb9172054b4087a40e8e0b5a5461e7c # v8.0.1
with:
name: release-certification
path: ./certification
- name: Verify and promote staged packages
shell: bash
run: |
set -euo pipefail
if [[ -z "$R2_ACCESS_KEY_ID" || -z "$R2_SECRET_ACCESS_KEY" || -z "$R2_ENDPOINT" || -z "$R2_BUCKET" ]]; then
echo "❌ R2 credentials missing; refusing to promote to the official channel" >&2
exit 1
fi
STAGING_PREFIX="s3://${R2_BUCKET}/artifacts/rustfs/packages/staging/${GITHUB_RUN_ID}/${GITHUB_RUN_ATTEMPT}/"
RELEASE_PATH="s3://${R2_BUCKET}/artifacts/rustfs/packages/release/"
LATEST_PATH="s3://${R2_BUCKET}/artifacts/rustfs/packages/latest/"
if ! command -v aws >/dev/null 2>&1; then
sudo apt-get update && sudo apt-get install -y awscli
fi
export AWS_ACCESS_KEY_ID="$R2_ACCESS_KEY_ID"
export AWS_SECRET_ACCESS_KEY="$R2_SECRET_ACCESS_KEY"
export AWS_DEFAULT_REGION="auto"
if ! command -v rpm2cpio >/dev/null 2>&1; then
sudo apt-get install -y rpm2cpio
fi
if ! command -v cpio >/dev/null 2>&1; then
sudo apt-get install -y cpio
fi
promoted=0
for package_file in packages/*.deb packages/*.rpm; do
[[ -f "$package_file" ]] || continue
name="$(basename "$package_file")"
case "$package_file" in
*.deb)
dpkg-deb --fsys-tarfile "$package_file" | tar -xOf - ./usr/bin/rustfs > "$RUNNER_TEMP/wrapped-rustfs"
;;
*.rpm)
rpm2cpio "$package_file" | cpio -i --to-stdout ./usr/bin/rustfs > "$RUNNER_TEMP/wrapped-rustfs"
;;
esac
python3 scripts/release/release_gate.py verify-promotion \
--plan .config/release-validation-plan.json \
--candidate certification/candidate.json \
--summary certification/release-gate-output \
--artifact "$RUNNER_TEMP/wrapped-rustfs" \
--artifact-role binary
staged="$(mktemp)"
aws s3 cp "${STAGING_PREFIX}${name}" "$staged" --endpoint-url "$R2_ENDPOINT" --only-show-errors
staged_sha="$(sha256sum "$staged" | awk '{print $1}')"
built_sha="$(sha256sum "$package_file" | awk '{print $1}')"
if [[ "$staged_sha" != "$built_sha" ]]; then
echo "❌ Staged package ${name} does not match the verified bytes" >&2
exit 1
fi
aws s3 cp "$staged" "${RELEASE_PATH}${name}" --endpoint-url "$R2_ENDPOINT" --only-show-errors
aws s3 cp "$staged" "${LATEST_PATH}${name}" --endpoint-url "$R2_ENDPOINT" --only-show-errors
rm -f "$staged"
promoted=$((promoted + 1))
echo "✅ Promoted ${name} (${built_sha})"
done
if [[ "$promoted" -eq 0 ]]; then
echo "No staged packages found to promote" >&2
exit 1
fi
# Summary
summary:
name: Summary
needs: [ resolve, package ]
# Every completed Build and Release run triggers this workflow; when
# resolve declined it (not a release tag), there is nothing to summarise.
if: always() && needs.resolve.result != 'skipped'
runs-on: ubuntu-latest
timeout-minutes: 5
steps:
- name: Print summary
shell: bash
sudo apt-get update
sudo apt-get install -y ruby ruby-dev build-essential rpm fakeroot cpio
sudo gem install fpm --version 1.17.0 --no-document
- name: Package accepted executable and verify contents
if: steps.reuse.outputs.reused != 'true'
env:
SUMMARY_VERSION: ${{ needs.resolve.outputs.version }}
SUMMARY_BUILD_TYPE: ${{ needs.resolve.outputs.build_type }}
SUMMARY_BUILD_RUN_ID: ${{ needs.resolve.outputs.build_run_id }}
SUMMARY_PACKAGE_STATUS: ${{ needs.package.result }}
run: |
{
echo "## 📦 Package Summary"
echo ""
echo "| Item | Value |"
echo "|------|-------|"
echo "| Version | \`${SUMMARY_VERSION}\` |"
echo "| Build Type | ${SUMMARY_BUILD_TYPE} |"
echo "| Build Run | #${SUMMARY_BUILD_RUN_ID} |"
echo "| Package Status | ${SUMMARY_PACKAGE_STATUS} |"
} >> "$GITHUB_STEP_SUMMARY"
GH_TOKEN: ${{ github.token }}
CANDIDATE_MANIFEST: ${{ inputs.candidate_manifest }}
TARGET: ${{ matrix.target }}
run: python3 scripts/release/deliver.py package --target "$TARGET" --output delivery
- uses: actions/upload-artifact@043fb46d1a93c77aae656e7c1c64a875d1fc6a0a # v7.0.1
if: steps.reuse.outputs.reused != 'true'
with:
name: packages-${{ matrix.target }}
path: delivery/*
if-no-files-found: error
compression-level: 0
retention-days: 30
+133
View File
@@ -0,0 +1,133 @@
# Copyright 2026 RustFS Team
# SPDX-License-Identifier: Apache-2.0
name: Release candidate (Linux)
on:
workflow_dispatch:
inputs:
version:
description: Final product version (no preview suffix)
type: string
required: true
source_sha:
description: Full reviewed source commit
type: string
required: true
permissions:
contents: read
actions: read
concurrency:
group: release-candidate-${{ inputs.version }}
cancel-in-progress: false
jobs:
freeze:
if: github.repository == 'rustfs/rustfs' && github.ref == 'refs/heads/main'
runs-on: ubuntu-latest
timeout-minutes: 15
outputs:
manifest: ${{ steps.freeze.outputs.manifest }}
steps:
- uses: actions/checkout@f548e57e544e1ff5a4c46bf1e1b8685f8e4a348a # v7
with:
fetch-depth: 0
persist-credentials: false
- name: Freeze or recover original inputs
id: freeze
env:
GH_TOKEN: ${{ github.token }}
VERSION: ${{ inputs.version }}
SOURCE_SHA: ${{ inputs.source_sha }}
run: python3 scripts/release/pipeline.py prepare --version "$VERSION" --source "$SOURCE_SHA" --directory frozen
- uses: actions/upload-artifact@043fb46d1a93c77aae656e7c1c64a875d1fc6a0a # v7.0.1
if: steps.freeze.outputs.reused != 'true'
with:
name: release-inputs
path: frozen/inputs.json
if-no-files-found: error
retention-days: 30
linux:
needs: freeze
uses: ./.github/workflows/build.yml
with:
candidate_manifest: ${{ needs.freeze.outputs.manifest }}
platforms: linux
upgrade:
needs: [freeze, linux]
runs-on: ubuntu-latest
timeout-minutes: 150
env:
RUSTUP_TOOLCHAIN: ${{ fromJson(needs.freeze.outputs.manifest).toolchain }}
steps:
- uses: actions/checkout@f548e57e544e1ff5a4c46bf1e1b8685f8e4a348a # v7
with:
ref: ${{ fromJson(needs.freeze.outputs.manifest).workflow_sha }}
persist-credentials: false
- uses: ./.github/actions/setup
with:
rust-version: ${{ fromJson(needs.freeze.outputs.manifest).toolchain }}
cache-shared-key: release-upgrade-harness-hosted
cache-save-if: ${{ github.ref == 'refs/heads/main' }}
install-build-packaging-tools: "false"
install-test-tools: "false"
- name: Verify archived server against previous releases
env:
GH_TOKEN: ${{ github.token }}
CANDIDATE_MANIFEST: ${{ needs.freeze.outputs.manifest }}
run: |
python3 - <<'PYVERIFY'
import sys
from pathlib import Path
sys.path.insert(0, 'scripts/release')
import candidate as c, pipeline, verify
inputs = pipeline.manifest_env()
run = pipeline.current_run('release-candidate.yml')
if any(a['name'] == 'release-upgrade' for a in c.artifact_list(run['id'])):
c.download_artifact(run, 'release-upgrade', 'receipts')
pipeline.verify_evidence(inputs, Path('receipts'), {run['id']: run})
c.emit_output('reused', 'true')
else:
target = 'linux-x86_64-gnu'
c.download_artifact(run, f"candidate-{inputs['candidate_id']}-{target}", 'binary')
verify.upgrade(inputs, Path('binary') / f"rustfs-{target}-v{inputs['version']}.zip", Path('receipts'))
PYVERIFY
id: verify
- uses: actions/upload-artifact@043fb46d1a93c77aae656e7c1c64a875d1fc6a0a # v7.0.1
if: steps.verify.outputs.reused != 'true'
with:
name: release-upgrade
path: receipts/*
if-no-files-found: error
retention-days: 30
- uses: actions/upload-artifact@043fb46d1a93c77aae656e7c1c64a875d1fc6a0a # v7.0.1
if: failure()
with:
name: release-upgrade-failure-${{ github.run_attempt }}
path: receipts/*
retention-days: 7
seal-linux:
needs: [freeze, linux, upgrade]
runs-on: ubuntu-latest
timeout-minutes: 30
steps:
- uses: actions/checkout@f548e57e544e1ff5a4c46bf1e1b8685f8e4a348a # v7
with:
persist-credentials: false
- name: Collect trusted receipts and seal Linux bytes
id: seal
env:
GH_TOKEN: ${{ github.token }}
run: |
python3 scripts/release/pipeline.py linux-seal --directory sealed
{
echo '## Linux acceptance'
echo 'Use this run ID and manifest SHA-256 when starting Release candidate (complete).'
sha256sum sealed/linux-manifest.json
echo 'Record cluster, performance, integrity and upgrade acceptance in backlog before continuing.'
} >> "$GITHUB_STEP_SUMMARY"
- uses: actions/upload-artifact@043fb46d1a93c77aae656e7c1c64a875d1fc6a0a # v7.0.1
if: steps.seal.outputs.reused != 'true'
with:
name: release-linux
path: sealed/*
if-no-files-found: error
compression-level: 0
retention-days: 30
+97
View File
@@ -0,0 +1,97 @@
# Copyright 2026 RustFS Team
# SPDX-License-Identifier: Apache-2.0
name: Release candidate (complete)
on:
workflow_dispatch:
inputs:
candidate_run_id:
description: Successful Linux candidate run ID
type: string
required: true
linux_manifest_sha256:
description: Exact accepted Linux manifest SHA-256
type: string
required: true
acceptance_report:
description: Backlog report accepting cluster, performance, integrity and upgrade results
type: string
required: true
permissions:
contents: read
actions: read
concurrency:
group: release-complete-${{ inputs.candidate_run_id }}
cancel-in-progress: false
jobs:
acceptance:
if: github.repository == 'rustfs/rustfs' && github.ref == 'refs/heads/main'
runs-on: ubuntu-latest
timeout-minutes: 30
environment: release-linux-acceptance
outputs:
manifest: ${{ steps.accept.outputs.manifest }}
steps:
- uses: actions/checkout@f548e57e544e1ff5a4c46bf1e1b8685f8e4a348a # v7
with:
fetch-depth: 0
persist-credentials: false
- name: Bind acceptance to the exact Linux manifest
id: accept
env:
GH_TOKEN: ${{ github.token }}
CANDIDATE_RUN: ${{ inputs.candidate_run_id }}
MANIFEST_SHA: ${{ inputs.linux_manifest_sha256 }}
REPORT: ${{ inputs.acceptance_report }}
run: python3 scripts/release/pipeline.py complete-prepare --run-id "$CANDIDATE_RUN" --digest "$MANIFEST_SHA" --report "$REPORT" --directory linux
desktop:
needs: acceptance
uses: ./.github/workflows/build.yml
with:
candidate_manifest: ${{ needs.acceptance.outputs.manifest }}
platforms: desktop
packages:
needs: acceptance
uses: ./.github/workflows/package.yml
with:
candidate_manifest: ${{ needs.acceptance.outputs.manifest }}
images:
needs: acceptance
uses: ./.github/workflows/docker.yml
with:
candidate_manifest: ${{ needs.acceptance.outputs.manifest }}
chart:
needs: acceptance
uses: ./.github/workflows/helm-package.yml
with:
candidate_manifest: ${{ needs.acceptance.outputs.manifest }}
seal:
needs: [acceptance, desktop, packages, images, chart]
runs-on: ubuntu-latest
timeout-minutes: 45
steps:
- uses: actions/checkout@f548e57e544e1ff5a4c46bf1e1b8685f8e4a348a # v7
with:
fetch-depth: 0
persist-credentials: false
- name: Verify every deliverable and seal the candidate
id: seal
env:
GH_TOKEN: ${{ github.token }}
CANDIDATE_RUN: ${{ inputs.candidate_run_id }}
MANIFEST_SHA: ${{ inputs.linux_manifest_sha256 }}
REPORT: ${{ inputs.acceptance_report }}
run: |
python3 scripts/release/pipeline.py complete-seal --run-id "$CANDIDATE_RUN" --digest "$MANIFEST_SHA" --report "$REPORT" --directory sealed
{
echo '## Final publication approval'
echo 'Review this manifest, then explicitly dispatch Publish accepted release with its SHA-256.'
sha256sum sealed/release-manifest.json
} >> "$GITHUB_STEP_SUMMARY"
- uses: actions/upload-artifact@043fb46d1a93c77aae656e7c1c64a875d1fc6a0a # v7.0.1
if: steps.seal.outputs.reused != 'true'
with:
name: release-seal
path: sealed/*
if-no-files-found: error
compression-level: 0
retention-days: 30
+193
View File
@@ -0,0 +1,193 @@
# Copyright 2026 RustFS Team
# SPDX-License-Identifier: Apache-2.0
name: Publish accepted release
on:
workflow_dispatch:
inputs:
complete_run_id:
description: Successful completed candidate run ID
type: string
required: true
manifest_sha256:
description: Exact approved final manifest SHA-256 (authorizes publication)
type: string
required: true
permissions:
contents: read
actions: read
concurrency:
group: release-publish-${{ inputs.complete_run_id }}
cancel-in-progress: false
jobs:
verify:
if: github.repository == 'rustfs/rustfs' && github.ref == 'refs/heads/main'
runs-on: ubuntu-latest
timeout-minutes: 45
outputs:
version: ${{ steps.verify.outputs.version }}
steps:
- uses: actions/checkout@f548e57e544e1ff5a4c46bf1e1b8685f8e4a348a # v7
with:
fetch-depth: 0
persist-credentials: false
- name: Recheck approved bytes and trusted certification
id: verify
env:
GH_TOKEN: ${{ github.token }}
COMPLETE_RUN: ${{ inputs.complete_run_id }}
MANIFEST_SHA: ${{ inputs.manifest_sha256 }}
run: python3 scripts/release/publish.py verify --run-id "$COMPLETE_RUN" --digest "$MANIFEST_SHA" --directory sealed
version:
needs: verify
runs-on: ubuntu-latest
timeout-minutes: 90
concurrency:
group: release-version-${{ needs.verify.outputs.version }}
cancel-in-progress: false
environment: release-publication
permissions:
contents: write
actions: read
packages: write
security-events: write
env:
GH_TOKEN: ${{ github.token }}
COMPLETE_RUN: ${{ inputs.complete_run_id }}
MANIFEST_SHA: ${{ inputs.manifest_sha256 }}
AWS_ACCESS_KEY_ID: ${{ secrets.R2_ACCESS_KEY_ID }}
AWS_SECRET_ACCESS_KEY: ${{ secrets.R2_SECRET_ACCESS_KEY }}
AWS_DEFAULT_REGION: auto
AWS_REQUEST_CHECKSUM_CALCULATION: when_required
AWS_RESPONSE_CHECKSUM_VALIDATION: when_required
AWS_EC2_METADATA_DISABLED: "true"
R2_ENDPOINT: ${{ secrets.R2_ENDPOINT }}
R2_BUCKET: ${{ secrets.R2_BUCKET }}
steps:
- uses: actions/checkout@f548e57e544e1ff5a4c46bf1e1b8685f8e4a348a # v7
with:
fetch-depth: 0
persist-credentials: false
- name: Install promotion tools
run: |
sudo apt-get update
sudo apt-get install -y skopeo
aws --version
- uses: docker/login-action@c94ce9fb468520275223c153574b00df6fe4bcc9 # v3
with:
username: rustfs
password: ${{ secrets.DOCKERHUB_TOKEN }}
- uses: docker/login-action@c94ce9fb468520275223c153574b00df6fe4bcc9 # v3
with:
registry: ghcr.io
username: ${{ secrets.GHCR_USERNAME }}
password: ${{ secrets.GHCR_PASSWORD }}
- uses: docker/login-action@c94ce9fb468520275223c153574b00df6fe4bcc9 # v3
with:
registry: quay.io
username: ${{ secrets.QUAY_USERNAME }}
password: ${{ secrets.QUAY_PASSWORD }}
- name: Share registry credentials with the OCI copier
run: |
install -m 600 "$HOME/.docker/config.json" "$RUNNER_TEMP/registry-auth.json"
echo "REGISTRY_AUTH_FILE=$RUNNER_TEMP/registry-auth.json" >> "$GITHUB_ENV"
- name: Promote verified version
run: python3 scripts/release/publish.py version --run-id "$COMPLETE_RUN" --digest "$MANIFEST_SHA" --directory sealed
- name: Publish accepted musl amd64 scan
uses: github/codeql-action/upload-sarif@99df26d4f13ea111d4ec1a7dddef6063f76b97e9 # v4
with:
sarif_file: sealed/scan-musl-amd64.sarif
category: container-image-musl-amd64
- name: Publish accepted musl arm64 scan
uses: github/codeql-action/upload-sarif@99df26d4f13ea111d4ec1a7dddef6063f76b97e9 # v4
with:
sarif_file: sealed/scan-musl-arm64.sarif
category: container-image-musl-arm64
- name: Publish accepted GNU amd64 scan
uses: github/codeql-action/upload-sarif@99df26d4f13ea111d4ec1a7dddef6063f76b97e9 # v4
with:
sarif_file: sealed/scan-gnu-amd64.sarif
category: container-image-gnu-amd64
- name: Publish accepted GNU arm64 scan
uses: github/codeql-action/upload-sarif@99df26d4f13ea111d4ec1a7dddef6063f76b97e9 # v4
with:
sarif_file: sealed/scan-gnu-arm64.sarif
category: container-image-gnu-arm64
- uses: actions/upload-artifact@043fb46d1a93c77aae656e7c1c64a875d1fc6a0a # v7.0.1
with:
name: release-version-receipt-${{ github.run_attempt }}
path: publication.json
if-no-files-found: error
retention-days: 90
channels:
needs: [verify, version]
runs-on: ubuntu-latest
timeout-minutes: 90
concurrency:
group: rustfs-release-channels
cancel-in-progress: false
permissions:
contents: write
actions: read
packages: write
env:
GH_TOKEN: ${{ github.token }}
COMPLETE_RUN: ${{ inputs.complete_run_id }}
MANIFEST_SHA: ${{ inputs.manifest_sha256 }}
AWS_ACCESS_KEY_ID: ${{ secrets.R2_ACCESS_KEY_ID }}
AWS_SECRET_ACCESS_KEY: ${{ secrets.R2_SECRET_ACCESS_KEY }}
AWS_DEFAULT_REGION: auto
AWS_REQUEST_CHECKSUM_CALCULATION: when_required
AWS_RESPONSE_CHECKSUM_VALIDATION: when_required
AWS_EC2_METADATA_DISABLED: "true"
R2_ENDPOINT: ${{ secrets.R2_ENDPOINT }}
R2_BUCKET: ${{ secrets.R2_BUCKET }}
steps:
- uses: actions/checkout@f548e57e544e1ff5a4c46bf1e1b8685f8e4a348a # v7
with:
fetch-depth: 0
persist-credentials: false
- name: Install promotion tools
run: |
sudo apt-get update
sudo apt-get install -y skopeo
aws --version
- uses: docker/login-action@c94ce9fb468520275223c153574b00df6fe4bcc9 # v3
with:
username: rustfs
password: ${{ secrets.DOCKERHUB_TOKEN }}
- uses: docker/login-action@c94ce9fb468520275223c153574b00df6fe4bcc9 # v3
with:
registry: ghcr.io
username: ${{ secrets.GHCR_USERNAME }}
password: ${{ secrets.GHCR_PASSWORD }}
- uses: docker/login-action@c94ce9fb468520275223c153574b00df6fe4bcc9 # v3
with:
registry: quay.io
username: ${{ secrets.QUAY_USERNAME }}
password: ${{ secrets.QUAY_PASSWORD }}
- name: Share registry credentials with the OCI copier
run: |
install -m 600 "$HOME/.docker/config.json" "$RUNNER_TEMP/registry-auth.json"
echo "REGISTRY_AUTH_FILE=$RUNNER_TEMP/registry-auth.json" >> "$GITHUB_ENV"
- uses: azure/setup-helm@b9e51907a09c216f16ebe8536097933489208112 # v4.3.0
- name: Install pinned OSS client
run: |
curl --fail --location --retry 3 --output "$RUNNER_TEMP/ossutil.zip" https://gosspublic.alicdn.com/ossutil/v2/2.1.1/ossutil-2.1.1-linux-amd64.zip
printf '%s %s\n' 60f3a808cbbdfd4b5269b8f967c6a66f564c9dacff52d93a5d21a588976ab5a2 "$RUNNER_TEMP/ossutil.zip" | sha256sum --check -
unzip -q "$RUNNER_TEMP/ossutil.zip" -d "$RUNNER_TEMP/ossutil-extract"
install -m 755 "$RUNNER_TEMP/ossutil-extract/ossutil-2.1.1-linux-amd64/ossutil" "$RUNNER_TEMP/ossutil"
echo "OSSUTIL_BIN=$RUNNER_TEMP/ossutil" >> "$GITHUB_ENV"
- name: Promote verified channels
env:
HELM_TOKEN: ${{ secrets.RUSTFS_HELM_PACKAGE }}
OSS_ACCESS_KEY_ID: ${{ secrets.ALICLOUDOSS_KEY_ID }}
OSS_ACCESS_KEY_SECRET: ${{ secrets.ALICLOUDOSS_KEY_SECRET }}
OSS_REGION: cn-beijing
OSS_ENDPOINT: https://oss-cn-beijing.aliyuncs.com
run: python3 scripts/release/publish.py channels --run-id "$COMPLETE_RUN" --digest "$MANIFEST_SHA" --directory sealed
- uses: actions/upload-artifact@043fb46d1a93c77aae656e7c1c64a875d1fc6a0a # v7.0.1
with:
name: release-channels-receipt-${{ github.run_attempt }}
path: publication.json
if-no-files-found: error
retention-days: 90
@@ -26,7 +26,13 @@ permissions:
jobs:
alert-on-incomplete-run:
name: Alert on incomplete scheduled run
if: github.repository == 'rustfs/rustfs'
# The registry still owns responsibility and cadence selection. Successful
# and PR runs can never produce an incomplete scheduled-run alert.
if: >-
github.repository == 'rustfs/rustfs' &&
github.event.workflow_run.conclusion != 'success' &&
(github.event.workflow_run.event == 'schedule' ||
github.event.workflow_run.event == 'workflow_run')
# Stay on GitHub-hosted runners: this monitor must keep running when the
# self-hosted sm-standard-* pool is down, which is exactly when the
# scheduled validations it watches stall.
Generated
+98 -101
View File
@@ -322,9 +322,9 @@ dependencies = [
[[package]]
name = "archive-trait"
version = "0.0.15"
version = "0.0.16"
source = "registry+https://github.com/rust-lang/crates.io-index"
checksum = "999bf88cc483664cab77700ca2fdfb0415a73412abe284ef5bc2319d00938e00"
checksum = "0b5040ca935c09888e7dfb04383b4d62fa3571e25bc66b6f345b591e97958524"
dependencies = [
"cap-std",
"thiserror 2.0.21",
@@ -811,9 +811,9 @@ dependencies = [
[[package]]
name = "async-nats"
version = "0.51.0"
version = "0.52.0"
source = "registry+https://github.com/rust-lang/crates.io-index"
checksum = "a4f92759f418774b78c36ef07885849150ce950c67d9bcdeff8a9c253f57d9a1"
checksum = "015f0a7a3b0b29701b78c76a4d4a122624c97c71fa30363381664eb729d619c7"
dependencies = [
"aws-lc-rs",
"base64 0.22.1",
@@ -3906,9 +3906,9 @@ dependencies = [
[[package]]
name = "dial9"
version = "0.5.7"
version = "0.5.8"
source = "registry+https://github.com/rust-lang/crates.io-index"
checksum = "2f3298a54fd82af243c9f122595762f5b0e48bd3d1ea8fe3f3eb5e95c130e881"
checksum = "c83757d61dc75b3e9d3579a3004c7fe0c42aff2f779ca9eeec7f0c8e1ae3fc10"
dependencies = [
"bon",
"dial9-core",
@@ -3990,9 +3990,9 @@ dependencies = [
[[package]]
name = "dial9-tokio-telemetry"
version = "0.5.7"
version = "0.5.8"
source = "registry+https://github.com/rust-lang/crates.io-index"
checksum = "0428cba60fbcf37ab82666f3a7ef3b6d0671e90dfb6a352376813b471319f061"
checksum = "6f0eb05627c12c57e40510e63f9a3d5f628ab4b32fec9ccdb0ae98544c13f15e"
dependencies = [
"bon",
"bytes",
@@ -4136,7 +4136,7 @@ checksum = "d0881ea181b1df73ff77ffaaf9c7544ecc11e82fba9b5f27b262a3c73a332555"
[[package]]
name = "e2e_test"
version = "1.0.2"
version = "1.0.5"
dependencies = [
"anyhow",
"async-compression",
@@ -4525,9 +4525,9 @@ dependencies = [
[[package]]
name = "flate2"
version = "1.1.10"
version = "1.1.11"
source = "registry+https://github.com/rust-lang/crates.io-index"
checksum = "6e634e2e0ebac1ee034020da1ca582e17ffe4e0f5e985823721e168928136dcb"
checksum = "626d4928326b646e70f65ed36fc7cf9f206bdb68a0019ee7c6b4709da4e084c4"
dependencies = [
"crc32fast",
"miniz_oxide",
@@ -4594,7 +4594,7 @@ checksum = "94e7099f6313ecacbe1256e8ff9d617b75d1bcb16a6fddef94866d225a01a14a"
dependencies = [
"io-lifetimes 2.0.4",
"rustix",
"windows-sys 0.59.0",
"windows-sys 0.52.0",
]
[[package]]
@@ -5385,9 +5385,9 @@ dependencies = [
[[package]]
name = "hotpath"
version = "0.30.3"
version = "0.31.1"
source = "registry+https://github.com/rust-lang/crates.io-index"
checksum = "d9632c1fd213e63c1b88d8f3e59543da4999b17f859cc1e1d67cf8f2207e8935"
checksum = "4039758c8bca06a34c475d17aaafc17ac1073ce2cecd8dba8c82e69edb300bbe"
dependencies = [
"arc-swap",
"async-channel",
@@ -5420,15 +5420,15 @@ dependencies = [
[[package]]
name = "hotpath-drain"
version = "0.30.3"
version = "0.31.1"
source = "registry+https://github.com/rust-lang/crates.io-index"
checksum = "738324f75aec847dea984f6806402f4f1bdfab437b4700fc8c3ebcad2dff6424"
checksum = "bbba0a3badfb9ebf95f22dc6cdb4afcd0ff07b91a85fa3196fd1060576ea3065"
[[package]]
name = "hotpath-macros"
version = "0.30.3"
version = "0.31.1"
source = "registry+https://github.com/rust-lang/crates.io-index"
checksum = "5d92fdfec803932e5764ea93c201cf7c6c2d8cff1871da44f64f5094c9c3ef8a"
checksum = "7d980014875338771b00ac44f1c9e2ea1a10650aa68ce54226233bc767714361"
dependencies = [
"proc-macro2",
"quote",
@@ -6412,9 +6412,9 @@ dependencies = [
[[package]]
name = "lzma-rust2"
version = "0.24.0"
version = "0.25.0"
source = "registry+https://github.com/rust-lang/crates.io-index"
checksum = "490750560fa7851fbbf5d3ab1a8750f09350ed361ed8aed85b1e1e40587812ee"
checksum = "ef1d73c59a56771343712b22ea5615092c612c16ba291c0979e3105dfeec4890"
dependencies = [
"sha2 0.11.1",
]
@@ -9520,7 +9520,7 @@ dependencies = [
[[package]]
name = "rustfs"
version = "1.0.2"
version = "1.0.5"
dependencies = [
"aes-gcm",
"anyhow",
@@ -9680,7 +9680,7 @@ dependencies = [
[[package]]
name = "rustfs-audit"
version = "1.0.2"
version = "1.0.5"
dependencies = [
"const-str",
"futures",
@@ -9702,7 +9702,7 @@ dependencies = [
[[package]]
name = "rustfs-checksums"
version = "1.0.2"
version = "1.0.5"
dependencies = [
"base64-simd",
"bytes",
@@ -9718,7 +9718,7 @@ dependencies = [
[[package]]
name = "rustfs-common"
version = "1.0.2"
version = "1.0.5"
dependencies = [
"arc-swap",
"hotpath",
@@ -9733,7 +9733,7 @@ dependencies = [
[[package]]
name = "rustfs-concurrency"
version = "1.0.2"
version = "1.0.5"
dependencies = [
"hotpath",
"insta",
@@ -9746,7 +9746,7 @@ dependencies = [
[[package]]
name = "rustfs-config"
version = "1.0.2"
version = "1.0.5"
dependencies = [
"const-str",
"hotpath",
@@ -9756,7 +9756,7 @@ dependencies = [
[[package]]
name = "rustfs-connect-symbol-catalog"
version = "1.0.2"
version = "1.0.5"
dependencies = [
"hex-simd",
"object 0.40.0",
@@ -9768,7 +9768,7 @@ dependencies = [
[[package]]
name = "rustfs-credentials"
version = "1.0.2"
version = "1.0.5"
dependencies = [
"base64-simd",
"hmac 0.13.0",
@@ -9782,7 +9782,7 @@ dependencies = [
[[package]]
name = "rustfs-crypto"
version = "1.0.2"
version = "1.0.5"
dependencies = [
"aes-gcm",
"argon2",
@@ -9803,7 +9803,7 @@ dependencies = [
[[package]]
name = "rustfs-data-usage"
version = "1.0.2"
version = "1.0.5"
dependencies = [
"hotpath",
"rmp-serde",
@@ -9813,7 +9813,7 @@ dependencies = [
[[package]]
name = "rustfs-ecstore"
version = "1.0.2"
version = "1.0.5"
dependencies = [
"arc-swap",
"async-channel",
@@ -9837,6 +9837,7 @@ dependencies = [
"google-cloud-auth",
"google-cloud-storage",
"h2",
"hashbrown 0.17.1",
"hex-simd",
"hmac 0.13.0",
"hostname",
@@ -9950,7 +9951,7 @@ dependencies = [
[[package]]
name = "rustfs-extension-schema"
version = "1.0.2"
version = "1.0.5"
dependencies = [
"hotpath",
"serde",
@@ -9960,7 +9961,7 @@ dependencies = [
[[package]]
name = "rustfs-filemeta"
version = "1.0.2"
version = "1.0.5"
dependencies = [
"arc-swap",
"base64-simd",
@@ -9989,7 +9990,7 @@ dependencies = [
[[package]]
name = "rustfs-heal"
version = "1.0.2"
version = "1.0.5"
dependencies = [
"async-trait",
"base64-simd",
@@ -10030,7 +10031,7 @@ dependencies = [
[[package]]
name = "rustfs-heal-contracts"
version = "1.0.2"
version = "1.0.5"
dependencies = [
"serde",
"serde_json",
@@ -10040,7 +10041,7 @@ dependencies = [
[[package]]
name = "rustfs-iam"
version = "1.0.2"
version = "1.0.5"
dependencies = [
"arc-swap",
"async-trait",
@@ -10090,7 +10091,7 @@ dependencies = [
[[package]]
name = "rustfs-io-core"
version = "1.0.2"
version = "1.0.5"
dependencies = [
"bytes",
"hotpath",
@@ -10103,7 +10104,7 @@ dependencies = [
[[package]]
name = "rustfs-io-metrics"
version = "1.0.2"
version = "1.0.5"
dependencies = [
"criterion",
"hotpath",
@@ -10167,7 +10168,7 @@ dependencies = [
[[package]]
name = "rustfs-keystone"
version = "1.0.2"
version = "1.0.5"
dependencies = [
"bytes",
"futures",
@@ -10193,7 +10194,7 @@ dependencies = [
[[package]]
name = "rustfs-kms"
version = "1.0.2"
version = "1.0.5"
dependencies = [
"aes-gcm",
"anyhow",
@@ -10243,14 +10244,14 @@ dependencies = [
[[package]]
name = "rustfs-license"
version = "1.0.2"
version = "1.0.5"
dependencies = [
"thiserror 2.0.21",
]
[[package]]
name = "rustfs-lifecycle"
version = "1.0.2"
version = "1.0.5"
dependencies = [
"async-trait",
"hotpath",
@@ -10272,7 +10273,7 @@ dependencies = [
[[package]]
name = "rustfs-lock"
version = "1.0.2"
version = "1.0.5"
dependencies = [
"async-trait",
"compact_str",
@@ -10296,7 +10297,7 @@ dependencies = [
[[package]]
name = "rustfs-log-analyzer"
version = "1.0.2"
version = "1.0.5"
dependencies = [
"chrono",
"flate2",
@@ -10315,7 +10316,7 @@ dependencies = [
[[package]]
name = "rustfs-madmin"
version = "1.0.2"
version = "1.0.5"
dependencies = [
"hotpath",
"http 1.5.0",
@@ -10335,25 +10336,25 @@ dependencies = [
[[package]]
name = "rustfs-mimalloc"
version = "0.7.3"
version = "0.7.4"
source = "registry+https://github.com/rust-lang/crates.io-index"
checksum = "216ecfc9341b5d6e2506d47ce57c9232ae1b8020bb10264ea05286fb234286e7"
checksum = "fc7914232a854336c62058c24a8362c553ac8e7adba2e2395ce9097db24d1510"
dependencies = [
"rustfs-mimalloc-sys",
]
[[package]]
name = "rustfs-mimalloc-sys"
version = "0.7.3"
version = "0.7.4"
source = "registry+https://github.com/rust-lang/crates.io-index"
checksum = "fb16ad68f96883407a75cdb88252b43dc2ea592428f0fc54ff7885756de4b814"
checksum = "8073624242781448512682f1931624ee67108b6b7873b22fb29c1a1df689ac0c"
dependencies = [
"cc",
]
[[package]]
name = "rustfs-notify"
version = "1.0.2"
version = "1.0.5"
dependencies = [
"arc-swap",
"async-trait",
@@ -10388,7 +10389,7 @@ dependencies = [
[[package]]
name = "rustfs-object-capacity"
version = "1.0.2"
version = "1.0.5"
dependencies = [
"criterion",
"futures",
@@ -10407,7 +10408,7 @@ dependencies = [
[[package]]
name = "rustfs-object-data-cache"
version = "1.0.2"
version = "1.0.5"
dependencies = [
"bytes",
"criterion",
@@ -10424,7 +10425,7 @@ dependencies = [
[[package]]
name = "rustfs-obs"
version = "1.0.2"
version = "1.0.5"
dependencies = [
"chrono",
"crossbeam-channel",
@@ -10487,7 +10488,7 @@ dependencies = [
[[package]]
name = "rustfs-policy"
version = "1.0.2"
version = "1.0.5"
dependencies = [
"async-trait",
"base64-simd",
@@ -10518,7 +10519,7 @@ dependencies = [
[[package]]
name = "rustfs-protocols"
version = "1.0.2"
version = "1.0.5"
dependencies = [
"async-compression",
"async-trait",
@@ -10581,7 +10582,7 @@ dependencies = [
[[package]]
name = "rustfs-protos"
version = "1.0.2"
version = "1.0.5"
dependencies = [
"flatbuffers",
"hotpath",
@@ -10606,7 +10607,7 @@ dependencies = [
[[package]]
name = "rustfs-replication"
version = "1.0.2"
version = "1.0.5"
dependencies = [
"byteorder",
"bytes",
@@ -10624,7 +10625,7 @@ dependencies = [
[[package]]
name = "rustfs-rio"
version = "1.0.2"
version = "1.0.5"
dependencies = [
"aes-gcm",
"arc-swap",
@@ -10670,7 +10671,7 @@ dependencies = [
[[package]]
name = "rustfs-rio-v2"
version = "1.0.2"
version = "1.0.5"
dependencies = [
"aes-gcm",
"bytes",
@@ -10693,7 +10694,7 @@ dependencies = [
[[package]]
name = "rustfs-s3-client"
version = "1.0.2"
version = "1.0.5"
dependencies = [
"base64-simd",
"bytes",
@@ -10737,7 +10738,7 @@ dependencies = [
[[package]]
name = "rustfs-s3-ops"
version = "1.0.2"
version = "1.0.5"
dependencies = [
"hotpath",
"rustfs-s3-types",
@@ -10745,7 +10746,7 @@ dependencies = [
[[package]]
name = "rustfs-s3-types"
version = "1.0.2"
version = "1.0.5"
dependencies = [
"hotpath",
"serde",
@@ -10754,20 +10755,25 @@ dependencies = [
[[package]]
name = "rustfs-s3select-api"
version = "1.0.2"
version = "1.0.5"
dependencies = [
"arc-swap",
"async-compression",
"async-trait",
"base64-simd",
"brotli 9.1.1",
"bytes",
"chrono",
"crc-fast",
"datafusion",
"flatbuffers",
"flate2",
"futures",
"futures-core",
"hotpath",
"http 1.5.0",
"lz4",
"lz4_flex",
"memchr",
"metrics",
"parking_lot 0.13.0",
@@ -10790,7 +10796,7 @@ dependencies = [
[[package]]
name = "rustfs-s3select-query"
version = "1.0.2"
version = "1.0.5"
dependencies = [
"async-trait",
"bytes",
@@ -10812,7 +10818,7 @@ dependencies = [
[[package]]
name = "rustfs-scanner"
version = "1.0.2"
version = "1.0.5"
dependencies = [
"async-trait",
"bytes",
@@ -10863,7 +10869,7 @@ dependencies = [
[[package]]
name = "rustfs-scanner-metrics"
version = "1.0.2"
version = "1.0.5"
dependencies = [
"chrono",
"jiff",
@@ -10878,7 +10884,7 @@ dependencies = [
[[package]]
name = "rustfs-security-governance"
version = "1.0.2"
version = "1.0.5"
dependencies = [
"hotpath",
"thiserror 2.0.21",
@@ -10886,7 +10892,7 @@ dependencies = [
[[package]]
name = "rustfs-signer"
version = "1.0.2"
version = "1.0.5"
dependencies = [
"base64-simd",
"bytes",
@@ -10905,7 +10911,7 @@ dependencies = [
[[package]]
name = "rustfs-storage-api"
version = "1.0.2"
version = "1.0.5"
dependencies = [
"async-trait",
"hotpath",
@@ -10920,7 +10926,7 @@ dependencies = [
[[package]]
name = "rustfs-targets"
version = "1.0.2"
version = "1.0.5"
dependencies = [
"arc-swap",
"async-nats",
@@ -10974,7 +10980,7 @@ dependencies = [
[[package]]
name = "rustfs-test-utils"
version = "1.0.2"
version = "1.0.5"
dependencies = [
"hotpath",
"rustfs-data-usage",
@@ -10990,7 +10996,7 @@ dependencies = [
[[package]]
name = "rustfs-tls-runtime"
version = "1.0.2"
version = "1.0.5"
dependencies = [
"arc-swap",
"hotpath",
@@ -11029,7 +11035,7 @@ dependencies = [
[[package]]
name = "rustfs-trusted-proxies"
version = "1.0.2"
version = "1.0.5"
dependencies = [
"async-trait",
"axum",
@@ -11066,7 +11072,7 @@ dependencies = [
[[package]]
name = "rustfs-utils"
version = "1.0.2"
version = "1.0.5"
dependencies = [
"aws-lc-rs",
"base64-simd",
@@ -11109,7 +11115,7 @@ dependencies = [
[[package]]
name = "rustfs-zip"
version = "1.0.2"
version = "1.0.5"
dependencies = [
"async-compression",
"futures",
@@ -11699,9 +11705,9 @@ dependencies = [
[[package]]
name = "serde_spanned"
version = "1.1.2"
version = "1.1.3"
source = "registry+https://github.com/rust-lang/crates.io-index"
checksum = "4e7523beb55eece201a2356bee0bbca0d1ab466c14c07703b2e0ee6d42cb0c2c"
checksum = "9c187e0a369471c550af028cba56d4f4fa2a489ad28151ae95a4d340a2a8b782"
dependencies = [
"serde_core",
]
@@ -12031,9 +12037,9 @@ dependencies = [
[[package]]
name = "smallvec"
version = "1.16.4"
version = "1.16.5"
source = "registry+https://github.com/rust-lang/crates.io-index"
checksum = "d0d5d4b1981b8bbbb7aed204cae5726d683d5d6ac7e74ec54122ea28f315e7c8"
checksum = "149309498a5f7334b12ddae0cad8aba81ce785fe91755068867e017cf98fee46"
dependencies = [
"serde",
]
@@ -12228,9 +12234,9 @@ dependencies = [
[[package]]
name = "starshard"
version = "2.4.0"
version = "2.4.1"
source = "registry+https://github.com/rust-lang/crates.io-index"
checksum = "fc45de5060fab0de00d7a42230877543ae00c37caf6f49975e69851089b023ac"
checksum = "b0883700b50fd0b62120ebc4c07230e02f0cf1ead474c3b3b44eaa13e0a999ea"
dependencies = [
"async-trait",
"hashbrown 0.17.1",
@@ -12479,9 +12485,9 @@ dependencies = [
[[package]]
name = "tar-codec"
version = "0.0.15"
version = "0.0.16"
source = "registry+https://github.com/rust-lang/crates.io-index"
checksum = "b9dad615e50d181afea4bafea23dacb8e33ed52dde06503813570634b932d467"
checksum = "6875ba507bbec6022500f46477159a03f07ebdffdb4ea043254974531df1dac2"
dependencies = [
"archive-trait",
"tar-framing",
@@ -12491,9 +12497,9 @@ dependencies = [
[[package]]
name = "tar-framing"
version = "0.0.15"
version = "0.0.16"
source = "registry+https://github.com/rust-lang/crates.io-index"
checksum = "563e8181b47c9cdcce569eae3ad431eec71543d6269dd60c7bd2350f70bcd233"
checksum = "49727b1a300514960d425d0e741a7da3e693170f3db39de3180fdbbc07e542c5"
dependencies = [
"thiserror 2.0.21",
"tokio",
@@ -12867,9 +12873,9 @@ dependencies = [
[[package]]
name = "toml"
version = "1.2.0+spec-1.1.0"
version = "1.2.1+spec-1.1.0"
source = "registry+https://github.com/rust-lang/crates.io-index"
checksum = "c5ffcdf8f50436dbbfd675c1fe1b8e0854f42b7defea8584bdebe1aee900e561"
checksum = "d5755c51863cdac711f1956baf8eb9588e060755f7d3c4c979e70ea1e1f32cd7"
dependencies = [
"indexmap 2.14.2",
"serde_core",
@@ -12882,9 +12888,9 @@ dependencies = [
[[package]]
name = "toml_datetime"
version = "1.1.2+spec-1.1.0"
version = "1.1.3+spec-1.1.0"
source = "registry+https://github.com/rust-lang/crates.io-index"
checksum = "2b86d767906c6c42421dcba507eb9d203e779497710a47782a224bb871653053"
checksum = "173f25c883cd2176a2d8212155638c0a60d484a9d84546d270866f40f56b6e49"
dependencies = [
"serde_core",
]
@@ -13777,15 +13783,6 @@ dependencies = [
"windows-targets 0.52.6",
]
[[package]]
name = "windows-sys"
version = "0.59.0"
source = "registry+https://github.com/rust-lang/crates.io-index"
checksum = "1e38bc4d79ed67fd075bcc251a1c39b32a1776bbe92e5bef1f0bf1f8c531853b"
dependencies = [
"windows-targets 0.52.6",
]
[[package]]
name = "windows-sys"
version = "0.60.2"
@@ -13958,7 +13955,7 @@ source = "registry+https://github.com/rust-lang/crates.io-index"
checksum = "3f3fd376f71958b862e7afb20cfe5a22830e1963462f3a17f49d82a6c1d1f42d"
dependencies = [
"bitflags 2.13.2",
"windows-sys 0.59.0",
"windows-sys 0.52.0",
]
[[package]]
@@ -14208,9 +14205,9 @@ dependencies = [
[[package]]
name = "zip"
version = "9.3.0"
version = "9.5.0"
source = "registry+https://github.com/rust-lang/crates.io-index"
checksum = "eaeb7f16d2605b49aef586996ded29e65642fab7259a3077e537351df6d80426"
checksum = "0700e6c427aff64acdbf02e85fbfd946c2ee12319d1af013092f3a2178914ff8"
dependencies = [
"aes 0.9.4",
"bzip2",
+56 -55
View File
@@ -75,7 +75,7 @@ edition = "2024"
license = "Apache-2.0"
repository = "https://github.com/rustfs/rustfs"
rust-version = "1.99.0"
version = "1.0.2"
version = "1.0.5"
homepage = "https://rustfs.com"
description = "RustFS is a high-performance distributed object storage software built using Rust, one of the most popular languages worldwide. "
keywords = ["RustFS", "Minio", "object-storage", "filesystem", "s3"]
@@ -92,56 +92,56 @@ redundant_clone = "warn"
[workspace.dependencies]
# RustFS Internal Crates
rustfs = { path = "./rustfs", version = "1.0.2" }
rustfs-heal = { path = "crates/heal", version = "1.0.2" }
rustfs-heal-contracts = { path = "crates/heal-contracts", version = "1.0.2" }
rustfs-scanner-metrics = { path = "crates/scanner-metrics", version = "1.0.2" }
rustfs-audit = { path = "crates/audit", version = "1.0.2" }
rustfs-checksums = { path = "crates/checksums", version = "1.0.2" }
rustfs-common = { path = "crates/common", version = "1.0.2" }
rustfs-data-usage = { path = "crates/data-usage", version = "1.0.2" }
rustfs-config = { path = "./crates/config", version = "1.0.2" }
rustfs-concurrency = { path = "./crates/concurrency", version = "1.0.2" }
rustfs-credentials = { path = "crates/credentials", version = "1.0.2" }
rustfs-crypto = { path = "crates/crypto", version = "1.0.2" }
rustfs-ecstore = { path = "crates/ecstore", version = "1.0.2" }
rustfs-filemeta = { path = "crates/filemeta", version = "1.0.2" }
rustfs-iam = { path = "crates/iam", version = "1.0.2" }
rustfs-keystone = { path = "crates/keystone", version = "1.0.2" }
rustfs-license = { path = "crates/license", version = "1.0.2" }
rustfs-lifecycle = { path = "crates/lifecycle", version = "1.0.2" }
rustfs-kms = { path = "crates/kms", version = "1.0.2" }
rustfs-lock = { path = "crates/lock", version = "1.0.2" }
rustfs-madmin = { path = "crates/madmin", version = "1.0.2" }
rustfs-notify = { path = "crates/notify", version = "1.0.2" }
rustfs-io-metrics = { path = "crates/io-metrics", version = "1.0.2" }
rustfs-io-core = { path = "crates/io-core", version = "1.0.2" }
rustfs-object-capacity = { path = "crates/object-capacity", version = "1.0.2" }
rustfs-object-data-cache = { path = "crates/object-data-cache", version = "1.0.2", default-features = false }
rustfs-log-analyzer = { path = "crates/log-analyzer", version = "1.0.2" }
rustfs-obs = { path = "crates/obs", version = "1.0.2" }
rustfs-policy = { path = "crates/policy", version = "1.0.2" }
rustfs-protos = { path = "crates/protos", version = "1.0.2" }
rustfs-protocols = { path = "crates/protocols", version = "1.0.2" }
rustfs-replication = { path = "crates/replication", version = "1.0.2" }
rustfs-rio = { path = "crates/rio", version = "1.0.2" }
rustfs-rio-v2 = { path = "crates/rio-v2", version = "1.0.2" }
rustfs-s3-client = { path = "crates/s3-client", version = "1.0.2" }
rustfs-s3-types = { path = "crates/s3-types", version = "1.0.2" }
rustfs-s3-ops = { path = "crates/s3-ops", version = "1.0.2" }
rustfs-s3select-api = { path = "crates/s3select-api", version = "1.0.2" }
rustfs-s3select-query = { path = "crates/s3select-query", version = "1.0.2" }
rustfs-scanner = { path = "crates/scanner", version = "1.0.2" }
rustfs-security-governance = { path = "crates/security-governance", version = "1.0.2" }
rustfs-extension-schema = { path = "crates/extension-schema", version = "1.0.2" }
rustfs-signer = { path = "crates/signer", version = "1.0.2" }
rustfs-storage-api = { path = "crates/storage-api", version = "1.0.2" }
rustfs-trusted-proxies = { path = "crates/trusted-proxies", version = "1.0.2" }
rustfs-targets = { path = "crates/targets", version = "1.0.2" }
rustfs-test-utils = { path = "crates/test-utils", version = "1.0.2" }
rustfs-tls-runtime = { path = "crates/tls-runtime", version = "1.0.2" }
rustfs-utils = { path = "crates/utils", version = "1.0.2" }
rustfs-zip = { path = "./crates/zip", version = "1.0.2" }
rustfs = { path = "./rustfs", version = "1.0.5" }
rustfs-heal = { path = "crates/heal", version = "1.0.5" }
rustfs-heal-contracts = { path = "crates/heal-contracts", version = "1.0.5" }
rustfs-scanner-metrics = { path = "crates/scanner-metrics", version = "1.0.5" }
rustfs-audit = { path = "crates/audit", version = "1.0.5" }
rustfs-checksums = { path = "crates/checksums", version = "1.0.5" }
rustfs-common = { path = "crates/common", version = "1.0.5" }
rustfs-data-usage = { path = "crates/data-usage", version = "1.0.5" }
rustfs-config = { path = "./crates/config", version = "1.0.5" }
rustfs-concurrency = { path = "./crates/concurrency", version = "1.0.5" }
rustfs-credentials = { path = "crates/credentials", version = "1.0.5" }
rustfs-crypto = { path = "crates/crypto", version = "1.0.5" }
rustfs-ecstore = { path = "crates/ecstore", version = "1.0.5" }
rustfs-filemeta = { path = "crates/filemeta", version = "1.0.5" }
rustfs-iam = { path = "crates/iam", version = "1.0.5" }
rustfs-keystone = { path = "crates/keystone", version = "1.0.5" }
rustfs-license = { path = "crates/license", version = "1.0.5" }
rustfs-lifecycle = { path = "crates/lifecycle", version = "1.0.5" }
rustfs-kms = { path = "crates/kms", version = "1.0.5" }
rustfs-lock = { path = "crates/lock", version = "1.0.5" }
rustfs-madmin = { path = "crates/madmin", version = "1.0.5" }
rustfs-notify = { path = "crates/notify", version = "1.0.5" }
rustfs-io-metrics = { path = "crates/io-metrics", version = "1.0.5" }
rustfs-io-core = { path = "crates/io-core", version = "1.0.5" }
rustfs-object-capacity = { path = "crates/object-capacity", version = "1.0.5" }
rustfs-object-data-cache = { path = "crates/object-data-cache", version = "1.0.5", default-features = false }
rustfs-log-analyzer = { path = "crates/log-analyzer", version = "1.0.5" }
rustfs-obs = { path = "crates/obs", version = "1.0.5" }
rustfs-policy = { path = "crates/policy", version = "1.0.5" }
rustfs-protos = { path = "crates/protos", version = "1.0.5" }
rustfs-protocols = { path = "crates/protocols", version = "1.0.5" }
rustfs-replication = { path = "crates/replication", version = "1.0.5" }
rustfs-rio = { path = "crates/rio", version = "1.0.5" }
rustfs-rio-v2 = { path = "crates/rio-v2", version = "1.0.5" }
rustfs-s3-client = { path = "crates/s3-client", version = "1.0.5" }
rustfs-s3-types = { path = "crates/s3-types", version = "1.0.5" }
rustfs-s3-ops = { path = "crates/s3-ops", version = "1.0.5" }
rustfs-s3select-api = { path = "crates/s3select-api", version = "1.0.5" }
rustfs-s3select-query = { path = "crates/s3select-query", version = "1.0.5" }
rustfs-scanner = { path = "crates/scanner", version = "1.0.5" }
rustfs-security-governance = { path = "crates/security-governance", version = "1.0.5" }
rustfs-extension-schema = { path = "crates/extension-schema", version = "1.0.5" }
rustfs-signer = { path = "crates/signer", version = "1.0.5" }
rustfs-storage-api = { path = "crates/storage-api", version = "1.0.5" }
rustfs-trusted-proxies = { path = "crates/trusted-proxies", version = "1.0.5" }
rustfs-targets = { path = "crates/targets", version = "1.0.5" }
rustfs-test-utils = { path = "crates/test-utils", version = "1.0.5" }
rustfs-tls-runtime = { path = "crates/tls-runtime", version = "1.0.5" }
rustfs-utils = { path = "crates/utils", version = "1.0.5" }
rustfs-zip = { path = "./crates/zip", version = "1.0.5" }
# Async Runtime and Networking
async-channel = "2.5.0"
@@ -149,7 +149,7 @@ async_zip = { default-features = false, version = "0.0.19" }
mysql_async = { default-features = false, version = "0.37.1" }
async-compression = { version = "0.4.52" }
async-trait = "0.1.92"
async-nats = { version = "0.51.0", default-features = false }
async-nats = { version = "0.52.0", default-features = false }
axum = "0.8.9"
futures = "0.3.34"
futures-core = "0.3.34"
@@ -249,8 +249,8 @@ anyhow = "1.0.104"
arc-swap = "1.9.3"
rustfs-tokio-tar = { version = "0.7.1" }
# Candidate Snowball parser versions exercised by rustfs-zip compatibility fixtures.
tar-codec = "0.0.15"
tar-framing = "0.0.15"
tar-codec = "0.0.16"
tar-framing = "0.0.16"
atoi = "3.1.1"
atomic_enum = "0.3.0"
aws-config = { version = "1.13.0" }
@@ -296,6 +296,7 @@ local-ip-address = "0.6.13"
log = "0.4"
memmap2 = "0.9.12"
lz4 = "1.28.1"
lz4_flex = { version = "=0.14.0", default-features = false, features = ["std", "frame"] }
matchit = "0.9.2"
md-5 = "0.11.0"
mime_guess = "2.0.5"
@@ -388,7 +389,7 @@ dav-server = "0.11.0"
# Performance Analysis and Memory Profiling
rustfs-mimalloc = { version = "0.7.3" }
# Preserve Unicode focus filters until rustfs/backlog#2302 is resolved.
hotpath = { version = "0.30.3", default-features = false }
hotpath = { version = "0.31.1", default-features = false }
# Snapshot testing for output format regression detection
insta = { version = "1.49" }
+11 -2
View File
@@ -12,7 +12,9 @@
# See the License for the specific language governing permissions and
# limitations under the License.
FROM alpine:3.24.2 AS build
ARG BINARY_SOURCE=download
FROM alpine:3.24.2 AS download
ARG TARGETARCH
ARG RELEASE=latest
@@ -74,6 +76,14 @@ COPY scripts/download_rc.sh /usr/local/bin/download_rc.sh
RUN sh /usr/local/bin/download_rc.sh "$TARGETARCH" musl "$RC_RELEASE" /build/rc
# Candidate packaging supplies the accepted executables from verified archives.
# The default download stage remains available for local image builds.
FROM scratch AS supplied
ARG TARGETARCH
COPY .release-inputs/${TARGETARCH}/ /build/
FROM ${BINARY_SOURCE} AS build
FROM alpine:3.24.2
ARG RELEASE=latest
@@ -103,7 +113,6 @@ RUN apk upgrade --no-cache && \
tzdata \
&& test "$(TZ=Asia/Kolkata date +%z)" = "+0530"
COPY --from=build /etc/ssl/certs/ca-certificates.crt /etc/ssl/certs/
COPY --from=build /build/rustfs /usr/bin/rustfs
COPY --from=build /build/rc /usr/bin/rc
COPY entrypoint.sh /etc/rustfs/entrypoint.sh
+11 -1
View File
@@ -12,7 +12,9 @@
# See the License for the specific language governing permissions and
# limitations under the License.
FROM ubuntu:26.04 AS build
ARG BINARY_SOURCE=download
FROM ubuntu:26.04 AS download
ARG TARGETARCH
ARG RELEASE=latest
@@ -80,6 +82,14 @@ RUN set -eux; \
COPY scripts/download_rc.sh /usr/local/bin/download_rc.sh
RUN sh /usr/local/bin/download_rc.sh "$TARGETARCH" gnu "$RC_RELEASE" /build/rc
# Candidate packaging supplies the accepted executables from verified archives.
# The default download stage remains available for local image builds.
FROM scratch AS supplied
ARG TARGETARCH
COPY .release-inputs/${TARGETARCH}/ /build/
FROM ${BINARY_SOURCE} AS build
FROM ubuntu:26.04
ARG RELEASE=latest
+3 -3
View File
@@ -145,7 +145,7 @@ chown -R 10001:10001 data logs
docker run -d -p 9000:9000 -p 9001:9001 -v $(pwd)/data:/data -v $(pwd)/logs:/logs rustfs/rustfs:latest
# Using specific version
docker run -d -p 9000:9000 -p 9001:9001 -v $(pwd)/data:/data -v $(pwd)/logs:/logs rustfs/rustfs:1.0.4
docker run -d -p 9000:9000 -p 9001:9001 -v $(pwd)/data:/data -v $(pwd)/logs:/logs rustfs/rustfs:1.0.5
```
If you use [podman](https://github.com/containers/podman) instead of docker, you can install the RustFS with the below command
@@ -224,7 +224,7 @@ For developers who want to build RustFS Docker images from source with multi-arc
./docker-buildx.sh --push
# Build specific version
./docker-buildx.sh --release 1.0.4 --push
./docker-buildx.sh --release 1.0.5 --push
# Build for custom registry
./docker-buildx.sh --registry your-registry.com --namespace yourname --push
@@ -242,7 +242,7 @@ You can also use Make targets for convenience:
```bash
make docker-buildx # Build locally
make docker-buildx-push # Build and push
make docker-buildx-version VERSION=1.0.4 # Build specific version
make docker-buildx-version VERSION=1.0.5 # Build specific version
make help-docker # Show all Docker-related commands
```
+3 -3
View File
@@ -138,7 +138,7 @@ chown -R 10001:10001 data logs
docker run -d -p 9000:9000 -p 9001:9001 -v $(pwd)/data:/data -v $(pwd)/logs:/logs rustfs/rustfs:latest
# 使用指定版本运行
docker run -d -p 9000:9000 -p 9001:9001 -v $(pwd)/data:/data -v $(pwd)/logs:/logs rustfs/rustfs:1.0.4
docker run -d -p 9000:9000 -p 9001:9001 -v $(pwd)/data:/data -v $(pwd)/logs:/logs rustfs/rustfs:1.0.5
```
如果您通过绑定挂载启用 TLS 证书目录,也请用同样方式准备该目录:
@@ -176,7 +176,7 @@ docker compose -f docker-compose-simple.yml up -d
./docker-buildx.sh --push
# 构建指定版本
./docker-buildx.sh --release 1.0.4 --push
./docker-buildx.sh --release 1.0.5 --push
# 构建并推送到自定义仓库
./docker-buildx.sh --registry your-registry.com --namespace yourname --push
@@ -193,7 +193,7 @@ docker compose -f docker-compose-simple.yml up -d
```bash
make docker-buildx # 本地构建
make docker-buildx-push # 构建并推送
make docker-buildx-version VERSION=1.0.4 # 构建指定版本
make docker-buildx-version VERSION=1.0.5 # 构建指定版本
make help-docker # 显示所有 Docker 相关命令
```
+100
View File
@@ -902,4 +902,104 @@ mod tests {
assert_eq!(raw.len(), 8);
assert_eq!(&raw[..], reference.digest().to_be_bytes().as_slice());
}
#[test]
fn checksums_streaming_chunks_and_header_encoding_match_oneshot_across_unaligned_boundaries() {
use crate::http::HttpChecksum;
let all = [
ChecksumAlgorithm::Crc32,
ChecksumAlgorithm::Crc32c,
ChecksumAlgorithm::Crc64Nvme,
ChecksumAlgorithm::Sha1,
ChecksumAlgorithm::Sha256,
ChecksumAlgorithm::Sha512,
ChecksumAlgorithm::Xxhash3,
ChecksumAlgorithm::Xxhash64,
ChecksumAlgorithm::Xxhash128,
];
for &payload_len in &[0usize, 1, 31, 33, 1024, 4096 + 37, 64 * 1024 + 19] {
let payload: Vec<u8> = (0..payload_len).map(|i| ((i * 19 + 5) % 253) as u8).collect();
for algorithm in all {
let mut one_shot = algorithm.into_impl();
one_shot.update(&payload);
let expected_raw = one_shot.finalize();
assert_eq!(expected_raw.len(), algorithm.raw_len());
let mut one_shot_hdr = algorithm.into_impl();
one_shot_hdr.update(&payload);
let expected_headers = one_shot_hdr.headers();
for &chunk_size in &[1usize, 31, 1021, 4096] {
if payload_len > 4096 && chunk_size == 1 {
continue;
}
let mut streaming: Box<dyn HttpChecksum> = algorithm.into_impl();
for chunk in payload.chunks(chunk_size) {
streaming.update(chunk);
}
let actual_headers = streaming.headers();
assert_eq!(
actual_headers.get(algorithm.http_header_name()),
expected_headers.get(algorithm.http_header_name()),
"{algorithm:?} streaming mismatch at payload_len={payload_len}, chunk_size={chunk_size}"
);
}
}
}
}
#[test]
#[ignore = "utility throughput microbenchmark; run via cargo test -p rustfs-checksums --lib --release -- --ignored --nocapture"]
fn microbench_checksums_streaming_chunks_and_header_encoding() {
use crate::http::HttpChecksum;
use std::hint::black_box;
use std::time::Instant;
let all = [
ChecksumAlgorithm::Crc32,
ChecksumAlgorithm::Crc32c,
ChecksumAlgorithm::Crc64Nvme,
ChecksumAlgorithm::Sha1,
ChecksumAlgorithm::Sha256,
ChecksumAlgorithm::Sha512,
ChecksumAlgorithm::Xxhash3,
ChecksumAlgorithm::Xxhash64,
ChecksumAlgorithm::Xxhash128,
];
let payload: Vec<u8> = (0..(256 * 1024)).map(|i| ((i * 19 + 5) % 253) as u8).collect();
for &chunk_size in &[4 * 1024usize, 64 * 1024, 256 * 1024] {
for algorithm in all {
let mut one_shot = algorithm.into_impl();
one_shot.update(&payload);
let expected_headers = one_shot.headers();
let iters = 8usize;
let t0 = Instant::now();
let mut last_headers = http::HeaderMap::new();
for _ in 0..iters {
let mut streaming: Box<dyn HttpChecksum> = algorithm.into_impl();
for chunk in payload.chunks(chunk_size) {
streaming.update(black_box(chunk));
}
last_headers = streaming.headers();
black_box(&last_headers);
}
let elapsed = t0.elapsed();
assert_eq!(
last_headers.get(algorithm.http_header_name()),
expected_headers.get(algorithm.http_header_name()),
"{algorithm:?} streaming chunk_size={chunk_size} header mismatch"
);
let total_mib = (payload.len() * iters) as f64 / (1024.0 * 1024.0);
let mibs = total_mib / elapsed.as_secs_f64().max(1e-9);
println!(
"[microbench][checksums] algo={algorithm:?} chunk_size={chunk_size}B throughput={mibs:.1} MiB/s (size={}B)",
algorithm.raw_len()
);
}
}
}
}
+21 -9
View File
@@ -184,15 +184,27 @@ const _: () = assert!(!DEFAULT_INTERNODE_RPC_REPLAY_SCOPE_STRICT);
/// Capacity (distinct nonces) of the process-local internode RPC replay cache that enforces
/// one-time consumption of authenticated RPC signatures.
///
/// The cache retains each nonce for the ~10-minute signature freshness envelope. Once peers use
/// replay-scoped v3 authentication, every authenticated RPC consumes one entry, so the steady
/// state holds roughly `authenticated RPC RPS x 601s` entries. This default is the minimum floor:
/// explicit operator values and resource-aware auto sizing both clamp upward to at least this
/// value. Overflow fails closed — legitimate signed traffic is the only thing that can fill the
/// cache (replays are rejected before insertion, and an attacker cannot mint valid nonces without
/// the shared secret) — and increments
/// `rustfs_system_network_internode_replay_cache_overflow_total`, so a sustained non-zero overflow
/// counter means this capacity is undersized for the node's peak authenticated RPC rate.
/// Nonces remain protected until their signed timestamp is outside the observed wall-clock
/// validity window, including after clock rollback. Current-time signatures live about five
/// minutes; the maximum accepted forward clock skew can extend that to ten minutes. A body-bound
/// v3 request consumes both its v3 and compatibility-v2 nonces, so size for receiver nonce intake,
/// not S3 request rate or CPU count.
///
/// Automatic admission budgets 13% of effective process memory for signed-time
/// hash tables, allocator overhead and simultaneous old/new tables during growth.
/// The reported count is an absolute UUID payload ceiling, not guaranteed throughput:
/// actual byte admission can reject earlier depending on timestamp distribution.
/// Tables grow with demand and are released together when their signed time expires.
/// CPU count, a historical minimum, and a fixed 32M ceiling do not constrain this budget.
/// If memory detection fails, the historical default remains a bounded count fallback.
///
/// Explicit values remain count ceilings with the historical minimum of 1,048,576
/// for configuration compatibility, and bypass automatic byte admission. That
/// compatibility floor does not apply to the automatically detected memory budget.
/// Zero never disables replay protection.
/// Overflow rejects the request without evicting a still-valid identity, increments
/// `rustfs_system_network_internode_replay_cache_overflow_total`, and returns resource
/// exhaustion rather than invalid credentials. Measure headroom before overriding.
pub const ENV_INTERNODE_RPC_REPLAY_CACHE_CAPACITY: &str = "RUSTFS_INTERNODE_RPC_REPLAY_CACHE_CAPACITY";
pub const DEFAULT_INTERNODE_RPC_REPLAY_CACHE_CAPACITY: usize = 1_048_576;
+3 -2
View File
@@ -32,8 +32,9 @@ pub const ENV_OBJECT_MEDIUM_CONCURRENCY_THRESHOLD: &str = "RUSTFS_OBJECT_MEDIUM_
/// Environment variable name for maximum concurrent disk reads for object operations.
/// - Purpose: Limit the number of concurrent disk read operations for object reads to prevent I/O saturation.
/// - Unit: request count (usize).
/// - Semantics: Throttling disk reads helps maintain overall system responsiveness under load.
/// - Unit: request count (usize) for GET; estimated drive fanout for native LIST.
/// - Semantics: GET bodies and native LIST walks use separate per-instance budgets of this size.
/// A LIST consumes up to one budget, including its disk-producer cleanup. `0` disables both caps.
/// - Example: `export RUSTFS_OBJECT_MAX_CONCURRENT_DISK_READS=16`
/// - Note: This setting may interact with OS-level I/O scheduling and should be tuned based on hardware capabilities.
pub const ENV_OBJECT_MAX_CONCURRENT_DISK_READS: &str = "RUSTFS_OBJECT_MAX_CONCURRENT_DISK_READS";
+6 -1
View File
@@ -91,7 +91,12 @@ pub const ENV_OBS_LOG_MATCH_MODE: &str = "RUSTFS_OBS_LOG_MATCH_MODE";
/// Default values for log cleanup
pub const DEFAULT_OBS_LOG_MAX_TOTAL_SIZE_BYTES: u64 = 2 * 1024 * 1024 * 1024; // 2 GiB
pub const DEFAULT_OBS_LOG_MAX_SINGLE_FILE_SIZE_BYTES: u64 = 0; // No single file limit
pub const DEFAULT_OBS_LOG_MAX_SINGLE_FILE_SIZE_BYTES: u64 = 0; // No per-file trim by the cleaner
/// Size at which the active log file is rolled when
/// `RUSTFS_OBS_LOG_MAX_SINGLE_FILE_SIZE_BYTES` is unset. Without a size roll
/// the active file is bounded only by the rotation period, and the cleaner
/// never touches the active file. An explicit `0` keeps size rolling off.
pub const DEFAULT_OBS_LOG_ACTIVE_FILE_ROLL_SIZE_BYTES: u64 = 256 * 1024 * 1024; // 256 MiB
pub const DEFAULT_OBS_LOG_COMPRESS_OLD_FILES: bool = true;
pub const DEFAULT_OBS_LOG_GZIP_COMPRESSION_LEVEL: u32 = 6;
pub const DEFAULT_OBS_LOG_COMPRESSION_ALGORITHM_GZIP: &str = "gzip";
+255 -2
View File
@@ -21,12 +21,15 @@
#[cfg(test)]
mod tests {
use crate::common::{RustFSTestEnvironment, init_logging};
use crate::common::{RustFSTestEnvironment, admin_create_user, init_logging};
use aws_sdk_s3::Client;
use aws_sdk_s3::primitives::ByteStream;
use aws_sdk_s3::types::{BucketVersioningStatus, Delete, ObjectIdentifier, VersioningConfiguration};
use std::error::Error;
use tracing::info;
type TestResult = Result<(), Box<dyn Error + Send + Sync>>;
async fn create_bucket(client: &Client, bucket: &str) -> Result<(), Box<dyn Error + Send + Sync>> {
client.create_bucket().bucket(bucket).send().await?;
Ok(())
@@ -90,7 +93,7 @@ mod tests {
Ok(())
}
/// Duplicate and repeated slashes after a leading slash collapse MinIO-style.
/// Duplicate and repeated slashes after a leading slash collapse MinIO-style across single and bulk operations.
#[tokio::test]
async fn test_put_object_with_duplicate_slashes_normalized() -> Result<(), Box<dyn Error + Send + Sync>> {
init_logging();
@@ -144,8 +147,258 @@ mod tests {
"GET after DELETE with raw key must return HTTP 404, got {error:?}"
);
assert_bulk_delete_normalizes_keys(&client).await?;
assert_bulk_delete_authorizes_normalized_keys(&env, &client).await?;
assert_bulk_delete_preserves_version_identity(&client).await?;
env.stop_server();
info!("Test completed successfully");
Ok(())
}
async fn assert_bulk_delete_normalizes_keys(client: &Client) -> TestResult {
let bucket = "test-bulk-leading-slash";
create_bucket(client, bucket).await?;
let cases = [
("/foo/bar", "foo/bar"),
("//keyname", "keyname"),
("/dir///sub//file", "dir/sub/file"),
("/folder///", "folder/"),
("/literal%2F+名", "literal%2F+名"),
("plain", "plain"),
];
let invalid = ["///", "/dir/../keep", "dir//keep"];
client
.put_object()
.bucket(bucket)
.key("keep")
.body(ByteStream::from_static(b"keep"))
.send()
.await?;
client
.put_object()
.bucket(bucket)
.key("dir/keep")
.body(ByteStream::from_static(b"keep"))
.send()
.await?;
for quiet in [false, true] {
for (raw, _) in cases {
client
.put_object()
.bucket(bucket)
.key(raw)
.body(ByteStream::from_static(b""))
.send()
.await?;
}
let listing = client.list_objects_v2().bucket(bucket).send().await?;
let stored: Vec<_> = listing.contents().iter().filter_map(|object| object.key()).collect();
let mut expected: Vec<_> = cases
.iter()
.map(|(_, normalized)| *normalized)
.chain(["dir/keep", "keep"])
.collect();
expected.sort_unstable();
assert_eq!(stored, expected, "PUT must store the exact normalized targets before bulk deletion");
let objects = cases
.iter()
.map(|(raw, _)| *raw)
.chain(invalid)
.map(|key| ObjectIdentifier::builder().key(key).build())
.collect::<Result<Vec<_>, _>>()?;
let delete = Delete::builder().set_objects(Some(objects)).quiet(quiet).build()?;
// A second identical batch also pins idempotent deletion of absent normalized keys.
for _ in 0..2 {
let output = client.delete_objects().bucket(bucket).delete(delete.clone()).send().await?;
let errors: Vec<_> = output.errors().iter().map(|error| (error.key(), error.code())).collect();
assert_eq!(
errors,
invalid.map(|key| (Some(key), Some("InvalidArgument"))),
"unexpected per-key errors: {output:?}"
);
let deleted: Vec<_> = output.deleted().iter().filter_map(|object| object.key()).collect();
let expected = if quiet {
vec![]
} else {
cases.iter().map(|(raw, _)| *raw).collect()
};
assert_eq!(deleted, expected, "responses must retain request keys, including directory slashes");
}
let listing = client.list_objects_v2().bucket(bucket).send().await?;
let remaining: Vec<_> = listing.contents().iter().filter_map(|object| object.key()).collect();
assert_eq!(
remaining,
["dir/keep", "keep"],
"bulk delete must remove normalized objects without resolving dot segments"
);
}
Ok(())
}
async fn assert_bulk_delete_authorizes_normalized_keys(env: &RustFSTestEnvironment, admin: &Client) -> TestResult {
let bucket = "test-bulk-slash-policy";
let user = "bulk-slash-user";
admin_create_user(env, user, "bulkSlashTestSecret123!").await?;
create_bucket(admin, bucket).await?;
for key in ["allowed/item", "protected/item"] {
admin
.put_object()
.bucket(bucket)
.key(key)
.body(ByteStream::from_static(b"keep"))
.send()
.await?;
}
let policy = serde_json::json!({
"Version": "2012-10-17",
"Statement": [
{"Effect": "Allow", "Principal": {"AWS": [user]}, "Action": ["s3:DeleteObject"],
"Resource": [format!("arn:aws:s3:::{bucket}/*")]},
{"Effect": "Deny", "Principal": {"AWS": [user]}, "Action": ["s3:DeleteObject"],
"Resource": [format!("arn:aws:s3:::{bucket}/protected/*")]}
]
});
admin
.put_bucket_policy()
.bucket(bucket)
.policy(policy.to_string())
.send()
.await?;
let client = env.create_s3_client_with_credentials(user, "bulkSlashTestSecret123!");
let output = client
.delete_objects()
.bucket(bucket)
.delete(
Delete::builder()
.objects(ObjectIdentifier::builder().key("/allowed///item").build()?)
.objects(ObjectIdentifier::builder().key("/protected///item").build()?)
.build()?,
)
.send()
.await?;
assert_eq!(output.deleted().len(), 1, "allowed normalized key must be deleted: {output:?}");
assert_eq!(output.deleted()[0].key(), Some("/allowed///item"));
assert_eq!(output.errors().len(), 1, "normalized resource denial must apply: {output:?}");
assert_eq!(output.errors()[0].key(), Some("/protected///item"));
assert_eq!(output.errors()[0].code(), Some("AccessDenied"));
let listing = admin.list_objects_v2().bucket(bucket).send().await?;
let remaining: Vec<_> = listing.contents().iter().filter_map(|object| object.key()).collect();
assert_eq!(remaining, ["protected/item"], "only the authorized object may be removed");
let protected = admin.get_object().bucket(bucket).key("protected/item").send().await?;
assert_eq!(protected.body.collect().await?.into_bytes().as_ref(), b"keep");
Ok(())
}
async fn assert_bulk_delete_preserves_version_identity(client: &Client) -> TestResult {
let bucket = "test-bulk-slash-versions";
let raw = "/history///file";
let normalized = "history/file";
create_bucket(client, bucket).await?;
client
.put_bucket_versioning()
.bucket(bucket)
.versioning_configuration(
VersioningConfiguration::builder()
.status(BucketVersioningStatus::Enabled)
.build(),
)
.send()
.await?;
let first = client
.put_object()
.bucket(bucket)
.key(raw)
.body(ByteStream::from_static(b"old"))
.send()
.await?;
let first_id = first.version_id().ok_or("first PUT omitted version ID")?;
let latest = client
.put_object()
.bucket(bucket)
.key(raw)
.body(ByteStream::from_static(b"latest"))
.send()
.await?;
let latest_id = latest.version_id().ok_or("latest PUT omitted version ID")?;
let invalid = client
.delete_objects()
.bucket(bucket)
.delete(
Delete::builder()
.objects(ObjectIdentifier::builder().key("///").build()?)
.build()?,
)
.send()
.await?;
assert!(invalid.deleted().is_empty(), "slash-only keys must not delete a versioned object");
assert_eq!(
invalid.errors().len(),
1,
"slash-only keys must remain invalid in versioned buckets: {invalid:?}"
);
assert_eq!(invalid.errors()[0].key(), Some("///"));
assert_eq!(invalid.errors()[0].code(), Some("InvalidArgument"));
let deleted = client
.delete_objects()
.bucket(bucket)
.delete(
Delete::builder()
.objects(ObjectIdentifier::builder().key(raw).version_id(first_id).build()?)
.build()?,
)
.send()
.await?;
assert!(deleted.errors().is_empty(), "historical delete failed: {deleted:?}");
assert_eq!(deleted.deleted().len(), 1);
assert_eq!(deleted.deleted()[0].key(), Some(raw));
assert_eq!(deleted.deleted()[0].version_id(), Some(first_id));
let listing = client.list_object_versions().bucket(bucket).send().await?;
assert_eq!(listing.versions().len(), 1, "only the latest data version should remain: {listing:?}");
assert_eq!(listing.versions()[0].version_id(), Some(latest_id));
let deleted = client
.delete_objects()
.bucket(bucket)
.delete(
Delete::builder()
.objects(ObjectIdentifier::builder().key(raw).build()?)
.build()?,
)
.send()
.await?;
assert!(deleted.errors().is_empty(), "delete marker creation failed: {deleted:?}");
assert_eq!(deleted.deleted().len(), 1);
assert_eq!(deleted.deleted()[0].key(), Some(raw));
assert_eq!(deleted.deleted()[0].delete_marker(), Some(true));
let marker = deleted.deleted()[0]
.delete_marker_version_id()
.ok_or("missing delete marker ID")?;
let listing = client.list_object_versions().bucket(bucket).send().await?;
assert_eq!(listing.delete_markers().len(), 1);
assert_eq!(listing.delete_markers()[0].key(), Some(normalized));
assert_eq!(listing.delete_markers()[0].version_id(), Some(marker));
let deleted = client
.delete_objects()
.bucket(bucket)
.delete(
Delete::builder()
.objects(ObjectIdentifier::builder().key(raw).version_id(marker).build()?)
.build()?,
)
.send()
.await?;
assert!(deleted.errors().is_empty(), "explicit marker removal failed: {deleted:?}");
assert_eq!(deleted.deleted().len(), 1);
assert_eq!(deleted.deleted()[0].key(), Some(raw));
assert_eq!(deleted.deleted()[0].version_id(), Some(marker));
let listing = client.list_object_versions().bucket(bucket).send().await?;
assert!(listing.delete_markers().is_empty());
let restored = client.get_object().bucket(bucket).key(normalized).send().await?;
assert_eq!(restored.version_id(), Some(latest_id));
assert_eq!(restored.body.collect().await?.into_bytes().as_ref(), b"latest");
Ok(())
}
}
@@ -114,10 +114,10 @@ fn staged_commit(metadata: &Value, location: &str) -> Value {
})
}
async fn location_claim_preserves_other_users_objects_after_drop_and_restart(backing: &str) -> TestResult {
async fn location_claim_preserves_other_users_objects_after_drop_and_restart() -> TestResult {
init_logging();
let mut env = RustFSTestEnvironment::new().await?;
let server_env = [("RUSTFS_TABLE_CATALOG_BACKING", backing)];
let server_env = [("RUSTFS_TABLE_CATALOG_BACKING", "object")];
env.start_rustfs_server_without_cleanup_with_env(&[("RUSTFS_TABLE_CATALOG_BACKING", "object")])
.await?;
env.wait_for_server_ready().await?;
@@ -147,39 +147,6 @@ async fn location_claim_preserves_other_users_objects_after_drop_and_restart(bac
StatusCode::OK,
)
.await?;
if backing == "durable-strong" {
catalog_request(
&env,
&root,
Method::POST,
&format!("/iceberg/v1/{bucket}/catalog/warehouse-index/backfill"),
None,
StatusCode::NO_CONTENT,
)
.await?;
let preflight = catalog_request(
&env,
&root,
Method::GET,
&format!("/iceberg/v1/{bucket}/catalog/migration"),
None,
StatusCode::OK,
)
.await?;
assert_eq!(preflight["blockers"], json!([]), "{preflight}");
let migration = catalog_request(
&env,
&root,
Method::POST,
&format!("/iceberg/v1/{bucket}/catalog/migration"),
None,
StatusCode::OK,
)
.await?;
assert_eq!(migration["ready_to_enable_durable_strong"], json!(true), "{migration}");
env.restart_server_preserving_data(vec![], &server_env).await?;
env.wait_for_server_ready().await?;
}
admin_create_user(&env, &creator.0, &creator.1).await?;
admin_create_user(&env, &reader.0, &reader.1).await?;
let create_statement = json!({
@@ -341,7 +308,7 @@ async fn location_claim_preserves_other_users_objects_after_drop_and_restart(bac
StatusCode::FORBIDDEN,
)
.await?;
if backing == "object" {
{
catalog_request(
&env,
&creator,
@@ -408,10 +375,5 @@ async fn location_claim_preserves_other_users_objects_after_drop_and_restart(bac
#[tokio::test]
async fn object_catalog_location_claim_preserves_objects_after_drop_and_restart() -> TestResult {
location_claim_preserves_other_users_objects_after_drop_and_restart("object").await
}
#[tokio::test]
async fn durable_strong_catalog_location_claim_preserves_objects_after_drop_and_restart() -> TestResult {
location_claim_preserves_other_users_objects_after_drop_and_restart("durable-strong").await
location_claim_preserves_other_users_objects_after_drop_and_restart().await
}
+1
View File
@@ -121,6 +121,7 @@ test-util = []
e2e-test-hooks = []
[dependencies]
hashbrown.workspace = true
hotpath.workspace = true
rustfs-filemeta.workspace = true
rustfs-utils = { workspace = true, features = ["full"] }
+11 -11
View File
@@ -640,17 +640,17 @@ pub mod rpc {
pub use crate::cluster::rpc::{
AuthenticatedChannel, KMS_SIGNAL_SUBSYSTEM, LocalPeerS3Client, MAX_NETWORK_PROBE_BYTES, MAX_NETWORK_PROBE_DURATION,
NetworkPeerProbeClient, NetworkPeerProbeError, NetworkPeerProbeMeasurement, NetworkPeerTarget, PEER_RESTDRY_RUN,
PEER_RESTSIGNAL, PEER_RESTSUB_SYS, PeerRestClient, PeerS3Client, S3PeerSys, SERVICE_SIGNAL_REFRESH_CONFIG,
SERVICE_SIGNAL_RELOAD_DYNAMIC, ScannerBucketListing, ScannerDirtyUsageAcknowledgement, ScannerPeerActivity,
ScannerPeerDirtyUsageBucket, ScannerPeerDirtyUsageSnapshot, ScannerPublicationLease, ScannerScopedDirtyUsageAckEntry,
TONIC_RPC_PREFIX, TonicInterceptor, build_put_file_auth_trailer, check_and_record_signed_rpc_nonce,
decode_heal_bucket_rpc_options, encode_heal_bucket_rpc_options, gen_signature_headers, gen_tonic_replay_scope_headers,
gen_tonic_signature_headers, gen_tonic_signature_interceptor, node_service_time_out_client,
node_service_time_out_client_no_auth, normalize_tonic_rpc_audience, rpc_background_admission,
set_tonic_canonical_body_digest, sign_ns_scanner_capability, sign_ns_scanner_capability_with_checkpoint_coverage,
sign_ns_scanner_capability_with_tier_registry_generation, sign_put_file_capability, sign_tonic_rpc_response_proof,
tonic_boot_epoch_challenge, tonic_boot_epoch_response_headers, tonic_rpc_auth_failure_reason,
verify_ns_scanner_capability, verify_ns_scanner_capability_with_checkpoint_coverage,
PEER_RESTSIGNAL, PEER_RESTSUB_SYS, PeerRestClient, PeerS3Client, RpcReplayCacheAdmissionError, S3PeerSys,
SERVICE_SIGNAL_REFRESH_CONFIG, SERVICE_SIGNAL_RELOAD_DYNAMIC, ScannerBucketListing, ScannerDirtyUsageAcknowledgement,
ScannerPeerActivity, ScannerPeerDirtyUsageBucket, ScannerPeerDirtyUsageSnapshot, ScannerPublicationLease,
ScannerScopedDirtyUsageAckEntry, TONIC_RPC_PREFIX, TonicInterceptor, build_put_file_auth_trailer,
check_and_record_signed_rpc_nonce, decode_heal_bucket_rpc_options, encode_heal_bucket_rpc_options, gen_signature_headers,
gen_tonic_replay_scope_headers, gen_tonic_signature_headers, gen_tonic_signature_interceptor,
node_service_time_out_client, node_service_time_out_client_no_auth, normalize_tonic_rpc_audience,
rpc_auth_is_resource_exhausted, rpc_background_admission, set_tonic_canonical_body_digest, sign_ns_scanner_capability,
sign_ns_scanner_capability_with_checkpoint_coverage, sign_ns_scanner_capability_with_tier_registry_generation,
sign_put_file_capability, sign_tonic_rpc_response_proof, tonic_boot_epoch_challenge, tonic_boot_epoch_response_headers,
tonic_rpc_auth_failure_reason, verify_ns_scanner_capability, verify_ns_scanner_capability_with_checkpoint_coverage,
verify_ns_scanner_capability_with_tier_registry_generation, verify_put_file_auth_trailer, verify_put_file_capability,
verify_rpc_signature, verify_tonic_boot_epoch_response, verify_tonic_canonical_body_digest,
verify_tonic_mutation_body_digest, verify_tonic_mutation_body_digest_reject_unsigned, verify_tonic_rpc_response_proof,
+3 -38
View File
@@ -17,7 +17,7 @@
use crate::bucket::metadata::BUCKET_METADATA_FILE;
use crate::bucket::replication::ReplicationMigrationBridge;
use crate::disk::{BUCKET_META_PREFIX, MIGRATING_META_BUCKET, RUSTFS_META_BUCKET};
use crate::error::{Error, Result, is_err_strict_not_found, is_err_strict_volume_not_found, to_object_err};
use crate::error::{Error, Result, is_err_strict_not_found, is_err_strict_volume_not_found};
use crate::object_api::{GetObjectReader, ObjectInfo, ObjectOptions, PutObjReader};
use crate::storage_api_contracts::{
bucket::{BucketInfo, BucketOperations, BucketOptions},
@@ -334,14 +334,6 @@ async fn migration_target_exists<S: EcstoreObjectOperations>(store: &S, path: &s
}
}
/// Names the target record in a failed migration write, as the existence
/// probe's error already does. The commit path returns bare disk errors, and on
/// Windows a blocked record is first seen here: reading through a regular-file
/// ancestor reports the record missing instead of refusing access to it.
fn migration_write_error(err: Error, path: &str) -> Error {
to_object_err(err, vec![RUSTFS_META_BUCKET, path])
}
async fn migrate_one_if_missing<S>(
store: Arc<S>,
opts: &ObjectOptions,
@@ -377,10 +369,7 @@ where
.unwrap_or(data);
let mut put_data = PutObjReader::from_vec(data);
store
.put_object(RUSTFS_META_BUCKET, path, &mut put_data, opts)
.await
.map_err(|err| migration_write_error(err, path))?;
store.put_object(RUSTFS_META_BUCKET, path, &mut put_data, opts).await?;
info!("Migrated {label}");
Ok(())
}
@@ -492,10 +481,7 @@ where
Err(_) => return Err(MigrationMetadataError::Incompatible(path.to_owned()).into()),
};
let mut put_data = PutObjReader::from_vec(data);
store
.put_object(RUSTFS_META_BUCKET, path, &mut put_data, &opts)
.await
.map_err(|err| migration_write_error(err, path))?;
store.put_object(RUSTFS_META_BUCKET, path, &mut put_data, &opts).await?;
info!("Migrated IAM config: {path}");
total_migrated += 1;
}
@@ -545,27 +531,6 @@ mod tests {
));
}
#[test]
fn migration_write_errors_name_the_target_record_without_fragmenting_groups() {
use super::{Error, migration_write_error};
let first = migration_write_error(Error::FileAccessDenied, "buckets/first/.metadata.bin");
let second = migration_write_error(Error::FileAccessDenied, "config/iam/format.json");
assert!(matches!(first, Error::PrefixAccessDenied(..)), "unexpected error: {first:?}");
assert!(
first.to_string().contains(".rustfs.sys/buckets/first/.metadata.bin"),
"the startup error must name the blocked record: {first}"
);
assert_eq!(first, second, "record paths must not fragment error grouping");
let unrelated = migration_write_error(Error::other("disk offline"), "buckets/first/.metadata.bin");
assert_eq!(
unrelated,
Error::other("disk offline"),
"errors without a record form pass through unchanged"
);
}
#[test]
fn migration_errors_group_by_cause_and_retain_typed_record_context() {
use super::{Error, MigrationMetadataError};
@@ -217,6 +217,8 @@ impl Drop for NotifyOnDrop {
#[derive(Default)]
pub struct ListPathRawOptions {
/// Retain native listing admission through producer cancellation/draining.
pub read_budget: Option<Arc<tokio::sync::OwnedSemaphorePermit>>,
pub disks: Vec<Option<DiskStore>>,
pub fallback_disks: Vec<Option<DiskStore>>,
pub bucket: String,
@@ -227,6 +229,7 @@ pub struct ListPathRawOptions {
pub hidden_entry_budget: usize,
pub pending_prefix: Option<String>,
pub filter_prefix: Option<String>,
pub identifier_tree_leaf: Option<String>,
pub forward_to: Option<String>,
pub min_disks: usize,
/// Deliver every replica to the partial callback, including matching headers.
@@ -255,6 +258,7 @@ pub struct ListPathRawOptions {
impl Clone for ListPathRawOptions {
fn clone(&self) -> Self {
Self {
read_budget: self.read_budget.clone(),
disks: self.disks.clone(),
fallback_disks: self.fallback_disks.clone(),
bucket: self.bucket.clone(),
@@ -265,6 +269,7 @@ impl Clone for ListPathRawOptions {
hidden_entry_budget: self.hidden_entry_budget,
pending_prefix: self.pending_prefix.clone(),
filter_prefix: self.filter_prefix.clone(),
identifier_tree_leaf: self.identifier_tree_leaf.clone(),
forward_to: self.forward_to.clone(),
min_disks: self.min_disks,
preserve_replica_metadata: self.preserve_replica_metadata,
@@ -297,6 +302,7 @@ fn walk_dir_options(opts: &ListPathRawOptions) -> WalkDirOptions {
pending_prefix: opts.pending_prefix.clone(),
report_notfound: opts.report_not_found,
filter_prefix: opts.filter_prefix.clone(),
identifier_tree_leaf: opts.identifier_tree_leaf.clone(),
forward_to: opts.forward_to.clone(),
limit: opts.per_disk_limit,
skip_total_timeout: opts.skip_walkdir_total_timeout,
@@ -389,7 +395,9 @@ async fn list_path_raw_inner(
let disk_io_context_clone = disk_io_context.clone();
let (rd, wr) = tokio::io::duplex(64 * 1024);
readers.push(MetacacheReader::new(rd));
let read_budget = opts.read_budget.clone();
let producer = with_optional_disk_io_context(disk_io_context_clone, async move {
let _read_budget = read_budget;
let mut wr = PublishedBytesWriter::new(wr);
#[cfg(test)]
if let Some(observer) = &test_disk_io_context_observed {
@@ -1228,6 +1236,21 @@ mod tests {
assert!(!options.skip_total_timeout);
}
#[test]
fn identifier_tree_selection_survives_job_clone_and_disk_request() {
let request = ListPathRawOptions {
path: "catalog/namespaces/".to_string(),
identifier_tree_leaf: Some("table-entry.json".to_string()),
recursive: true,
..Default::default()
};
let cloned = request.clone();
let walk = walk_dir_options(&cloned);
assert_eq!(walk.identifier_tree_leaf, request.identifier_tree_leaf);
assert_eq!(walk.base_dir, request.path);
assert!(walk.recursive);
}
#[test]
fn cloned_reader_options_preserve_hidden_scan_budget_and_pending_prefix() {
let request = ListPathRawOptions {
@@ -1897,6 +1920,42 @@ mod tests {
assert_eq!(seen.lock().expect("seen mutex poisoned").as_slice(), &["bucket/object".to_string()]);
}
#[tokio::test]
async fn list_read_budget_is_retained_by_raw_producer_until_cancellation() {
let semaphore = Arc::new(tokio::sync::Semaphore::new(4));
let grant = Arc::new(semaphore.clone().acquire_many_owned(4).await.expect("listing grant"));
let weak_grant = Arc::downgrade(&grant);
let entered = Arc::new(tokio::sync::Notify::new());
let stopped = Arc::new(tokio::sync::Notify::new());
let worker = tokio::spawn(list_path_raw(
CancellationToken::new(),
ListPathRawOptions {
read_budget: Some(grant),
disks: vec![None],
min_disks: 1,
test_reader_behaviors: vec![TestReaderBehavior::StallWithSignals {
entered: entered.clone(),
stopped: stopped.clone(),
}],
..Default::default()
},
));
timeout(Duration::from_secs(5), entered.notified())
.await
.expect("producer entered");
assert!(weak_grant.strong_count() >= 2, "producer must retain a grant independently of the parent");
assert_eq!(semaphore.available_permits(), 0);
worker.abort();
assert!(worker.await.expect_err("parent cancelled").is_cancelled());
timeout(Duration::from_secs(5), stopped.notified())
.await
.expect("producer stopped");
let _next = timeout(Duration::from_secs(5), semaphore.acquire_many(4))
.await
.expect("cancelled listing releases all permits")
.expect("budget stays open");
}
#[tokio::test]
async fn list_path_raw_aborts_unresponsive_producer_after_quorum_eof() {
let result = timeout(
File diff suppressed because it is too large Load Diff
+11 -11
View File
@@ -35,17 +35,17 @@ pub use client::{
// Re-exported through `api::rpc`; not every item is consumed inside this crate.
#[allow(unused_imports)]
pub use http_auth::{
RPC_BACKGROUND_ADMISSION_HEADER, TONIC_RPC_PREFIX, build_auth_headers, build_put_file_auth_trailer,
check_and_record_signed_rpc_nonce, gen_signature_headers, gen_tonic_replay_scope_headers, gen_tonic_signature_headers,
normalize_tonic_rpc_audience, rpc_background_admission, set_tonic_canonical_body_digest, set_tonic_mutation_body_digest,
set_tonic_rolling_canonical_body_digest, set_tonic_rolling_mutation_body_digest, sign_ns_scanner_capability,
sign_ns_scanner_capability_with_checkpoint_coverage, sign_ns_scanner_capability_with_tier_registry_generation,
sign_put_file_capability, sign_tonic_rpc_response_proof, tonic_boot_epoch_challenge, tonic_boot_epoch_response_headers,
tonic_rpc_auth_failure_reason, verify_ns_scanner_capability, verify_ns_scanner_capability_with_checkpoint_coverage,
verify_ns_scanner_capability_with_tier_registry_generation, verify_put_file_auth_trailer, verify_put_file_capability,
verify_rpc_signature, verify_tonic_boot_epoch_response, verify_tonic_canonical_body_digest,
verify_tonic_mutation_body_digest, verify_tonic_mutation_body_digest_reject_unsigned, verify_tonic_rpc_response_proof,
verify_tonic_rpc_signature, verify_tonic_rpc_signature_with_bootstrap,
RPC_BACKGROUND_ADMISSION_HEADER, RpcReplayCacheAdmissionError, TONIC_RPC_PREFIX, build_auth_headers,
build_put_file_auth_trailer, check_and_record_signed_rpc_nonce, gen_signature_headers, gen_tonic_replay_scope_headers,
gen_tonic_signature_headers, normalize_tonic_rpc_audience, rpc_auth_is_resource_exhausted, rpc_background_admission,
set_tonic_canonical_body_digest, set_tonic_mutation_body_digest, set_tonic_rolling_canonical_body_digest,
set_tonic_rolling_mutation_body_digest, sign_ns_scanner_capability, sign_ns_scanner_capability_with_checkpoint_coverage,
sign_ns_scanner_capability_with_tier_registry_generation, sign_put_file_capability, sign_tonic_rpc_response_proof,
tonic_boot_epoch_challenge, tonic_boot_epoch_response_headers, tonic_rpc_auth_failure_reason, verify_ns_scanner_capability,
verify_ns_scanner_capability_with_checkpoint_coverage, verify_ns_scanner_capability_with_tier_registry_generation,
verify_put_file_auth_trailer, verify_put_file_capability, verify_rpc_signature, verify_tonic_boot_epoch_response,
verify_tonic_canonical_body_digest, verify_tonic_mutation_body_digest, verify_tonic_mutation_body_digest_reject_unsigned,
verify_tonic_rpc_response_proof, verify_tonic_rpc_signature, verify_tonic_rpc_signature_with_bootstrap,
};
#[cfg(test)]
pub(crate) use internode_data_transport::TcpHttpInternodeDataTransport;
@@ -18,7 +18,7 @@ use crate::cluster::rpc::client::{
node_service_time_out_client,
};
use crate::cluster::rpc::set_tonic_mutation_body_digest;
use crate::core::pools::{PoolMeta, PoolMetaWriteState};
use crate::core::pools::{PoolMeta, PoolMetaSaveGate, PoolMetaSaveGateWriteGuard};
use crate::disk::error::DiskError;
use crate::disk::error::{Error, Result};
use crate::disk::error_reduce::{BUCKET_OP_IGNORED_ERRS, is_all_buckets_not_found, reduce_write_quorum_errs};
@@ -798,12 +798,12 @@ fn try_acquire_bucket_heal_movement_guard<'a>(
}
async fn acquire_bucket_heal_write_guard<'a>(
gate: Option<&'a tokio::sync::Mutex<PoolMetaWriteState>>,
) -> Result<Option<tokio::sync::MutexGuard<'a, PoolMetaWriteState>>> {
gate: Option<&'a PoolMetaSaveGate>,
) -> Result<Option<PoolMetaSaveGateWriteGuard<'a>>> {
let Some(gate) = gate else {
return Ok(None);
};
let guard = gate.lock().await;
let guard = gate.write().await;
guard
.ensure_write_safe("bucket heal cannot run while pool metadata requires recovery")
.map_err(|e| e.narrow_to_disk().unwrap_or_else(Error::other))?;
@@ -2388,8 +2388,8 @@ mod tests {
#[tokio::test]
async fn local_bucket_heal_refuses_receiver_with_unsafe_pool_metadata() {
let gate = tokio::sync::Mutex::new(PoolMetaWriteState::default());
gate.lock().await.observe_replicas(PoolMetaReplicaState {
let gate = PoolMetaSaveGate::default();
gate.write().await.observe_replicas(PoolMetaReplicaState {
needs_repair: true,
repair_write_safe: false,
});
+54 -1
View File
@@ -376,6 +376,21 @@ struct WalkCopyError {
source: io::Error,
}
impl WalkCopyError {
fn into_disk_error(self) -> DiskError {
// Only a local output write/flush proves the metacache consumer closed.
// An identical I/O kind from the remote input remains a transport failure.
if matches!(self.reason, RemoteWalkFailureReason::Write | RemoteWalkFailureReason::Flush)
&& self.source.kind() == io::ErrorKind::BrokenPipe
&& self.source.get_ref().is_none()
{
DiskError::metacache_output_stream_closed()
} else {
DiskError::Io(self.source)
}
}
}
async fn copy_stream_with_buffer<R, W>(
reader: &mut R,
writer: &mut W,
@@ -2096,6 +2111,7 @@ impl RemoteDisk {
failure_health_action: FailureHealthAction,
) {
if let Err(err) = operation_result
&& !err.is_metacache_output_stream_closed()
&& is_network_like_disk_error(err)
{
counter!(
@@ -4056,7 +4072,7 @@ impl DiskAPI for RemoteDisk {
Ok(_) => return Ok(()),
Err(error) => {
failure_reason = Some(error.reason);
return Err(DiskError::Io(error.source));
return Err(error.into_disk_error());
}
}
}
@@ -9687,6 +9703,43 @@ mod tests {
}
}
#[tokio::test]
async fn remote_walk_preserves_local_metacache_consumer_close() {
let transport =
RetryingWalkDirInternodeDataTransport::with_steps(vec![WalkDirTestStep::Data(b"metacache payload".to_vec())]);
let disk = new_remote_disk_with_transport(Arc::new(transport.clone())).await;
let (mut writer, reader) = tokio::io::duplex(1);
drop(reader);
let error = disk
.walk_dir(WalkDirOptions::default(), &mut writer)
.await
.expect_err("a dropped metacache consumer must stop the producer");
assert!(
error.is_metacache_output_stream_closed(),
"preserve the benign completion marker: {error:?}"
);
assert_eq!(transport.calls().len(), 1, "local consumer closure is not a remote open retry");
assert!(!disk.health.is_faulty());
}
#[tokio::test]
async fn remote_walk_keeps_remote_read_broken_pipe_as_transport_failure() {
let transport = RetryingWalkDirInternodeDataTransport::with_steps(vec![WalkDirTestStep::PartialDataThenError {
data: b"partial metacache".to_vec(),
error: io::ErrorKind::BrokenPipe.into(),
}]);
let disk = new_remote_disk_with_transport(Arc::new(transport.clone())).await;
let mut writer = Vec::new();
let error = disk
.walk_dir(WalkDirOptions::default(), &mut writer)
.await
.expect_err("an upstream close after partial output is a failed walk");
assert!(!error.is_metacache_output_stream_closed());
assert!(error.contains_io_error_kind(io::ErrorKind::BrokenPipe));
assert_eq!(writer, b"partial metacache");
assert_eq!(transport.calls().len(), 1, "partial output must not be replayed");
}
#[tokio::test]
async fn walk_stream_backpressure_limits_read_ahead_and_is_cancellable() {
let mut reader = Cursor::new(vec![5; 4 * DEFAULT_READ_BUFFER_SIZE]);
@@ -423,11 +423,15 @@ impl RemoteClient {
/// peer toward the offline threshold. See issue #4567.
fn is_transport_failure(status: &tonic::Status) -> bool {
use tonic::Code;
// A valid stale-epoch request is rejected before execution. Its signed
// response refreshes the peer epoch on the existing healthy channel.
let epoch_rejection = status.code() == Code::Unavailable && status.message() == "RPC boot epoch changed";
std::error::Error::source(status).is_some()
|| matches!(
status.code(),
Code::Unavailable | Code::DeadlineExceeded | Code::Unknown | Code::Cancelled
)
|| (!epoch_rejection
&& matches!(
status.code(),
Code::Unavailable | Code::DeadlineExceeded | Code::Unknown | Code::Cancelled
))
}
async fn evict_connection(&self, op: &'static str, reason: &str, resource_summary: &str) {
@@ -2827,6 +2831,37 @@ mod tests {
}
}
#[tokio::test]
async fn stale_boot_epoch_rejections_do_not_open_lock_request_breaker() {
let addr = "http://stale-boot-epoch-lock-breaker:9000";
let client = RemoteClient::new(addr.to_string());
reset_lock_peer_health_for_test(addr);
for _ in 0..LOCK_RPC_REQUEST_BACKOFF_THRESHOLD {
let error = client
.execute_rpc::<(), _>(
"lock",
"bucket/new-object",
Duration::from_secs(1),
async { Err(tonic::Status::unavailable("RPC boot epoch changed")) },
RpcCleanup::new(None, None, None),
)
.await
.expect_err("a stale epoch still rejects the operation before execution");
assert!(error.to_string().contains("RPC boot epoch changed"));
}
let health = lock_peer_health_for_test(addr);
assert_eq!(health.request_consecutive_timeouts, 0);
assert!(health.request_backoff_until.is_none());
assert!(health.last_eviction.is_none());
let admission = client
.admit_lock_request("lock")
.expect("authenticated epoch refresh may proceed immediately");
client.release_lock_request("lock", Some(admission));
reset_lock_peer_health_for_test(addr);
}
#[test]
#[serial_test::serial]
fn test_remote_client_rpc_timeout_honors_configured_deadline() {
@@ -628,7 +628,7 @@ async fn pool_meta_native_physical_runtime_fork_after_apply_latches_original_gat
let fork = identity_epoch_fork(&fixture).await;
let since = OffsetDateTime::from_unix_timestamp(1_700_000_000).expect("fixed original failure timestamp should be valid");
{
let mut state = fixture.store.pool_meta_save_gate.lock().await;
let mut state = fixture.store.pool_meta_save_gate.write().await;
assert_eq!(state.active_transactions.load(Ordering::SeqCst), 0);
state.write_blocked = false;
state.block_context = None;
@@ -685,7 +685,7 @@ async fn pool_meta_native_physical_runtime_fork_after_apply_latches_original_gat
);
assert_committed(&fixture).await;
{
let state = fixture.store.pool_meta_save_gate.lock().await;
let state = fixture.store.pool_meta_save_gate.write().await;
assert!(
state.write_blocked,
"candidate validation must propagate its permanent latch to the original state"
@@ -706,7 +706,7 @@ async fn pool_meta_native_physical_runtime_fork_after_apply_latches_original_gat
.expect("permanently blocked recovery should return without mutation"),
"ordinary-readable repaired heads must not clear the original known-fork latch"
);
let state = fixture.store.pool_meta_save_gate.lock().await;
let state = fixture.store.pool_meta_save_gate.write().await;
assert!(state.write_blocked);
assert_eq!(
state
@@ -979,7 +979,7 @@ async fn pool_meta_native_heal_v1_with_all_upgrade_gates_disabled_keeps_exact_ra
async {
let (fixture, raw) = legacy_native_owner_fixture(POOL_META_V1_VERSION, false).await;
let guard = owner(&fixture.store).await;
let mut state = fixture.store.pool_meta_save_gate.lock().await;
let mut state = fixture.store.pool_meta_save_gate.write().await;
let result = tokio::time::timeout(
std::time::Duration::from_secs(30),
fixture.store.heal_pool_meta_replica_under_fence(0, guard, &mut state),
@@ -1036,7 +1036,7 @@ async fn pool_meta_native_heal_rejects_legacy_source_with_trailing_bytes_without
let before = snapshots(&fixture.store).await;
let identities = owner_repair_identity_heads(&fixture.store).await;
let guard = owner(&fixture.store).await;
let mut state = fixture.store.pool_meta_save_gate.lock().await;
let mut state = fixture.store.pool_meta_save_gate.write().await;
let result = tokio::time::timeout(
std::time::Duration::from_secs(30),
fixture.store.heal_pool_meta_replica_under_fence(1, guard, &mut state),
@@ -1110,7 +1110,7 @@ async fn pool_meta_native_heal_mixed_v1_v2_preserves_observed_format_floor() {
assert_eq!(selection.revision.version, POOL_META_VERSION, "the observed V2 format floor must be retained");
assert_eq!(selection.canonical.as_deref(), Some(semantic_v2.as_slice()));
let guard = owner(&fixture.store).await;
let mut state = fixture.store.pool_meta_save_gate.lock().await;
let mut state = fixture.store.pool_meta_save_gate.write().await;
tokio::time::timeout(std::time::Duration::from_secs(30),
fixture.store.heal_pool_meta_replica_under_fence(target_pool, guard, &mut state))
.await.expect("mixed legacy heal must terminate")
@@ -1199,7 +1199,7 @@ async fn pool_meta_native_heal_rejects_v1_and_v3_prepare_in_either_pool_order()
let before = snapshots(&fixture.store).await;
let identities = owner_repair_identity_heads(&fixture.store).await;
let guard = owner(&fixture.store).await;
let mut state = fixture.store.pool_meta_save_gate.lock().await;
let mut state = fixture.store.pool_meta_save_gate.write().await;
let result = tokio::time::timeout(
std::time::Duration::from_secs(30),
fixture
@@ -1301,7 +1301,7 @@ async fn pool_meta_native_heal_rejects_v1_legacy_dont_save_true_without_writes()
let before = snapshots(&fixture.store).await;
let identities = owner_repair_identity_heads(&fixture.store).await;
let guard = owner(&fixture.store).await;
let mut state = fixture.store.pool_meta_save_gate.lock().await;
let mut state = fixture.store.pool_meta_save_gate.write().await;
let result = tokio::time::timeout(
std::time::Duration::from_secs(30),
fixture.store.heal_pool_meta_replica_under_fence(1, guard, &mut state),
@@ -1410,7 +1410,7 @@ async fn pool_meta_native_heal_rejects_target_v1_dont_save_true_without_writes()
let before = snapshots(&fixture.store).await;
let identities = owner_repair_identity_heads(&fixture.store).await;
let guard = owner(&fixture.store).await;
let mut state = fixture.store.pool_meta_save_gate.lock().await;
let mut state = fixture.store.pool_meta_save_gate.write().await;
let result = tokio::time::timeout(
std::time::Duration::from_secs(30),
fixture.store.heal_pool_meta_replica_under_fence(1, guard, &mut state),
File diff suppressed because it is too large Load Diff
+37 -6
View File
@@ -527,8 +527,8 @@ mod decommission_lock_order_tests {
rebalance_meta: tokio::sync::RwLock::new(other_store.rebalance_meta.read().await.clone()),
decommission_cancelers: tokio::sync::RwLock::new(vec![None; other_store.pools.len()]),
start_gate: tokio::sync::Mutex::new(()),
pool_meta_save_gate: tokio::sync::Mutex::new(
other_store.pool_meta_save_gate.lock().await.independent_clone_for_test(),
pool_meta_save_gate: crate::core::pools::PoolMetaSaveGate::new(
other_store.pool_meta_save_gate.write().await.independent_clone_for_test(),
),
ctx,
bucket_fence_registry: Arc::default(),
@@ -1616,7 +1616,7 @@ mod decommission_lock_order_tests {
run_routine_to_a_deferral_pause(
&store,
1,
"target pool 1 no longer fits the selected mutation: source pool 0 reservation estimate is exhausted",
"target pool 1 no longer fits the selected mutation: target pool 1 reservation requires",
)
.await;
});
@@ -1647,7 +1647,7 @@ mod decommission_lock_order_tests {
run_routine_to_a_deferral_pause(
&store,
1,
"target pool 1 no longer fits the selected mutation: source pool 0 reservation estimate is exhausted",
"target pool 1 no longer fits the selected mutation: target pool 1 reservation requires",
)
.await;
});
@@ -6062,6 +6062,12 @@ mod decommission_lock_order_tests {
)
.await
.expect("read the next source object for the multipart capacity oracle");
// Reconciled bytes remain charged. With no physical room for a new
// reservation, a distinct mutation must fail instead of reusing them.
let mut exhausted_capacity = capacity_snapshot();
exhausted_capacity[2] = DecommissionPoolCapacityInfo::for_test(2, layout, 0, target_total, target_total);
set_decommission_capacity_info_overrides_for_test(other_store.id, vec![exhausted_capacity]);
let next_result = tokio::time::timeout(
Duration::from_secs(30),
data_movement::migrate_decommission_object(
@@ -7827,6 +7833,12 @@ mod decommission_lock_order_tests {
.expect("same-mutation retry should publish a readable target");
assert_eq!(retry_info.size, i64::try_from(body.len()).expect("test body size fits i64"));
// Reconciled bytes remain charged. With no physical room for a new
// reservation, a distinct mutation must fail instead of reusing them.
let mut exhausted_capacity = capacity_snapshot();
exhausted_capacity[2] = DecommissionPoolCapacityInfo::for_test(2, layout, 0, target_total, target_total);
set_decommission_capacity_info_overrides_for_test(other_store.id, vec![exhausted_capacity]);
let next_object = "data-movement-put-after-retry.bin";
let next_version_id = uuid::Uuid::new_v4().to_string();
let mut next_opts = ObjectOptions {
@@ -7841,7 +7853,11 @@ mod decommission_lock_order_tests {
let next_result = other_store
.put_object_for_data_movement(RUSTFS_META_BUCKET, next_object, &mut next_data, &next_opts, None)
.await;
assert!(next_result.is_err(), "a different mutation must not reuse A's recovered capacity");
let next_error = match next_result {
Err(error) | Ok((_, Err(error))) => error,
Ok((_, Ok(_))) => panic!("a different mutation must not reuse A's recovered capacity"),
};
assert!(next_error.to_string().contains("decommission_capacity_blocked"), "{next_error}");
let next_target_err = other_store.pools[2]
.get_object_info(
RUSTFS_META_BUCKET,
@@ -9929,6 +9945,12 @@ mod decommission_lock_order_tests {
)
.await
.expect("read the next source object for the capacity oracle");
// Reconciled bytes remain charged. With no physical room for a new
// reservation, a distinct mutation must fail instead of reusing them.
let mut exhausted_capacity = capacity_snapshot();
exhausted_capacity[2] = DecommissionPoolCapacityInfo::for_test(2, layout, 0, target_total, target_total);
set_decommission_capacity_info_overrides_for_test(other_store.id, vec![exhausted_capacity]);
let next_result = tokio::time::timeout(
Duration::from_secs(30),
data_movement::migrate_decommission_object(
@@ -10190,11 +10212,20 @@ mod decommission_lock_order_tests {
assert_eq!(reconciled_reservation.pending_target_physical_bytes, 0);
assert_eq!(reconciled_reservation.consumed_target_physical_bytes, 1);
// Reconciled bytes remain charged. With no physical room for a new
// reservation, a distinct mutation must fail instead of reusing them.
let mut exhausted_capacity = capacity_snapshot();
exhausted_capacity[2] = DecommissionPoolCapacityInfo::for_test(2, layout, 1, 2, 1);
set_decommission_capacity_info_overrides_for_test(other_store.id, vec![exhausted_capacity]);
let next_after_reconcile = other_store
.delete_object(&bucket, next_object, next_opts)
.await
.expect_err("the consumed exact-fit capacity must reject the next marker");
assert!(next_after_reconcile.to_string().contains("decommission_capacity_blocked"));
assert!(
next_after_reconcile.to_string().contains("decommission_capacity_blocked"),
"{next_after_reconcile}"
);
let next_target_err = other_store.pools[2]
.get_object_info(
&bucket,
@@ -85,6 +85,7 @@ impl ScannerPauseBacklogRetirementPhase {
pub struct ScannerPauseBacklogRetirementReplica {
pub pool_index: usize,
pub set_index: usize,
pub set_id: Option<uuid::Uuid>,
/// False for retiring members retained only to verify the old commit proof.
pub writable: bool,
pub data: Option<Vec<u8>>,
@@ -140,6 +141,7 @@ async fn read_replica((set, writable): (Arc<SetDisks>, bool)) -> Result<ScannerP
let mut replica = ScannerPauseBacklogRetirementReplica {
pool_index: set.pool_index,
set_index: set.set_index,
set_id: Some(set.persistent_set_id()?),
writable,
data: None,
};
@@ -222,6 +224,7 @@ pub(crate) fn plan_scanner_pause_backlog_retirement(
.map(|read| ScannerPauseBacklogRetirementReplica {
pool_index: read.replica.pool_index,
set_index: read.replica.set_index,
set_id: read.replica.set_id,
writable: read.replica.writable,
data: read.replica.data.clone(),
})
+61 -31
View File
@@ -685,12 +685,8 @@ async fn clear_bucket_usage_memory(bucket: &str, guard: Option<&rustfs_lock::Nam
}
pub async fn remove_bucket_usage_from_backend(store: Arc<ECStore>, bucket: &str) -> Result<(), Error> {
remove_bucket_usage_for_namespace_change(store.as_ref(), bucket).await
}
pub(crate) async fn remove_bucket_usage_for_namespace_change(store: &ECStore, bucket: &str) -> Result<(), Error> {
prepare_bucket_usage_for_namespace_change(bucket, None).await?;
remove_bucket_usage_from_backend_with_guard_fenced(store, bucket, None).await
remove_bucket_usage_with_owned_namespace(store, bucket, None).await
}
pub(crate) async fn prepare_bucket_usage_for_namespace_change(
@@ -730,11 +726,37 @@ where
pub(crate) async fn remove_bucket_usage_from_backend_with_guard_fenced(
store: &ECStore,
bucket: &str,
guard: Option<&rustfs_lock::NamespaceLockGuard>,
guard: Option<&Arc<rustfs_lock::NamespaceLockGuard>>,
) -> Result<(), Error> {
let result = remove_bucket_usage_from_backend_with_store_and_guard_and_publication(store, bucket, guard, Some(store)).await;
invalidate_bucket_usage_snapshot_caches(guard, bucket).await?;
result
let owned_store = crate::bucket::metadata_sys::object_store_in(&store.ctx).await?;
if !std::ptr::eq(store, owned_store.as_ref()) {
return Err(Error::other("usage cleanup requires its own metadata store"));
}
remove_bucket_usage_with_owned_namespace(owned_store, bucket, guard.cloned()).await
}
async fn remove_bucket_usage_with_owned_namespace(
store: Arc<ECStore>,
bucket: &str,
guard: Option<Arc<rustfs_lock::NamespaceLockGuard>>,
) -> Result<(), Error> {
let bucket = bucket.to_owned();
let ctx = Arc::clone(&store.ctx);
// Retain the bucket generation and movement/object guards through every
// physical commit even when the namespace request stops waiting.
ctx.run_detached_mutation(async move {
let result = remove_bucket_usage_from_backend_with_store_and_guard_and_publication(
store.as_ref(),
&bucket,
guard.as_deref(),
Some(store.as_ref()),
)
.await;
invalidate_bucket_usage_snapshot_caches(guard.as_deref(), &bucket).await?;
result
})
.await
.map_err(Error::from)?
}
async fn invalidate_bucket_usage_snapshot_caches(
@@ -849,9 +871,9 @@ where
let primary_seed_epoch = match publication_store {
Some(publication_store) => Some(
publication_store
.scanner_data_usage_publication_epoch()
.data_usage_namespace_cleanup_epoch()
.await
.ok_or_else(|| Error::other("data usage publication is blocked by data movement"))?,
.ok_or_else(|| Error::other("data usage namespace cleanup epoch is unavailable"))?,
),
None => None,
};
@@ -871,9 +893,9 @@ where
let backup_seed_epoch = match publication_store {
Some(publication_store) => Some(
publication_store
.scanner_data_usage_publication_epoch()
.data_usage_namespace_cleanup_epoch()
.await
.ok_or_else(|| Error::other("data usage publication is blocked by data movement"))?,
.ok_or_else(|| Error::other("data usage namespace cleanup epoch is unavailable"))?,
),
None => None,
};
@@ -985,9 +1007,9 @@ where
let read_epoch = match publication {
Some((publication_store, expected_publication_epoch)) => {
let epoch = publication_store
.scanner_data_usage_publication_epoch()
.data_usage_namespace_cleanup_epoch()
.await
.ok_or_else(|| Error::other("data usage publication is blocked by data movement"))?;
.ok_or_else(|| Error::other("data usage namespace cleanup epoch is unavailable"))?;
if expected_publication_epoch.is_some_and(|expected| expected != epoch) {
return Err(Error::other("data usage publication epoch changed before snapshot read"));
}
@@ -1022,9 +1044,8 @@ where
ensure_bucket_namespace_guard(guard, bucket, "data usage snapshot commit")?;
let publication_guard = match publication {
Some((publication_store, expected_publication_epoch)) => {
let Some((guard, publication_epoch)) = publication_store.scanner_data_usage_publication_admission_guard().await
else {
return Err(Error::other("data usage publication is blocked by data movement"));
let Some((guard, publication_epoch)) = publication_store.data_usage_namespace_cleanup_guard().await else {
return Err(Error::other("data usage namespace cleanup epoch is unavailable"));
};
if expected_publication_epoch
.or(read_epoch)
@@ -1036,18 +1057,27 @@ where
}
None => None,
};
let save_result = store
.put_object(
RUSTFS_META_BUCKET,
object,
&mut put_data,
&ObjectOptions {
max_parity: true,
http_preconditions: Some(http_preconditions),
..Default::default()
},
)
.await;
let save_result = match publication {
Some((publication_store, _)) => {
publication_store
.put_usage_snapshot_for_namespace_cleanup(object, revision.as_deref(), &mut put_data, guard)
.await
}
None => {
store
.put_object(
RUSTFS_META_BUCKET,
object,
&mut put_data,
&ObjectOptions {
max_parity: true,
http_preconditions: Some(http_preconditions),
..Default::default()
},
)
.await
}
};
drop(publication_guard);
match save_result.map_err(|err| crate::config::com::map_system_metadata_write_error(err, object)) {
Ok(_) => return Ok(()),
@@ -3436,7 +3466,7 @@ mod tests {
rebalance_meta: RwLock::new(None),
decommission_cancelers: RwLock::new(Vec::new()),
start_gate: TokioMutex::new(()),
pool_meta_save_gate: TokioMutex::default(),
pool_meta_save_gate: crate::core::pools::PoolMetaSaveGate::default(),
ctx,
bucket_fence_registry: Arc::default(),
})
@@ -682,15 +682,11 @@ async fn get_pools_info(all_disks: &[Disk]) -> Result<HashMap<i32, HashMap<i32,
return Err(Error::other("ServerNotInitialized"));
};
let mut pools_info: HashMap<i32, HashMap<i32, ErasureSetInfo>> = HashMap::new();
for d in all_disks {
let pool_info = pools_info.entry(d.pool_index).or_default();
let erasure_set = pool_info.entry(d.set_index).or_default();
if erasure_set.id == 0 {
erasure_set.id = d.set_index;
collect_pools_info(all_disks, |pool_index, set_index| {
let store = store.clone();
async move {
let load = load_data_usage_cache(
store.pools[d.pool_index as usize].disk_set[d.set_index as usize].as_ref(),
store.pools[pool_index as usize].disk_set[set_index as usize].as_ref(),
DATA_USAGE_CACHE_NAME,
);
#[cfg(feature = "diagnostic-read-version-caller")]
@@ -698,11 +694,34 @@ async fn get_pools_info(all_disks: &[Disk]) -> Result<HashMap<i32, HashMap<i32,
rustfs_io_metrics::internode_metrics::ReadVersionEntrySource::ServerInfoPools,
load,
);
match load.await {
Ok(cache) => apply_erasure_set_usage(&cache, erasure_set),
Err(_) => erasure_set.usage_error = Some(DATA_USAGE_UNAVAILABLE_ERROR.to_string()),
}
load.await
}
})
.await
}
async fn collect_pools_info<F, Fut>(all_disks: &[Disk], mut load_usage: F) -> Result<HashMap<i32, HashMap<i32, ErasureSetInfo>>>
where
F: FnMut(i32, i32) -> Fut,
Fut: std::future::Future<Output = Result<rustfs_data_usage::DataUsageCache>>,
{
let mut pools_info: HashMap<i32, HashMap<i32, ErasureSetInfo>> = HashMap::new();
for d in all_disks {
let pool_info = pools_info.entry(d.pool_index).or_default();
let erasure_set = match pool_info.entry(d.set_index) {
std::collections::hash_map::Entry::Occupied(entry) => entry.into_mut(),
std::collections::hash_map::Entry::Vacant(entry) => {
let erasure_set = entry.insert(ErasureSetInfo {
id: d.set_index,
..Default::default()
});
match load_usage(d.pool_index, d.set_index).await {
Ok(cache) => apply_erasure_set_usage(&cache, erasure_set),
Err(_) => erasure_set.usage_error = Some(DATA_USAGE_UNAVAILABLE_ERROR.to_string()),
}
erasure_set
}
};
erasure_set.raw_capacity += d.total_space;
erasure_set.raw_usage += d.used_space;
@@ -1241,6 +1260,59 @@ mod tests {
assert_eq!(info.deployment_id, expected_deployment_id);
}
#[tokio::test]
async fn pool_usage_is_loaded_once_per_set_including_set_zero() {
let mut disks = Vec::new();
for (pool, set, count) in [(0, 0, 16), (1, 0, 16), (1, 1, 4)] {
disks.extend((0..count).map(|_| Disk {
pool_index: pool,
set_index: set,
used_space: 10,
total_space: 100,
..Default::default()
}));
}
let mut loads = Vec::new();
let info = super::collect_pools_info(&disks, |pool, set| {
loads.push((pool, set));
async move {
if pool == 1 && set == 0 {
return Err(crate::error::Error::other("usage unavailable"));
}
let mut cache = rustfs_data_usage::DataUsageCache::default();
cache.info.name = DATA_USAGE_ROOT.to_owned();
cache.info.last_update = Some(std::time::SystemTime::UNIX_EPOCH);
cache.info.snapshot_complete = true;
cache.replace(
DATA_USAGE_ROOT,
"",
rustfs_data_usage::DataUsageEntry {
size: 25,
objects: 2,
..Default::default()
},
);
Ok(cache)
}
})
.await
.expect("collect mixed successful and unavailable usage");
assert_eq!(loads, [(0, 0), (1, 0), (1, 1)]);
for (pool, set, count) in [(0, 0, 16), (1, 0, 16), (1, 1, 4)] {
let set_info = &info[&pool][&set];
assert_eq!(set_info.id, set);
assert_eq!(set_info.raw_usage, count * 10);
assert_eq!(set_info.raw_capacity, count * 100);
if pool == 1 && set == 0 {
assert_eq!(set_info.usage_error.as_deref(), Some(DATA_USAGE_UNAVAILABLE_ERROR));
} else {
assert_eq!(set_info.usage, 25, "logical usage must not multiply by drive count");
assert_eq!(set_info.objects_count, 2);
assert!(set_info.usage_error.is_none());
}
}
}
#[serial]
#[tokio::test]
async fn local_server_property_includes_runtime_stats_without_endpoint_pools() {
+498 -130
View File
@@ -562,12 +562,59 @@ fn remove_dir_all_if_exists(path: &Path) -> std::io::Result<()> {
fn snapshot_part_transaction_file(src: &Path, backup: &Path, absent: &Path) -> std::io::Result<bool> {
match std::fs::symlink_metadata(src) {
Ok(metadata) if metadata.is_file() => std::fs::hard_link(src, backup).map(|()| true),
Ok(metadata) if metadata.file_type().is_symlink() => {
Err(std::io::Error::new(ErrorKind::InvalidInput, "multipart transaction source is a symlink"))
}
Ok(_) => Err(std::io::Error::new(ErrorKind::InvalidData, "multipart transaction source is not a file")),
Err(err) if err.kind() == ErrorKind::NotFound => std::fs::write(absent, []).map(|()| false),
Err(err) => Err(err),
}
}
struct PreparedPartTxnReceipt {
src_volume_dir: PathBuf,
dst_volume_dir: PathBuf,
src_file_path: PathBuf,
dst_file_path: PathBuf,
dst_meta_path: PathBuf,
transaction_path: PathBuf,
meta: Bytes,
had_old_data: bool,
had_old_meta: bool,
}
struct PublishedPartTxnReceipt {
current_data_path: PathBuf,
current_meta_path: PathBuf,
transaction_path: PathBuf,
had_old_data: bool,
had_old_meta: bool,
}
const PART_TXN_RECEIPTS_PER_SHARD: usize = 64;
static PREPARED_PART_TXN_RECEIPTS: std::sync::LazyLock<os::ShardedPathCache<PreparedPartTxnReceipt>> =
std::sync::LazyLock::new(|| os::ShardedPathCache::new(PART_TXN_RECEIPTS_PER_SHARD));
static PUBLISHED_PART_TXN_RECEIPTS: std::sync::LazyLock<os::ShardedPathCache<PublishedPartTxnReceipt>> =
std::sync::LazyLock::new(|| os::ShardedPathCache::new(PART_TXN_RECEIPTS_PER_SHARD));
fn put_prepared_part_txn_receipt(transaction_path: &Path, entry: PreparedPartTxnReceipt) {
PREPARED_PART_TXN_RECEIPTS.insert(transaction_path, entry);
}
fn take_prepared_part_txn_receipt(transaction_path: &Path) -> Option<PreparedPartTxnReceipt> {
PREPARED_PART_TXN_RECEIPTS.remove(transaction_path)
}
fn put_published_part_txn_receipt(transaction_path: &Path, entry: PublishedPartTxnReceipt) {
PUBLISHED_PART_TXN_RECEIPTS.insert(transaction_path, entry);
}
fn take_published_part_txn_receipt(transaction_path: &Path) -> Option<PublishedPartTxnReceipt> {
PUBLISHED_PART_TXN_RECEIPTS.remove(transaction_path)
}
fn restore_part_transaction_file(current: &Path, backup: &Path, absent: &Path, restore: &Path) -> std::io::Result<()> {
match std::fs::symlink_metadata(backup) {
Ok(metadata) if metadata.is_file() => {
@@ -932,6 +979,7 @@ async fn restore_delete_rollback_with_namespace_owner(
.insert(backup_path.clone());
os::rename_all_with_owner(&backup_path, xl_path, object_dir, publication_root, namespace_owner.clone()).await
};
let restored_backup = restored.is_ok();
match restored {
Ok(()) => {}
Err(DiskError::FileNotFound) if remove_new_metadata => {
@@ -961,6 +1009,16 @@ async fn restore_delete_rollback_with_namespace_owner(
os::fsync_dir_with_owner(object_dir, namespace_owner.clone())
.await
.map_err(to_file_error)?;
}
if restored_backup {
// Renaming two hardlinks to the same inode can leave the backup in place.
match os::remove_file_with_owner(&backup_path, namespace_owner.clone()).await {
Ok(()) => {}
Err(err) if err.kind() == ErrorKind::NotFound => {}
Err(err) => return Err(to_file_error(err).into()),
}
}
if effective_durability(volume).syncs_commit_metadata() {
os::fsync_dir_with_owner(&rollback_path, namespace_owner.clone())
.await
.map_err(to_file_error)?;
@@ -3160,6 +3218,10 @@ pub(super) fn has_rename_data_test_failpoints(dst_path: &str) -> bool {
|| os::has_dir_fsync_test_failures()
|| os::file_sync_probe::is_active()
|| os::prepared_publication_test_hooks::has_any_hooks()
|| !OWNED_FILE_BEFORE_OPEN
.lock()
.expect("test file open hook lock should not be poisoned")
.is_empty()
}
#[cfg(all(not(test), feature = "test-util", target_os = "linux"))]
@@ -3745,6 +3807,25 @@ async fn read_dir_entries_with_walk_stall(path: &Path, count: i32, stall: Option
const WALK_DIR_BATCH_SIZE: usize = 32;
#[cfg(not(test))]
const WALK_DIR_BATCH_SIZE: usize = 256;
async fn read_identifier_tree_entries(path: &Path, leaf: &str, stall: Option<Duration>) -> Result<Vec<String>> {
let mut entries = read_dir_with_walk_stall(path.to_path_buf(), stall).await?;
let mut names = Vec::new();
loop {
let (next, batch, eof) = read_dir_entry_batch_with_walk_stall(entries, READ_DIR_BATCH_SIZE, stall).await?;
entries = next;
// Filter before retaining or sorting the inventory. A report flood
// costs directory iteration, not one retained name/metadata read per report.
names.extend(batch.into_iter().filter(|name| {
name.strip_suffix(SLASH_SEPARATOR)
.is_some_and(|component| component == leaf || super::is_identifier_tree_component(component))
}));
if eof {
return Ok(names);
}
}
}
const READ_DIR_BATCH_SIZE: usize = 512;
async fn read_dir_with_walk_stall(path: PathBuf, stall: Option<Duration>) -> Result<std::fs::ReadDir> {
@@ -4239,20 +4320,28 @@ fn take_presynced_tmp_file(file: &Path) -> bool {
PRESYNCED_TMP_FILES.remove(file).is_some()
}
fn open_lazy_local_file_std(file_path: &Path, volume_dir: &Path, _reclaim_on_shutdown: bool) -> std::io::Result<std::fs::File> {
if let Some(parent) = file_path.parent()
&& (volume_dir.as_os_str().is_empty() || !volume_dir.starts_with(parent))
&& let Err(err) = super::fs::create_dir_all_std(parent)
&& err.kind() != std::io::ErrorKind::AlreadyExists
{
return Err(to_file_error(err));
}
let file = std::fs::OpenOptions::new()
.write(true)
.create(true)
.truncate(true)
.open(file_path)
.map_err(to_file_error)?;
fn open_lazy_local_file_std(
file_path: &Path,
volume_dir: &Path,
_reclaim_on_shutdown: bool,
parent_exists: bool,
) -> std::io::Result<std::fs::File> {
let mut open_opts = std::fs::OpenOptions::new();
open_opts.write(true).create(true).truncate(true);
let file = match parent_exists.then(|| open_opts.open(file_path)) {
Some(Ok(file)) => file,
Some(Err(err)) if err.kind() != std::io::ErrorKind::NotFound => return Err(to_file_error(err)),
_ => {
if let Some(parent) = file_path.parent()
&& (volume_dir.as_os_str().is_empty() || !volume_dir.starts_with(parent))
&& let Err(err) = super::fs::create_dir_all_std(parent)
&& err.kind() != std::io::ErrorKind::AlreadyExists
{
return Err(to_file_error(err));
}
open_opts.open(file_path).map_err(to_file_error)?
}
};
#[cfg(target_os = "macos")]
if _reclaim_on_shutdown {
let _ = set_std_fd_nocache(&file);
@@ -4267,6 +4356,7 @@ struct LazyLocalFileWriter {
reclaim_on_shutdown: bool,
sync_on_close: bool,
mark_dir_presynced: bool,
parent_exists: bool,
bytes_written: usize,
state: LazyLocalFileWriterState,
}
@@ -4287,6 +4377,7 @@ impl LazyLocalFileWriter {
reclaim_on_shutdown: bool,
sync_on_close: bool,
mark_dir_presynced: bool,
parent_exists: bool,
) -> Self {
Self {
file_path,
@@ -4295,6 +4386,7 @@ impl LazyLocalFileWriter {
reclaim_on_shutdown,
sync_on_close,
mark_dir_presynced,
parent_exists,
bytes_written: 0,
state: LazyLocalFileWriterState::Unopened,
}
@@ -4307,12 +4399,13 @@ impl LazyLocalFileWriter {
let reclaim_on_shutdown = self.reclaim_on_shutdown;
let sync_on_close = self.sync_on_close;
let mark_dir_presynced = self.mark_dir_presynced;
let parent_exists = self.parent_exists;
match std::mem::replace(&mut self.state, LazyLocalFileWriterState::Closed) {
LazyLocalFileWriterState::Unopened => {
let file_path = self.file_path.clone();
let volume_dir = self.volume_dir.clone();
self.state = LazyLocalFileWriterState::Writing(tokio::task::spawn_blocking(move || {
let mut file = open_lazy_local_file_std(&file_path, &volume_dir, reclaim_on_shutdown)?;
let mut file = open_lazy_local_file_std(&file_path, &volume_dir, reclaim_on_shutdown, parent_exists)?;
std::io::Write::write_all(&mut file, &payload).map_err(to_file_error)?;
if close_after {
if sync_on_close {
@@ -4454,8 +4547,9 @@ impl AsyncWrite for LazyLocalFileWriter {
let file_path = self.file_path.clone();
let volume_dir = self.volume_dir.clone();
let reclaim_on_shutdown = self.reclaim_on_shutdown;
let parent_exists = self.parent_exists;
self.state = LazyLocalFileWriterState::Closing(tokio::task::spawn_blocking(move || {
open_lazy_local_file_std(&file_path, &volume_dir, reclaim_on_shutdown).map(drop)
open_lazy_local_file_std(&file_path, &volume_dir, reclaim_on_shutdown, parent_exists).map(drop)
}));
}
LazyLocalFileWriterState::Open(file) => {
@@ -5504,11 +5598,14 @@ impl LocalIoBackend for StdBackend {
let file_path = local_disk_object_path(self.io_root(), volume, path)?;
check_path_length(file_path.to_string_lossy().as_ref())?;
let mut mark_dir_presynced = false;
let mut parent_exists = false;
if let Some(parent) = file_path.parent() {
check_path_length(parent.to_string_lossy().as_ref())?;
parent_exists = parent == volume_dir;
if volume == RUSTFS_META_TMP_BUCKET {
unmark_presynced_tmp_file(&file_path);
let parent_was_presynced = unmark_presynced_tmp_data_dir(parent);
parent_exists |= parent_was_presynced;
let file_name = file_path.file_name().and_then(|n| n.to_str()).unwrap_or_default();
mark_dir_presynced = sync_on_close
&& (file_name == "part.1"
@@ -5559,6 +5656,7 @@ impl LocalIoBackend for StdBackend {
reclaim_on_shutdown,
sync_on_close,
mark_dir_presynced,
parent_exists,
)))
}
WriteMode::Append => {
@@ -9220,15 +9318,14 @@ impl LocalDisk {
.await;
}
let volume_dir = self.io_get_bucket_path(volume)?;
let file_path = self.io_get_object_path(volume, path)?;
let (volume_dir, file_path) = self.io_get_bucket_and_object_path(volume, path)?;
check_path_length(file_path.to_string_lossy().as_ref())?;
let xl_path = path_join(&[file_path.as_path(), Path::new(STORAGE_FORMAT_FILE)]);
let object_path = self.get_object_path(volume, path)?;
let mut namespace_owner: Option<Arc<dyn Send + Sync>> =
Some(os::acquire_metadata_mutation_lease(&self.get_object_path(volume, path)?, namespace_owner).await);
Some(os::acquire_metadata_mutation_lease(&object_path, namespace_owner).await);
if opts.recoverable_delete || (opts.undo_write && opts.undo_delete) {
let volume_guard = os::disk_volume_mutation_lock(&self.root, volume).read_owned().await;
namespace_owner = Some(Arc::new((namespace_owner, volume_guard)));
@@ -9253,7 +9350,15 @@ impl LocalDisk {
let rollback_dir = opts.old_data_dir;
if let Some(rollback_dir) = rollback_dir
&& let Some(()) = self
.try_fused_recoverable_delete_version(volume, path, &fi, &opts, rollback_dir, namespace_owner.clone())
.try_fused_recoverable_delete_version_with_paths(
volume,
path,
&fi,
&opts,
rollback_dir,
namespace_owner.clone(),
Some((object_path, volume_dir.clone(), file_path.clone())),
)
.await?
{
return Ok(());
@@ -11416,9 +11521,17 @@ impl LocalDisk {
admission: Option<multipart_staging::MultipartSourceAdmission>,
) -> Result<()> {
multipart_staging::reject_reserved_anchor(path)?;
if volume == RUSTFS_META_MULTIPART_BUCKET {
self.multipart_index.record_head(path, &buf).await?;
}
let fuse_with_admission = admission.is_some() && !crash_inject::is_armed(CrashPoint::MetaWriteAfterTmpBeforeRename, path);
let prepared_index = if volume == RUSTFS_META_MULTIPART_BUCKET {
if fuse_with_admission {
self.multipart_index.prepare_record_head(path, &buf).await
} else {
self.multipart_index.record_head(path, &buf).await?;
None
}
} else {
None
};
let volume_dir = self.io_get_bucket_path(volume)?;
let file_path = self.io_get_object_path(volume, path)?;
check_path_length(file_path.to_string_lossy().as_ref())?;
@@ -11434,11 +11547,14 @@ impl LocalDisk {
{
multipart_source_state::invalidate_active_multipart_source_head(admission.source());
}
if admission.is_some() && !crash_inject::is_armed(CrashPoint::MetaWriteAfterTmpBeforeRename, path) {
if fuse_with_admission {
let precondition = admission.as_ref().map(|a| a.precondition());
let root = self.publication_root.clone();
let lease = os::acquire_namespace_mutation_lease_with_owner(&file_path, namespace_owner).await;
os::run_blocking_namespace_operation_with_sync(lease, do_sync, move || {
if let Some(record) = prepared_index {
record.run_std()?;
}
super::fs::create_dir_all_std(&tmp_volume_dir)?;
#[cfg(test)]
run_owned_file_before_open(&tmp_file_path);
@@ -11690,6 +11806,14 @@ impl LocalDisk {
W: AsyncWrite + Unpin + Send + 'a,
{
Box::pin(async move {
if opts.identifier_tree_leaf.is_some() {
let relative = current
.strip_prefix(&opts.base_dir)
.ok_or_else(|| DiskError::other("identifier-tree walk escaped its root"))?;
if relative.split('/').filter(|component| !component.is_empty()).count() > 512 {
return Err(DiskError::other("identifier-tree walk exceeds its directory depth limit"));
}
}
// The part of forward_to below this directory, taken before `current`
// loses its trailing slash.
let forward_rest = opts
@@ -11722,7 +11846,12 @@ impl LocalDisk {
// as a single logical read in API semantics.
let read_dir_started = rustfs_io_metrics::get_stage_metrics_enabled().then(std::time::Instant::now);
let dir_path_abs = self.io_get_object_path(&opts.bucket, current.trim_start_matches(SLASH_SEPARATOR))?;
let read_dir_result = match read_dir_entries_with_walk_stall(&dir_path_abs, -1, stall).await {
let read_dir_result = if let Some(leaf) = opts.identifier_tree_leaf.as_deref() {
read_identifier_tree_entries(&dir_path_abs, leaf, stall).await
} else {
read_dir_entries_with_walk_stall(&dir_path_abs, -1, stall).await
};
let read_dir_result = match read_dir_result {
Err(err) if err == Error::FileNotFound && !skip_access_checks(&opts.bucket) => {
let volume_dir = self.io_get_bucket_path(&opts.bucket)?;
if let Err(access_err) = cached_access(&volume_dir).await {
@@ -12019,6 +12148,16 @@ impl LocalDisk {
return Ok(true);
}
if let Some(leaf) = opts.identifier_tree_leaf.as_deref()
&& entry != leaf
{
// Identifier directories can be namespaces at any depth,
// including components named tables or maintenance. Their
// own object metadata is outside this selected leaf scan.
schedule_dir(&mut dir_stack, format!("{name}/"), true, None, true, false);
continue;
}
let mut meta = MetaCacheEntry {
name,
..Default::default()
@@ -12068,7 +12207,8 @@ impl LocalDisk {
} else {
adaptive_batch_cap
};
let batch_names = if batch_limit > 1
let batch_names = if opts.identifier_tree_leaf.is_none()
&& batch_limit > 1
&& dir_objes.is_empty()
&& dir_stack.iter().all(|(_, _, _, scan_required, _)| !*scan_required)
{
@@ -12121,6 +12261,11 @@ impl LocalDisk {
write_metacache_obj(out, &meta).await?;
}
if opts.identifier_tree_leaf.is_some() {
// A dotted leaf cannot contain identifier-tree descendants.
continue;
}
let has_namespace_child_candidate = read.has_namespace_child_candidate;
let dir_to_skip: HashSet<String> = if has_namespace_child_candidate || !dir_objes.is_empty() {
read.data_dirs.into_iter().map(|dir| dir.to_string()).collect()
@@ -12167,6 +12312,9 @@ impl LocalDisk {
}
Err(err) => {
if err == Error::FileNotFound || err == Error::IsNotRegular {
if opts.identifier_tree_leaf.is_some() {
continue;
}
adaptive_batch_cap = 1;
// NOT an object, append to stack (with slash)
// If dirObject, but no metadata (which is unexpected) we skip it.
@@ -14307,6 +14455,7 @@ impl DiskAPI for LocalDisk {
if bucket == RUSTFS_META_MULTIPART_BUCKET && !paths.is_empty() {
let mut cached_parts = Vec::with_capacity(paths.len());
let mut all_cached = true;
let mut verified_source: Option<(&str, PathBuf)> = None;
for path_str in paths {
let Some(num) = Path::new(path_str)
.file_name()
@@ -14318,11 +14467,21 @@ impl DiskAPI for LocalDisk {
all_cached = false;
break;
};
let Ok(source) = self.io_get_object_path(bucket, multipart_staging::upload_scope_root(path_str, true)) else {
all_cached = false;
break;
let scope_root = multipart_staging::upload_scope_root(path_str, true);
let (source, head_already_verified) = match verified_source.as_ref() {
Some((prev_root, prev_source)) if *prev_root == scope_root => (prev_source, true),
_ => {
let Ok(source) = self.io_get_object_open_path(bucket, scope_root) else {
all_cached = false;
break;
};
verified_source = Some((scope_root, source));
(&verified_source.as_ref().expect("just initialized").1, false)
}
};
let Some(part) = multipart_source_state::get_active_multipart_source_part(&source, num) else {
let Some(part) =
multipart_source_state::get_active_multipart_source_part_with_head(source, num, head_already_verified)
else {
all_cached = false;
break;
};
@@ -14445,8 +14604,8 @@ impl DiskAPI for LocalDisk {
multipart_staging::reject_noncanonical_scope(&self.root, self.io_root(), src_volume, src_path)?;
let source_owner = self.multipart_source_owner(dst_volume, dst_path, true, false, None).await?;
let admission = self.multipart_source_admission(dst_volume, dst_path, true, false)?;
let src_volume_dir = self.io_get_bucket_path(src_volume)?;
let dst_volume_dir = self.io_get_bucket_path(dst_volume)?;
let (src_volume_dir, src_file_path) = self.io_get_bucket_and_object_path(src_volume, src_path)?;
let (dst_volume_dir, dst_file_path) = self.io_get_bucket_and_object_path(dst_volume, dst_path)?;
if !skip_access_checks(src_volume) {
super::fs::access_std(&src_volume_dir).map_err(|err| to_access_error(err, DiskError::VolumeAccessDenied))?;
}
@@ -14454,10 +14613,8 @@ impl DiskAPI for LocalDisk {
super::fs::access_std(&dst_volume_dir).map_err(|err| to_access_error(err, DiskError::VolumeAccessDenied))?;
}
let src_file_path = self.io_get_object_path(src_volume, src_path)?;
let dst_file_path = self.io_get_object_path(dst_volume, dst_path)?;
let dst_meta_path = self.io_get_object_path(dst_volume, &format!("{dst_path}.meta"))?;
let transaction_path = self.io_get_object_path(dst_volume, &crate::disk::part_transaction_path(dst_path))?;
let dst_meta_path = self.io_get_object_open_path(dst_volume, &format!("{dst_path}.meta"))?;
let transaction_path = self.io_get_object_open_path(dst_volume, &crate::disk::part_transaction_path(dst_path))?;
for path in [&src_file_path, &dst_file_path, &dst_meta_path, &transaction_path] {
check_path_length(path.to_string_lossy().as_ref())?;
}
@@ -14502,9 +14659,13 @@ impl DiskAPI for LocalDisk {
if !source.is_file() {
return Ok(Err(DiskError::FileAccessDenied));
}
if transaction_path.exists() {
return Ok(Err(DiskError::FileAccessDenied));
match std::fs::symlink_metadata(&transaction_path) {
Err(err) if err.kind() == ErrorKind::NotFound => {}
Ok(m) if m.file_type().is_symlink() => return Ok(Err(DiskError::InvalidPath)),
Ok(_) => return Ok(Err(DiskError::FileAccessDenied)),
Err(err) => return Ok(Err(to_file_error(err).into())),
}
let mut created_transaction_parent = false;
if let Some((integrity, source, destination, expected_len)) = integrity_prep {
let stat = match std::fs::symlink_metadata(&source) {
Ok(stat) => stat,
@@ -14513,6 +14674,7 @@ impl DiskAPI for LocalDisk {
if !stat.is_file() || stat.len() != expected_len {
return Ok(Err(DiskError::FileCorrupt));
}
let presynced_proof = take_presynced_tmp_file(&source);
let verify_and_sync = (|| -> std::io::Result<bool> {
let mut file = os::open_file_for_sync_std(&source)?;
let mut header = [0u8; rustfs_filemeta::shard_integrity::INDEX_HEADER_SIZE];
@@ -14520,7 +14682,7 @@ impl DiskAPI for LocalDisk {
if header != integrity.index_header() {
return Ok(false);
}
if durability.syncs_data_shards() && !take_presynced_tmp_file(&source) {
if durability.syncs_data_shards() && !presynced_proof {
file.sync_all()?;
}
Ok(true)
@@ -14530,7 +14692,13 @@ impl DiskAPI for LocalDisk {
Ok(false) => return Ok(Err(DiskError::FileCorrupt)),
Err(err) => return Ok(Err(to_file_error(err).into())),
}
let precondition = admission.as_ref().map(|check| check.precondition());
#[cfg(all(any(test, feature = "test-util"), not(windows)))]
let has_hooks = os::prepared_publication_test_hooks::has_any_hooks();
#[cfg(not(all(any(test, feature = "test-util"), not(windows))))]
let has_hooks = false;
let precondition = has_hooks
.then(|| admission.as_ref().map(|check| check.precondition()))
.flatten();
// Publish the immutable generation before preparing the part switch.
// Rollback retains the old generation; an unreferenced new index is
// reclaimed with the upload directory if preparation is interrupted.
@@ -14547,12 +14715,13 @@ impl DiskAPI for LocalDisk {
) {
return Ok(Err(to_file_error(err).into()));
}
created_transaction_parent = same_parent;
}
let Some(transaction_parent) = transaction_path.parent() else {
return Ok(Err(DiskError::InvalidPath));
};
if let Err(err) = std::fs::create_dir_all(transaction_parent) {
if !created_transaction_parent && let Err(err) = std::fs::create_dir_all(transaction_parent) {
return Ok(Err(to_file_error(err).into()));
}
let staging_path = transaction_parent.join(format!(".part-txn-{}", Uuid::new_v4()));
@@ -14560,22 +14729,41 @@ impl DiskAPI for LocalDisk {
return Ok(Err(to_file_error(err).into()));
}
let prepare_result = (|| -> std::io::Result<()> {
let prepare_result = (|| -> std::io::Result<(bool, bool)> {
let old_data_absent = staging_path.join(PART_TRANSACTION_OLD_DATA_ABSENT);
let old_meta_absent = staging_path.join(PART_TRANSACTION_OLD_META_ABSENT);
let had_old_data = snapshot_part_transaction_file(
&dst_file_path,
&staging_path.join(PART_TRANSACTION_OLD_DATA),
&staging_path.join(PART_TRANSACTION_OLD_DATA_ABSENT),
)?;
let had_old_meta = snapshot_part_transaction_file(
&dst_meta_path,
&staging_path.join(PART_TRANSACTION_OLD_META),
&staging_path.join(PART_TRANSACTION_OLD_META_ABSENT),
&old_data_absent,
)?;
let had_old_meta = match std::fs::symlink_metadata(&dst_meta_path) {
Ok(metadata) if metadata.is_file() => {
std::fs::hard_link(&dst_meta_path, staging_path.join(PART_TRANSACTION_OLD_META))?;
true
}
Ok(metadata) if metadata.file_type().is_symlink() => {
return Err(std::io::Error::new(ErrorKind::InvalidInput, "multipart transaction source is a symlink"));
}
Ok(_) => {
return Err(std::io::Error::new(ErrorKind::InvalidData, "multipart transaction source is not a file"));
}
Err(err) if err.kind() == ErrorKind::NotFound => {
if !had_old_data && std::fs::hard_link(&old_data_absent, &old_meta_absent).is_ok() {
false
} else {
std::fs::write(&old_meta_absent, [])?;
false
}
}
Err(err) => return Err(err),
};
let new_meta_path = staging_path.join(PART_TRANSACTION_NEW_META);
let mut new_meta = std::fs::OpenOptions::new()
.create_new(true)
.write(true)
.open(staging_path.join(PART_TRANSACTION_NEW_META))?;
.open(&new_meta_path)?;
std::io::Write::write_all(&mut new_meta, &meta)?;
if durability.syncs_commit_metadata() {
new_meta.sync_data()?;
@@ -14583,6 +14771,7 @@ impl DiskAPI for LocalDisk {
// Windows rejects renaming a directory while one of its children is
// still open, even when the child handle shares delete access.
drop(new_meta);
let _ = std::fs::hard_link(&new_meta_path, staging_path.join(PART_TRANSACTION_PUBLISH_META));
let sync_overwrite_backup = durability.syncs_commit_metadata() && (had_old_data || had_old_meta);
if sync_overwrite_backup {
os::fsync_dir_std(&staging_path)?;
@@ -14591,13 +14780,30 @@ impl DiskAPI for LocalDisk {
if sync_overwrite_backup {
os::fsync_dir_std(transaction_parent)?;
}
Ok(())
Ok((had_old_data, had_old_meta))
})();
if let Err(err) = prepare_result {
let _ = remove_dir_all_if_exists(&staging_path);
return Ok(Err(to_file_error(err).into()));
}
let (had_old_data, had_old_meta) = match prepare_result {
Ok(flags) => flags,
Err(err) => {
let _ = remove_dir_all_if_exists(&staging_path);
return Ok(Err(to_file_error(err).into()));
}
};
put_prepared_part_txn_receipt(
&transaction_path,
PreparedPartTxnReceipt {
src_volume_dir,
dst_volume_dir,
src_file_path,
dst_file_path,
dst_meta_path,
transaction_path: transaction_path.clone(),
meta,
had_old_data,
had_old_meta,
},
);
Ok(Ok(()))
})
@@ -14616,13 +14822,30 @@ impl DiskAPI for LocalDisk {
let source_owner = self.multipart_source_owner(volume, path, true, false, None).await?;
let admission = self.multipart_source_admission(volume, path, true, false)?;
let admission_source = admission.as_ref().map(|a| a.source().to_path_buf());
self.io_get_bucket_path(volume)?;
let current_data_path = self.io_get_object_path(volume, path)?;
let current_meta_path = self.io_get_object_path(volume, &format!("{path}.meta"))?;
let transaction_path = self.io_get_object_path(volume, &crate::disk::part_transaction_path(path))?;
for candidate in [&current_data_path, &current_meta_path, &transaction_path] {
check_path_length(candidate.to_string_lossy().as_ref())?;
}
let txn_rel_path = crate::disk::part_transaction_path(path);
let cached_published = self
.io_get_object_open_path(volume, &txn_rel_path)
.ok()
.and_then(|txn_open| take_published_part_txn_receipt(&txn_open));
let (current_data_path, current_meta_path, transaction_path, skip_obsolete_read_on_commit, cached_old_flags) =
match cached_published {
Some(entry) => (
entry.current_data_path,
entry.current_meta_path,
entry.transaction_path,
!entry.had_old_meta,
Some((entry.had_old_data, entry.had_old_meta)),
),
None => {
let (_, current_data_path) = self.io_get_bucket_and_object_path(volume, path)?;
let current_meta_path = self.io_get_object_path(volume, &format!("{path}.meta"))?;
let transaction_path = self.io_get_object_path(volume, &txn_rel_path)?;
for candidate in [&current_data_path, &current_meta_path, &transaction_path] {
check_path_length(candidate.to_string_lossy().as_ref())?;
}
(current_data_path, current_meta_path, transaction_path, false, None)
}
};
let durability = effective_durability(volume);
let sync = durability.syncs_commit_metadata();
let part_number = current_data_path
@@ -14688,7 +14911,8 @@ impl DiskAPI for LocalDisk {
PartTransactionAction::Commit => (PART_TRANSACTION_NEW_META, PART_TRANSACTION_OLD_META),
PartTransactionAction::Rollback => (PART_TRANSACTION_OLD_META, PART_TRANSACTION_NEW_META),
};
if let Ok(obsolete) = std::fs::read(transaction_path.join(obsolete_name))
if !(action == PartTransactionAction::Commit && skip_obsolete_read_on_commit && !cfg!(test))
&& let Ok(obsolete) = std::fs::read(transaction_path.join(obsolete_name))
&& let Ok(obsolete) = ObjectPartInfo::unmarshal(&obsolete)
&& let Some(integrity) = obsolete.integrity
&& integrity.validate().is_ok()
@@ -14720,7 +14944,26 @@ impl DiskAPI for LocalDisk {
{
return Ok(Err(to_file_error(err).into()));
}
if let Err(err) = remove_dir_all_if_exists(&cleanup_path) {
if let Some((had_old_data, had_old_meta)) = cached_old_flags.filter(|_| action == PartTransactionAction::Commit) {
let old_data_name = if had_old_data {
PART_TRANSACTION_OLD_DATA
} else {
PART_TRANSACTION_OLD_DATA_ABSENT
};
let old_meta_name = if had_old_meta {
PART_TRANSACTION_OLD_META
} else {
PART_TRANSACTION_OLD_META_ABSENT
};
for name in [PART_TRANSACTION_NEW_META, old_data_name, old_meta_name] {
let _ = std::fs::remove_file(cleanup_path.join(name));
}
if std::fs::remove_dir(&cleanup_path).is_err()
&& let Err(err) = remove_dir_all_if_exists(&cleanup_path)
{
return Ok(Err(to_file_error(err).into()));
}
} else if let Err(err) = remove_dir_all_if_exists(&cleanup_path) {
return Ok(Err(to_file_error(err).into()));
}
Ok(Ok(true))
@@ -14754,15 +14997,6 @@ impl DiskAPI for LocalDisk {
let admission_source = admission.as_ref().map(|a| a.source().to_path_buf());
pool_meta_native::reject_raw_path(&self.root, src_volume, src_path)?;
pool_meta_native::reject_raw_path(&self.root, dst_volume, dst_path)?;
let src_volume_dir = self.io_get_bucket_path(src_volume)?;
let dst_volume_dir = self.io_get_bucket_path(dst_volume)?;
if !skip_access_checks(src_volume) {
super::fs::access_std(&src_volume_dir).map_err(|e| to_access_error(e, DiskError::VolumeAccessDenied))?
}
if !skip_access_checks(dst_volume) {
super::fs::access_std(&dst_volume_dir).map_err(|e| to_access_error(e, DiskError::VolumeAccessDenied))?
}
let src_is_dir = has_suffix(src_path, SLASH_SEPARATOR);
let dst_is_dir = has_suffix(dst_path, SLASH_SEPARATOR);
@@ -14779,25 +15013,74 @@ impl DiskAPI for LocalDisk {
return Err(DiskError::FileAccessDenied);
}
let src_file_path = self.io_get_object_path(src_volume, src_path)?;
let dst_file_path = self.io_get_object_path(dst_volume, dst_path)?;
// warn!("rename_part src_file_path:{:?}, dst_file_path:{:?}", &src_file_path, &dst_file_path);
check_path_length(src_file_path.to_string_lossy().as_ref())?;
check_path_length(dst_file_path.to_string_lossy().as_ref())?;
let dst_meta_path = if src_is_dir {
None
let cached_prepared = if !src_is_dir {
let txn_rel = crate::disk::part_transaction_path(dst_path);
self.io_get_object_open_path(dst_volume, &txn_rel)
.ok()
.and_then(|txn_open| take_prepared_part_txn_receipt(&txn_open))
.filter(|entry| {
entry.meta == meta
&& self
.io_get_object_open_path(src_volume, src_path)
.is_ok_and(|src_open| src_open == entry.src_file_path)
})
} else {
let path = self.io_get_object_path(dst_volume, &format!("{dst_path}.meta"))?;
check_path_length(path.to_string_lossy().as_ref())?;
Some(path)
None
};
let transaction_path = if src_is_dir {
None
} else {
Some(self.io_get_object_path(dst_volume, &crate::disk::part_transaction_path(dst_path))?)
let (
src_volume_dir,
dst_volume_dir,
src_file_path,
dst_file_path,
dst_meta_path,
transaction_path,
preverified_meta_and_old_flags,
) = match cached_prepared {
Some(entry) => (
entry.src_volume_dir,
entry.dst_volume_dir,
entry.src_file_path,
entry.dst_file_path,
Some(entry.dst_meta_path),
Some(entry.transaction_path),
Some((entry.had_old_data, entry.had_old_meta)),
),
None => {
let (src_volume_dir, src_file_path) = self.io_get_bucket_and_object_path(src_volume, src_path)?;
let (dst_volume_dir, dst_file_path) = self.io_get_bucket_and_object_path(dst_volume, dst_path)?;
if !skip_access_checks(src_volume) {
super::fs::access_std(&src_volume_dir).map_err(|e| to_access_error(e, DiskError::VolumeAccessDenied))?
}
if !skip_access_checks(dst_volume) {
super::fs::access_std(&dst_volume_dir).map_err(|e| to_access_error(e, DiskError::VolumeAccessDenied))?
}
check_path_length(src_file_path.to_string_lossy().as_ref())?;
check_path_length(dst_file_path.to_string_lossy().as_ref())?;
let dst_meta_path = if src_is_dir {
None
} else {
let path = self.io_get_object_path(dst_volume, &format!("{dst_path}.meta"))?;
check_path_length(path.to_string_lossy().as_ref())?;
Some(path)
};
let transaction_path = if src_is_dir {
None
} else {
Some(self.io_get_object_path(dst_volume, &crate::disk::part_transaction_path(dst_path))?)
};
(
src_volume_dir,
dst_volume_dir,
src_file_path,
dst_file_path,
dst_meta_path,
transaction_path,
None,
)
}
};
// UploadPart is acknowledged once this rename lands, so the part data and
@@ -14856,35 +15139,43 @@ impl DiskAPI for LocalDisk {
let transaction_publish_meta = if let Some(transaction_path) = transaction_path.as_ref() {
let transaction_meta_path = transaction_path.join(PART_TRANSACTION_NEW_META);
match std::fs::File::open(&transaction_meta_path) {
Ok(mut file) => {
let file_len = file.metadata().map_err(to_file_error)?.len();
if file_len != meta.len() as u64 {
return Err(DiskError::FileCorrupt.into());
}
let mut expected_meta = Vec::with_capacity(meta.len());
std::io::Read::read_to_end(&mut std::io::Read::take(&mut file, file_len), &mut expected_meta)
.map_err(to_file_error)?;
drop(file);
if expected_meta.as_slice() != meta.as_ref() {
return Err(DiskError::FileCorrupt.into());
}
let publish_meta_path = transaction_path.join(PART_TRANSACTION_PUBLISH_META);
if !cfg!(test)
&& preverified_meta_and_old_flags.is_some()
&& publish_meta_path.is_file()
&& std::fs::metadata(&transaction_meta_path).is_ok_and(|m| m.is_file() && m.len() == meta.len() as u64)
{
Some(publish_meta_path)
} else {
match std::fs::File::open(&transaction_meta_path) {
Ok(mut file) => {
let file_len = file.metadata().map_err(to_file_error)?.len();
if file_len != meta.len() as u64 {
return Err(DiskError::FileCorrupt.into());
}
let mut expected_meta = Vec::with_capacity(meta.len());
std::io::Read::read_to_end(&mut std::io::Read::take(&mut file, file_len), &mut expected_meta)
.map_err(to_file_error)?;
drop(file);
if expected_meta.as_slice() != meta.as_ref() {
return Err(DiskError::FileCorrupt.into());
}
let publish_meta_path = transaction_path.join(PART_TRANSACTION_PUBLISH_META);
remove_file_if_exists(&publish_meta_path).map_err(to_file_error)?;
std::fs::hard_link(&transaction_meta_path, &publish_meta_path).map_err(to_file_error)?;
Some(publish_meta_path)
remove_file_if_exists(&publish_meta_path).map_err(to_file_error)?;
std::fs::hard_link(&transaction_meta_path, &publish_meta_path).map_err(to_file_error)?;
Some(publish_meta_path)
}
// Old peers know only RenamePart. The new coordinator never
// reaches rename unless prepare succeeded, so an absent
// transaction directory identifies the rolling-upgrade legacy
// path. A present directory without new.meta is corruption.
Err(err) if err.kind() == ErrorKind::NotFound => match std::fs::metadata(transaction_path) {
Ok(_) => return Err(DiskError::FileCorrupt.into()),
Err(meta_err) if meta_err.kind() == ErrorKind::NotFound => None,
Err(meta_err) => return Err(to_file_error(meta_err)),
},
Err(err) => return Err(to_file_error(err)),
}
// Old peers know only RenamePart. The new coordinator never
// reaches rename unless prepare succeeded, so an absent
// transaction directory identifies the rolling-upgrade legacy
// path. A present directory without new.meta is corruption.
Err(err) if err.kind() == ErrorKind::NotFound => match std::fs::metadata(transaction_path) {
Ok(_) => return Err(DiskError::FileCorrupt.into()),
Err(meta_err) if meta_err.kind() == ErrorKind::NotFound => None,
Err(meta_err) => return Err(to_file_error(meta_err)),
},
Err(err) => return Err(to_file_error(err)),
}
} else {
None
@@ -14894,7 +15185,13 @@ impl DiskAPI for LocalDisk {
os::sync_file(&src_file_path).map_err(to_file_error)?;
}
let precondition = admission.as_ref().map(|check| check.precondition());
#[cfg(all(any(test, feature = "test-util"), not(windows)))]
let has_hooks = os::prepared_publication_test_hooks::has_any_hooks();
#[cfg(not(all(any(test, feature = "test-util"), not(windows))))]
let has_hooks = false;
let precondition = has_hooks
.then(|| admission.as_ref().map(|check| check.precondition()))
.flatten();
os::rename_all_with_precondition_and_parent_sync_std(
&src_file_path,
@@ -14906,17 +15203,19 @@ impl DiskAPI for LocalDisk {
)
.map_err(to_file_error)?;
let dst_meta = lstat_std(&dst_file_path).map_err(to_file_error)?;
if src_is_dir != dst_meta.is_dir() {
warn!(
event = EVENT_DISK_LOCAL_RENAME_REJECTED,
component = LOG_COMPONENT_ECSTORE,
subsystem = LOG_SUBSYSTEM_DISK_LOCAL,
reason = "dst_type_changed_after_rename",
path = ?dst_file_path,
"Disk local rename rejected"
);
return Err(DiskError::FileAccessDenied.into());
if cfg!(test) || src_is_dir {
let dst_meta = lstat_std(&dst_file_path).map_err(to_file_error)?;
if src_is_dir != dst_meta.is_dir() {
warn!(
event = EVENT_DISK_LOCAL_RENAME_REJECTED,
component = LOG_COMPONENT_ECSTORE,
subsystem = LOG_SUBSYSTEM_DISK_LOCAL,
reason = "dst_type_changed_after_rename",
path = ?dst_file_path,
"Disk local rename rejected"
);
return Err(DiskError::FileAccessDenied.into());
}
}
if let Some(transaction_publish_meta) = transaction_publish_meta {
@@ -14947,6 +15246,20 @@ impl DiskAPI for LocalDisk {
}
}
}
if let (Some((had_old_data, had_old_meta)), Some(txn_path), Some(meta_path)) =
(preverified_meta_and_old_flags, transaction_path.as_ref(), dst_meta_path.as_path().into())
{
put_published_part_txn_receipt(
txn_path,
PublishedPartTxnReceipt {
current_data_path: dst_file_path,
current_meta_path: meta_path.to_path_buf(),
transaction_path: txn_path.clone(),
had_old_data,
had_old_meta,
},
);
}
Ok(true)
} else {
Ok(false)
@@ -15278,6 +15591,12 @@ impl DiskAPI for LocalDisk {
super::lifecycle_test_io::observe(&opts.bucket, None);
self.wait_for_startup_cleanup().await;
if let Some(leaf) = opts.identifier_tree_leaf.as_deref()
&& (!opts.recursive || !opts.base_dir.ends_with('/') || !super::is_identifier_tree_leaf(leaf))
{
return Err(DiskError::other("invalid identifier-tree walk options"));
}
// Callers that do not pin a stall budget inherit the configured default, so
// every local walk keeps a liveness bound even when the wrapper-level total
// timeout is skipped.
@@ -15304,7 +15623,7 @@ impl DiskAPI for LocalDisk {
let mut skip_current_dir_object = false;
let mut multipart_dir_to_skip: HashSet<String> = HashSet::new();
if opts.base_dir.ends_with(SLASH_SEPARATOR) {
if opts.identifier_tree_leaf.is_none() && opts.base_dir.ends_with(SLASH_SEPARATOR) {
if let Ok(data) = with_walk_stall_timeout(
stall,
self.read_listing_metadata(
@@ -19254,6 +19573,55 @@ mod test {
}
}
#[tokio::test]
async fn identifier_tree_walk_filters_report_batches_and_bounds_depth() {
let dir = tempfile::tempdir().expect("create disk fixture");
let root = dir.path().join("catalog/prefix");
fs::create_dir_all(root.join("sales"))
.await
.expect("create identifier directory");
fs::create_dir_all(root.join("table-entry.json"))
.await
.expect("create selected leaf");
for index in 0..READ_DIR_BATCH_SIZE * 3 + 1 {
fs::create_dir(root.join(format!("report-{index}.json")))
.await
.expect("create report directory");
}
let mut names = read_identifier_tree_entries(&root, "table-entry.json", Some(Duration::from_secs(5)))
.await
.expect("all directory batches should be filtered");
names.sort();
assert_eq!(names, ["sales/", "table-entry.json/"]);
let endpoint = Endpoint::try_from(dir.path().to_str().expect("UTF-8 fixture path")).expect("valid endpoint");
let disk = LocalDisk::new(&endpoint, false).await.expect("create local disk");
let options = WalkDirOptions {
bucket: "catalog".to_string(),
base_dir: "prefix/".to_string(),
recursive: true,
identifier_tree_leaf: Some("table-entry.json".to_string()),
..Default::default()
};
let mut output = Vec::new();
let mut writer = MetacacheWriter::new(&mut output);
let mut count = 0;
let error = disk
.scan_dir(
format!("prefix/{}", "a/".repeat(513)),
String::new(),
&options,
&mut writer,
&mut count,
true,
None,
)
.await
.expect_err("excessive depth must fail before filesystem I/O");
assert!(error.to_string().contains("directory depth limit"));
assert_eq!(count, 0);
}
// #4644: the stall budget bounds a single drive read, never the walk as a
// whole. A walk that keeps making progress must survive even when the total
// time spent blocked on a slow consumer dwarfs the stall timeout.
+154 -48
View File
@@ -15,8 +15,6 @@
//! Single-disk object rename publication and rollback. The shared execution core
//! retains instrumentation, mutation leases, and commit guards through the syscall.
#[cfg(target_os = "linux")]
use super::has_rename_data_test_failpoints;
#[cfg(all(test, windows))]
use super::run_destination_commit_directory_preparation;
#[cfg(any(not(windows), test))]
@@ -24,8 +22,8 @@ use super::should_fail_local_inline_rollback_hardlink;
use super::{
EVENT_DISK_LOCAL_ACCESS_FAILED, EVENT_DISK_LOCAL_HEAL_PURGE_FAILED, EVENT_DISK_LOCAL_RENAME_REJECTED, LOG_COMPONENT_ECSTORE,
LOG_SUBSYSTEM_DISK_LOCAL, LocalDisk, SyncMode, effective_durability, has_inline_commit_test_failpoints,
inline_metadata_rollback_dir, observe_old_current, remove_dir_all_if_exists, remove_dst_base_before_commit,
remove_file_if_exists, rename_data_versions_signature, run_inline_preparation_before_backup,
has_rename_data_test_failpoints, inline_metadata_rollback_dir, observe_old_current, remove_dir_all_if_exists,
remove_dst_base_before_commit, remove_file_if_exists, rename_data_versions_signature, run_inline_preparation_before_backup,
should_fail_after_metadata_commit, should_fail_before_old_metadata_backup, should_fail_commit_rename,
should_remove_staged_meta_before_commit, skip_access_checks, tmp_staging_mutation_path,
};
@@ -172,6 +170,44 @@ impl RenameCommitContext<'_> {
}
}
#[cfg(any(not(windows), test))]
fn inline_rollback_backup_path(staging_file_path: &Path) -> std::io::Result<PathBuf> {
let Some(staging_parent) = staging_file_path.parent() else {
return Err(std::io::Error::new(ErrorKind::InvalidInput, "missing staging metadata parent"));
};
let backup_path = if staging_file_path.file_name().and_then(|n| n.to_str()) == Some(STORAGE_FORMAT_FILE) {
staging_parent.join(STORAGE_FORMAT_FILE_BACKUP)
} else {
staging_file_path.with_extension("meta.bkp")
};
remove_file_if_exists(&backup_path)?;
Ok(backup_path)
}
#[cfg(any(not(windows), test))]
fn write_local_inline_rollback_backup_from_bytes(
staging_file_path: &Path,
old_metadata: &[u8],
sync: bool,
) -> std::io::Result<PathBuf> {
let backup_path = inline_rollback_backup_path(staging_file_path)?;
let write_res = (|| -> std::io::Result<()> {
#[cfg(test)]
run_owned_file_before_open(&backup_path);
let mut file = std::fs::File::create(&backup_path)?;
std::io::Write::write_all(&mut file, old_metadata)?;
if sync && !old_metadata.is_empty() {
file.sync_data()?;
}
Ok(())
})();
if let Err(err) = write_res {
let _ = remove_file_if_exists(&backup_path);
return Err(err);
}
Ok(backup_path)
}
#[cfg(any(not(windows), test))]
pub(super) fn create_local_inline_rollback_backup(
dst_file_path: &Path,
@@ -179,27 +215,12 @@ pub(super) fn create_local_inline_rollback_backup(
old_metadata: &[u8],
sync: bool,
) -> std::io::Result<PathBuf> {
let Some(staging_parent) = staging_file_path.parent() else {
return Err(std::io::Error::new(ErrorKind::InvalidInput, "missing staging metadata parent"));
};
let backup_path = staging_parent.join(STORAGE_FORMAT_FILE_BACKUP);
remove_file_if_exists(&backup_path)?;
let backup_path = inline_rollback_backup_path(staging_file_path)?;
if should_fail_local_inline_rollback_hardlink(dst_file_path) || std::fs::hard_link(dst_file_path, &backup_path).is_err() {
let write_res = (|| -> std::io::Result<()> {
#[cfg(test)]
run_owned_file_before_open(&backup_path);
let mut file = std::fs::File::create(&backup_path)?;
std::io::Write::write_all(&mut file, old_metadata)?;
if sync && !old_metadata.is_empty() {
file.sync_data()?;
}
Ok(())
})();
if let Err(err) = write_res {
let _ = remove_file_if_exists(&backup_path);
return Err(err);
}
} else if sync && !old_metadata.is_empty() {
return write_local_inline_rollback_backup_from_bytes(staging_file_path, old_metadata, sync);
} else if cfg!(test) && sync && !old_metadata.is_empty() {
// Skip redundant fdatasync on hardlinked rollback backups in production;
// keep a Windows-safe sync open in tests.
os::open_file_for_sync_std(&backup_path)?.sync_data()?;
}
Ok(backup_path)
@@ -265,12 +286,15 @@ pub(super) fn fsync_inline_commit_std(
if destination_was_absent {
let mut ancestor = dst.parent().and_then(Path::parent);
let mut ancestor_step = 0usize;
while let Some(ancestor_dir) = ancestor {
if !ancestor_dir.starts_with(dst_volume_dir) {
break;
}
let fsync_started = rustfs_io_metrics::put_stage_timer();
if let Err(err) = os::fsync_dir_limited_std(ancestor_dir) {
if let Err(err) =
os::fsync_commit_ancestor_directory_limited_std(ancestor_dir, dst_volume_dir, ancestor_step, commit_guard)
{
rustfs_io_metrics::record_put_object_stage_duration_from(
rustfs_io_metrics::PUT_STAGE_SET_DISK_RENAME_ANCESTOR_DIR_FSYNC,
fsync_started,
@@ -289,6 +313,7 @@ pub(super) fn fsync_inline_commit_std(
break;
}
ancestor = ancestor_dir.parent();
ancestor_step += 1;
}
}
Ok(())
@@ -658,7 +683,14 @@ impl LocalDisk {
})
.collect()
};
let src_volume_dir = self.io_get_bucket_path(src_volume)?;
let src_volume_dir = if cfg!(target_os = "linux")
&& src_volume == super::super::RUSTFS_META_TMP_BUCKET
&& skip_access_checks(src_volume)
{
self.io_root().join(src_volume)
} else {
self.io_get_bucket_path(src_volume)?
};
if !skip_access_checks(src_volume)
&& let Err(e) = super::super::fs::access_std(&src_volume_dir)
{
@@ -702,8 +734,24 @@ impl LocalDisk {
} else {
STORAGE_FORMAT_FILE
};
let flat_inline_tmp = cfg!(target_os = "linux")
&& !no_inline
&& src_volume == super::super::RUSTFS_META_TMP_BUCKET
&& !preserve_upload_metadata
&& multipart_publication.is_none()
&& pool_meta_condition.is_none()
&& !fi_healing
&& !os::dst_dir_fsync_group_commit_enabled()
&& !os::file_fdatasync_group_commit_enabled()
&& !has_rename_data_test_failpoints(dst_path);
let src_file_path = match multipart_publication.as_ref() {
Some(publication) => publication.canonical_head(),
None if flat_inline_tmp => {
self.io_get_object_open_path(src_volume, format!("{src_path}.{source_metadata_name}").as_str())?
}
None if cfg!(target_os = "linux") => {
self.io_get_object_open_path(src_volume, format!("{src_path}/{source_metadata_name}").as_str())?
}
None => self.io_get_object_path(src_volume, format!("{src_path}/{source_metadata_name}").as_str())?,
};
let dst_file_path = self.io_get_object_open_path(dst_volume, format!("{}/{}", dst_path, STORAGE_FORMAT_FILE).as_str())?;
@@ -714,14 +762,13 @@ impl LocalDisk {
.data_dir
.map(|dir| rustfs_utils::path::retain_slash(dir.to_string().as_str()))
{
let src_data_path = self.io_get_object_path(
src_volume,
rustfs_utils::path::retain_slash(format!("{}/{}", src_path, data_dir).as_str()).as_str(),
)?;
let dst_data_path = self.io_get_object_path(
dst_volume,
rustfs_utils::path::retain_slash(format!("{}/{}", dst_path, data_dir).as_str()).as_str(),
)?;
let src_rel = rustfs_utils::path::retain_slash(format!("{}/{}", src_path, data_dir).as_str());
let src_data_path = if cfg!(target_os = "linux") && multipart_publication.is_none() && !fi_healing {
self.io_get_object_open_path(src_volume, src_rel.as_str())?
} else {
self.io_get_object_path(src_volume, src_rel.as_str())?
};
let dst_data_path = destination_object_path.join(data_dir.trim_matches('/'));
Some((src_data_path, dst_data_path))
} else {
@@ -804,7 +851,7 @@ impl LocalDisk {
let fused_attempt = move |is_cancelled: &dyn Fn() -> bool| -> std::io::Result<FusedCommitAttempt> {
let (guard, has_dst_buf) = prepare_rename_commit_and_destination_metadata_std(
!no_inline,
!no_inline && !flat_inline_tmp,
&src_parent_owned,
&dst_parent_owned,
&dst_file_path_owned,
@@ -876,7 +923,16 @@ impl LocalDisk {
run_owned_file_before_open(&src_file_path_owned);
let mut prepared_metadata_source =
os::create_prepared_rename_source_with_commit_guard(&src_file_path_owned, &dst_file_path_owned, &guard)?;
prepared_metadata_source.write_all(&new_dst_buf, false)?;
let cleanup_flat_tmp = || {
if flat_inline_tmp {
let _ = remove_file_if_exists(&src_file_path_owned);
}
};
if let Err(err) = prepared_metadata_source.write_all(&new_dst_buf, false) {
drop(prepared_metadata_source);
cleanup_flat_tmp();
return Err(err);
}
if !no_inline {
run_inline_preparation_before_backup(&dst_path_str);
}
@@ -884,14 +940,21 @@ impl LocalDisk {
let mut staged_rollback_path = None;
if (!no_inline || rollback_data_dir.is_none())
&& let Some(ref old_metadata) = has_dst_buf
&& (rollback_data_dir.is_some() || sync_commit_metadata || cfg!(test))
&& (rollback_data_dir.is_some() || cfg!(test))
{
let backup_path = create_local_inline_rollback_backup(
let backup_path = match create_local_inline_rollback_backup(
&dst_file_path_owned,
&src_file_path_owned,
old_metadata,
sync_commit_metadata,
)?;
) {
Ok(path) => path,
Err(err) => {
drop(prepared_metadata_source);
cleanup_flat_tmp();
return Err(err);
}
};
staged_rollback_path = Some(StagedRollbackBackup(backup_path));
}
@@ -920,6 +983,8 @@ impl LocalDisk {
return Err(err);
}
} else if sync_commit_metadata && let Err(err) = prepared_metadata_source.sync_data() {
drop(prepared_metadata_source);
cleanup_flat_tmp();
if let Some(backup_path) = staged_rollback_path.as_ref().map(StagedRollbackBackup::path) {
let _ = remove_file_if_exists(backup_path);
}
@@ -928,6 +993,7 @@ impl LocalDisk {
if is_cancelled() {
drop(prepared_metadata_source);
cleanup_flat_tmp();
if let Some(backup_path) = staged_rollback_path.as_ref().map(StagedRollbackBackup::path) {
let _ = remove_file_if_exists(backup_path);
}
@@ -944,6 +1010,8 @@ impl LocalDisk {
) {
Ok(guard) => guard,
Err(err) => {
drop(prepared_metadata_source);
cleanup_flat_tmp();
if let Some(backup_path) = staged_rollback_path.as_ref().map(StagedRollbackBackup::path) {
let _ = remove_file_if_exists(backup_path);
}
@@ -1027,6 +1095,8 @@ impl LocalDisk {
None,
sync_commit_metadata,
) {
drop(prepared_metadata_source);
cleanup_flat_tmp();
let _ = remove_file_if_exists(staged_backup);
return Ok(FusedCommitAttempt::CommitErr(to_file_error(err).into()));
}
@@ -1047,6 +1117,7 @@ impl LocalDisk {
)
};
if let Err(err) = rename_res {
cleanup_flat_tmp();
if let Some(backup_path) = staged_rollback_path.as_ref().map(StagedRollbackBackup::path) {
let _ = remove_file_if_exists(backup_path);
}
@@ -1075,8 +1146,38 @@ impl LocalDisk {
return Ok(FusedCommitAttempt::CommitErr(DiskError::Unexpected));
}
if sync_commit_metadata
&& let Err(err) = fsync_inline_commit_std(
if sync_commit_metadata {
if staged_rollback_path.is_none()
&& rollback_data_dir.is_none()
&& let Some(ref old_metadata) = has_dst_buf
&& (!no_inline || rollback_data_dir.is_none())
{
let fsync_res = (|| -> std::io::Result<()> {
if let Some(dst_parent) = dst_file_path_owned.parent() {
os::fsync_commit_directory_limited_std(dst_parent, &guard)?;
}
Ok(())
})();
if let Err(err) = fsync_res {
let rollback_ok =
write_local_inline_rollback_backup_from_bytes(&src_file_path_owned, old_metadata, true)
.and_then(|backup_path| {
let backup = StagedRollbackBackup(backup_path);
let res =
rollback_inline_metadata_std(&dst_file_path_owned, rollback_data_dir, Some(&backup));
let _ = remove_file_if_exists(backup.path());
res
})
.is_ok();
if no_inline {
if rollback_ok && let Some((_, new_data_path)) = has_data_dir_path_owned.as_ref() {
let _ = remove_dir_all_if_exists(new_data_path);
}
return Ok(FusedCommitAttempt::CommitErr(to_file_error(err).into()));
}
return Ok(FusedCommitAttempt::CommitErr(DiskError::from(err)));
}
} else if let Err(err) = fsync_inline_commit_std(
&dst_file_path_owned,
&dst_volume_dir_owned,
destination_was_absent,
@@ -1087,19 +1188,19 @@ impl LocalDisk {
.filter(|_| no_inline)
.map(|(_, new_data_path)| new_data_path.as_path()),
&guard,
)
{
if no_inline {
return Ok(FusedCommitAttempt::CommitErr(to_file_error(err).into()));
) {
if no_inline {
return Ok(FusedCommitAttempt::CommitErr(to_file_error(err).into()));
}
return Ok(FusedCommitAttempt::CommitErr(DiskError::from(err)));
}
return Ok(FusedCommitAttempt::CommitErr(DiskError::from(err)));
}
if let Some(backup_path) = staged_rollback_path.as_ref().map(StagedRollbackBackup::path) {
let _ = remove_file_if_exists(backup_path);
}
if let Some(parent) = src_file_path_owned.parent() {
if !flat_inline_tmp && let Some(parent) = src_file_path_owned.parent() {
let _ = std::fs::remove_dir(parent);
}
@@ -1364,7 +1465,12 @@ impl LocalDisk {
}
}
}
match publication_owned.finish_on_thread(sync_commit_metadata, staging_cleanup) {
match publication_owned.finish_on_thread(
sync_commit_metadata,
!cfg!(test),
staging_cleanup,
Some(&directory_guard_owned),
) {
Ok(result) => Ok(MultipartFusedCommitOutcome::Committed(Box::new(RenameDataResp {
multipart_source: Some(Box::new(result)),
old_data_dir: has_old_data_dir,
File diff suppressed because it is too large Load Diff
@@ -372,29 +372,32 @@ impl IndexOwner {
#[cfg(unix)]
{
use rustix::fs::{Mode, OFlags, mkdirat, openat};
let mut directories = vec![self._identity_pin.try_clone()?];
for component in [".rustfs.sys", "multipart-index-v1", &sha(bucket)] {
let parent = directories.last().ok_or_else(|| io::Error::other("index parent missing"))?;
match mkdirat(parent, component, Mode::RWXU | Mode::RWXG | Mode::RWXO) {
Ok(()) | Err(rustix::io::Errno::EXIST) => {}
Err(error) => return Err(error.into()),
let open_dir = |parent: rustix::fd::BorrowedFd<'_>, component: &str| -> io::Result<rustix::fd::OwnedFd> {
let flags = OFlags::RDONLY | OFlags::DIRECTORY | OFlags::NOFOLLOW | OFlags::CLOEXEC;
match openat(parent, component, flags, Mode::empty()) {
Ok(dir) => Ok(dir),
Err(rustix::io::Errno::NOENT) => {
match mkdirat(parent, component, Mode::RWXU | Mode::RWXG | Mode::RWXO) {
Ok(()) | Err(rustix::io::Errno::EXIST) => {}
Err(error) => return Err(error.into()),
}
openat(parent, component, flags, Mode::empty()).map_err(Into::into)
}
Err(error) => Err(error.into()),
}
let directory = openat(
parent,
component,
OFlags::RDONLY | OFlags::DIRECTORY | OFlags::NOFOLLOW | OFlags::CLOEXEC,
Mode::empty(),
)?;
directories.push(std::fs::File::from(directory));
}
let parent = directories.last().ok_or_else(|| io::Error::other("index parent missing"))?;
};
use std::os::fd::AsFd;
let bucket_sha = sha(bucket);
let sys_dir = open_dir(self._identity_pin.as_fd(), ".rustfs.sys")?;
let index_dir = open_dir(sys_dir.as_fd(), "multipart-index-v1")?;
let parent = open_dir(index_dir.as_fd(), &bucket_sha)?;
// The index is boot-local: every start rescans the staging area
// before serving it, so a marker lost to a crash is recreated before
// it can be consulted. Creation is enough; no fsync of the marker or
// its ancestors. An existing marker is touched so pruning sees the
// new upload's age, not the previous one's.
match openat(
parent,
&parent,
source_root,
OFlags::CREATE | OFlags::EXCL | OFlags::WRONLY | OFlags::NOFOLLOW | OFlags::CLOEXEC,
Mode::RUSR | Mode::WUSR,
@@ -409,7 +412,7 @@ impl IndexOwner {
// the previous upload's age, which only shortens the prune
// grace for this key; index completeness does not depend on it.
if let Ok(marker) =
openat(parent, source_root, OFlags::WRONLY | OFlags::NOFOLLOW | OFlags::CLOEXEC, Mode::empty())
openat(&parent, source_root, OFlags::WRONLY | OFlags::NOFOLLOW | OFlags::CLOEXEC, Mode::empty())
{
let _ = std::fs::File::from(marker).set_modified(std::time::SystemTime::now());
}
@@ -426,10 +429,22 @@ impl IndexOwner {
os::replace_multipart_source_metadata(&marker, b"1", &self.base, &root)?;
os::sync_multipart_owned_record_parents(&self.base.join(".rustfs.sys"), &directory, true, &self.base, &root)?;
}
self.ensure_current_root()
Ok(())
}
fn ensure_current_root(&self) -> io::Result<()> {
#[cfg(unix)]
let current = {
use std::os::unix::fs::MetadataExt;
let metadata = std::fs::symlink_metadata(&*self.root_path)?;
if !metadata.is_dir() {
return Err(io::Error::other("multipart index physical root is not a directory"));
}
let mut inode = [0; 16];
inode[..8].copy_from_slice(&metadata.ino().to_le_bytes());
(metadata.dev(), inode)
};
#[cfg(not(unix))]
let current = os::PublicationRoot::new(&self.root_path)?.physical_identity()?;
if current != self.physical {
self.disabled.store(true, Ordering::Release);
@@ -532,13 +547,10 @@ impl IndexOwner {
Ok(())
}
pub(super) async fn record_head(self: &Arc<Self>, path: &str, bytes: &[u8]) -> Result<()> {
let Some(upload) = path
pub(super) async fn prepare_record_head(self: &Arc<Self>, path: &str, bytes: &[u8]) -> Option<PreparedHeadRecord> {
let upload = path
.strip_suffix("/xl.meta")
.filter(|upload| canonical_upload(upload).is_some())
else {
return Ok(());
};
.filter(|upload| canonical_upload(upload).is_some())?;
let info = match get_file_info(
bytes,
RUSTFS_META_MULTIPART_BUCKET,
@@ -553,10 +565,33 @@ impl IndexOwner {
Ok(info) => info,
Err(_) => {
self.disabled.store(true, Ordering::Release);
return Ok(());
return None;
}
};
self.record_info(upload, &info).await
let marker = info
.metadata
.get(crate::disk::multipart_index::RUSTFS_MULTIPART_BUCKET_KEY)
.map(|bucket| {
self.base
.join(INDEX)
.join(sha(bucket))
.join(upload.split('/').next().unwrap_or_default())
})
.unwrap_or_else(|| self.base.join(INDEX));
let lease = os::acquire_namespace_mutation_lease_with_owner(&marker, None).await;
Some(PreparedHeadRecord {
owner: Arc::clone(self),
upload: upload.to_owned(),
info,
_lease: lease,
})
}
pub(super) async fn record_head(self: &Arc<Self>, path: &str, bytes: &[u8]) -> Result<()> {
let Some(prepared) = self.prepare_record_head(path, bytes).await else {
return Ok(());
};
prepared.run().await
}
async fn page(self: &Arc<Self>, bucket: &str, marker: Option<&str>, disk_id: Uuid) -> Result<MultipartIndexPage> {
@@ -678,6 +713,37 @@ impl IndexOwner {
}
}
pub(super) struct PreparedHeadRecord {
owner: Arc<IndexOwner>,
upload: String,
info: FileInfo,
_lease: Arc<os::NamespaceMutationLease>,
}
impl PreparedHeadRecord {
pub(super) fn run_std(self) -> io::Result<()> {
let Self {
owner,
upload,
info,
_lease,
} = self;
match owner.record(&upload, &info) {
Err(error) if error.kind() == io::ErrorKind::InvalidData => {
owner.disabled.store(true, Ordering::Release);
Ok(())
}
result => result,
}
}
async fn run(self) -> Result<()> {
let lease = Arc::clone(&self._lease);
os::run_blocking_namespace_operation_with_sync(lease, true, move || self.run_std()).await?;
Ok(())
}
}
impl LocalDisk {
#[cfg(any(test, feature = "test-util"))]
pub(in crate::disk) async fn wait_for_multipart_index_bootstrap(&self) {
@@ -428,7 +428,6 @@ impl LocalDisk {
let root = store.publication_root.clone();
let lease = os::acquire_namespace_mutation_lease_with_owner(&destination, Some(store.lease.clone())).await;
os::run_blocking_namespace_operation_with_sync(lease, true, move || {
permit.validate()?;
os::rename_multipart_source_owned_std(&source, &destination, &base, &root, || permit.validate())
})
.await?;
@@ -295,10 +295,17 @@ pub(super) fn put_active_multipart_source_head(source: impl AsRef<Path>, raw: By
}
fn stamp_matches_active_disk(source: &Path, expected_head_stamp: FileStamp) -> bool {
FileStamp::from_path(&source.join(STORAGE_FORMAT_FILE)) == Some(expected_head_stamp)
&& !source.join(STATE).exists()
&& !source.join(crate::disk::MULTIPART_RECOVERY_DIR).exists()
&& !source.join(".mpu-completions").exists()
if FileStamp::from_path(&source.join(STORAGE_FORMAT_FILE)) != Some(expected_head_stamp) {
return false;
}
#[cfg(test)]
if source.join(STATE).exists()
|| source.join(crate::disk::MULTIPART_RECOVERY_DIR).exists()
|| source.join(".mpu-completions").exists()
{
return false;
}
true
}
pub(super) fn get_active_multipart_source_head(source: &Path) -> Option<(Bytes, FileInfo)> {
@@ -375,7 +382,11 @@ pub(super) fn finish_active_multipart_part_transaction(
ACTIVE_MULTIPART_SOURCE_HEADS.remove_if_mut(source, |entry| !entry.unsettled_parts.remove(&part_number));
}
pub(super) fn get_active_multipart_source_part(source: &Path, part_number: usize) -> Option<rustfs_filemeta::ObjectPartInfo> {
pub(super) fn get_active_multipart_source_part_with_head(
source: &Path,
part_number: usize,
head_already_verified: bool,
) -> Option<rustfs_filemeta::ObjectPartInfo> {
let (head_stamp, data_dir, expected_stamp, part) = ACTIVE_MULTIPART_SOURCE_HEADS
.with_get(source, |entry| {
if !entry.unsettled_parts.is_empty() {
@@ -386,7 +397,7 @@ pub(super) fn get_active_multipart_source_part(source: &Path, part_number: usize
Some((entry.head_stamp, data_dir, *expected_stamp, part.clone()))
})
.flatten()?;
if !stamp_matches_active_disk(source, head_stamp) {
if !head_already_verified && !stamp_matches_active_disk(source, head_stamp) {
ACTIVE_MULTIPART_SOURCE_HEADS.remove(source);
return None;
}
@@ -609,25 +620,45 @@ impl SourceStore {
raw: &[u8],
terminal: Option<&Path>,
sync_source_before: bool,
) -> std::io::Result<()> {
Self::persist_state_with_guard_std(source, base, root, raw, terminal, sync_source_before, None)
}
fn persist_state_with_guard_std(
source: &Path,
base: &Path,
root: &os::PublicationRoot,
raw: &[u8],
terminal: Option<&Path>,
sync_source_before: bool,
directory_guard: Option<&os::RenameCommitGuard>,
) -> std::io::Result<()> {
invalidate_active_multipart_source_head(source);
#[cfg(unix)]
let mut pinned = os::pin_multipart_source_directory(source, base, root)?;
#[cfg(not(unix))]
#[cfg(not(target_os = "linux"))]
let _pinned = os::pin_multipart_source_directory(source, base, root)?;
if sync_source_before {
#[cfg(unix)]
#[cfg(target_os = "linux")]
{
let mut pinned = os::pin_multipart_source_directory(source, base, root)?;
let opened = std::fs::File::from(pinned.pop().ok_or_else(|| IoError::other("empty source directory guard"))?);
os::fsync_directory_handle_std(source, Some(&opened))?;
}
#[cfg(all(unix, not(target_os = "linux")))]
{
let opened = std::fs::File::from(
_pinned
.last()
.ok_or_else(|| IoError::other("empty source directory guard"))?
.try_clone()?,
);
os::fsync_directory_handle_std(source, Some(&opened))?;
}
#[cfg(not(unix))]
os::fsync_dir_std(source)?;
}
if let Some(path) = terminal {
os::replace_multipart_source_metadata(path, raw, base, root)?;
}
os::replace_multipart_source_metadata(&source.join(STATE), raw, base, root)
os::replace_multipart_source_metadata_with_guard(&source.join(STATE), raw, base, root, directory_guard)
}
async fn write_owned_backup_and_persist(&self, owner: Uuid, target: bool, bytes: &[u8], state: &SourceState) -> Result<()> {
if owner.is_nil() {
@@ -724,17 +755,39 @@ impl SourceStore {
bytes: &[u8],
sync_source_root: bool,
) -> std::io::Result<()> {
Self::write_owned_backup_inner_std(source, base, root, owner, target, bytes, sync_source_root, false, true, false)
}
#[expect(clippy::too_many_arguments)]
fn write_owned_backup_inner_std(
source: &Path,
base: &Path,
root: &os::PublicationRoot,
owner: Uuid,
target: bool,
bytes: &[u8],
sync_source_root: bool,
fresh_owner: bool,
sync_parents: bool,
source_already_pinned: bool,
) -> std::io::Result<()> {
invalidate_active_multipart_source_head(source);
let path = if target {
target_backup_path(source, owner)
} else {
original_path(source, owner)
};
let _pin = os::pin_multipart_source_directory(source, base, root)?;
let _pin = if source_already_pinned {
None
} else {
Some(os::pin_multipart_source_directory(source, base, root)?)
};
let parent = path
.parent()
.ok_or_else(|| std::io::Error::other("source backup has no parent"))?;
let _parents = os::mkdir_all_below_existing_base_std(parent, base, root)?;
let existed = if target {
let existed = if fresh_owner {
false
} else if target {
match os::observe_multipart_source_metadata(&path, base, root, bytes.len() as u64) {
Ok((length, sha)) if length == bytes.len() as u64 && sha == raw_sha(bytes) => true,
Ok(_) => {
@@ -754,7 +807,10 @@ impl SourceStore {
if !existed {
os::replace_multipart_source_metadata(&path, bytes, base, root)?;
}
os::sync_multipart_owned_record_parents_options(source, parent, existed, sync_source_root, base, root)
if sync_parents {
os::sync_multipart_owned_record_parents_options(source, parent, existed, sync_source_root, base, root)?;
}
Ok(())
}
pub async fn replace_head(&self, bytes: &[u8]) -> Result<()> {
self.multipart_index
@@ -1979,25 +2035,28 @@ impl NativeMultipartPublication {
let canonical = self.canonical_head();
let prepared = (|| -> std::io::Result<os::PreparedRenameSource> {
if let Some((ref original_bytes, _, _)) = deferred_initial {
let current = load_source(source, base, root, path, disk_id, physical_slot, disk_ids).map_err(IoError::other)?;
let snapshot = current
.snapshot
.as_ref()
.ok_or_else(|| IoError::other(DiskError::ConditionalDeleteConflict))?;
let current_head = current
.head
.as_ref()
.and_then(SourceHead::raw)
.ok_or_else(|| IoError::other(DiskError::ConditionalDeleteConflict))?;
if !current.head_matches_state
|| snapshot.phase != MultipartSourcePhase::Active
|| snapshot.capability_sha256.is_some()
|| snapshot.lineage != expected_lineage
|| snapshot.generation != expected_generation
|| snapshot.revision.checked_add(1) != Some(expected_revision)
|| current_head != original_bytes.as_slice()
{
return Err(IoError::other(DiskError::ConditionalDeleteConflict));
if cfg!(test) || !can_mark_presynced {
let current =
load_source(source, base, root, path, disk_id, physical_slot, disk_ids).map_err(IoError::other)?;
let snapshot = current
.snapshot
.as_ref()
.ok_or_else(|| IoError::other(DiskError::ConditionalDeleteConflict))?;
let current_head = current
.head
.as_ref()
.and_then(SourceHead::raw)
.ok_or_else(|| IoError::other(DiskError::ConditionalDeleteConflict))?;
if !current.head_matches_state
|| snapshot.phase != MultipartSourcePhase::Active
|| snapshot.capability_sha256.is_some()
|| snapshot.lineage != expected_lineage
|| snapshot.generation != expected_generation
|| snapshot.revision.checked_add(1) != Some(expected_revision)
|| current_head != original_bytes.as_slice()
{
return Err(IoError::other(DiskError::ConditionalDeleteConflict));
}
}
} else {
let current = load_source(source, base, root, path, disk_id, physical_slot, disk_ids).map_err(IoError::other)?;
@@ -2021,11 +2080,41 @@ impl NativeMultipartPublication {
}
let setup_res: std::io::Result<()> = (|| {
if let Some((original_bytes, directory, anchor_bytes)) = deferred_initial {
SourceStore::write_owned_backup_std(source, base, root, backup_owner, false, &original_bytes, false)?;
SourceStore::write_owned_backup_inner_std(
source,
base,
root,
backup_owner,
false,
&original_bytes,
false,
true,
old_metadata.is_none(),
true,
)?;
if let Some(previous) = old_metadata {
SourceStore::write_owned_backup_std(source, base, root, backup_owner, true, previous, false)?;
SourceStore::write_owned_backup_inner_std(
source,
base,
root,
backup_owner,
true,
previous,
false,
true,
true,
true,
)?;
}
SourceStore::persist_state_std(source, base, root, &next_state_raw, terminal.as_deref(), false)?;
SourceStore::persist_state_with_guard_std(
source,
base,
root,
&next_state_raw,
terminal.as_deref(),
false,
Some(directory_guard),
)?;
let _parent = os::mkdir_all_below_existing_base_std(&directory, base, root)?;
os::replace_multipart_source_metadata(&directory.join(RETIREMENT_ANCHOR), &anchor_bytes, base, root)?;
#[cfg(target_os = "linux")]
@@ -2034,9 +2123,28 @@ impl NativeMultipartPublication {
}
} else {
if let Some(previous) = old_metadata {
SourceStore::write_owned_backup_std(source, base, root, backup_owner, true, previous, false)?;
SourceStore::write_owned_backup_inner_std(
source,
base,
root,
backup_owner,
true,
previous,
false,
false,
true,
true,
)?;
}
SourceStore::persist_state_std(source, base, root, &next_state_raw, terminal.as_deref(), false)?;
SourceStore::persist_state_with_guard_std(
source,
base,
root,
&next_state_raw,
terminal.as_deref(),
false,
Some(directory_guard),
)?;
}
Ok(())
})();
@@ -2050,7 +2158,7 @@ impl NativeMultipartPublication {
let _ = std::fs::remove_file(prepared_path);
return Err(err);
}
os::install_multipart_prepared_head(prepared_metadata_source, &canonical, base, root)
os::install_multipart_prepared_head(prepared_metadata_source, &canonical, base, root, Some(directory_guard))
})()
.map_err(DiskError::from)?;
*self.state.lock() = state;
@@ -2160,7 +2268,7 @@ impl NativeMultipartPublication {
let mut prepared_metadata_source =
os::create_prepared_rename_source_with_commit_guard(&prepared_path, &dst_file_path, &owned_directory_guard)?;
prepared_metadata_source.write_all(&new_bytes, true)?;
os::install_multipart_prepared_head(prepared_metadata_source, &canonical, &base, &root)
os::install_multipart_prepared_head(prepared_metadata_source, &canonical, &base, &root, Some(&owned_directory_guard))
})
.await
.map_err(DiskError::from)?;
@@ -2207,7 +2315,7 @@ impl NativeMultipartPublication {
let base = self.store.base.clone();
let root = self.store.publication_root.clone();
os::run_blocking_namespace_operation(self.store.lease.clone(), move || {
os::install_multipart_prepared_head(prepared, &canonical, &base, &root)
os::install_multipart_prepared_head(prepared, &canonical, &base, &root, None)
})
.await
.map_err(DiskError::from)
@@ -2264,7 +2372,9 @@ impl NativeMultipartPublication {
pub(super) fn finish_on_thread(
&self,
target_already_synced: bool,
fused_in_place: bool,
staging_cleanup: Option<(PathBuf, Arc<std::sync::atomic::AtomicBool>)>,
directory_guard: Option<&os::RenameCommitGuard>,
) -> Result<MultipartPublicationResult> {
let mut state = self.state.lock().clone();
let expected_state_raw = encode(&state)?;
@@ -2293,62 +2403,72 @@ impl NativeMultipartPublication {
let disk_ids = &self.store.disk_ids;
let terminal = terminal_record_path(source, &state);
(|| -> std::io::Result<()> {
let current = load_source(source, base, root, path, disk_id, physical_slot, disk_ids).map_err(IoError::other)?;
let current_state = current
.state
.as_ref()
.ok_or_else(|| IoError::other(DiskError::ConditionalDeleteConflict))?;
let current_raw = encode(current_state).map_err(IoError::other)?;
if current.head.is_some() || current_raw != expected_state_raw {
return Err(IoError::other(DiskError::ConditionalDeleteConflict));
}
if target_already_synced {
let pinned = os::pin_multipart_source_directory(&target_dir, base, root).map_err(|error| {
if error.kind() == std::io::ErrorKind::NotFound {
IoError::other(DiskError::ConditionalDeleteConflict)
} else {
error
}
})?;
let target = os::read_multipart_source_metadata_in_pinned(
&pinned,
&target_dir,
std::ffi::OsStr::new(STORAGE_FORMAT_FILE),
base,
root,
)
.map_err(|error| {
if error.kind() == std::io::ErrorKind::NotFound {
IoError::other(DiskError::ConditionalDeleteConflict)
} else {
error
}
})?;
if target.len() as u64 != expected_length || raw_sha(&target) != expected_sha256 {
if !fused_in_place {
let current = load_source(source, base, root, path, disk_id, physical_slot, disk_ids).map_err(IoError::other)?;
let current_state = current
.state
.as_ref()
.ok_or_else(|| IoError::other(DiskError::ConditionalDeleteConflict))?;
let current_raw = encode(current_state).map_err(IoError::other)?;
if current.head.is_some() || current_raw != expected_state_raw {
return Err(IoError::other(DiskError::ConditionalDeleteConflict));
}
} else {
os::read_and_sync_multipart_target_metadata(
&target_dir,
std::ffi::OsStr::new(STORAGE_FORMAT_FILE),
base,
root,
|target| {
if target.len() as u64 != expected_length || raw_sha(target) != expected_sha256 {
return Err(IoError::other(DiskError::ConditionalDeleteConflict));
if target_already_synced {
let pinned = os::pin_multipart_source_directory(&target_dir, base, root).map_err(|error| {
if error.kind() == std::io::ErrorKind::NotFound {
IoError::other(DiskError::ConditionalDeleteConflict)
} else {
error
}
Ok(())
},
)
.map_err(|error| {
if error.kind() == std::io::ErrorKind::NotFound {
IoError::other(DiskError::ConditionalDeleteConflict)
} else {
error
})?;
let target = os::read_multipart_source_metadata_in_pinned(
&pinned,
&target_dir,
std::ffi::OsStr::new(STORAGE_FORMAT_FILE),
base,
root,
)
.map_err(|error| {
if error.kind() == std::io::ErrorKind::NotFound {
IoError::other(DiskError::ConditionalDeleteConflict)
} else {
error
}
})?;
if target.len() as u64 != expected_length || raw_sha(&target) != expected_sha256 {
return Err(IoError::other(DiskError::ConditionalDeleteConflict));
}
})?;
} else {
os::read_and_sync_multipart_target_metadata(
&target_dir,
std::ffi::OsStr::new(STORAGE_FORMAT_FILE),
base,
root,
|target| {
if target.len() as u64 != expected_length || raw_sha(target) != expected_sha256 {
return Err(IoError::other(DiskError::ConditionalDeleteConflict));
}
Ok(())
},
)
.map_err(|error| {
if error.kind() == std::io::ErrorKind::NotFound {
IoError::other(DiskError::ConditionalDeleteConflict)
} else {
error
}
})?;
}
}
SourceStore::persist_state_std(source, base, root, &next_state_raw, terminal.as_deref(), true)?;
SourceStore::persist_state_with_guard_std(
source,
base,
root,
&next_state_raw,
terminal.as_deref(),
!fused_in_place,
directory_guard,
)?;
if let Some((io_path, cleaned)) = staging_cleanup {
match std::fs::remove_dir(&io_path) {
Ok(()) => cleaned.store(true, std::sync::atomic::Ordering::Release),
@@ -2381,7 +2501,7 @@ impl NativeMultipartPublication {
) -> Result<MultipartPublicationResult> {
let this = Arc::clone(self);
os::run_blocking_namespace_operation_with_sync(target_lease, true, move || {
this.finish_on_thread(target_already_synced, staging_cleanup)
this.finish_on_thread(target_already_synced, false, staging_cleanup, None)
.map_err(IoError::other)
})
.await
@@ -455,8 +455,11 @@ impl LocalDisk {
}
let meta_name = format!("part.{}.meta", part.number);
let actual = if let Some(cached_part) =
super::multipart_source_state::get_active_multipart_source_part(&source_object, part.number)
{
super::multipart_source_state::get_active_multipart_source_part_with_head(
&source_object,
part.number,
from_active_cache,
) {
cached_part
} else {
if pinned_source_data.is_none() {
+101 -1
View File
@@ -2222,6 +2222,36 @@ pub struct WalkDirOptions {
// Override the remote stream stall timeout for long background walks.
#[serde(default)]
pub stall_timeout_ms: Option<u64>,
/// Select this leaf below identifier-like directory components. The root is
/// `base_dir`; unrelated object metadata must not be read or returned.
/// Older peers ignore this additive field, so consumers must also filter.
#[serde(default, skip_serializing_if = "Option::is_none")]
pub identifier_tree_leaf: Option<String>,
}
pub(crate) fn is_identifier_tree_component(value: &str) -> bool {
!value.is_empty()
&& value
.bytes()
.all(|byte| byte.is_ascii_alphanumeric() || matches!(byte, b'_' | b'-'))
}
pub(crate) fn is_identifier_tree_leaf(value: &str) -> bool {
value.len() <= 255
&& value.contains('.')
&& !matches!(value, "." | "..")
&& value
.bytes()
.all(|byte| byte.is_ascii_alphanumeric() || matches!(byte, b'_' | b'-' | b'.'))
}
pub(crate) fn identifier_tree_leaf_matches(prefix: &str, leaf: &str, object: &str) -> bool {
let Some(relative) = object.strip_prefix(prefix) else {
return false;
};
let mut components = relative.split('/');
components.next_back() == Some(leaf) && components.all(is_identifier_tree_component)
}
impl WalkDirOptions {
@@ -2540,7 +2570,13 @@ pub struct BoundedDeleteRecoveryDiscovery {
pub complete: bool,
}
pub const DELETE_RECOVERY_MAX_DISCOVERY_ENTRIES: u32 = 4096;
/// Entries one bounded discovery call may charge on one drive. Object-scoped
/// discovery charges each transaction-shaped (UUID) child of the key's
/// directory once and every name that cannot be delete evidence at
/// 1/`PLAIN_CHILDREN_PER_BUDGET_ENTRY`, so this admits a missing-key delete of a
/// folder prefix with this many UUID-named children per drive. The request
/// budget (`DeleteRecoveryWorkBudget`) still bounds the total across keys.
pub const DELETE_RECOVERY_MAX_DISCOVERY_ENTRIES: u32 = 262_144;
pub const DELETE_RECOVERY_MAX_INSPECTION_ENTRIES: u32 = 65536;
#[derive(Debug, Clone, Serialize, Deserialize)]
@@ -3157,6 +3193,7 @@ mod tests {
skip_total_timeout: false,
timeout_ms: Some(10_000),
stall_timeout_ms: Some(20_000),
identifier_tree_leaf: None,
};
assert_eq!(opts.bucket, "test-bucket");
@@ -3191,6 +3228,69 @@ mod tests {
assert!(!decoded.skip_hidden_prefix_check);
}
#[test]
fn identifier_tree_leaf_selection_preserves_nested_identifier_directories() {
let prefix = "catalog/namespaces/";
let leaf = "table-entry.json";
for relative in [
"sales/tables/orders/table-entry.json",
"sales/tables/orders/maintenance/tables/owner/table-entry.json",
"maintenance/tables/tables/table-entry.json",
] {
assert!(identifier_tree_leaf_matches(prefix, leaf, &format!("{prefix}{relative}")));
}
for relative in [
"sales/tables/orders/not-table-entry.json",
"sales/tables/orders/table-entry.json/child",
"sales/report.json/tables/owner/table-entry.json",
"sales//tables/owner/table-entry.json",
"sales/../tables/owner/table-entry.json",
] {
assert!(!identifier_tree_leaf_matches(prefix, leaf, &format!("{prefix}{relative}")));
}
assert!(!identifier_tree_leaf_matches(prefix, leaf, "other/sales/tables/orders/table-entry.json"));
assert!(is_identifier_tree_leaf(leaf));
for invalid in ["", ".", "..", "table-entry", "table/entry.json", "table\\entry.json"] {
assert!(!is_identifier_tree_leaf(invalid), "invalid selector {invalid}");
}
}
#[test]
fn identifier_tree_walk_options_are_additive_on_json_and_messagepack_maps() {
#[derive(serde::Deserialize)]
struct LegacyView {
bucket: String,
base_dir: String,
recursive: bool,
}
let options = WalkDirOptions {
bucket: "catalog".to_string(),
base_dir: "namespaces/".to_string(),
recursive: true,
identifier_tree_leaf: Some("table-entry.json".to_string()),
..Default::default()
};
let json = serde_json::to_vec(&options).expect("encode current HTTP options");
let messagepack = rmp_serde::to_vec_named(&options).expect("encode legacy RPC map");
for old in [
serde_json::from_slice::<LegacyView>(&json).expect("old JSON readers ignore the additive field"),
rmp_serde::from_slice::<LegacyView>(&messagepack).expect("old MessagePack readers ignore the additive field"),
] {
assert_eq!(old.bucket, options.bucket);
assert_eq!(old.base_dir, options.base_dir);
assert!(old.recursive);
}
let mut old = serde_json::to_value(&options).expect("encode legacy options");
old.as_object_mut().expect("options are a map").remove("identifier_tree_leaf");
let from_json: WalkDirOptions = serde_json::from_value(old.clone()).expect("decode old HTTP options");
let from_messagepack: WalkDirOptions =
rmp_serde::from_slice(&rmp_serde::to_vec_named(&old).expect("encode old RPC map")).expect("decode old RPC map");
assert!(from_json.identifier_tree_leaf.is_none());
assert!(from_messagepack.identifier_tree_leaf.is_none());
assert_eq!(from_json.base_dir, options.base_dir);
assert_eq!(from_messagepack.base_dir, options.base_dir);
}
/// Test DeleteOptions structure
#[test]
fn test_delete_options() {
+104 -19
View File
@@ -1280,6 +1280,28 @@ pub(crate) fn fsync_dir_limited_std(dir: &Path) -> io::Result<()> {
fsync_dir_std(dir)
}
pub(crate) fn fsync_commit_ancestor_directory_limited_std(
dir: &Path,
base_dir: &Path,
step_from_destination: usize,
guard: &RenameCommitGuard,
) -> io::Result<()> {
#[cfg(target_os = "linux")]
if base_dir == guard.directories.destination_base
&& guard.directories.destination_path.ancestors().nth(1 + step_from_destination) == Some(dir)
&& let Some(ancestors) = guard.directories.destination_ancestors.get()
&& let Some(idx) = ancestors.len().checked_sub(1 + step_from_destination)
&& let Some(file) = ancestors.get(idx)
{
#[cfg(test)]
fsync_dir_recorder::record_limited(dir);
return fsync_directory_handle_std(dir, Some(file));
}
#[cfg(not(target_os = "linux"))]
let _ = (base_dir, step_from_destination, guard);
fsync_dir_limited_std(dir)
}
/// Reuse the published directory without bypassing group commit or sync admission.
pub(crate) async fn fsync_commit_directory(
dir: &Path,
@@ -2200,6 +2222,7 @@ pub(crate) const MAX_FSYNC_WORKERS_PER_DISK: usize = 8;
// Return busy workers to the shared pool so other disks can acquire a slot.
const MAX_BLOCKING_JOBS_PER_WORKER: usize = 64;
const WORKER_IDLE_SPIN_ITERS: usize = 256;
const FSYNC_WORKER_IDLE_SPIN_ITERS: usize = 1024;
const FALLBACK_NAMESPACE_QUEUE_SHARDS: usize = 8;
#[derive(Clone, Copy, PartialEq, Eq)]
@@ -2408,7 +2431,7 @@ impl DiskBlockingQueue {
.ready
.store(!state.jobs.is_empty(), std::sync::atomic::Ordering::Relaxed);
if job.is_none() {
if state.idle_workers > 0 {
if lane != DiskBlockingLane::Fsync && state.idle_workers > 0 {
state.active_workers = state.active_workers.saturating_sub(1);
guard.state = DiskBlockingWorkerState::Retired;
return;
@@ -2426,7 +2449,12 @@ impl DiskBlockingQueue {
}
idle_polled = false;
} else {
for _ in 0..WORKER_IDLE_SPIN_ITERS {
let spin_iters = if lane == DiskBlockingLane::Fsync {
FSYNC_WORKER_IDLE_SPIN_ITERS
} else {
WORKER_IDLE_SPIN_ITERS
};
for _ in 0..spin_iters {
if queue.lane(lane).ready.load(std::sync::atomic::Ordering::Relaxed) {
break;
}
@@ -3798,7 +3826,7 @@ pub(crate) fn rename_all_with_precondition_and_parent_sync_std(
#[cfg(all(test, not(windows)))]
prepared_publication_test_hooks::run_rename_destination(source, destination);
#[cfg(all(any(test, feature = "test-util"), not(windows)))]
{
if prepared_publication_test_hooks::has_any_hooks() {
prepared_publication_test_hooks::run(prepared_publication_test_hooks::Stage::Rename, source);
prepared_publication_test_hooks::run(prepared_publication_test_hooks::Stage::Rename, destination);
if let Some(check) = precondition {
@@ -5738,6 +5766,7 @@ struct LinuxCommitDirectories {
destination_path: PathBuf,
source: LinuxCommitDirectory,
destination: std::sync::OnceLock<LinuxCommitDirectory>,
destination_ancestors: std::sync::OnceLock<Vec<std::fs::File>>,
destination_base: PathBuf,
}
@@ -5754,6 +5783,8 @@ impl LinuxCommitDirectories {
.ok_or_else(|| io::Error::other("missing destination directory guard"))?,
);
let directory = LinuxCommitDirectory::new(directory)?;
let ancestors = parents.into_iter().map(std::fs::File::from).collect();
let _ = self.destination_ancestors.set(ancestors);
Ok(self.destination.get_or_init(|| directory))
}
@@ -5763,7 +5794,7 @@ impl LinuxCommitDirectories {
.ok_or_else(|| io::Error::new(io::ErrorKind::InvalidInput, "file parent does not match its commit guard"))
}
fn validate(&self) -> io::Result<()> {
fn validate(&self, check_destination: bool) -> io::Result<()> {
use std::os::unix::fs::MetadataExt;
// A pinned directory may have been unlinked or replaced. Do not report a
// successful publication into a detached object tree.
@@ -5772,8 +5803,9 @@ impl LinuxCommitDirectories {
return Err(io::Error::new(io::ErrorKind::InvalidData, "commit directory identity changed"));
}
let destination = self.prepare_destination()?;
if (self.source.device, self.source.inode) != (destination.device, destination.inode)
|| self.source_path.as_os_str() != self.destination_path.as_os_str()
if check_destination
&& ((self.source.device, self.source.inode) != (destination.device, destination.inode)
|| self.source_path.as_os_str() != self.destination_path.as_os_str())
{
let current = self.root.open_directory(&self.destination_path)?.metadata()?;
if (current.dev(), current.ino()) != (destination.device, destination.inode) {
@@ -5786,8 +5818,9 @@ impl LinuxCommitDirectories {
fn rename(&self, source: &Path, destination: &Path, prepared: Option<&PreparedRenameSource>) -> io::Result<()> {
let source_name = self.file_name(source, true)?;
let destination_name = self.file_name(destination, false)?;
let destination_was_prepared = self.destination.get().is_some();
let destination_parent = self.prepare_destination()?;
self.validate()?;
self.validate(destination_was_prepared)?;
let mut attempt = 0;
loop {
if let Some(prepared) = prepared {
@@ -6009,7 +6042,7 @@ pub(crate) fn multipart_payload_commit_guard_std(
source_parent: &Path,
) -> io::Result<RenameCommitGuard> {
let original = &metadata_guard.directories;
original.validate()?;
original.validate(true)?;
let source = LinuxCommitDirectory::new(original.root.open_directory(source_parent)?)?;
let destination = original.prepare_destination()?;
// Clone the pinned file identity, never recapture the target pathname
@@ -6026,6 +6059,7 @@ pub(crate) fn multipart_payload_commit_guard_std(
destination_path: original.destination_path.clone(),
source,
destination: std::sync::OnceLock::from(destination),
destination_ancestors: std::sync::OnceLock::new(),
destination_base: original.destination_base.clone(),
}),
})
@@ -6107,6 +6141,7 @@ pub(crate) fn prepare_rename_commit_guard(
destination_path: destination_parent.to_path_buf(),
source,
destination,
destination_ancestors: std::sync::OnceLock::new(),
destination_base: destination_base.to_path_buf(),
}),
})
@@ -7192,6 +7227,40 @@ pub(crate) fn pin_multipart_source_directory(
}
}
fn same_parent_commit_guard(
parent: &Path,
base: &Path,
publication_root: &PublicationRoot,
directory_guard: Option<&RenameCommitGuard>,
) -> io::Result<RenameCommitGuard> {
#[cfg(target_os = "linux")]
if let Some(directory_guard) = directory_guard.filter(|g| g.directories.source_path == parent) {
let dir = &directory_guard.directories.source;
return Ok(RenameCommitGuard {
directories: Arc::new(LinuxCommitDirectories {
root: directory_guard.directories.root.clone(),
source_path: parent.to_path_buf(),
destination_path: parent.to_path_buf(),
source: LinuxCommitDirectory {
file: Arc::clone(&dir.file),
device: dir.device,
inode: dir.inode,
},
destination: std::sync::OnceLock::from(LinuxCommitDirectory {
file: Arc::clone(&dir.file),
device: dir.device,
inode: dir.inode,
}),
destination_ancestors: std::sync::OnceLock::new(),
destination_base: base.to_path_buf(),
}),
});
}
#[cfg(not(target_os = "linux"))]
let _ = directory_guard;
prepare_rename_commit_guard(parent, parent, base, publication_root)
}
/// A small control record always gets its own inode and an atomic replacement.
/// The caller owns the upload SourceRW and keeps its namespace lease alive.
pub(crate) fn replace_multipart_source_metadata(
@@ -7199,11 +7268,21 @@ pub(crate) fn replace_multipart_source_metadata(
bytes: &[u8],
base: &Path,
publication_root: &PublicationRoot,
) -> io::Result<()> {
replace_multipart_source_metadata_with_guard(path, bytes, base, publication_root, None)
}
pub(crate) fn replace_multipart_source_metadata_with_guard(
path: &Path,
bytes: &[u8],
base: &Path,
publication_root: &PublicationRoot,
directory_guard: Option<&RenameCommitGuard>,
) -> io::Result<()> {
let parent = path.parent().ok_or_else(|| io::Error::other("MPU record has no parent"))?;
#[cfg(not(target_os = "linux"))]
let _pinned = pin_multipart_source_directory(parent, base, publication_root)?;
let guard = prepare_rename_commit_guard(parent, parent, base, publication_root)?;
let guard = same_parent_commit_guard(parent, base, publication_root, directory_guard)?;
let staged = parent.join(format!(".mpu-record-{}", uuid::Uuid::new_v4()));
let mut prepared = create_prepared_rename_source_with_commit_guard(&staged, path, &guard)?;
prepared.write_all(bytes, true)?;
@@ -7443,6 +7522,7 @@ pub(crate) fn install_multipart_prepared_head(
canonical_head: &Path,
base: &Path,
publication_root: &PublicationRoot,
directory_guard: Option<&RenameCommitGuard>,
) -> io::Result<PreparedRenameSource> {
let original_path = prepared.path.clone();
let parent = canonical_head
@@ -7453,7 +7533,7 @@ pub(crate) fn install_multipart_prepared_head(
}
#[cfg(not(target_os = "linux"))]
let _pinned = pin_multipart_source_directory(parent, base, publication_root)?;
let guard = prepare_rename_commit_guard(parent, parent, base, publication_root)?;
let guard = same_parent_commit_guard(parent, base, publication_root, directory_guard)?;
rename_prepared_source_std(&prepared, &original_path, canonical_head, base, publication_root, &guard)?;
prepared.path = canonical_head.to_path_buf();
fsync_commit_directory_limited_std(parent, &guard)?;
@@ -7755,8 +7835,7 @@ pub(crate) fn hardlink_multipart_source_files(
}
#[cfg(unix)]
{
use rustix::fs::{AtFlags, Mode, OFlags, linkat, openat, unlinkat};
use std::os::unix::fs::MetadataExt;
use rustix::fs::{AtFlags, FileType, linkat, statat, unlinkat};
#[cfg(test)]
let source_path = source.to_path_buf();
#[cfg(test)]
@@ -7771,20 +7850,26 @@ pub(crate) fn hardlink_multipart_source_files(
.ok_or_else(|| io::Error::other("missing multipart attempt directory"))?;
#[cfg(test)]
prepared_publication_test_hooks::run_hardlink_destination(&source_path, &destination_path);
let flags = OFlags::RDONLY | OFlags::NOFOLLOW | OFlags::CLOEXEC;
for name in names {
if cancelled() {
return Err(io::Error::new(io::ErrorKind::Interrupted, "multipart staging cancelled"));
}
let original = std::fs::File::from(openat(source, name.as_str(), flags, Mode::empty())?);
let expected = original.metadata()?;
if !expected.is_file() {
let expected = statat(source, name.as_str(), AtFlags::SYMLINK_NOFOLLOW)?;
let expected_type = FileType::from_raw_mode(expected.st_mode);
if expected_type.is_symlink() {
return Err(io::Error::from_raw_os_error(libc::ELOOP));
}
if !expected_type.is_file() {
return Err(io::Error::new(io::ErrorKind::InvalidData, "multipart shard is not a regular file"));
}
linkat(source, name.as_str(), destination, name.as_str(), AtFlags::empty())?;
let linked = std::fs::File::from(openat(destination, name.as_str(), flags, Mode::empty())?);
let actual = linked.metadata()?;
if actual.dev() != expected.dev() || actual.ino() != expected.ino() {
let actual = statat(destination, name.as_str(), AtFlags::SYMLINK_NOFOLLOW)?;
if !FileType::from_raw_mode(actual.st_mode).is_file()
|| actual.st_dev != expected.st_dev
|| actual.st_ino != expected.st_ino
|| actual.st_size != expected.st_size
|| actual.st_mtime != expected.st_mtime
{
let _ = unlinkat(destination, name.as_str(), AtFlags::empty());
return Err(io::Error::new(io::ErrorKind::InvalidData, "multipart shard changed during staging"));
}
-24
View File
@@ -2149,15 +2149,6 @@ mod test {
}
}
/// Runs endpoint parsing cases whose local paths do not exist. Unix leaves
/// missing paths out of the physical-disk independence check, but Windows
/// resolves them to their volume, so paths such as `/d1` and `/d2` count as
/// one shared device there. These cases cover parsing, not disk
/// independence, which has its own tests below.
async fn with_disk_independence_check_bypassed<F: std::future::Future<Output = ()>>(cases: F) {
async_with_vars([(ENV_UNSAFE_BYPASS_DISK_CHECK, Some("true"))], cases).await;
}
#[test]
fn endpoint_parse_errors_do_not_echo_url_credentials() {
let err = Endpoints::try_from(["http://:topsecret@server/path"].as_slice()).unwrap_err();
@@ -2166,13 +2157,8 @@ mod test {
assert!(!err.to_string().contains("topsecret"));
}
#[serial]
#[tokio::test]
async fn explicit_local_endpoint_host_selects_only_exact_host_and_server_port() {
with_disk_independence_check_bypassed(explicit_local_endpoint_host_selects_only_exact_host_and_server_port_cases()).await;
}
async fn explicit_local_endpoint_host_selects_only_exact_host_and_server_port_cases() {
let args = vec![
"http://rustfs-0.rustfs-headless.ns.svc.cluster.local./data0",
"http://rustfs-0.rustfs-headless.ns.svc.cluster.local.:9443/data1",
@@ -2935,13 +2921,8 @@ mod test {
}
}
#[serial]
#[tokio::test]
async fn test_create_pool_endpoints() {
with_disk_independence_check_bypassed(test_create_pool_endpoints_cases()).await;
}
async fn test_create_pool_endpoints_cases() {
#[derive(Default)]
struct TestCase<'a> {
num: usize,
@@ -3560,13 +3541,8 @@ mod test {
(urls, local_flags)
}
#[serial]
#[tokio::test]
async fn test_create_server_endpoints() {
with_disk_independence_check_bypassed(test_create_server_endpoints_cases()).await;
}
async fn test_create_server_endpoints_cases() {
let test_cases = [
// Invalid input.
("", vec![], false),
+5 -3
View File
@@ -975,9 +975,11 @@ pub struct DeleteRecoveryWorkBudget {
remaining: std::sync::atomic::AtomicU32,
}
/// Entries one disk may visit per recovery step (discovery plus inspection).
/// Every disk of every set a request fans out to is admitted up to this much
/// and charged on completion, so a budget must cover that many disks.
/// Per-drive unit a request budget is sized by (a typical discovery plus one
/// inspection). Admission charges what a call visited, not this unit, so it only
/// scales the request total with the drives a request can reach; it is kept
/// independent of the per-call caps, which may let one key's probe use more of
/// the same total on a large folder.
const DELETE_RECOVERY_RESERVATION_PER_DISK: u32 = 4096 + 65_536;
impl Default for DeleteRecoveryWorkBudget {
+132 -1
View File
@@ -58,7 +58,7 @@ use std::sync::{
Arc, OnceLock,
atomic::{AtomicBool, AtomicU8, AtomicU64, Ordering},
};
use tokio::sync::{Mutex, Notify, OnceCell, OwnedRwLockReadGuard, RwLock};
use tokio::sync::{Mutex, Notify, OnceCell, OwnedRwLockReadGuard, OwnedSemaphorePermit, RwLock, Semaphore};
use tokio::time::Instant;
use tokio_util::sync::CancellationToken;
use tokio_util::task::TaskTracker;
@@ -187,6 +187,38 @@ impl DecommissionCountLedger {
}
}
#[derive(Debug)]
struct ListReadBudget {
permits: Arc<Semaphore>,
capacity: usize,
}
impl ListReadBudget {
fn new(capacity: usize) -> Self {
let capacity = capacity
.min(Semaphore::MAX_PERMITS)
.min(usize::try_from(u32::MAX).unwrap_or(usize::MAX));
Self {
permits: Arc::new(Semaphore::new(capacity)),
capacity,
}
}
async fn acquire(&self, fanout: usize) -> Result<Option<Arc<OwnedSemaphorePermit>>, tokio::sync::AcquireError> {
if self.capacity == 0 {
return Ok(None);
}
// A single walk must remain admissible even when the topology exceeds
// the configured budget. Tokio accepts at most u32::MAX in one grant.
let weight = u32::try_from(fanout.max(1).min(self.capacity)).unwrap_or(u32::MAX);
self.permits
.clone()
.acquire_many_owned(weight)
.await
.map(|permit| Some(Arc::new(permit)))
}
}
/// Runtime state owned by a single `ECStore` instance.
///
/// This is intentionally minimal in the first migration slice; subsequent
@@ -201,6 +233,7 @@ pub struct InstanceContext {
/// drift out of sync). Stored as one value so the three predicates can
/// never observe a torn intermediate state.
erasure_kind: RwLock<SetupType>,
list_read_budget: OnceLock<ListReadBudget>,
/// This instance's namespace lock manager (Phase 5 Slice 3, backlog#939).
///
/// Owned per-instance so two instances no longer share a lock namespace
@@ -354,6 +387,24 @@ impl InstanceContext {
Self::with_lock_manager(Arc::new(GlobalLockManager::new()))
}
/// Bound native listing fanout independently of GET body permits. A walk
/// keeps its shared grant until all of its disk producers have terminated.
/// Acquire before creating the merge channels or taking object locks.
pub(crate) async fn acquire_list_read_budget(
&self,
fanout: usize,
) -> Result<Option<Arc<OwnedSemaphorePermit>>, tokio::sync::AcquireError> {
self.list_read_budget
.get_or_init(|| {
ListReadBudget::new(rustfs_utils::get_env_usize(
rustfs_config::ENV_OBJECT_MAX_CONCURRENT_DISK_READS,
rustfs_config::DEFAULT_OBJECT_MAX_CONCURRENT_DISK_READS,
))
})
.acquire(fanout)
.await
}
/// Build a context bound to a specific lock manager.
///
/// [`bootstrap_ctx`] uses this to alias the process-global lock manager so
@@ -362,6 +413,7 @@ impl InstanceContext {
fn with_lock_manager(lock_manager: Arc<GlobalLockManager>) -> Self {
Self {
erasure_kind: RwLock::new(SetupType::Unknown),
list_read_budget: OnceLock::new(),
lock_manager,
region: OnceLock::new(),
deployment_id: OnceLock::new(),
@@ -1111,6 +1163,85 @@ pub fn bootstrap_ctx() -> Arc<InstanceContext> {
mod tests {
use super::*;
#[tokio::test]
async fn list_read_budget_weights_and_clamps_fanout() {
let budget = ListReadBudget::new(8);
let first = budget.acquire(3).await.expect("first walk").expect("bounded grant");
assert_eq!(budget.permits.available_permits(), 5);
let mut blocked = std::pin::pin!(budget.acquire(6));
assert!(futures::poll!(&mut blocked).is_pending());
drop(first);
let second = blocked.await.expect("queued walk").expect("bounded grant");
assert_eq!(budget.permits.available_permits(), 2);
drop(second);
let oversized = budget
.acquire(usize::MAX)
.await
.expect("oversized walk")
.expect("bounded grant");
assert_eq!(budget.permits.available_permits(), 0);
drop(oversized);
assert_eq!(budget.permits.available_permits(), 8);
}
#[tokio::test]
async fn list_read_budget_zero_preserves_unbounded_configuration() {
let budget = ListReadBudget::new(0);
assert!(budget.acquire(usize::MAX).await.expect("disabled budget").is_none());
}
#[tokio::test]
async fn list_read_budget_cancelled_waiter_returns_partial_grant() {
let budget = ListReadBudget::new(4);
let held = budget.acquire(3).await.expect("first walk");
{
let mut waiter = std::pin::pin!(budget.acquire(4));
assert!(futures::poll!(&mut waiter).is_pending());
}
assert_eq!(budget.permits.available_permits(), 1);
drop(held);
assert!(budget.acquire(4).await.expect("next walk").is_some());
}
#[tokio::test]
async fn list_read_budget_shared_grant_survives_parent_drop() {
let budget = ListReadBudget::new(4);
let parent = budget.acquire(4).await.expect("first walk").expect("bounded grant");
let producer = parent.clone();
drop(parent);
assert_eq!(budget.permits.available_permits(), 0);
drop(producer);
assert_eq!(budget.permits.available_permits(), 4);
}
#[tokio::test]
async fn list_read_budget_is_shared_by_context_and_isolated_between_instances() {
let first = Arc::new(InstanceContext::new());
first.list_read_budget.set(ListReadBudget::new(4)).expect("fresh context");
let same = first.clone();
let other = InstanceContext::new();
other.list_read_budget.set(ListReadBudget::new(4)).expect("fresh context");
let held = first.acquire_list_read_budget(4).await.expect("first context");
let mut waiter = std::pin::pin!(same.acquire_list_read_budget(1));
assert!(futures::poll!(&mut waiter).is_pending());
assert!(
other
.acquire_list_read_budget(4)
.await
.expect("independent context")
.is_some()
);
drop(held);
assert!(waiter.await.expect("shared context resumed").is_some());
}
#[test]
fn list_read_budget_extreme_configuration_is_bounded_without_panicking() {
let budget = ListReadBudget::new(usize::MAX);
assert!(budget.capacity <= Semaphore::MAX_PERMITS);
assert!(u32::try_from(budget.capacity).is_ok());
}
#[test]
fn decommission_count_ledger_bounds_retained_entries() {
let mut ledger = super::DecommissionCountLedger::default();
@@ -593,7 +593,7 @@ impl ECStore {
#[cfg(test)]
crate::core::pools::observe_pool_activation_start_attempt(crate::core::pools::PoolActivationStartKind::Rebalance);
let fleet_proof = acquire_pool_activation_fleet_proof(&self.ctx).await?;
let mut pool_meta_guard = self.pool_meta_save_gate.lock().await;
let mut pool_meta_guard = self.pool_meta_save_gate.write().await;
pool_meta_guard.ensure_write_safe(stage)?;
let activation_fence = acquire_pool_rebalance_activation_locks(pool.clone(), fleet_proof).await?;
let pool_meta = self
@@ -625,7 +625,7 @@ impl ECStore {
{
// Lock order: pool_meta_save_gate -> rebalance.bin -> pool.bin. These
// shared fences admit a peer without waiting for the run's own entries.
let mut pool_meta_guard = self.pool_meta_save_gate.lock().await;
let mut pool_meta_guard = self.pool_meta_save_gate.write().await;
pool_meta_guard.ensure_write_safe("rebalance worker activation")?;
let mut worker_fence = acquire_pool_rebalance_worker_locks(pool.clone()).await?;
let pool_meta = self
@@ -654,7 +654,7 @@ impl ECStore {
{
// The admission read fences must already be dropped. Reacquire in the
// original order and revalidate every durable boundary across that gap.
let mut pool_meta_guard = self.pool_meta_save_gate.lock().await;
let mut pool_meta_guard = self.pool_meta_save_gate.write().await;
pool_meta_guard.ensure_write_safe("rebalance worker metadata update")?;
let mut activation_fence = acquire_pool_rebalance_activation_locks(pool.clone(), None).await?;
let pool_meta = self
@@ -1714,7 +1714,7 @@ mod tests {
persist_pool_meta_identity_for_startup(store.pools.clone(), &mut write_state, true)
.await
.expect("initialized pool metadata identity should persist");
*store.pool_meta_save_gate.lock().await = write_state;
*store.pool_meta_save_gate.write().await = write_state;
// Simulate physical loss; raw deletion of authoritative pool.bin is
// deliberately rejected by the native metadata mutation guard.
for pool in &store.pools {
+2 -2
View File
@@ -100,7 +100,7 @@ pub async fn test_store_with_persisted_rebalance_meta(
rebalance_meta: tokio::sync::RwLock::new(Some(meta)),
decommission_cancelers: tokio::sync::RwLock::new(vec![None]),
start_gate: tokio::sync::Mutex::new(()),
pool_meta_save_gate: tokio::sync::Mutex::default(),
pool_meta_save_gate: crate::core::pools::PoolMetaSaveGate::default(),
ctx,
bucket_fence_registry: std::sync::Arc::default(),
});
@@ -335,7 +335,7 @@ async fn test_pool_stores_with_contexts(
rebalance_meta: tokio::sync::RwLock::new(rebalance_meta.clone()),
decommission_cancelers: tokio::sync::RwLock::new(vec![None; pool_count]),
start_gate: tokio::sync::Mutex::new(()),
pool_meta_save_gate: tokio::sync::Mutex::new(pool_meta_write_state.independent_clone_for_test()),
pool_meta_save_gate: crate::core::pools::PoolMetaSaveGate::new(pool_meta_write_state.independent_clone_for_test()),
ctx: store_ctx,
bucket_fence_registry: std::sync::Arc::default(),
})
@@ -3412,7 +3412,7 @@ fn test_store_with_rebalance_meta(meta: RebalanceMeta) -> Arc<crate::store::ECSt
rebalance_meta: tokio::sync::RwLock::new(Some(meta)),
decommission_cancelers: tokio::sync::RwLock::new(Vec::new()),
start_gate: tokio::sync::Mutex::new(()),
pool_meta_save_gate: tokio::sync::Mutex::default(),
pool_meta_save_gate: crate::core::pools::PoolMetaSaveGate::default(),
ctx: crate::runtime::instance::bootstrap_ctx(),
bucket_fence_registry: std::sync::Arc::default(),
})
+263 -1
View File
@@ -93,6 +93,7 @@ use super::{
delete_tier_mutation_intent_record_if_current, list_tier_coordinator_mutation_intent_records,
list_tier_mutation_intent_records, load_tier_coordinator_mutation_intent_record_with_etag,
load_tier_mutation_intent_record_with_etag, save_tier_coordinator_mutation_intent_record_if_absent,
save_tier_mutation_intent_record_if_current,
},
warm_backend::WarmBackendImpl,
};
@@ -620,7 +621,11 @@ struct CommittedMutationRecoveryPlan {
impl TierMutationRecoverySnapshot {
fn requires_config_lock(&self) -> bool {
self.has_committed_mutation_blocks || self.intents.iter().any(|recovered| recovered.has_coordinator_record)
self.has_committed_mutation_blocks
|| self.intents.iter().any(|recovered| {
recovered.has_coordinator_record
|| (recovered.has_peer_record && recovered.intent.state == TierMutationIntentState::Prepared)
})
}
fn allowance(&self) -> MutationBlockAllowance {
@@ -5102,6 +5107,9 @@ impl TierConfigMgr {
Self::reconcile_terminal_dual_prefix_mutation_intents(api.clone(), &mut snapshot).await?;
Self::recover_prepared_coordinator_mutation_intents(api.clone(), &mut snapshot.coordinator_intents).await?;
if config_lock.is_some() {
Self::recover_expired_peer_only_prepares(api.clone(), &snapshot).await?;
}
snapshot = Self::load_and_reconcile_mutation_recovery_snapshot(handle, api.clone(), true).await?;
if snapshot.requires_config_lock() && config_lock.is_none() {
@@ -5502,6 +5510,58 @@ impl TierConfigMgr {
}
}
async fn recover_expired_peer_only_prepares<S>(api: Arc<S>, snapshot: &TierMutationRecoverySnapshot) -> io::Result<()>
where
S: EcstoreObjectIO,
{
let now = i64::try_from(OffsetDateTime::now_utc().unix_timestamp_nanos()).unwrap_or(i64::MAX);
let orphaned = snapshot
.intents
.iter()
.filter(|recovered| {
recovered.has_peer_record
&& !recovered.has_coordinator_record
&& recovered.intent.state == TierMutationIntentState::Prepared
&& Self::tier_mutation_tombstone_retention_elapsed(&recovered.intent, now)
})
.map(|recovered| recovered.intent.clone())
.collect::<Vec<_>>();
if orphaned.is_empty() {
return Ok(());
}
// The caller holds the tier-config namespace lock. Only the exact
// persisted candidate proves a commit; an old or different config
// leaves the peer fence intact.
let loaded = load_tier_config_for_recovery(api.clone()).await?;
let Some(config_etag) = loaded.etag else {
return Ok(());
};
for intent in orphaned {
if loaded.persisted_digest != Some(intent.candidate_digest) {
continue;
}
let _mutation_guard = acquire_tier_mutation_mutex(intent.mutation_id).await;
let (mut current, record_etag) =
match load_tier_mutation_intent_record_with_etag(api.clone(), intent.mutation_id).await {
Ok(record) => record,
Err(Error::ConfigNotFound) => continue,
Err(err) => return Err(tier_mutation_replay_error(err)),
};
if current != intent {
continue;
}
current
.advance(TierMutationIntentState::Committed, Some(config_etag.clone()))
.map_err(io::Error::other)?;
match save_tier_mutation_intent_record_if_current(api.clone(), &current, &record_etag).await {
Ok(()) | Err(Error::PreconditionFailed) => {}
Err(err) => return Err(tier_mutation_replay_error(err)),
}
}
Ok(())
}
async fn recover_prepared_coordinator_mutation_intents<S>(api: Arc<S>, intents: &mut [TierMutationIntent]) -> io::Result<()>
where
S: EcstoreObjectIO,
@@ -13379,6 +13439,208 @@ mod tests {
);
}
#[tokio::test]
async fn reload_recovers_expired_peer_only_prepare_for_persisted_add() {
use crate::services::tier::tier_mutation_intent::save_tier_mutation_intent_record;
let store = Arc::new(CasConfigStore::default());
let previous = empty_mgr();
let mut candidate = empty_mgr();
candidate.tiers.insert("COLD-A".to_string(), build_rustfs_tier("COLD-A"));
let targets = build_tier_mutation_affected_targets(
TierMutationIntentKind::Add,
HashSet::from(["COLD-A".to_string()]),
&previous,
&candidate,
)
.expect("added tier should have a target");
let mut intent = build_coordinator_tier_mutation_intent(TierMutationIntentKind::Add, None, &candidate, targets)
.expect("intent should build")
.expect("add should require an intent");
intent.expires_at_unix_nanos = 1;
candidate
.save_tiering_config_if_current(store.clone(), None)
.await
.expect("candidate should persist");
save_tier_mutation_intent_record(store.clone(), &intent)
.await
.expect("peer prepare should persist without a coordinator record");
let manager = TierConfigMgr::new();
TIER_MUTATION_TEST_PEERS
.scope(Vec::new(), async {
TIER_DRIVER_TEST_FACTORY
.scope(healthy_driver_factory(), async {
TierConfigMgr::reload_handle_with(&manager, store.clone())
.await
.expect("matching saved candidate should recover");
TierConfigMgr::acquire_operation_lease(&manager, "COLD-A")
.await
.expect("recovered tier should be readable");
})
.await;
})
.await;
assert!(
TierConfigMgr::load_tier_mutation_intents(store)
.await
.expect("peer scan should succeed")
.iter()
.all(|record| record.state != TierMutationIntentState::Prepared),
"the orphaned prepare must no longer block later reloads"
);
}
#[tokio::test]
async fn reload_keeps_peer_only_prepare_without_commit_proof() {
use crate::services::tier::tier_mutation_intent::{
save_tier_mutation_intent_record, tier_mutation_intent_record_object_name,
};
for case in [
"unexpired",
"recently_expired",
"different_candidate",
"missing_config",
"concurrent_change",
] {
let store = Arc::new(CasConfigStore::default());
let previous = empty_mgr();
let mut candidate = empty_mgr();
candidate.tiers.insert("COLD-A".to_string(), build_rustfs_tier("COLD-A"));
let targets = build_tier_mutation_affected_targets(
TierMutationIntentKind::Add,
HashSet::from(["COLD-A".to_string()]),
&previous,
&candidate,
)
.expect("added tier should have a target");
let mut intent = build_coordinator_tier_mutation_intent(TierMutationIntentKind::Add, None, &candidate, targets)
.expect("intent should build")
.expect("add should require an intent");
intent.expires_at_unix_nanos = match case {
"unexpired" => tier_mutation_intent_expiry_unix_nanos(),
"recently_expired" => {
i64::try_from(OffsetDateTime::now_utc().unix_timestamp_nanos()).expect("test clock should fit")
- 1_000_000_000
}
_ => 1,
};
if case == "different_candidate" {
intent.candidate_digest = [0x51; 32];
}
if case != "missing_config" {
candidate
.save_tiering_config_if_current(store.clone(), None)
.await
.expect("candidate should persist");
}
save_tier_mutation_intent_record(store.clone(), &intent)
.await
.expect("peer prepare should persist");
let expected = if case == "concurrent_change" {
let mut changed = intent.clone();
changed.candidate_digest = [0x52; 32];
let object = tier_mutation_intent_record_object_name(intent.mutation_id).expect("record path should build");
store
.rewrite_on_next_if_match(object, changed.encode().expect("changed intent should encode"))
.await;
changed
} else {
intent
};
let manager = TierConfigMgr::new();
TIER_MUTATION_TEST_PEERS
.scope(Vec::new(), async {
TierConfigMgr::reload_handle_with(&manager, store.clone())
.await
.expect("unproven prepare should remain fenced");
})
.await;
let error = match TierConfigMgr::acquire_operation_lease(&manager, "COLD-A").await {
Ok(_) => panic!("{case}: unproven prepare must block reads"),
Err(error) => error,
};
assert!(error.message.contains("being replaced"), "{case}: {error}");
let records = TierConfigMgr::load_tier_mutation_intents(store)
.await
.expect("peer scan should succeed");
assert_eq!(records, vec![expected], "{case}: prepared evidence must remain unchanged");
}
}
#[tokio::test]
async fn peer_only_prepare_recovery_retains_fence_until_peer_commit_succeeds() {
use crate::services::tier::tier_mutation_intent::save_tier_mutation_intent_record;
let store = Arc::new(CasConfigStore::default());
let previous = empty_mgr();
let mut candidate = empty_mgr();
candidate.tiers.insert("COLD-A".to_string(), build_rustfs_tier("COLD-A"));
let targets = build_tier_mutation_affected_targets(
TierMutationIntentKind::Add,
HashSet::from(["COLD-A".to_string()]),
&previous,
&candidate,
)
.expect("added tier should have a target");
let mut intent = build_coordinator_tier_mutation_intent(TierMutationIntentKind::Add, None, &candidate, targets)
.expect("intent should build")
.expect("add should require an intent");
intent.expires_at_unix_nanos = 1;
candidate
.save_tiering_config_if_current(store.clone(), None)
.await
.expect("candidate should persist");
save_tier_mutation_intent_record(store.clone(), &intent)
.await
.expect("peer prepare should persist");
let manager = TierConfigMgr::new();
let calls = Arc::new(Mutex::new(Vec::new()));
TIER_MUTATION_TEST_PEERS
.scope(vec![FakeTierMutationPeer::boxed("peer-a", calls.clone(), Err("offline"))], async {
TierConfigMgr::reload_handle_with(&manager, store.clone())
.await
.expect_err("failed peer commit must leave recovery incomplete");
})
.await;
let retained = TierConfigMgr::load_tier_mutation_intents(store.clone())
.await
.expect("peer scan should succeed");
assert_eq!(retained.len(), 1);
assert_eq!(retained[0].state, TierMutationIntentState::Committed);
let error = match TierConfigMgr::acquire_operation_lease(&manager, "COLD-A").await {
Ok(_) => panic!("failed peer commit must retain the fence"),
Err(error) => error,
};
assert!(error.message.contains("being replaced"), "{error}");
TIER_MUTATION_TEST_PEERS
.scope(
vec![FakeTierMutationPeer::boxed(
"peer-a",
calls.clone(),
Ok(PeerTierMutationState::Committed),
)],
async {
TIER_DRIVER_TEST_FACTORY
.scope(healthy_driver_factory(), async {
TierConfigMgr::reload_handle_with(&manager, store.clone())
.await
.expect("peer commit retry should recover");
TierConfigMgr::acquire_operation_lease(&manager, "COLD-A")
.await
.expect("tier should be readable after the retry");
})
.await;
},
)
.await;
assert_eq!(lock_unpoisoned(&calls).len(), 2, "recovery should retry the peer commit");
}
#[tokio::test]
async fn coordinator_mutation_recovery_scan_blocks_new_operation_lease() {
let store = Arc::new(CasConfigStore::default());
+27 -2
View File
@@ -4608,6 +4608,19 @@ fn marker_purge_receipt_path(
}
impl SetDisks {
/// The first disk slot UUID is assigned when this erasure set is formatted.
/// It survives pool reordering and replacement of a disk in that slot.
pub fn persistent_set_id(&self) -> Result<Uuid> {
self.format
.erasure
.sets
.get(self.set_index)
.and_then(|disks| disks.first())
.copied()
.filter(|id| !id.is_nil())
.ok_or_else(|| Error::other("erasure set has no persistent format identity"))
}
/// Persist a quorum DELETE receipt outside the object marker itself. The
/// receipt is written to the metadata bucket so it survives a node restart
/// while an inaccessible member still holds the stale marker.
@@ -11027,7 +11040,12 @@ mod tests {
let live = Uuid::new_v4();
let orphan = Uuid::new_v4();
let object_dir = dir.path().join("bucket/obj");
write_object_meta_with_data_dirs(&object_dir, "bucket", "obj", &[live]).await;
// Part files are inert to discovery and cost 1/64 of an entry, so the
// first step is exhausted by what still costs a full entry each: the
// data directories of many versions, all referenced by the metadata.
let versions = (0..4200).map(|_| Uuid::new_v4()).collect::<Vec<_>>();
let referenced = std::iter::once(live).chain(versions.iter().copied()).collect::<Vec<_>>();
write_object_meta_with_data_dirs(&object_dir, "bucket", "obj", &referenced).await;
let live_dir = object_dir.join(live.to_string());
let orphan_dir = object_dir.join(orphan.to_string());
fs::create_dir_all(&live_dir).await.expect("create live multipart directory");
@@ -11036,10 +11054,14 @@ mod tests {
.await
.expect("write orphan payload");
let parts_dir = live_dir.clone();
let versions_root = object_dir.clone();
tokio::task::spawn_blocking(move || {
for part in 1..=4200 {
std::fs::write(parts_dir.join(format!("part.{part}")), b"live").expect("write live multipart entry");
}
for version in versions {
std::fs::create_dir(versions_root.join(version.to_string())).expect("create referenced version directory");
}
})
.await
.expect("create multipart fixture");
@@ -11049,7 +11071,10 @@ mod tests {
.await
.expect("read first discovery page");
assert!(first.transactions.is_empty());
assert!(first.next.is_some(), "ordinary multipart entries must exhaust the first scan budget");
assert!(
first.next.is_some(),
"the data directories of many versions must exhaust the first scan step"
);
let last = disk
.discover_delete_transactions("bucket", Some("obj"), first.next, 1)
.await
@@ -1709,6 +1709,37 @@ mod tests {
);
}
#[tokio::test]
async fn delete_recovery_missing_key_naming_a_folder_of_uuid_named_keys_completes() {
let bucket = "recover-uuid-named-folder";
let (dirs, store, _) = fixture(bucket).await;
// More UUID-named child keys per drive than the former 4096-entry probe
// could cover once each child's `xl.meta` and data directory were charged
// (three entries per child).
const CHILDREN: usize = 1500;
for dir in &dirs {
let prefix = dir.path().join(bucket).join("uploads");
for _ in 0..CHILDREN {
let child = prefix.join(Uuid::new_v4().to_string());
std::fs::create_dir_all(child.join(Uuid::new_v4().to_string())).expect("child key data dir");
std::fs::write(child.join(crate::disk::STORAGE_FORMAT_FILE), b"meta").expect("child key metadata");
}
}
let error = store
.delete_object(bucket, "uploads", ObjectOptions::default())
.await
.expect_err("the folder's own key does not exist");
assert!(
crate::error::is_err_object_not_found(&error),
"a complete inventory keeps the missing-key no-op semantics: {error:?}"
);
assert!(dirs.iter().all(|dir| {
std::fs::read_dir(dir.path().join(bucket).join("uploads"))
.map(|entries| entries.count() == CHILDREN)
.unwrap_or(false)
}));
}
#[tokio::test]
async fn delete_recovery_batch_budget_survives_pool_and_set_fanout() {
use crate::layout::endpoint::Endpoint;
+39 -2
View File
@@ -2979,10 +2979,11 @@ fn build_legacy_per_part_fallback_reader(
part_length: usize,
part_size: usize,
read_repair: Option<(DecodeReadRepairContext, bool)>,
) -> Box<dyn AsyncRead + Unpin + Send + Sync> {
) -> Box<LegacyPerPartDecodeReader> {
let buffer = adaptive_duplex_buffer_size(part_size as i64);
let (read_half, mut write_half) = tokio::io::duplex(buffer);
let (read_half, write_half) = tokio::io::duplex(buffer);
let decode = tokio::spawn(async move {
let mut write_half = GetObjectDownstreamWriter::new(write_half);
let outcome = erasure
.decode_with_stripe_handles_and_reopeners_with_diagnostics(
&mut write_half,
@@ -8908,6 +8909,42 @@ mod tests {
assert!(err.to_string().contains("inconsistent read source shards"));
}
#[tokio::test]
async fn legacy_per_part_output_close_retains_downstream_classification() {
let erasure = coding::Erasure::new(2, 2, 64);
let data = (0..64u8).collect::<Vec<_>>();
let setup =
setup_codec_data_blocks_first_encoded_bitrot_readers(&erasure, &data, &[0], &[], &[], HashAlgorithm::HighwayHash256)
.await;
let mut reader = build_legacy_per_part_fallback_reader(
erasure,
setup.readers,
setup.deferred_stripe_handles,
Vec::new(),
None,
0,
data.len(),
data.len(),
None,
);
// Closing the actual output pipe models the body consumer disappearing
// while retaining the executor long enough to inspect its typed result.
assert!(!reader.finished);
let (replacement, _unused) = tokio::io::duplex(1);
drop(std::mem::replace(&mut reader.inner, replacement));
let handle = reader.decode.take().expect("fallback decode task");
let error = handle
.await
.expect("join the fallback task")
.expect("closed pipe must interrupt output");
assert_eq!(error.kind(), std::io::ErrorKind::BrokenPipe);
assert_eq!(
crate::diagnostics::get::classify_io_error(&error),
GetObjectFailureReason::DownstreamClosed,
"fallback output must carry the same marker as the normal GET path"
);
}
// backlog#879: a later multipart part that loses a shard (read quorum still
// holds) must be reconstructed in place through the legacy per-part decode
// bridge instead of failing the stream. This exercises the exact reader the
+21 -17
View File
@@ -443,6 +443,7 @@ impl ECStore {
},
other => StorageError::Lock(other),
})?;
let ns_guard = Arc::new(ns_guard);
let recover = Box::pin(async {
self.ensure_bucket_volume_absent_on_all_sets(bucket).await?;
@@ -930,15 +931,16 @@ impl ECStore {
Ok(())
}
async fn cleanup_bucket_usage(&self, bucket: &str, guard: Option<&rustfs_lock::NamespaceLockGuard>) -> Result<()> {
run_bucket_usage_cleanup(guard, bucket, async {
crate::data_usage::prepare_bucket_usage_for_namespace_change(bucket, guard).await?;
async fn cleanup_bucket_usage(&self, bucket: &str, guard: Option<&Arc<rustfs_lock::NamespaceLockGuard>>) -> Result<()> {
let namespace_guard = guard.map(Arc::as_ref);
run_bucket_usage_cleanup(namespace_guard, bucket, async {
crate::data_usage::prepare_bucket_usage_for_namespace_change(bucket, namespace_guard).await?;
crate::data_usage::remove_bucket_usage_from_backend_with_guard_fenced(self, bucket, guard).await
})
.await
}
async fn cleanup_bucket_usage_best_effort(&self, bucket: &str, guard: Option<&rustfs_lock::NamespaceLockGuard>) {
async fn cleanup_bucket_usage_best_effort(&self, bucket: &str, guard: Option<&Arc<rustfs_lock::NamespaceLockGuard>>) {
if let Err(err) = self.cleanup_bucket_usage(bucket, guard).await {
warn!(
bucket = %bucket,
@@ -1029,6 +1031,7 @@ impl ECStore {
} else {
None
};
let ns_guard = ns_guard.map(Arc::new);
let existing_bucket_info = match self.get_bucket_info_from_sets(bucket, &BucketOptions::default()).await {
Ok(info) => Some(info),
@@ -1146,7 +1149,7 @@ impl ECStore {
metadata_sys::set_new_bucket_metadata_intent_in(&self.ctx, meta.clone()).await?;
if bucket_lifecycle_guard.as_ref().is_some_and(|guard| guard.is_lock_lost())
|| metadata_transaction_guard.as_ref().is_some_and(|guard| guard.is_lock_lost())
|| ns_guard.as_ref().is_some_and(|guard| guard.is_lock_lost())
|| ns_guard.as_deref().is_some_and(|guard| guard.is_lock_lost())
{
return Err(Error::other(format!("bucket metadata creation intent lock was lost: {bucket}")));
}
@@ -1162,7 +1165,7 @@ impl ECStore {
if let Err(err) = await_bucket_lifecycle_operation(
bucket_lifecycle_guard.as_ref(),
ns_guard.as_ref(),
ns_guard.as_deref(),
bucket,
"physical bucket creation",
await_bucket_namespace_operation(
@@ -1191,9 +1194,9 @@ impl ECStore {
{
warn!("best-effort bucket heal after BucketExists failed: {heal_err}");
}
if confirmed_missing && !is_err_bucket_exists(&err) && ns_guard.as_ref().is_none_or(|guard| !guard.is_lock_lost()) {
if confirmed_missing && !is_err_bucket_exists(&err) && ns_guard.as_deref().is_none_or(|guard| !guard.is_lock_lost()) {
error!("make bucket failed: {err}");
self.rollback_failed_bucket_creation(bucket, bucket_lifecycle_guard.as_ref(), ns_guard.as_ref())
self.rollback_failed_bucket_creation(bucket, bucket_lifecycle_guard.as_ref(), ns_guard.as_deref())
.await;
}
return Err(err);
@@ -1214,10 +1217,10 @@ impl ECStore {
.await;
let metadata_lock_lost = bucket_lifecycle_guard.as_ref().is_some_and(|guard| guard.is_lock_lost())
|| metadata_transaction_guard.as_ref().is_some_and(|guard| guard.is_lock_lost())
|| ns_guard.as_ref().is_some_and(|guard| guard.is_lock_lost());
|| ns_guard.as_deref().is_some_and(|guard| guard.is_lock_lost());
if let Err(err) = metadata_result {
if !metadata_lock_lost {
self.rollback_failed_bucket_creation(bucket, bucket_lifecycle_guard.as_ref(), ns_guard.as_ref())
self.rollback_failed_bucket_creation(bucket, bucket_lifecycle_guard.as_ref(), ns_guard.as_deref())
.await;
}
return Err(err);
@@ -1580,6 +1583,7 @@ impl ECStore {
} else {
None
};
let ns_guard = ns_guard.map(Arc::new);
let sr_mark_delete = opts.srdelete_op == SRBucketDeleteOp::MarkDelete;
let sr_purge = opts.srdelete_op == SRBucketDeleteOp::Purge;
@@ -1615,7 +1619,7 @@ impl ECStore {
crate::bucket::metadata_sys::get_object_lock_config_and_incarnation_from_disk_with_guard_in(
&self.ctx,
bucket,
Some(ns_guard.as_ref().unwrap_or(&guard)),
Some(ns_guard.as_deref().unwrap_or(&guard)),
)
.await?;
match state {
@@ -1641,7 +1645,7 @@ impl ECStore {
if sr_mark_delete {
await_bucket_lifecycle_operation(
bucket_lifecycle_guard.as_ref(),
ns_guard.as_ref(),
ns_guard.as_deref(),
bucket,
"bucket delete marker creation",
await_bucket_namespace_operation(
@@ -1680,7 +1684,7 @@ impl ECStore {
object_lock_metadata_guard.as_ref().map(|(guard, _)| guard),
bucket,
"Object Lock configuration fence during bucket deletion",
run_physical_bucket_deletion(ns_guard.as_ref(), bucket, async {
run_physical_bucket_deletion(ns_guard.as_deref(), bucket, async {
self.delete_bucket_on_sets(bucket, &delete_opts)
.await
.map_err(|err| to_object_err(err, vec![bucket]))
@@ -1709,12 +1713,12 @@ impl ECStore {
if let Some(guard) = bucket_lifecycle_guard.as_ref() {
record_opts.add_bucket_lifecycle_lock_guard(guard);
}
if let Some(guard) = ns_guard.as_ref() {
if let Some(guard) = ns_guard.as_deref() {
record_opts.add_namespace_lock_guard(guard);
}
await_bucket_lifecycle_operation(
bucket_lifecycle_guard.as_ref(),
ns_guard.as_ref(),
ns_guard.as_deref(),
bucket,
"bucket retirement publication",
crate::bucket::retirement::commit_retirement(store.clone(), bucket, incarnation, &record_opts),
@@ -1724,7 +1728,7 @@ impl ECStore {
// V1 ledger must not block quota admission after same-name recreation.
await_bucket_lifecycle_operation(
bucket_lifecycle_guard.as_ref(),
ns_guard.as_ref(),
ns_guard.as_deref(),
bucket,
"retired bucket quota ledger cleanup",
crate::bucket::quota::reservation::cleanup_retired_bucket_ledger(store, bucket, incarnation),
@@ -1735,7 +1739,7 @@ impl ECStore {
self.cleanup_bucket_usage_best_effort(bucket, ns_guard.as_ref()).await;
if let Err(err) = self
.cleanup_deleted_bucket_metadata(bucket, sr_purge, ns_guard.as_ref())
.cleanup_deleted_bucket_metadata(bucket, sr_purge, ns_guard.as_deref())
.await
{
warn!(
+27 -19
View File
@@ -527,7 +527,7 @@ impl ECStore {
async fn acquire_heal_format_fence(
&self,
) -> Result<(
tokio::sync::MutexGuard<'_, PoolMetaWriteState>,
crate::core::pools::PoolMetaSaveGateWriteGuard<'_>,
NamespaceLockGuard,
NamespaceLockGuard,
PoolMeta,
@@ -538,7 +538,7 @@ impl ECStore {
.first()
.cloned()
.ok_or_else(|| Error::other("heal format requires at least one storage pool"))?;
let mut write_state = self.pool_meta_save_gate.lock().await;
let mut write_state = self.pool_meta_save_gate.write().await;
write_state.ensure_write_safe("heal format fence failed")?;
// Metadata fence order is part of the decommission/rebalance protocol:
@@ -1079,7 +1079,7 @@ impl ECStore {
pub(super) async fn handle_heal_bucket(&self, bucket: &str, opts: &HealOpts) -> Result<HealResultItem> {
let movement_gate = self.ctx.data_movement_operation_gate();
let _movement_guard = movement_gate.read().await;
let save_guard = self.pool_meta_save_gate.lock().await;
let save_guard = self.pool_meta_save_gate.write().await;
save_guard.ensure_write_safe("bucket heal cannot run while pool metadata requires recovery")?;
let mut fenced_pools = BTreeSet::new();
{
@@ -2527,7 +2527,7 @@ mod tests {
rebalance_meta: RwLock::new(None),
decommission_cancelers: RwLock::new(Vec::new()),
start_gate: Mutex::new(()),
pool_meta_save_gate: Mutex::default(),
pool_meta_save_gate: crate::core::pools::PoolMetaSaveGate::default(),
ctx: crate::runtime::instance::bootstrap_ctx(),
bucket_fence_registry: std::sync::Arc::default(),
}
@@ -3550,10 +3550,14 @@ mod tests {
async fn full_format_heal_preblocked_pool_metadata_never_writes_format() {
let (temp_dir, store, shutdown) = multi_pool_heal_store().await;
remove_heal_test_format(&temp_dir, &store, 0, 3).await;
store.pool_meta_save_gate.lock().await.observe_replicas(PoolMetaReplicaState {
needs_repair: true,
repair_write_safe: false,
});
store
.pool_meta_save_gate
.write()
.await
.observe_replicas(PoolMetaReplicaState {
needs_repair: true,
repair_write_safe: false,
});
let err = store
.handle_heal_format(false)
@@ -3688,7 +3692,7 @@ mod tests {
let target = remove_heal_test_format(&temp_dir, &store, 0, 3).await;
store.pools[0].disk_set[0].disks.write().await[3] = None;
let barrier = HealFormatAfterSaveBarrier::install(&store.pools[0], 3);
let recovery_latch = store.pool_meta_save_gate.lock().await.aborted_transaction_latch_for_test();
let recovery_latch = store.pool_meta_save_gate.write().await.aborted_transaction_latch_for_test();
let mut heal = tokio::spawn({
let store = Arc::clone(&store);
async move { store.handle_heal_format(false).await }
@@ -3848,7 +3852,7 @@ mod tests {
}
}
let endpoints: EndpointServerPools = pools.iter().map(|pool| pool.endpoints.clone()).collect::<Vec<_>>().into();
let state = source.pool_meta_save_gate.lock().await.independent_clone_for_test();
let state = source.pool_meta_save_gate.write().await.independent_clone_for_test();
Arc::new(ECStore {
id: Uuid::new_v4(),
disk_map: Default::default(),
@@ -3858,7 +3862,7 @@ mod tests {
rebalance_meta: tokio::sync::RwLock::new(None),
decommission_cancelers: tokio::sync::RwLock::new(vec![None; 2]),
start_gate: tokio::sync::Mutex::new(()),
pool_meta_save_gate: tokio::sync::Mutex::new(state),
pool_meta_save_gate: crate::core::pools::PoolMetaSaveGate::new(state),
ctx: Arc::clone(&source.ctx),
bucket_fence_registry: Arc::default(),
})
@@ -4040,7 +4044,7 @@ mod tests {
let missing = remove_pool_meta_shard(&store, 0).await;
let (guard, admissions, mut state) = store.acquire_pool_meta_object_heal_fence(&[0]).await.expect("heal authority");
assert!(admissions.into_iter().all(|admission| admission.is_ok()));
assert!(store.pool_meta_save_gate.try_lock().is_err(), "heal must retain the earlier save gate");
assert!(store.pool_meta_save_gate.try_write().is_err(), "heal must retain the earlier save gate");
let writer_store = Arc::clone(&store);
let writer = tokio::spawn(async move { writer_store.save_current_pool_meta_for_test(&[0]).await });
tokio::task::yield_now().await;
@@ -4687,10 +4691,14 @@ mod tests {
#[tokio::test]
async fn bucket_heal_blocks_before_dispatch_after_unreadable_pool_meta_replica() {
let store = minimal_heal_store().await;
store.pool_meta_save_gate.lock().await.observe_replicas(PoolMetaReplicaState {
needs_repair: true,
repair_write_safe: false,
});
store
.pool_meta_save_gate
.write()
.await
.observe_replicas(PoolMetaReplicaState {
needs_repair: true,
repair_write_safe: false,
});
let err = store
.handle_heal_bucket("bucket", &HealOpts::default())
@@ -4724,7 +4732,7 @@ mod tests {
.await
.expect("peer dispatch should start");
assert!(
store.pool_meta_save_gate.try_lock().is_ok(),
store.pool_meta_save_gate.try_write().is_ok(),
"coordinator must release its local save gate before waiting for peers"
);
assert!(
@@ -5116,7 +5124,7 @@ mod tests {
rebalance_meta: RwLock::new(None),
decommission_cancelers: RwLock::new(Vec::new()),
start_gate: Mutex::new(()),
pool_meta_save_gate: Mutex::default(),
pool_meta_save_gate: crate::core::pools::PoolMetaSaveGate::default(),
ctx: crate::runtime::instance::bootstrap_ctx(),
bucket_fence_registry: std::sync::Arc::default(),
};
@@ -5136,7 +5144,7 @@ mod tests {
.save_for_startup(store.pools.clone())
.await
.expect("pool metadata should be persisted before format heal");
*store.pool_meta_save_gate.lock().await = PoolMetaWriteState::default();
*store.pool_meta_save_gate.write().await = crate::core::pools::PoolMetaWriteState::default();
let replacement_endpoint = store.pools[1].endpoints.endpoints.as_ref()[2].clone();
tokio::fs::remove_file(
+2 -2
View File
@@ -676,7 +676,7 @@ impl ECStore {
rebalance_meta: RwLock::new(None),
decommission_cancelers,
start_gate: Mutex::new(()),
pool_meta_save_gate: Mutex::new(pool_meta_write_state),
pool_meta_save_gate: crate::core::pools::PoolMetaSaveGate::new(pool_meta_write_state),
// Adopt the caller's context (the process bootstrap one on the
// legacy path) so startup writes (erasure type recorded before
// this point) and later reads share one cell.
@@ -725,7 +725,7 @@ impl ECStore {
// Lock order is save gate -> distributed metadata writer, matching
// runtime mutations. RPC locks are available before ECStore publication
// and do not depend on the later fleet proof.
let mut startup_write_state = self.pool_meta_save_gate.lock().await;
let mut startup_write_state = self.pool_meta_save_gate.write().await;
let startup_pool_meta_guard = if should_persist_pool_meta {
let pool = self
.pools
+309 -18
View File
@@ -127,28 +127,30 @@ impl AuthoritativeListOperations for ECStore {
marker: Option<String>,
max_keys: i32,
) -> Result<ListObjectVersionsInfo> {
if max_keys <= 0 || max_keys > MAX_OBJECT_LIST {
self.list_unversioned_authoritative_page(bucket, prefix, marker, max_keys, None)
.await
}
async fn list_identifier_tree_leaves_authoritative(
self: Arc<Self>,
bucket: &str,
prefix: &str,
leaf: &str,
marker: Option<String>,
max_keys: i32,
) -> Result<ListObjectVersionsInfo> {
let valid_prefix = prefix
.strip_suffix('/')
.is_some_and(|root| !root.is_empty() && root.split('/').all(|part| !part.is_empty() && !matches!(part, "." | "..")));
if !valid_prefix || !crate::disk::is_identifier_tree_leaf(leaf) {
return Err(StorageError::InvalidArgument(
bucket.to_string(),
prefix.to_string(),
"authoritative page size is out of range".to_string(),
"identifier-tree listing requires a directory prefix and a dotted leaf name".to_string(),
));
}
let opts = ListPathOptions {
bucket: bucket.to_string(),
prefix: prefix.to_string(),
marker: marker.clone(),
limit: max_keys,
recursive: true,
incl_deleted: true,
versioned: true,
authoritative: true,
caller: ListObjectsGatherCaller::S3ListVersions,
..Default::default()
};
let mut page = self.list_path(&opts).await?;
page.entries.forward_past(marker);
unversioned_authoritative_page(bucket, page)
self.list_unversioned_authoritative_page(bucket, prefix, marker, max_keys, Some(leaf.to_string()))
.await
}
async fn walk_authoritative(
@@ -368,6 +370,9 @@ fn log_list_path_finished(
#[derive(Debug, Default, Clone)]
pub struct ListPathOptions {
pub caller: ListObjectsGatherCaller,
/// Internal native-walk admission, shared with every disk producer so
/// aborting an HTTP request cannot release its budget ahead of cleanup.
pub read_budget: Option<Arc<tokio::sync::OwnedSemaphorePermit>>,
pub id: Option<String>,
// Bucket of the listing.
@@ -433,6 +438,8 @@ pub struct ListPathOptions {
// Internal bounded pages require GET-equivalent per-set resolution.
pub authoritative: bool,
pub identifier_tree_leaf: Option<String>,
pub stop_disk_at_limit: bool,
pub pool_idx: Option<usize>,
@@ -1098,6 +1105,39 @@ impl ListPathOptions {
}
impl ECStore {
async fn list_unversioned_authoritative_page(
self: Arc<Self>,
bucket: &str,
prefix: &str,
marker: Option<String>,
max_keys: i32,
identifier_tree_leaf: Option<String>,
) -> Result<ListObjectVersionsInfo> {
if max_keys <= 0 || max_keys > MAX_OBJECT_LIST {
return Err(StorageError::InvalidArgument(
bucket.to_string(),
prefix.to_string(),
"authoritative page size is out of range".to_string(),
));
}
let opts = ListPathOptions {
bucket: bucket.to_string(),
prefix: prefix.to_string(),
marker: marker.clone(),
limit: max_keys,
recursive: true,
incl_deleted: true,
versioned: true,
authoritative: true,
identifier_tree_leaf,
caller: ListObjectsGatherCaller::S3ListVersions,
..Default::default()
};
let mut page = self.list_path(&opts).await?;
page.entries.forward_past(marker);
unversioned_authoritative_page(bucket, page)
}
#[allow(clippy::too_many_arguments)]
// @continuation_token marker
// @start_after as marker when continuation_token empty
@@ -1478,6 +1518,11 @@ impl ECStore {
if o.transient {
o.create = false;
}
let fanout = self.pools.iter().fold(0usize, |total, pool| {
total.saturating_add(pool.disk_set.len().saturating_mul(pool.set_drive_count))
});
o.read_budget = self.ctx.acquire_list_read_budget(fanout).await.map_err(Error::other)?;
let log_context = ListPathLogContext::from_options(&o);
// cancel channel
@@ -2878,6 +2923,12 @@ impl SetDisks {
if o.transient {
o.create = false;
}
o.read_budget = self
.instance_ctx()
.acquire_list_read_budget(self.set_drive_count)
.await
.map_err(Error::other)?;
let log_context = ListPathLogContext::from_options(&o);
let cancel = CancellationToken::new();
@@ -2922,6 +2973,33 @@ impl SetDisks {
Ok(page)
}
fn resolve_authoritative_listed_entry(
bucket: &str,
selection: Option<(&str, &str)>,
entries: MetaCacheEntries,
errors: &[Option<DiskError>],
disk_count: usize,
parity: usize,
) -> crate::disk::error::Result<Option<MetaCacheEntry>> {
if disk_count == 0 || entries.0.len() > disk_count || entries.0.len() != errors.len() {
return Err(DiskError::ErasureReadQuorum);
}
// An old peer may ignore the optional disk selector. Its noncandidate
// rows must be ignored before metadata parsing. Selected keys retain
// the original resolver and all of its quorum/corruption checks.
if selection.is_some_and(|(prefix, leaf)| {
entries
.0
.iter()
.flatten()
.next()
.is_some_and(|entry| !crate::disk::identifier_tree_leaf_matches(prefix, leaf, &entry.name))
}) {
return Ok(None);
}
Self::resolve_listed_versions(bucket, entries, errors, disk_count, parity)
}
/// Version scan for set-scoped callers that act on absence: set walks
/// (tier reference proofs, tier-delete journals, pool maintenance) and the
/// recursive prefix-delete scan. Every online drive is read and each
@@ -2945,10 +3023,13 @@ impl SetDisks {
let first_error = Arc::new(tokio::sync::Mutex::new(None));
let error_sink = Arc::clone(&first_error);
let bucket = opts.bucket.clone();
let prefix = opts.prefix.clone();
let identifier_tree_leaf = opts.identifier_tree_leaf.clone();
let cancel = rx.clone();
let result = list_path_raw_with_claim_tracker(
rx,
ListPathRawOptions {
read_budget: opts.read_budget.clone(),
disks: disks.into_iter().map(Some).collect(),
bucket: opts.bucket,
path: opts.base_dir,
@@ -2961,6 +3042,7 @@ impl SetDisks {
forward_to: opts.marker,
min_disks: read_quorum,
preserve_replica_metadata: true,
identifier_tree_leaf: opts.identifier_tree_leaf,
// A reader's local page may be consumed by stale entries. Only
// the resolved, merged page may stop the authoritative walk.
per_disk_limit: 0,
@@ -2968,7 +3050,14 @@ impl SetDisks {
walkdir_timeout: opts.walkdir_timeout,
walkdir_stall_timeout: opts.walkdir_stall_timeout,
partial: Some(Box::new(move |entries, errors| {
let resolved = Self::resolve_listed_versions(&bucket, entries, errors, disk_count, parity);
let resolved = Self::resolve_authoritative_listed_entry(
&bucket,
identifier_tree_leaf.as_deref().map(|leaf| (prefix.as_str(), leaf)),
entries,
errors,
disk_count,
parity,
);
let sender = sender.clone();
let cancel = cancel.clone();
let first_error = Arc::clone(&error_sink);
@@ -3107,6 +3196,7 @@ impl SetDisks {
let result = list_path_raw_with_outcome(
rx,
ListPathRawOptions {
read_budget: opts.read_budget.clone(),
disks: disks.into_iter().map(Some).collect(),
fallback_disks: fallback_disks.into_iter().map(Some).collect(),
bucket: opts.bucket,
@@ -6788,6 +6878,58 @@ mod test {
(dirs, store)
}
#[tokio::test]
async fn list_read_budget_bounds_store_and_set_entrypoints_and_recovers_after_errors() {
let (_dirs, store) = single_set_store(4).await;
let held = store
.ctx
.acquire_list_read_budget(usize::MAX)
.await
.expect("occupy native budget");
let semaphore = held.as_ref().expect("bounded instance budget").semaphore();
let owners = Arc::strong_count(semaphore);
let opts = ListPathOptions {
bucket: "list-budget-absent-bucket".into(),
limit: 1,
authoritative: true,
..Default::default()
};
let set = store.pools[0].disk_set[0].clone();
set.disks.write().await.fill(None);
{
let mut store_walk = std::pin::pin!(store.clone().list_path(&opts));
let mut set_walk = std::pin::pin!(set.clone().list_path_result(&opts));
assert!(futures::poll!(&mut store_walk).is_pending(), "store walk must await its budget");
assert_eq!(Arc::strong_count(semaphore), owners + 1, "store must queue on the shared budget");
assert!(futures::poll!(&mut set_walk).is_pending(), "set walk shares the same instance budget");
assert_eq!(Arc::strong_count(semaphore), owners + 2, "set must queue on the same budget");
let empty = store
.clone()
.list_path(&ListPathOptions {
limit: 0,
..opts.clone()
})
.await
.expect("empty page needs no disk admission");
assert!(empty.exhausted);
}
assert_eq!(Arc::strong_count(semaphore), owners, "cancelled entry points release the queue ownership");
drop(held);
let result = timeout(Duration::from_secs(5), store.clone().list_path(&opts))
.await
.expect("failed walk terminates");
assert_eq!(
result.err(),
Some(DiskError::ErasureReadQuorum.into()),
"offline drives retain their quorum error"
);
let next = timeout(Duration::from_secs(5), store.ctx.acquire_list_read_budget(usize::MAX))
.await
.expect("failed and cancelled walks release their grants")
.expect("budget remains open");
assert!(next.is_some());
}
fn versioned_object_metadata(bucket: &str, name: &str, versions: &[(u128, i64)], slot: usize) -> Bytes {
let mut meta = FileMeta::new();
for (id, modified) in versions {
@@ -6929,6 +7071,155 @@ mod test {
);
}
#[test]
fn authoritative_identifier_tree_filters_old_peer_rows_before_metadata_resolution() {
let prefix = "table-catalog/namespaces/";
let leaf = "table-entry.json";
let corrupt = |name: &str| {
MetaCacheEntries(vec![
Some(MetaCacheEntry {
name: format!("{prefix}{name}"),
metadata: b"corrupt metadata".to_vec(),
..Default::default()
});
4
])
};
let report = "sales/tables/orders/maintenance/hash/jobs/job.json";
let errors = vec![None; 4];
assert!(
SetDisks::resolve_authoritative_listed_entry(RUSTFS_META_BUCKET, None, corrupt(report), &errors, 4, 2,).is_err(),
"the control must actually parse and reject the corrupt metadata"
);
assert!(
SetDisks::resolve_authoritative_listed_entry(
RUSTFS_META_BUCKET,
Some((prefix, leaf)),
corrupt(report),
&errors,
4,
2,
)
.expect("an old peer's unrelated report is outside the selected scan")
.is_none()
);
assert!(
SetDisks::resolve_authoritative_listed_entry(
RUSTFS_META_BUCKET,
Some((prefix, leaf)),
corrupt("sales/tables/orders/maintenance/tables/owner/table-entry.json"),
&errors,
4,
2,
)
.is_err(),
"a matching nested owner must still fail closed on corruption"
);
}
#[tokio::test]
async fn authoritative_identifier_tree_pages_preserve_nested_owners_and_read_quorum() {
let (dirs, store) = single_set_store(4).await;
let prefix = "table-catalog/identifier-tree/";
let leaf = "table-entry.json";
for (root, selected) in [
("table-catalog/identifier-tree", leaf),
(prefix, "../entry.json"),
(prefix, "table-entry"),
] {
assert!(matches!(
store
.clone()
.list_identifier_tree_leaves_authoritative(RUSTFS_META_BUCKET, root, selected, None, 2)
.await,
Err(StorageError::InvalidArgument(..))
));
}
let names = [
format!("{prefix}a/tables/first/{leaf}"),
format!("{prefix}sales/tables/orders/maintenance/tables/owner/{leaf}"),
format!("{prefix}z/tables/last/{leaf}"),
];
for name in &names {
write_version_replicas(&dirs, RUSTFS_META_BUCKET, name, |slot| Some(unversioned_catalog_metadata(name, slot))).await;
}
for name in [
format!("{prefix}sales"),
format!("{prefix}sales/tables/orders/maintenance/hash/jobs/job.json"),
] {
write_version_replicas(&dirs, RUSTFS_META_BUCKET, &name, |_| Some(b"corrupt noncandidate metadata".to_vec())).await;
}
assert!(
store
.clone()
.list_unversioned_objects_authoritative(RUSTFS_META_BUCKET, prefix, None, 100)
.await
.is_err(),
"the ordinary authoritative scan must expose the corrupt control objects"
);
let first = store
.clone()
.list_identifier_tree_leaves_authoritative(RUSTFS_META_BUCKET, prefix, leaf, None, 2)
.await
.expect("noncandidate metadata must not be read");
assert!(first.is_truncated);
assert_eq!(
first.objects.iter().map(|object| &object.name).collect::<Vec<_>>(),
names[..2].iter().collect::<Vec<_>>()
);
let second = store
.clone()
.list_identifier_tree_leaves_authoritative(RUSTFS_META_BUCKET, prefix, leaf, first.next_marker, 2)
.await
.expect("the next page must retain the final owner");
assert!(!second.is_truncated);
assert_eq!(second.objects.iter().map(|object| &object.name).collect::<Vec<_>>(), vec![&names[2]]);
write_version_replicas(&dirs, RUSTFS_META_BUCKET, &names[0], |_| Some(b"corrupt selected metadata".to_vec())).await;
assert!(
store
.clone()
.list_identifier_tree_leaves_authoritative(RUSTFS_META_BUCKET, prefix, leaf, None, 2)
.await
.is_err(),
"corruption at a selected key cannot be skipped to fill the page"
);
write_version_replicas(&dirs, RUSTFS_META_BUCKET, &names[0], |slot| {
Some(unversioned_catalog_metadata(&names[0], slot))
})
.await;
write_version_replicas(&dirs, RUSTFS_META_BUCKET, &names[0], |slot| {
Some(versioned_object_metadata(RUSTFS_META_BUCKET, &names[0], &[(2, 1_700_000_002)], slot).to_vec())
})
.await;
assert!(
store
.clone()
.list_identifier_tree_leaves_authoritative(RUSTFS_META_BUCKET, prefix, leaf, None, 2)
.await
.is_err(),
"a versioned ownership entry cannot be accepted as an unversioned proof"
);
write_version_replicas(&dirs, RUSTFS_META_BUCKET, &names[0], |slot| {
Some(unversioned_catalog_metadata(&names[0], slot))
})
.await;
let set = &store.pools[0].disk_set[0];
for disk in set.disks.write().await.iter_mut().skip(1) {
*disk = None;
}
assert!(
matches!(
store
.clone()
.list_identifier_tree_leaves_authoritative(RUSTFS_META_BUCKET, prefix, leaf, None, 2)
.await,
Err(StorageError::ErasureReadQuorum)
),
"unavailable replicas cannot prove owner absence"
);
}
#[tokio::test]
async fn authoritative_store_walk_includes_reserved_prefix_versions() {
let bucket = "catalog-cleanup-versions";
+22 -6
View File
@@ -34,7 +34,7 @@ use crate::bucket::utils::check_put_object_part_args;
use crate::bucket::utils::{check_valid_bucket_name, check_valid_bucket_name_strict, is_meta_bucketname};
use crate::cluster::rpc::{RemoteClient, S3PeerSys};
use crate::config::storageclass;
use crate::core::pools::{DecommissionCanceler, PoolMeta, PoolMetaWriteState};
use crate::core::pools::{DecommissionCanceler, PoolMeta, PoolMetaSaveGate};
use crate::disk::endpoint::{Endpoint, EndpointType};
use crate::disk::{DiskAPI, DiskInfo, DiskInfoOptions};
use crate::error::{Error, Result};
@@ -480,9 +480,13 @@ pub struct ECStore {
/// Serializes full-document pool metadata saves and retains a fail-closed
/// write block after startup observes an unreadable replica.
///
/// Immutable admission proofs share authority; publication, recovery and
/// authority changes retain exclusive access through their existing I/O.
/// Each writer publishes its final state on release, and health
/// inspection reads that committed state without acquiring the gate.
/// Lock order: acquire `pool_meta_save_gate`, then the distributed
/// `pool.bin` fence, then clone `pool_meta` under a short read lock.
pub(crate) pool_meta_save_gate: Mutex<PoolMetaWriteState>,
pub(crate) pool_meta_save_gate: PoolMetaSaveGate,
/// Per-instance runtime state (Phase 5, backlog#939).
///
/// Carries this instance's identity/runtime out of the process globals so
@@ -555,7 +559,6 @@ impl Default for ScannerDataMovementPauseStatus {
#[derive(Clone, Debug, PartialEq, Eq, serde::Serialize)]
pub struct PoolMetaWriteGateStatus {
pub writes_ready: bool,
pub check_timed_out: bool,
pub write_blocked: bool,
pub transaction_aborted: bool,
pub pool_meta_absent: bool,
@@ -571,7 +574,6 @@ impl Default for PoolMetaWriteGateStatus {
fn default() -> Self {
Self {
writes_ready: true,
check_timed_out: false,
write_blocked: false,
transaction_aborted: false,
pool_meta_absent: false,
@@ -1118,6 +1120,20 @@ impl ECStore {
Some(epoch)
}
/// Namespace cleanup edits an existing usage baseline; it does not certify
/// a scan. Retain the movement fence without requiring scanning to resume.
pub(crate) async fn data_usage_namespace_cleanup_guard(&self) -> Option<(tokio::sync::OwnedRwLockReadGuard<()>, u64)> {
let operation_guard = self.ctx.data_movement_operation_gate().read_owned().await;
if self.ctx.data_movement_operation_epoch_exhausted() || self.ctx.data_movement_generation_exhausted() {
return None;
}
Some((operation_guard, self.ctx.data_movement_operation_epoch()))
}
pub(crate) async fn data_usage_namespace_cleanup_epoch(&self) -> Option<u64> {
self.data_usage_namespace_cleanup_guard().await.map(|(_, epoch)| epoch)
}
/// Acquire a storage-owned read admission for a coordinator's final
/// scanner publication. The guard remains in the context's lease table,
/// so a local movement writer cannot pass the peer while its authoritative
@@ -1936,7 +1952,7 @@ mod tests {
rebalance_meta: RwLock::new(None),
decommission_cancelers: RwLock::new(Vec::new()),
start_gate: Mutex::new(()),
pool_meta_save_gate: Mutex::default(),
pool_meta_save_gate: crate::core::pools::PoolMetaSaveGate::default(),
ctx,
bucket_fence_registry: Arc::default(),
};
@@ -2011,7 +2027,7 @@ mod tests {
rebalance_meta: RwLock::new(None),
decommission_cancelers: RwLock::new(Vec::new()),
start_gate: Mutex::new(()),
pool_meta_save_gate: Mutex::default(),
pool_meta_save_gate: crate::core::pools::PoolMetaSaveGate::default(),
ctx,
bucket_fence_registry: Arc::default(),
})
+1 -1
View File
@@ -2133,7 +2133,7 @@ mod tests {
rebalance_meta: RwLock::new(None),
decommission_cancelers: RwLock::new(Vec::new()),
start_gate: Mutex::new(()),
pool_meta_save_gate: Mutex::default(),
pool_meta_save_gate: crate::core::pools::PoolMetaSaveGate::default(),
ctx: crate::runtime::instance::bootstrap_ctx(),
bucket_fence_registry: std::sync::Arc::default(),
}
+374 -3
View File
@@ -4937,6 +4937,87 @@ impl ECStore {
Ok((idx, result))
}
/// Compare the logical revision across all pools before publishing a
/// namespace-only usage update to a writable pool. A suspended source is
/// retained for decommission to reconcile against the newer target.
pub(crate) async fn put_usage_snapshot_for_namespace_cleanup(
&self,
object: &str,
expected_revision: Option<&str>,
data: &mut PutObjReader,
namespace_guard: Option<&rustfs_lock::NamespaceLockGuard>,
) -> Result<ObjectInfo> {
// The caller owns the bucket namespace and movement guards. Acquire
// the fixed object domain, then every physical domain in pool order.
let guards = self
.acquire_all_physical_object_write_locks("usage_namespace_cleanup", RUSTFS_META_BUCKET, object)
.await?;
let mut opts = ObjectOptions {
no_lock: true,
max_parity: true,
write_completion: crate::object_api::WriteCompletion::TailDrained,
suppress_read_repair: true,
..Default::default()
};
for guard in &guards {
guard.add_namespace_lock_fence(&mut opts);
}
if let Some(guard) = namespace_guard {
opts.add_namespace_lock_guard(guard);
}
let source = match self
.get_latest_object_info_for_conditional_write(RUSTFS_META_BUCKET, object, &opts)
.await
.and_then(|(info, idx)| resolve_latest_object_access(RUSTFS_META_BUCKET, object, info, idx, &opts))
{
Ok((info, _)) => Some(info),
Err(err) if is_err_object_not_found(&err) || is_err_version_not_found(&err) => None,
Err(err) => return Err(err),
};
if source.as_ref().and_then(|info| info.etag.as_deref()) != expected_revision {
return Err(StorageError::PreconditionFailed);
}
if source.as_ref().is_some_and(|info| {
info.version_id.is_some_and(|version| !version.is_nil()) || info.etag.as_ref().is_none_or(String::is_empty)
}) {
return Err(StorageError::other("usage cleanup requires an unversioned snapshot with a revision"));
}
let target = self.get_pool_idx_no_lock(RUSTFS_META_BUCKET, object, data.size()).await?;
let target_info = match self.pools[target].get_object_info(RUSTFS_META_BUCKET, object, &opts).await {
Ok(info) => Some(info),
Err(err) if is_err_object_not_found(&err) || is_err_version_not_found(&err) => None,
Err(err) => return Err(err),
};
opts.http_preconditions = Some(match target_info.as_ref() {
Some(info) => crate::storage_api_contracts::object::HTTPPreconditions {
if_match: Some(
info.etag
.as_ref()
.filter(|etag| !etag.is_empty())
.cloned()
.ok_or_else(|| StorageError::other("usage cleanup target has no revision"))?,
),
..Default::default()
},
None => crate::storage_api_contracts::object::HTTPPreconditions {
if_none_match: Some("*".to_string()),
..Default::default()
},
});
let latest = source
.iter()
.chain(target_info.iter())
.filter_map(|info| info.mod_time)
.fold(OffsetDateTime::now_utc(), std::cmp::max);
opts.mod_time = Some(
latest
.checked_add(time::Duration::nanoseconds(1))
.ok_or_else(|| StorageError::other("usage cleanup snapshot timestamp is exhausted"))?,
);
opts.decommission_capacity_admission = crate::bucket::metadata_sys::object_store_if_initialized_in(&self.ctx).await;
self.pools[target].put_object(RUSTFS_META_BUCKET, object, data, &opts).await
}
#[instrument(level = "debug", skip(self, data))]
#[hotpath::measure(impl_type = "ECStore")]
pub(super) async fn handle_put_object(
@@ -7005,7 +7086,7 @@ mod tests {
};
use crate::set_disk::same_distributed_lock_domain;
use crate::set_disk::{SetDisks, disk_call_counters};
use crate::storage_api_contracts::bucket::MakeBucketOptions;
use crate::storage_api_contracts::bucket::{BucketOptions, MakeBucketOptions};
use crate::storage_api_contracts::lifecycle::TransitionedObject;
use crate::storage_api_contracts::list::ListOperations as _;
use bytes::Bytes;
@@ -10003,7 +10084,7 @@ mod tests {
rebalance_meta: RwLock::new(None),
decommission_cancelers: RwLock::new(Vec::new()),
start_gate: Mutex::new(()),
pool_meta_save_gate: Mutex::default(),
pool_meta_save_gate: crate::core::pools::PoolMetaSaveGate::default(),
ctx: crate::runtime::instance::bootstrap_ctx(),
bucket_fence_registry: std::sync::Arc::default(),
}
@@ -10095,7 +10176,7 @@ mod tests {
rebalance_meta: RwLock::new(None),
decommission_cancelers: RwLock::new(Vec::new()),
start_gate: Mutex::new(()),
pool_meta_save_gate: Mutex::default(),
pool_meta_save_gate: crate::core::pools::PoolMetaSaveGate::default(),
ctx,
bucket_fence_registry: std::sync::Arc::default(),
}
@@ -10347,6 +10428,296 @@ mod tests {
multipool_delete_test_store(bucket, true).await
}
#[tokio::test]
async fn create_fresh_bucket_after_canceled_decommission() {
let (_dirs, store) = multipool_version_test_store("canceled-decommission-existing").await;
let mut canceled = prepared_pool_test_status(1, false);
canceled.decommission = Some(PoolDecommissionInfo {
canceled: true,
..Default::default()
});
*store.pool_meta.write().await = PoolMeta {
pools: vec![prepared_pool_test_status(0, false), canceled],
..Default::default()
};
store
.handle_make_bucket("canceled-decommission-fresh", &MakeBucketOptions::default())
.await
.expect("canceled decommission must not block creation of a new bucket");
assert!(
store.scanner_data_usage_publication_blocked().await,
"scanner publication must remain fenced"
);
}
#[tokio::test]
async fn create_bucket_after_failed_decommission() {
let (_dirs, store) = multipool_version_test_store("decommission-existing-bucket").await;
let bucket = "decommission-new-bucket";
let snapshot_time = std::time::SystemTime::now() - Duration::from_secs(60);
let mut snapshot = rustfs_data_usage::DataUsageInfo {
last_update: Some(snapshot_time),
scanner_cycle: Some(37),
scanner_epoch: Some(2),
buckets_count: 1,
usage_snapshot_complete: true,
..Default::default()
};
snapshot.buckets_usage.insert(
bucket.to_string(),
rustfs_data_usage::BucketUsageInfo {
size: 42,
objects_count: 1,
versions_count: 1,
..Default::default()
},
);
snapshot.bucket_sizes.insert(bucket.to_string(), 42);
snapshot.calculate_totals();
crate::data_usage::store_data_usage_in_backend(snapshot, store.clone())
.await
.expect("store a stale bucket-usage baseline before the failed decommission");
let (old_snapshot, usage_owner) = store
.get_latest_accessible_object_info_with_idx(
RUSTFS_META_BUCKET,
crate::data_usage::DATA_USAGE_OBJ_NAME_PATH.as_str(),
&ObjectOptions::default(),
)
.await
.expect("identify the pool holding the old usage baseline");
let mut pool_statuses = vec![prepared_pool_test_status(0, false), prepared_pool_test_status(1, false)];
pool_statuses[usage_owner].decommission = Some(PoolDecommissionInfo {
failed: true,
..Default::default()
});
*store.pool_meta.write().await = PoolMeta {
pools: pool_statuses,
..Default::default()
};
store
.handle_make_bucket(bucket, &MakeBucketOptions::default())
.await
.expect("a failed decommission must not block creation of an unrelated bucket");
store
.handle_get_bucket_info(bucket, &BucketOptions::default())
.await
.expect("the created bucket must be visible");
let usage = crate::data_usage::load_data_usage_from_backend(store.clone())
.await
.expect("load the cleaned usage baseline");
assert!(!usage.buckets_usage.contains_key(bucket), "the new bucket must not inherit old usage");
assert_eq!(usage.scanner_cycle, Some(37), "namespace cleanup must not certify a new scan");
assert_eq!(usage.scanner_epoch, Some(2), "namespace cleanup must preserve the scanner authority");
let object = crate::data_usage::DATA_USAGE_OBJ_NAME_PATH.as_str();
let (_, current_owner) = store
.get_latest_accessible_object_info_with_idx(RUSTFS_META_BUCKET, object, &ObjectOptions::default())
.await
.expect("read the new snapshot authority");
assert_ne!(current_owner, usage_owner, "cleanup must publish to a writable pool");
let retained_source = store.pools[usage_owner]
.get_object_info(RUSTFS_META_BUCKET, object, &ObjectOptions::default())
.await
.expect("decommission retains its original source for reconciliation");
assert_eq!(retained_source.etag, old_snapshot.etag);
let stale_update = store
.put_usage_snapshot_for_namespace_cleanup(
object,
old_snapshot.etag.as_deref(),
&mut PutObjReader::from_vec(b"stale snapshot".to_vec()),
None,
)
.await;
assert!(matches!(stale_update, Err(StorageError::PreconditionFailed)));
let usage = crate::data_usage::load_data_usage_from_backend(store.clone())
.await
.expect("a stale source CAS must not replace the writable target");
assert!(!usage.buckets_usage.contains_key(bucket));
assert!(
store.scanner_data_usage_publication_blocked().await,
"scanner publication must remain fenced"
);
}
#[tokio::test]
async fn canceled_create_retains_usage_cleanup_namespace_owner() {
use crate::set_disk::{PutObjectCommitBarrier, PutObjectCommitPause};
let (_dirs, store) = multipool_version_test_store("cleanup-cancellation-existing").await;
let bucket = "cleanup-cancellation-new";
let mut snapshot = rustfs_data_usage::DataUsageInfo {
last_update: Some(std::time::SystemTime::now()),
usage_snapshot_complete: true,
..Default::default()
};
snapshot.buckets_usage.insert(
bucket.to_owned(),
rustfs_data_usage::BucketUsageInfo {
size: 42,
objects_count: 1,
versions_count: 1,
..Default::default()
},
);
snapshot.bucket_sizes.insert(bucket.to_owned(), 42);
snapshot.calculate_totals();
crate::data_usage::store_data_usage_in_backend(snapshot, store.clone())
.await
.expect("seed the stale namespace usage");
let object = crate::data_usage::DATA_USAGE_OBJ_NAME_PATH.as_str();
let (_, source) = store
.get_latest_accessible_object_info_with_idx(RUSTFS_META_BUCKET, object, &ObjectOptions::default())
.await
.expect("find the source snapshot");
let mut pools = vec![prepared_pool_test_status(0, false), prepared_pool_test_status(1, false)];
pools[source].decommission = Some(PoolDecommissionInfo {
canceled: true,
..Default::default()
});
*store.pool_meta.write().await = PoolMeta {
pools,
..Default::default()
};
let barrier = PutObjectCommitBarrier::install(RUSTFS_META_BUCKET, object, PutObjectCommitPause::AfterNamespace);
let create_store = store.clone();
let create = tokio::spawn(async move { create_store.handle_make_bucket(bucket, &MakeBucketOptions::default()).await });
tokio::time::timeout(Duration::from_secs(10), barrier.wait_until_paused())
.await
.expect("cleanup reaches its physical commit with the namespace held");
create.abort();
assert!(create.await.expect_err("the request waiter was canceled").is_cancelled());
let namespace = store
.new_ns_lock(bucket, bucket)
.await
.expect("open the bucket namespace lock");
assert!(
tokio::time::timeout(Duration::from_millis(50), namespace.get_write_lock(Duration::from_secs(1)))
.await
.is_err(),
"cancellation must not release the namespace before its cleanup finishes"
);
barrier.release();
drop(barrier);
tokio::time::timeout(Duration::from_secs(10), store.handle_make_bucket(bucket, &MakeBucketOptions::default()))
.await
.expect("the resumed create must not remain blocked")
.expect("a new request resumes after the old cleanup releases its namespace");
let usage = crate::data_usage::load_data_usage_from_backend(store.clone())
.await
.expect("read the cleaned baseline");
assert!(!usage.buckets_usage.contains_key(bucket));
assert!(store.scanner_data_usage_publication_blocked().await);
}
#[tokio::test]
async fn usage_cleanup_rejects_an_unreadable_pool() {
let (_dirs, store) = multipool_version_test_store("cleanup-unreadable-pool").await;
let snapshot = rustfs_data_usage::DataUsageInfo {
last_update: Some(std::time::SystemTime::now()),
usage_snapshot_complete: true,
..Default::default()
};
crate::data_usage::store_data_usage_in_backend(snapshot, store.clone())
.await
.expect("seed the usage baseline");
let object = crate::data_usage::DATA_USAGE_OBJ_NAME_PATH.as_str();
let (before, source) = store
.get_latest_accessible_object_info_with_idx(RUSTFS_META_BUCKET, object, &ObjectOptions::default())
.await
.expect("find the baseline owner");
let unavailable = 1 - source;
for disk in store.pools[unavailable]
.get_disks_by_key(object)
.disks
.read()
.await
.iter()
.flatten()
{
disk.close().await.expect("stop disk health monitoring");
disk.force_offline_for_test();
}
let update = store
.put_usage_snapshot_for_namespace_cleanup(
object,
before.etag.as_deref(),
&mut PutObjReader::from_vec(b"unproven authority".to_vec()),
None,
)
.await;
assert!(update.is_err(), "an unreadable pool may hold a newer snapshot");
let after = store.pools[source]
.get_object_info(RUSTFS_META_BUCKET, object, &ObjectOptions::default())
.await
.expect("the readable baseline remains intact");
assert_eq!(after.etag, before.etag);
}
#[tokio::test]
async fn usage_cleanup_rejects_namespace_loss_before_physical_commit() {
use crate::set_disk::{PutObjectCommitBarrier, PutObjectCommitPause};
use rustfs_lock::{LocalClient, LockRequest, LockType, NamespaceLock, ObjectKey};
let (_dirs, store) = multipool_version_test_store("cleanup-fence-existing").await;
let snapshot = rustfs_data_usage::DataUsageInfo {
last_update: Some(std::time::SystemTime::now()),
usage_snapshot_complete: true,
..Default::default()
};
crate::data_usage::store_data_usage_in_backend(snapshot, store.clone())
.await
.expect("seed the protected baseline");
let object = crate::data_usage::DATA_USAGE_OBJ_NAME_PATH.as_str();
let (before, _) = store
.get_latest_accessible_object_info_with_idx(RUSTFS_META_BUCKET, object, &ObjectOptions::default())
.await
.expect("read the baseline revision");
let ttl = Duration::from_secs(60);
let namespace = NamespaceLock::new("usage-cleanup-commit-fence".to_owned(), Arc::new(LocalClient::new()));
let request = LockRequest::new(ObjectKey::new("cleanup-fence-existing", ""), LockType::Exclusive, "usage-cleanup")
.with_acquire_timeout(Duration::from_secs(1))
.with_ttl(ttl)
.with_refresh_interval(ttl);
let guard = Arc::new(
namespace
.acquire_guard(&request)
.await
.expect("acquire the namespace lease")
.expect("namespace lease is available"),
);
let barrier = PutObjectCommitBarrier::install(RUSTFS_META_BUCKET, object, PutObjectCommitPause::AfterNamespace);
let update_store = store.clone();
let update_guard = guard.clone();
let revision = before.etag.clone();
let update = tokio::spawn(async move {
let _movement = update_store
.data_usage_namespace_cleanup_guard()
.await
.expect("admit the namespace-only update");
update_store
.put_usage_snapshot_for_namespace_cleanup(
object,
revision.as_deref(),
&mut PutObjReader::from_vec(b"must not commit".to_vec()),
Some(update_guard.as_ref()),
)
.await
});
tokio::time::timeout(Duration::from_secs(10), barrier.wait_until_paused())
.await
.expect("the update reaches its commit barrier");
tokio::time::pause();
tokio::time::advance(ttl + Duration::from_millis(1)).await;
assert!(guard.is_lock_lost(), "the parent namespace lease has expired");
tokio::time::resume();
barrier.release();
assert!(update.await.expect("the cleanup task completes").is_err());
let (after, _) = store
.get_latest_accessible_object_info_with_idx(RUSTFS_META_BUCKET, object, &ObjectOptions::default())
.await
.expect("the original baseline remains readable");
assert_eq!(after.etag, before.etag, "a lost namespace lease must fence the physical write");
}
async fn multipool_delete_test_store(bucket: &str, versioned: bool) -> (Vec<tempfile::TempDir>, Arc<ECStore>) {
let ctx = Arc::new(crate::runtime::instance::InstanceContext::new());
let (mut dirs, first_set) = make_local_set_disks_with_ctx(4, 2, Arc::clone(&ctx)).await;
+1 -1
View File
@@ -58,7 +58,7 @@ pub enum Error {
TaskExecutionFailed { message: String },
/// Only the qualified MRF Deep check may produce this terminal result.
#[error("Heal task execution failed: Missing exact storage proof for durable MRF repair {target}")]
#[error("Independent verification unavailable for legacy object {target}; durable MRF responsibility retained")]
LegacyMrfUnverified { target: String },
#[error("Replacement failure could not be persisted: {failure}; persistence error: {persistence}")]
+17
View File
@@ -292,6 +292,20 @@ impl HealManager {
"Heal scheduler task completed"
);
}
Err(e @ Error::LegacyMrfUnverified { .. }) => {
warn!(
target: "rustfs::heal::manager",
event = EVENT_HEAL_SCHEDULER_STATE,
component = LOG_COMPONENT_HEAL,
subsystem = LOG_SUBSYSTEM_MANAGER,
state = "verification_unavailable",
task_id,
heal_type = %task_type_label_for_spawn,
set = %task_set_label_for_spawn,
detail = %e,
"Heal attempt completed without independent verification"
);
}
Err(e) => {
let will_retry = retry_request.is_some();
if will_retry {
@@ -483,6 +497,9 @@ impl HealManager {
// update statistics
let mut stats = statistics_clone.write().await;
match completed_status {
HealTaskStatus::Completed if matches!(&result, Err(Error::LegacyMrfUnverified { .. })) => {
stats.update_unverified_task_completion();
}
HealTaskStatus::Completed => {
stats.update_task_completion(true);
stats.add_healed_objects(completed_progress.objects_healed, completed_progress.bytes_processed);
+63
View File
@@ -1106,6 +1106,22 @@ impl HealStorageAPI for MockStorage {
if bucket.starts_with("legacy-ambiguity-scheduler-") {
return Err(Error::Storage(EcstoreError::DeleteRecoveryLegacyAmbiguous));
}
if bucket == "legacy-held-pending-scheduler" {
let drives = vec![rustfs_madmin::heal_commands::HealDriveInfo {
state: "ok".to_owned(),
..Default::default()
}];
return Ok((
HealResultItem {
legacy_identity_unverified: true,
detail: rustfs_heal_contracts::heal_channel::LEGACY_OBJECT_IDENTITY_UNVERIFIED_DETAIL.to_owned(),
before: rustfs_madmin::heal_commands::Infos { drives: drives.clone() },
after: rustfs_madmin::heal_commands::Infos { drives },
..Default::default()
},
None,
));
}
if bucket == "completed-retention-failed" {
return Err(Error::TaskExecutionFailed {
message: "retention fixture failure".to_string(),
@@ -2153,6 +2169,53 @@ fn only_typed_mrf_result_can_pause_unverified_legacy() {
}
}
#[tokio::test(flavor = "current_thread")]
async fn mrf_legacy_completed_attempt_retains_responsibility_without_failed_statistics() {
use crate::heal::failure_logging::test_support::FailureLogCapture;
let capture = FailureLogCapture::default();
let _subscriber = tracing::subscriber::set_default(capture.clone());
let (target, _) = unverified_legacy_fixture("legacy-held-pending-scheduler");
let bucket = target.bucket.clone();
let anchor = target.durable_anchor.clone().expect("durable MRF anchor");
let manager = HealManager::new_without_root_recovery_for_test(Arc::new(MockStorage), None);
let mut request = HealRequest::object(bucket.to_string(), target.object.to_string(), None);
request.source = HealRequestSource::Mrf;
request.expected_mrf_bucket_incarnation_id = Some(anchor.bucket_incarnation_id);
request.options.pool_index = Some(1);
request.options.set_index = Some(2);
let receipt = manager
.submit_mrf_heal_request_with_receipt_and_identity(request, target)
.await
.expect("admit durable MRF check");
assert_eq!(receipt.result, HealAdmissionResult::Accepted);
process_manager_queue_once(&manager).await;
tokio::time::timeout(Duration::from_secs(2), async {
while manager.get_statistics().await.unverified_tasks == 0 {
tokio::task::yield_now().await;
}
})
.await
.expect("publish completed unverified attempt");
let stats = manager.get_statistics().await;
assert_eq!(stats.unverified_tasks, 1);
assert_eq!(stats.failed_tasks, 0);
assert_eq!(stats.successful_tasks, 0);
assert_eq!(stats.total_objects_healed, 0);
let report = manager.get_task_report(&receipt.task_id).await.expect("retained task report");
assert_eq!(report.status, HealTaskStatus::Completed);
let outcome = report.outcome.expect("retained unknown outcome");
assert_eq!(outcome.counters.unknown, 1);
assert_eq!(outcome.counters.failed, 0);
assert!(rustfs_common::mrf_channel::take_mrf_verified_repair_events_for(&bucket).is_empty());
let held = rustfs_common::mrf_channel::take_mrf_unverified_legacy_events_for(&bucket);
assert_eq!(held.len(), 1);
assert_eq!(held[0].anchor, anchor);
assert!(manager.retrying_heals.lock().await.is_empty());
assert_eq!(capture.count(tracing::Level::ERROR, Some("rustfs::heal::manager")), 0);
assert_eq!(capture.count(tracing::Level::WARN, Some("rustfs::heal::manager")), 1);
}
#[test]
fn completed_mrf_notice_publishes_only_verified_positive_events() {
use crate::heal::outcome::{HealObjectIdentity, HealObjectKind, HealObjectOutcome, HealTaskOutcome};
+24
View File
@@ -506,6 +506,9 @@ pub struct HealStatistics {
pub successful_tasks: u64,
/// Failed tasks
pub failed_tasks: u64,
/// Completed attempts without independent verification; not the current durable MRF backlog.
#[serde(default)]
pub unverified_tasks: u64,
/// Running tasks
pub running_tasks: u64,
/// Total healed objects
@@ -528,6 +531,7 @@ impl HealStatistics {
total_tasks: 0,
successful_tasks: 0,
failed_tasks: 0,
unverified_tasks: 0,
running_tasks: 0,
total_objects_healed: 0,
total_bytes_healed: 0,
@@ -549,6 +553,11 @@ impl HealStatistics {
self.last_update_time = SystemTime::now();
}
pub(crate) fn update_unverified_task_completion(&mut self) {
self.unverified_tasks = self.unverified_tasks.saturating_add(1);
self.last_update_time = SystemTime::now();
}
pub fn add_healed_objects(&mut self, count: u64, bytes: u64) {
self.total_objects_healed += count;
self.total_bytes_healed += bytes;
@@ -915,6 +924,21 @@ mod tests {
assert_eq!(stats.failed_tasks, 0);
}
#[test]
fn unverified_statistics_preserve_old_snapshot_compatibility() {
let mut stats = HealStatistics::new();
stats.update_unverified_task_completion();
assert_eq!(stats.unverified_tasks, 1);
assert_eq!(stats.successful_tasks, 0);
assert_eq!(stats.failed_tasks, 0);
let mut wire = serde_json::to_value(&stats).expect("serialize task statistics");
let current: HealStatistics = serde_json::from_value(wire.clone()).expect("read new statistics");
assert_eq!(current.unverified_tasks, 1);
wire.as_object_mut().expect("statistics object").remove("unverified_tasks");
let old: HealStatistics = serde_json::from_value(wire).expect("read statistics without new field");
assert_eq!(old.unverified_tasks, 0);
}
#[test]
fn test_heal_statistics_update_task_completion_success() {
let mut stats = HealStatistics::new();
+23 -2
View File
@@ -291,7 +291,7 @@ pub enum HealTaskStatus {
Running,
/// Retrying after a recoverable failure
Retrying { error: String, retry_attempt: u32 },
/// Completed
/// Execution completed; the outcome and durable MRF ledger retain any unverified responsibility.
Completed,
/// Failed
Failed { error: String },
@@ -1232,6 +1232,22 @@ impl HealTask {
"Heal task completed"
});
}
Err(Error::LegacyMrfUnverified { .. }) => {
// The qualified Deep attempt finished, but its Unknown outcome
// cannot discharge the separately owned durable responsibility.
self.progress.write().await.mark_completed();
*self.status.write().await = HealTaskStatus::Completed;
debug!(
target: "rustfs::heal::task",
event = EVENT_HEAL_TASK_STATE,
component = LOG_COMPONENT_HEAL,
subsystem = LOG_SUBSYSTEM_TASK,
state = "verification_unavailable",
task_id = %self.id,
heal_type = self.heal_type.kind_label(),
"Heal attempt completed without independent verification"
);
}
Err(Error::TaskCancelled) => {
let mut status = self.status.write().await;
*status = HealTaskStatus::Cancelled;
@@ -1281,11 +1297,16 @@ impl HealTask {
let terminal_state = match &result {
Ok(_) => "completed",
Err(Error::LegacyMrfUnverified { .. }) => "verification_unavailable",
Err(Error::TaskCancelled) => "cancelled",
Err(Error::TaskTimeout) => "timed_out",
Err(_) => "failed",
};
self.emit_trace_task_state(terminal_state, start_instant.elapsed(), result.as_ref().err());
let terminal_error = result
.as_ref()
.err()
.filter(|error| !matches!(error, Error::LegacyMrfUnverified { .. }));
self.emit_trace_task_state(terminal_state, start_instant.elapsed(), terminal_error);
result
}
+29 -1
View File
@@ -3285,6 +3285,29 @@ async fn durable_mrf_unverified_object_names_cannot_change_retry_classification(
}
}
#[tokio::test]
async fn mrf_legacy_verification_pending_is_a_completed_unverified_attempt() {
let (task, _) = mrf_task_for_item("legacy-pending", healthy_legacy_heal_item());
let error = task.execute().await.expect_err("unverified result must retain MRF ownership");
assert!(matches!(error, Error::LegacyMrfUnverified { .. }));
assert!(!error.is_recoverable_heal());
assert_eq!(task.get_status().await, HealTaskStatus::Completed);
let outcome = task.get_outcome().await;
assert_eq!(outcome.execution, HealExecutionOutcome::Completed);
assert_eq!(outcome.counters.unknown, 1);
assert_eq!(outcome.counters.failed, 0);
assert_eq!(outcome.counters.healed, 0);
assert_eq!(outcome.counters.unchanged, 0);
assert!(matches!(outcome.objects[0].disposition, HealObjectDisposition::Unknown));
let (status, detail) = outcome.legacy_status("finished", None);
assert_eq!(status, "finished");
assert!(
detail
.expect("visible unverified result")
.contains("authoritative storage proof is unavailable")
);
}
#[tokio::test]
async fn mrf_legacy_object_names_cannot_change_retry_classification() {
for object in [
@@ -3304,7 +3327,9 @@ async fn mrf_legacy_object_names_cannot_change_retry_classification() {
assert!(matches!(error, Error::LegacyMrfUnverified { .. }));
assert_eq!(
error.to_string(),
format!("Heal task execution failed: Missing exact storage proof for durable MRF repair bucket-a/{object}")
format!(
"Independent verification unavailable for legacy object bucket-a/{object}; durable MRF responsibility retained"
)
);
}
}
@@ -3362,11 +3387,14 @@ async fn mrf_legacy_pause_requires_storage_owned_healthy_deep_result() {
let outcome = task.get_outcome().await;
assert_eq!(outcome.counters.healed, 0);
if case == "qualified" {
assert_eq!(task.get_status().await, HealTaskStatus::Completed);
assert_eq!(outcome.counters.unknown, 1);
assert_eq!(outcome.objects[0].detail.as_deref(), Some("updated diagnostic wording"));
let opts = storage.object_heal_opts.lock().expect("storage options");
assert_eq!(opts.len(), 1);
assert_eq!(opts[0].scan_mode, HealScanMode::Deep);
} else {
assert!(matches!(task.get_status().await, HealTaskStatus::Failed { .. }), "case={case}");
}
}
}
@@ -30,6 +30,78 @@ use storage_api::integration::{
WriteCommitGuard, WriteCompletion,
};
async fn complete_fixture_replicas(env: &TestECStoreEnv, bucket: &str, object: &str) {
// TailDrained completes fan-out, but a successful quorum write may still
// have minority failures. Fault injection needs a fully repaired baseline.
let result = ECStoreHealStorage::new(env.ecstore.clone())
.heal_object_with_receipt(
bucket,
object,
None,
&HealOpts {
scan_mode: HealScanMode::Deep,
..Default::default()
},
)
.await
.expect("complete protected fixture replicas before fault injection");
assert!(result.error.is_none(), "fixture repair failed: {:?}", result.error);
assert_eq!(result.item.after.drives.len(), env.disk_paths.len());
assert!(
result.item.after.drives.iter().all(|drive| drive.state == "ok"),
"fixture requires every replica, not only read/write quorum: {:?}",
result.item.after.drives
);
assert!(result.item.integrity_verified, "fixture must retain protected payload verification");
}
#[tokio::test]
#[serial]
async fn mixed_batch_fixture_repairs_a_missing_minority_replica() {
let root = tempfile::tempdir().expect("minority replica fixture");
let env = TestECStoreEnv::builder().base_dir(root.path()).build().await;
let bucket = "mixed-fixture-baseline";
let object = "object";
env.make_bucket(bucket, false).await;
let expected = vec![0x5a; 1024 * 1024 + 123];
env.ecstore
.put_object(
bucket,
object,
&mut PutObjReader::from_vec(expected.clone()),
&ObjectOptions {
shard_integrity_write_mode: Some(ShardIntegrityWriteMode::Protected),
write_completion: WriteCompletion::TailDrained,
..Default::default()
},
)
.await
.expect("quorum commit fixture");
// Guarantee a minority failure regardless of whether the original write
// succeeded on every disk, then repair it before any test fault is applied.
let replica = env.disk_paths[0].join(bucket).join(object);
if replica.exists() {
tokio::fs::remove_dir_all(&replica)
.await
.expect("remove one complete replica");
}
complete_fixture_replicas(&env, bucket, object).await;
let disks = env.ecstore.pools[0].get_disks(0);
for disk in disks.disks.read().await.iter().flatten() {
disk.read_version("", bucket, object, "", &ReadOptions::default())
.await
.expect("every fixture replica has metadata after completion");
}
let mut reader = env
.ecstore
.get_object_reader(bucket, object, None, Default::default(), &ObjectOptions::default())
.await
.expect("read the completed fixture");
let mut actual = Vec::new();
reader.stream.read_to_end(&mut actual).await.expect("completed fixture body");
assert_eq!(actual, expected);
}
#[test]
fn disabled_lock_reports_only_quorum_selection_and_deep_check_observations() {
const CHILD_ENV: &str = "RUSTFS_TEST_DISABLED_LOCK_HEAL_CHILD";
@@ -704,6 +776,7 @@ fn admin_mixed_batch_retains_repairs_after_object_retry_exhaustion() {
)
.await
.expect("commit complete fixture");
complete_fixture_replicas(&env, bucket, object).await;
for (slot, disk) in disks.iter().enumerate() {
let meta = disk
.read_version("", bucket, object, "", &ReadOptions::default())
+45 -4
View File
@@ -66,11 +66,26 @@ pub fn select_effective_memory(host_total: u64, host_available: u64, cgroup: Opt
/// Resolves the effective memory from an already-refreshed system handle.
///
/// `cgroup_limits()` is computed fresh on each call and is only implemented on
/// Linux (it returns `None` elsewhere), so non-Linux hosts always use host
/// values.
/// On Linux, the current process cgroup and its ancestors are read on each call,
/// including limits imposed by systemd services. System-wide cgroup limits are
/// used if process information is unavailable. Other platforms use host values.
pub fn effective_memory_from_system(system: &System) -> EffectiveMemory {
let cgroup = system.cgroup_limits().map(|limits| (limits.total_memory, limits.free_memory));
#[cfg(target_os = "linux")]
let limits = sysinfo::get_current_pid()
.ok()
.and_then(|pid| {
let mut processes = System::new().ok()?;
processes.refresh_processes_specifics(
sysinfo::ProcessesToUpdate::Some(&[pid]),
false,
sysinfo::ProcessRefreshKind::nothing().with_memory(),
);
processes.process(pid)?.cgroup_limits()
})
.or_else(|| system.cgroup_limits());
#[cfg(not(target_os = "linux"))]
let limits = system.cgroup_limits();
let cgroup = limits.map(|limits| (limits.total_memory, limits.free_memory));
select_effective_memory(system.total_memory(), system.available_memory(), cgroup)
}
@@ -89,6 +104,32 @@ mod tests {
const GIB: u64 = 1024 * 1024 * 1024;
#[cfg(target_os = "linux")]
#[test]
fn effective_memory_uses_current_process_cgroup() {
use sysinfo::{ProcessRefreshKind, ProcessesToUpdate, System, get_current_pid};
let pid = get_current_pid().expect("current process ID");
let mut system = System::new().expect("system memory handle");
system.refresh_memory();
let mut process_system = System::new().expect("process memory handle");
process_system.refresh_processes_specifics(
ProcessesToUpdate::Some(&[pid]),
false,
ProcessRefreshKind::nothing().with_memory(),
);
let limits = process_system.process(pid).expect("current process entry").cgroup_limits();
let expected = select_effective_memory(
system.total_memory(),
system.available_memory(),
limits.map(|limits| (limits.total_memory, limits.free_memory)),
);
let actual = super::effective_memory_from_system(&system);
assert_eq!(actual.basis, expected.basis);
assert_eq!(actual.total_bytes, expected.total_bytes);
assert!(actual.available_bytes <= actual.total_bytes);
}
#[test]
fn select_effective_memory_prefers_constraining_cgroup() {
let effective = select_effective_memory(64 * GIB, 40 * GIB, Some((2 * GIB, GIB)));
+14 -6
View File
@@ -145,7 +145,7 @@ With OTLP → Collector → Prometheus, the time from a failure to an alert also
| Variable | Default | Description |
|---------------------------------|--------------|-----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------|
| `RUSTFS_OBS_LOGGER_LEVEL` | `info` | Log level; `RUST_LOG` syntax supported |
| `RUSTFS_OBS_LOGGER_LEVEL` | `error` | Log level; `RUST_LOG` syntax supported |
| `RUSTFS_OBS_LOG_STDOUT_ENABLED` | `false` | When file logging is active, also mirror to stdout |
| `RUSTFS_OBS_LOG_DIRECTORY` | _(empty)_ | **Directory for rolling log files. When empty, logs go to stdout only** |
| `RUSTFS_OBS_LOG_FILENAME` | `rustfs.log` | Base filename for rolling logs. Rotated archives include a high-precision timestamp and counter. With the default `RUSTFS_OBS_LOG_MATCH_MODE=suffix`, names look like `<timestamp>-<counter>.rustfs.log` (e.g., `20231027103001.123456-0.rustfs.log`); with `prefix`, they look like `rustfs.log.<timestamp>-<counter>` (e.g., `rustfs.log.20231027103001.123456-0`). |
@@ -157,23 +157,31 @@ With OTLP → Collector → Prometheus, the time from a failure to an alert also
| Variable | Default | Description |
|-------------------------------------------------|--------------|-------------------------------------------------------------|
| `RUSTFS_OBS_LOG_MAX_TOTAL_SIZE_BYTES` | `2147483648` | Hard cap on total log directory size (2 GiB) |
| `RUSTFS_OBS_LOG_MAX_SINGLE_FILE_SIZE_BYTES` | `0` | Per-file size cap; `0` = unlimited |
| `RUSTFS_OBS_LOG_MAX_TOTAL_SIZE_BYTES` | `2147483648` | Byte budget (2 GiB), applied to rolled logs and to compressed archives separately; `0` = no budget. See [Disk usage bound](#disk-usage-bound) |
| `RUSTFS_OBS_LOG_MAX_SINGLE_FILE_SIZE_BYTES` | unset | Unset or unparsable: the active file rolls at 256 MiB, or at a quarter of the byte budget when that is smaller (at least 1 MiB). A value: the active file rolls at that size and the cleaner also processes rolled files above it. `0` = no size roll |
| `RUSTFS_OBS_LOG_COMPRESS_OLD_FILES` | `true` | Compress files before deleting |
| `RUSTFS_OBS_LOG_GZIP_COMPRESSION_LEVEL` | `6` | Gzip level `1` (fastest) – `9` (best) |
| `RUSTFS_OBS_LOG_COMPRESSION_ALGORITHM` | `zstd` | Compression codec: `zstd` or `gzip` |
| `RUSTFS_OBS_LOG_PARALLEL_COMPRESS` | `true` | Enable work-stealing parallel compression |
| `RUSTFS_OBS_LOG_PARALLEL_WORKERS` | `6` | Number of cleaner worker threads |
| `RUSTFS_OBS_LOG_ZSTD_COMPRESSION_LEVEL` | `8` | Zstd level `1` (fastest) – `21` (best ratio) |
| `RUSTFS_OBS_LOG_ZSTD_COMPRESSION_LEVEL` | `8` | Zstd level `1` (fastest) – `22` (best ratio); `0` = codec default |
| `RUSTFS_OBS_LOG_ZSTD_FALLBACK_TO_GZIP` | `true` | Fallback to gzip when zstd compression fails |
| `RUSTFS_OBS_LOG_ZSTD_WORKERS` | `1` | zstdmt worker threads per compression task |
| `RUSTFS_OBS_LOG_COMPRESSED_FILE_RETENTION_DAYS` | `30` | Delete `.gz` / `.zst` archives older than N days; `0` = keep forever |
| `RUSTFS_OBS_LOG_EXCLUDE_PATTERNS` | _(empty)_ | Comma-separated glob patterns to never clean up |
| `RUSTFS_OBS_LOG_DELETE_EMPTY_FILES` | `true` | Remove zero-byte files |
| `RUSTFS_OBS_LOG_MIN_FILE_AGE_SECONDS` | `3600` | Minimum file age (seconds) before cleanup |
| `RUSTFS_OBS_LOG_CLEANUP_INTERVAL_SECONDS` | `1800` | How often the cleanup task runs (0.5 hours) |
| `RUSTFS_OBS_LOG_MIN_FILE_AGE_SECONDS` | `3600` | Minimum file age (seconds) before cleanup; not applied while rolled logs exceed the byte budget |
| `RUSTFS_OBS_LOG_CLEANUP_INTERVAL_SECONDS` | `1800` | How often the cleanup task runs (0.5 hours); a pass also runs after every roll |
| `RUSTFS_OBS_LOG_DRY_RUN` | `false` | Report deletions without actually removing files |
### Disk usage bound
With the defaults the log directory holds at most the byte budget of rolled logs, the same budget of compressed archives, and one active file of up to 256 MiB. The active file is rolled by size as well as by time, and the cleaner runs a pass after every roll in addition to its interval, so the budget is enforced when a file is rolled. While the rolled logs exceed the budget the cleaner processes the oldest ones, as many as the budget requires, even if they are younger than `RUSTFS_OBS_LOG_MIN_FILE_AGE_SECONDS`; `RUSTFS_OBS_LOG_KEEP_FILES` still applies only to files past that age.
The bound is not instantaneous: the directory can exceed it by what is written while one cleanup pass runs, archives created by a pass are trimmed by the next one, and a cleaner that compresses more slowly than logs are produced falls behind. Setting `RUSTFS_OBS_LOG_MAX_TOTAL_SIZE_BYTES=0` removes the budget and `RUSTFS_OBS_LOG_MAX_SINGLE_FILE_SIZE_BYTES=0` turns size rolling off, which leaves the active file bounded only by the rotation period. Neither setting limits how fast logs are produced.
The cleaner recognizes only its own process's active file. Do not let two processes write log files that match the same pattern in one directory: a file another process is still writing looks like a rolled log, and the minimum file age no longer shields it once the budget is exceeded.
---
## Cleaner & Rotation Metrics
+70 -11
View File
@@ -149,6 +149,7 @@ impl LogCleaner {
let LogScanResult {
mut logs,
recent_logs,
mut compressed_archives,
} = scan_log_directory(
&self.log_dir,
@@ -161,8 +162,9 @@ impl LogCleaner {
self.dry_run,
)?;
logs.sort_by_key(|f| f.modified);
let mut to_delete = Vec::new();
if !logs.is_empty() {
logs.sort_by_key(|f| f.modified);
let total_size: u64 = logs.iter().map(|f| f.size).sum();
info!(
@@ -178,16 +180,35 @@ impl LogCleaner {
// Select the oldest files first, then additionally trim any files
// that still violate configured size constraints.
let to_delete = self.select_files_to_process(&logs, total_size);
if !to_delete.is_empty() {
let (deleted, freed) = if self.parallel_compress && self.compress_old_files {
self.parallel_stealing_compress(&to_delete)?
} else {
self.serial_compress_and_delete(&to_delete)?
};
total_deleted += deleted;
total_freed += freed;
}
to_delete = self.select_files_to_process(&logs, total_size);
}
// `min_file_age_seconds` keeps freshly rolled files readable for a
// while, but those files fill the disk like any other. Once the rolled
// files that would remain exceed the byte budget, the budget wins:
// the oldest of them go, fresh or not, and only as many as it takes.
let over_budget = self.select_files_over_budget(&logs, &to_delete, recent_logs);
if !over_budget.is_empty() {
warn!(
event = EVENT_LOG_CLEANER_STATE,
component = LOG_COMPONENT_OBS,
subsystem = LOG_SUBSYSTEM_LOG_CLEANER,
result = "byte_budget_overrides_min_file_age",
log_files = over_budget.len(),
max_total_size_bytes = self.max_total_size_bytes,
"log cleaner state changed"
);
to_delete.extend(over_budget);
}
if !to_delete.is_empty() {
let (deleted, freed) = if self.parallel_compress && self.compress_old_files {
self.parallel_stealing_compress(&to_delete)?
} else {
self.serial_compress_and_delete(&to_delete)?
};
total_deleted += deleted;
total_freed += freed;
}
if !compressed_archives.is_empty() {
@@ -217,6 +238,44 @@ impl LogCleaner {
Ok((total_deleted, total_freed))
}
/// Byte-budget trim that also counts the rolled files still inside
/// `min_file_age_seconds`.
///
/// `aged` is sorted oldest first and `selected` is what the regular
/// selection already took from it. When the files that would remain,
/// aged and recent together, exceed `max_total_size_bytes`, the oldest of
/// them are returned until the rest fits. `keep_files` and the per-file
/// limit are not applied to recent files: only the bytes matter here.
pub(super) fn select_files_over_budget(
&self,
aged: &[FileInfo],
selected: &[FileInfo],
mut recent: Vec<FileInfo>,
) -> Vec<FileInfo> {
if self.max_total_size_bytes == 0 || recent.is_empty() {
return Vec::new();
}
recent.sort_by_key(|file| file.modified);
let selected: std::collections::HashSet<&std::path::Path> = selected.iter().map(|file| file.path.as_path()).collect();
// Aged files are older than every recent one, so chaining keeps the
// oldest-first order.
let remaining: Vec<&FileInfo> = aged
.iter()
.filter(|file| !selected.contains(file.path.as_path()))
.chain(recent.iter())
.collect();
let mut occupied = remaining.iter().fold(0u64, |total, file| total.saturating_add(file.size));
let mut over_budget = Vec::new();
for file in remaining {
if occupied <= self.max_total_size_bytes {
break;
}
occupied = occupied.saturating_sub(file.size);
over_budget.push(file.clone());
}
over_budget
}
/// Choose regular log files that should be compressed and/or deleted.
///
/// The `files` slice must already be sorted from oldest to newest. The
+165
View File
@@ -444,6 +444,171 @@ mod tests {
Ok(())
}
/// Rolled files inside `min_file_age_seconds` occupy the disk too: once
/// they push the directory over the byte budget, the oldest are processed
/// regardless of age until the budget holds again.
#[test]
fn byte_budget_overrides_min_age_for_fresh_logs() -> std::io::Result<()> {
let tmp = TempDir::new()?;
let dir = tmp.path().to_path_buf();
for i in 0..6u64 {
let name = format!("app.log.2024-01-0{i}");
create_log_file(&dir, &name, 1024)?;
// All well inside the age gate, oldest first by name.
set_mtime_ago(&dir.join(&name), Duration::from_secs(60 - i * 5));
}
create_log_file(&dir, "app.log", 1024)?;
let cleaner = LogCleaner::builder(dir.clone(), "app.log.".to_string(), "app.log".to_string())
.match_mode(FileMatchMode::Prefix)
.keep_files(30)
.max_total_size_bytes(3 * 1024)
.min_file_age_seconds(3600)
.build();
let (deleted, freed) = cleaner.cleanup()?;
assert_eq!((deleted, freed), (3, 3 * 1024), "the three oldest fresh logs go; the budget then holds");
for i in 0..3 {
assert!(!dir.join(format!("app.log.2024-01-0{i}")).exists());
}
for i in 3..6 {
assert!(dir.join(format!("app.log.2024-01-0{i}")).exists(), "the newest fresh logs stay");
}
assert!(dir.join("app.log").exists(), "the active file is never touched");
Ok(())
}
/// Within the byte budget the age gate is unchanged: fresh logs are not
/// touched, even above `keep_files`.
#[test]
fn min_age_still_protects_fresh_logs_within_the_byte_budget() -> std::io::Result<()> {
let tmp = TempDir::new()?;
let dir = tmp.path().to_path_buf();
for i in 0..4 {
create_log_file(&dir, &format!("app.log.2024-01-0{i}"), 1024)?;
}
let cleaner = LogCleaner::builder(dir.clone(), "app.log.".to_string(), "app.log".to_string())
.match_mode(FileMatchMode::Prefix)
.keep_files(1)
.max_total_size_bytes(4 * 1024)
.min_file_age_seconds(3600)
.build();
assert_eq!(cleaner.cleanup()?, (0, 0), "exactly at the budget nothing is over it");
for i in 0..4 {
assert!(dir.join(format!("app.log.2024-01-0{i}")).exists());
}
// No byte budget configured: fresh files are never counted.
let unbounded = LogCleaner::builder(dir, "app.log.".to_string(), "app.log".to_string())
.match_mode(FileMatchMode::Prefix)
.keep_files(1)
.max_total_size_bytes(0)
.min_file_age_seconds(3600)
.build();
assert_eq!(unbounded.cleanup()?, (0, 0));
Ok(())
}
/// The budget override is a byte trim only: `keep_files` is not applied
/// to fresh files, so one byte over the budget costs one file, not all
/// but `keep_files` of them.
#[test]
fn byte_budget_override_removes_only_what_the_budget_needs() -> std::io::Result<()> {
let tmp = TempDir::new()?;
let dir = tmp.path().to_path_buf();
for i in 0..5u64 {
let name = format!("app.log.2024-01-0{i}");
create_log_file(&dir, &name, 1024)?;
set_mtime_ago(&dir.join(&name), Duration::from_secs(50 - i * 10));
}
let cleaner = LogCleaner::builder(dir.clone(), "app.log.".to_string(), "app.log".to_string())
.match_mode(FileMatchMode::Prefix)
.keep_files(1)
.max_single_file_size_bytes(512)
.max_total_size_bytes(4 * 1024)
.min_file_age_seconds(3600)
.build();
let (deleted, freed) = cleaner.cleanup()?;
assert_eq!(
(deleted, freed),
(1, 1024),
"neither keep_files nor the per-file limit applies to fresh files"
);
assert!(!dir.join("app.log.2024-01-00").exists(), "the oldest fresh file goes");
for i in 1..5 {
assert!(dir.join(format!("app.log.2024-01-0{i}")).exists());
}
Ok(())
}
/// Aged files keep their regular selection; the byte trim then counts the
/// fresh files and takes the oldest remaining files first.
#[test]
fn regular_selection_and_budget_override_compose() -> std::io::Result<()> {
let tmp = TempDir::new()?;
let dir = tmp.path().to_path_buf();
for i in 0..3u64 {
let name = format!("app.log.aged-{i}");
create_log_file(&dir, &name, 1024)?;
set_mtime_ago(&dir.join(&name), Duration::from_secs((5 - i) * 3600));
}
for i in 0..3u64 {
let name = format!("app.log.fresh-{i}");
create_log_file(&dir, &name, 1024)?;
set_mtime_ago(&dir.join(&name), Duration::from_secs(30 - i * 10));
}
// keep_files(2) takes aged-0. Five files remain against a budget of
// three, so the two oldest of them go as well: aged-1 and aged-2.
let cleaner = LogCleaner::builder(dir.clone(), "app.log.".to_string(), "app.log".to_string())
.match_mode(FileMatchMode::Prefix)
.keep_files(2)
.max_total_size_bytes(3 * 1024)
.min_file_age_seconds(3600)
.build();
let (deleted, freed) = cleaner.cleanup()?;
assert_eq!((deleted, freed), (3, 3 * 1024));
for i in 0..3 {
assert!(!dir.join(format!("app.log.aged-{i}")).exists());
assert!(dir.join(format!("app.log.fresh-{i}")).exists());
}
Ok(())
}
/// Aged and fresh logs share one budget: aged files alone fit, the fresh
/// ones put the directory over it, and the oldest files overall go first.
#[test]
fn aged_and_fresh_logs_share_the_byte_budget() -> std::io::Result<()> {
let tmp = TempDir::new()?;
let dir = tmp.path().to_path_buf();
create_log_file(&dir, "app.log.aged-0", 1024)?;
create_log_file(&dir, "app.log.aged-1", 1024)?;
set_mtime_ago(&dir.join("app.log.aged-0"), Duration::from_secs(3 * 3600));
set_mtime_ago(&dir.join("app.log.aged-1"), Duration::from_secs(2 * 3600));
create_log_file(&dir, "app.log.fresh-0", 1024)?;
create_log_file(&dir, "app.log.fresh-1", 1024)?;
set_mtime_ago(&dir.join("app.log.fresh-0"), Duration::from_secs(30));
let cleaner = LogCleaner::builder(dir.clone(), "app.log.".to_string(), "app.log".to_string())
.match_mode(FileMatchMode::Prefix)
.keep_files(30)
.max_total_size_bytes(2 * 1024)
.min_file_age_seconds(3600)
.build();
let (deleted, _) = cleaner.cleanup()?;
assert_eq!(deleted, 2);
assert!(!dir.join("app.log.aged-0").exists());
assert!(!dir.join("app.log.aged-1").exists());
assert!(dir.join("app.log.fresh-0").exists());
assert!(dir.join("app.log.fresh-1").exists());
Ok(())
}
#[test]
fn active_file_matching_pattern_is_protected() -> std::io::Result<()> {
let tmp = TempDir::new()?;
+48 -4
View File
@@ -48,6 +48,10 @@ const ORPHAN_TMP_GRACE_SECONDS: u64 = 60;
pub(super) struct LogScanResult {
/// Regular log files eligible for deletion/compression.
pub logs: Vec<FileInfo>,
/// Regular log files younger than `min_file_age_seconds`. They occupy the
/// directory like any other rolled file, so the caller counts them against
/// the byte budget and may process them when that budget is exceeded.
pub recent_logs: Vec<FileInfo>,
/// Already compressed files eligible for expiry deletion.
pub compressed_archives: Vec<FileInfo>,
}
@@ -83,11 +87,13 @@ pub(super) fn scan_log_directory(
if file_pattern.is_empty() {
return Ok(LogScanResult {
logs: Vec::new(),
recent_logs: Vec::new(),
compressed_archives: Vec::new(),
});
}
let mut logs = Vec::new();
let mut recent_logs = Vec::new();
let mut compressed_archives = Vec::new();
let now = SystemTime::now();
@@ -99,6 +105,7 @@ pub(super) fn scan_log_directory(
// as "no files found" instead of failing the whole cleanup pass.
return Ok(LogScanResult {
logs,
recent_logs,
compressed_archives,
});
}
@@ -230,10 +237,7 @@ pub(super) fn scan_log_directory(
// 6. Age gate regular logs only.
// Compressed files deliberately bypass this check because archive
// expiry is driven by a dedicated retention horizon in the caller.
if !is_compressed && !is_old_enough_for_cleanup(age, min_file_age_seconds) {
// Too young to be touched.
continue;
}
let recent = !is_compressed && !is_old_enough_for_cleanup(age, min_file_age_seconds);
let info = FileInfo {
path,
@@ -244,6 +248,9 @@ pub(super) fn scan_log_directory(
if is_compressed {
compressed_archives.push(info);
} else if recent {
// Too young to be touched while the directory is within budget.
recent_logs.push(info);
} else {
logs.push(info);
}
@@ -251,6 +258,7 @@ pub(super) fn scan_log_directory(
Ok(LogScanResult {
logs,
recent_logs,
compressed_archives,
})
}
@@ -311,6 +319,42 @@ mod tests {
Ok(())
}
#[test]
fn fresh_nonempty_log_is_reported_as_recent_not_as_a_candidate() -> std::io::Result<()> {
let tmp = TempDir::new()?;
let dir = tmp.path();
std::fs::write(dir.join("fresh.rustfs.log"), b"fresh")?;
std::fs::write(dir.join("aged.rustfs.log"), b"aged!!")?;
let old = SystemTime::now() - Duration::from_secs(7200);
File::options()
.write(true)
.open(dir.join("aged.rustfs.log"))?
.set_modified(old)?;
std::fs::write(dir.join("active.rustfs.log"), b"active")?;
let result = scan_log_directory(
dir,
".rustfs.log",
Some("active.rustfs.log"),
FileMatchMode::Suffix,
&[],
3600,
true,
false,
)?;
let names = |files: &[FileInfo]| -> Vec<String> {
files
.iter()
.map(|file| file.path.file_name().expect("file name").to_string_lossy().into_owned())
.collect()
};
assert_eq!(names(&result.logs), vec!["aged.rustfs.log"]);
assert_eq!(names(&result.recent_logs), vec!["fresh.rustfs.log"], "the active file is in neither list");
assert_eq!(result.recent_logs[0].size, 5);
Ok(())
}
#[test]
fn orphan_tmp_archive_is_deleted() -> std::io::Result<()> {
let tmp = TempDir::new()?;
+47 -22
View File
@@ -26,22 +26,21 @@ use rustfs_config::observability::{
DEFAULT_OBS_ENVIRONMENT_PRODUCTION, DEFAULT_OBS_LOG_CLEANUP_INTERVAL_SECONDS, DEFAULT_OBS_LOG_COMPRESS_OLD_FILES,
DEFAULT_OBS_LOG_COMPRESSED_FILE_RETENTION_DAYS, DEFAULT_OBS_LOG_COMPRESSION_ALGORITHM, DEFAULT_OBS_LOG_DELETE_EMPTY_FILES,
DEFAULT_OBS_LOG_DRY_RUN, DEFAULT_OBS_LOG_GZIP_COMPRESSION_LEVEL, DEFAULT_OBS_LOG_MATCH_MODE,
DEFAULT_OBS_LOG_MAX_SINGLE_FILE_SIZE_BYTES, DEFAULT_OBS_LOG_MAX_TOTAL_SIZE_BYTES, DEFAULT_OBS_LOG_MIN_FILE_AGE_SECONDS,
DEFAULT_OBS_LOG_PARALLEL_COMPRESS, DEFAULT_OBS_LOG_PARALLEL_WORKERS, DEFAULT_OBS_LOG_ZSTD_COMPRESSION_LEVEL,
DEFAULT_OBS_LOG_ZSTD_FALLBACK_TO_GZIP, DEFAULT_OBS_LOG_ZSTD_WORKERS, ENV_OBS_ENDPOINT, ENV_OBS_ENDPOINT_HEADERS,
ENV_OBS_ENDPOINT_LOGS_HEADERS, ENV_OBS_ENDPOINT_LOGS_TIMEOUT_MILLIS, ENV_OBS_ENDPOINT_METRICS_HEADERS,
ENV_OBS_ENDPOINT_METRICS_TIMEOUT_MILLIS, ENV_OBS_ENDPOINT_TIMEOUT_MILLIS, ENV_OBS_ENDPOINT_TRACES_HEADERS,
ENV_OBS_ENDPOINT_TRACES_TIMEOUT_MILLIS, ENV_OBS_ENVIRONMENT, ENV_OBS_LOG_CLEANUP_INTERVAL_SECONDS,
ENV_OBS_LOG_COMPRESS_OLD_FILES, ENV_OBS_LOG_COMPRESSED_FILE_RETENTION_DAYS, ENV_OBS_LOG_COMPRESSION_ALGORITHM,
ENV_OBS_LOG_DELETE_EMPTY_FILES, ENV_OBS_LOG_DIRECTORY, ENV_OBS_LOG_DRY_RUN, ENV_OBS_LOG_ENDPOINT,
ENV_OBS_LOG_EXCLUDE_PATTERNS, ENV_OBS_LOG_FILENAME, ENV_OBS_LOG_GZIP_COMPRESSION_LEVEL, ENV_OBS_LOG_KEEP_FILES,
ENV_OBS_LOG_MATCH_MODE, ENV_OBS_LOG_MAX_SINGLE_FILE_SIZE_BYTES, ENV_OBS_LOG_MAX_TOTAL_SIZE_BYTES,
ENV_OBS_LOG_MIN_FILE_AGE_SECONDS, ENV_OBS_LOG_PARALLEL_COMPRESS, ENV_OBS_LOG_PARALLEL_WORKERS, ENV_OBS_LOG_ROTATION_TIME,
ENV_OBS_LOG_STDOUT_ENABLED, ENV_OBS_LOG_ZSTD_COMPRESSION_LEVEL, ENV_OBS_LOG_ZSTD_FALLBACK_TO_GZIP, ENV_OBS_LOG_ZSTD_WORKERS,
ENV_OBS_LOGGER_LEVEL, ENV_OBS_LOGS_EXPORT_ENABLED, ENV_OBS_METER_INTERVAL, ENV_OBS_METRIC_ENDPOINT,
ENV_OBS_METRICS_EXPORT_ENABLED, ENV_OBS_PROFILING_ENDPOINT, ENV_OBS_PROFILING_EXPORT_ENABLED, ENV_OBS_SAMPLE_RATIO,
ENV_OBS_SERVICE_NAME, ENV_OBS_SERVICE_VERSION, ENV_OBS_TLS_CA_FILE, ENV_OBS_TRACE_ENDPOINT, ENV_OBS_TRACES_EXPORT_ENABLED,
ENV_OBS_USE_STDOUT,
DEFAULT_OBS_LOG_MAX_TOTAL_SIZE_BYTES, DEFAULT_OBS_LOG_MIN_FILE_AGE_SECONDS, DEFAULT_OBS_LOG_PARALLEL_COMPRESS,
DEFAULT_OBS_LOG_PARALLEL_WORKERS, DEFAULT_OBS_LOG_ZSTD_COMPRESSION_LEVEL, DEFAULT_OBS_LOG_ZSTD_FALLBACK_TO_GZIP,
DEFAULT_OBS_LOG_ZSTD_WORKERS, ENV_OBS_ENDPOINT, ENV_OBS_ENDPOINT_HEADERS, ENV_OBS_ENDPOINT_LOGS_HEADERS,
ENV_OBS_ENDPOINT_LOGS_TIMEOUT_MILLIS, ENV_OBS_ENDPOINT_METRICS_HEADERS, ENV_OBS_ENDPOINT_METRICS_TIMEOUT_MILLIS,
ENV_OBS_ENDPOINT_TIMEOUT_MILLIS, ENV_OBS_ENDPOINT_TRACES_HEADERS, ENV_OBS_ENDPOINT_TRACES_TIMEOUT_MILLIS,
ENV_OBS_ENVIRONMENT, ENV_OBS_LOG_CLEANUP_INTERVAL_SECONDS, ENV_OBS_LOG_COMPRESS_OLD_FILES,
ENV_OBS_LOG_COMPRESSED_FILE_RETENTION_DAYS, ENV_OBS_LOG_COMPRESSION_ALGORITHM, ENV_OBS_LOG_DELETE_EMPTY_FILES,
ENV_OBS_LOG_DIRECTORY, ENV_OBS_LOG_DRY_RUN, ENV_OBS_LOG_ENDPOINT, ENV_OBS_LOG_EXCLUDE_PATTERNS, ENV_OBS_LOG_FILENAME,
ENV_OBS_LOG_GZIP_COMPRESSION_LEVEL, ENV_OBS_LOG_KEEP_FILES, ENV_OBS_LOG_MATCH_MODE, ENV_OBS_LOG_MAX_SINGLE_FILE_SIZE_BYTES,
ENV_OBS_LOG_MAX_TOTAL_SIZE_BYTES, ENV_OBS_LOG_MIN_FILE_AGE_SECONDS, ENV_OBS_LOG_PARALLEL_COMPRESS,
ENV_OBS_LOG_PARALLEL_WORKERS, ENV_OBS_LOG_ROTATION_TIME, ENV_OBS_LOG_STDOUT_ENABLED, ENV_OBS_LOG_ZSTD_COMPRESSION_LEVEL,
ENV_OBS_LOG_ZSTD_FALLBACK_TO_GZIP, ENV_OBS_LOG_ZSTD_WORKERS, ENV_OBS_LOGGER_LEVEL, ENV_OBS_LOGS_EXPORT_ENABLED,
ENV_OBS_METER_INTERVAL, ENV_OBS_METRIC_ENDPOINT, ENV_OBS_METRICS_EXPORT_ENABLED, ENV_OBS_PROFILING_ENDPOINT,
ENV_OBS_PROFILING_EXPORT_ENABLED, ENV_OBS_SAMPLE_RATIO, ENV_OBS_SERVICE_NAME, ENV_OBS_SERVICE_VERSION, ENV_OBS_TLS_CA_FILE,
ENV_OBS_TRACE_ENDPOINT, ENV_OBS_TRACES_EXPORT_ENABLED, ENV_OBS_USE_STDOUT,
};
use rustfs_config::{
APP_NAME, DEFAULT_LOG_KEEP_FILES, DEFAULT_LOG_LEVEL, DEFAULT_LOG_ROTATION_TIME, DEFAULT_OBS_LOG_FILENAME,
@@ -189,9 +188,13 @@ pub struct OtelConfig {
pub log_keep_files: Option<usize>,
// ── Log cleanup ───────────────────────────────────────────────────────────
/// Hard ceiling on the total size (bytes) of all log files (default: 2 GiB).
/// Byte budget for rolled log files, and separately for compressed
/// archives (default: 2 GiB); `0` means no budget. The active file is not
/// counted.
pub log_max_total_size_bytes: Option<u64>,
/// Per-file size ceiling (bytes); `0` means unlimited (default: `0`).
/// Per-file size ceiling (bytes). Unset or unparsable: the active file
/// rolls at 256 MiB (less under a small byte budget) and the cleaner
/// applies no per-file trim. `0`: no size roll and no trim.
pub log_max_single_file_size_bytes: Option<u64>,
/// Compress eligible files with gzip before deletion (default: `true`).
pub log_compress_old_files: Option<bool>,
@@ -344,10 +347,8 @@ impl OtelConfig {
log_keep_files,
// Log cleanup
log_max_total_size_bytes: Some(get_env_u64(ENV_OBS_LOG_MAX_TOTAL_SIZE_BYTES, DEFAULT_OBS_LOG_MAX_TOTAL_SIZE_BYTES)),
log_max_single_file_size_bytes: Some(get_env_u64(
ENV_OBS_LOG_MAX_SINGLE_FILE_SIZE_BYTES,
DEFAULT_OBS_LOG_MAX_SINGLE_FILE_SIZE_BYTES,
)),
// Unset stays `None`: the appender and the cleaner default differently.
log_max_single_file_size_bytes: get_env_opt_u64(ENV_OBS_LOG_MAX_SINGLE_FILE_SIZE_BYTES),
log_compress_old_files: Some(get_env_bool(ENV_OBS_LOG_COMPRESS_OLD_FILES, DEFAULT_OBS_LOG_COMPRESS_OLD_FILES)),
log_gzip_compression_level: Some(get_env_u64(
ENV_OBS_LOG_GZIP_COMPRESSION_LEVEL,
@@ -480,6 +481,30 @@ mod tests {
OtelConfig::extract_otel_config_from_env(None).profiling_export_enabled
}
#[test]
fn single_file_size_limit_distinguishes_unset_from_an_explicit_zero() {
with_profiling_env_lock(|| {
temp_env::with_var_unset(ENV_OBS_LOG_MAX_SINGLE_FILE_SIZE_BYTES, || {
let config = OtelConfig::extract_otel_config_from_env(None);
assert_eq!(config.log_max_single_file_size_bytes, None, "unset leaves the default to each consumer");
});
temp_env::with_var(ENV_OBS_LOG_MAX_SINGLE_FILE_SIZE_BYTES, Some("0"), || {
let config = OtelConfig::extract_otel_config_from_env(None);
assert_eq!(config.log_max_single_file_size_bytes, Some(0), "an explicit 0 is kept as unlimited");
});
for unparsable in ["", "256MiB"] {
temp_env::with_var(ENV_OBS_LOG_MAX_SINGLE_FILE_SIZE_BYTES, Some(unparsable), || {
let config = OtelConfig::extract_otel_config_from_env(None);
assert_eq!(config.log_max_single_file_size_bytes, None, "{unparsable:?} is treated as unset");
});
}
temp_env::with_var(ENV_OBS_LOG_MAX_SINGLE_FILE_SIZE_BYTES, Some("1048576"), || {
let config = OtelConfig::extract_otel_config_from_env(None);
assert_eq!(config.log_max_single_file_size_bytes, Some(1_048_576));
});
});
}
#[test]
fn service_version_uses_application_version_when_environment_is_unset() {
with_profiling_env_lock(|| {
+227 -19
View File
@@ -40,12 +40,13 @@ use crate::telemetry::filter::{build_env_filter, effective_verbose_logging, pyro
use crate::telemetry::rolling::{RollingAppender, Rotation};
use metrics::counter;
use rustfs_config::observability::{
DEFAULT_OBS_LOG_CLEANUP_INTERVAL_SECONDS, DEFAULT_OBS_LOG_COMPRESS_OLD_FILES, DEFAULT_OBS_LOG_COMPRESSED_FILE_RETENTION_DAYS,
DEFAULT_OBS_LOG_COMPRESSION_ALGORITHM, DEFAULT_OBS_LOG_DELETE_EMPTY_FILES, DEFAULT_OBS_LOG_DRY_RUN,
DEFAULT_OBS_LOG_GZIP_COMPRESSION_LEVEL, DEFAULT_OBS_LOG_MATCH_MODE, DEFAULT_OBS_LOG_MAX_SINGLE_FILE_SIZE_BYTES,
DEFAULT_OBS_LOG_MAX_TOTAL_SIZE_BYTES, DEFAULT_OBS_LOG_MIN_FILE_AGE_SECONDS, DEFAULT_OBS_LOG_PARALLEL_COMPRESS,
DEFAULT_OBS_LOG_PARALLEL_WORKERS, DEFAULT_OBS_LOG_SPAN_EVENTS, DEFAULT_OBS_LOG_ZSTD_COMPRESSION_LEVEL,
DEFAULT_OBS_LOG_ZSTD_FALLBACK_TO_GZIP, DEFAULT_OBS_LOG_ZSTD_WORKERS, ENV_OBS_LOG_SPAN_EVENTS,
DEFAULT_OBS_LOG_ACTIVE_FILE_ROLL_SIZE_BYTES, DEFAULT_OBS_LOG_CLEANUP_INTERVAL_SECONDS, DEFAULT_OBS_LOG_COMPRESS_OLD_FILES,
DEFAULT_OBS_LOG_COMPRESSED_FILE_RETENTION_DAYS, DEFAULT_OBS_LOG_COMPRESSION_ALGORITHM, DEFAULT_OBS_LOG_DELETE_EMPTY_FILES,
DEFAULT_OBS_LOG_DRY_RUN, DEFAULT_OBS_LOG_GZIP_COMPRESSION_LEVEL, DEFAULT_OBS_LOG_MATCH_MODE,
DEFAULT_OBS_LOG_MAX_SINGLE_FILE_SIZE_BYTES, DEFAULT_OBS_LOG_MAX_TOTAL_SIZE_BYTES, DEFAULT_OBS_LOG_MIN_FILE_AGE_SECONDS,
DEFAULT_OBS_LOG_PARALLEL_COMPRESS, DEFAULT_OBS_LOG_PARALLEL_WORKERS, DEFAULT_OBS_LOG_SPAN_EVENTS,
DEFAULT_OBS_LOG_ZSTD_COMPRESSION_LEVEL, DEFAULT_OBS_LOG_ZSTD_FALLBACK_TO_GZIP, DEFAULT_OBS_LOG_ZSTD_WORKERS,
ENV_OBS_LOG_SPAN_EVENTS,
};
use rustfs_config::{APP_NAME, DEFAULT_LOG_KEEP_FILES, DEFAULT_LOG_ROTATION_TIME};
use rustfs_utils::get_env_str;
@@ -159,6 +160,58 @@ fn validate_distinct_file_identities(
Ok(())
}
/// Smallest roll size derived from a small byte budget.
const MIN_DERIVED_ROLL_SIZE_BYTES: u64 = 1024 * 1024;
/// Size at which the active log file is rolled. An explicit limit is used as
/// is, and an explicit `0` keeps size rolling off. An unset limit rolls at the
/// shipped default so the active file is bounded, lowered to a quarter of the
/// byte budget when the budget is small: a rolled file larger than the whole
/// budget would be removed by the pass its own roll starts.
pub(crate) fn resolve_active_file_roll_size(config: &OtelConfig) -> u64 {
if let Some(configured) = config.log_max_single_file_size_bytes {
return configured;
}
match config
.log_max_total_size_bytes
.unwrap_or(DEFAULT_OBS_LOG_MAX_TOTAL_SIZE_BYTES)
{
0 => DEFAULT_OBS_LOG_ACTIVE_FILE_ROLL_SIZE_BYTES,
budget => DEFAULT_OBS_LOG_ACTIVE_FILE_ROLL_SIZE_BYTES.min((budget / 4).max(MIN_DERIVED_ROLL_SIZE_BYTES)),
}
}
/// Per-file limit above which the cleaner processes a rolled file. Unset stays
/// off: the appender's default roll size is not a reason to trim files that an
/// earlier release rolled by time only.
pub(crate) fn resolve_cleanup_single_file_limit(config: &OtelConfig) -> u64 {
config
.log_max_single_file_size_bytes
.unwrap_or(DEFAULT_OBS_LOG_MAX_SINGLE_FILE_SIZE_BYTES)
}
/// Open the rolling appender for the configured log file together with the
/// signal it raises after every roll. The caller hands that signal to
/// [`spawn_cleanup_task`].
pub(crate) fn open_rolling_appender(
config: &OtelConfig,
log_directory: &str,
log_filename: &str,
rotation: Rotation,
match_mode: FileMatchMode,
) -> std::io::Result<(RollingAppender, Arc<tokio::sync::Notify>)> {
let roll_signal = Arc::new(tokio::sync::Notify::new());
let appender = RollingAppender::new(
log_directory,
log_filename.to_string(),
rotation,
resolve_active_file_roll_size(config),
match_mode,
)?
.with_roll_signal(roll_signal.clone());
Ok((appender, roll_signal))
}
fn resolve_log_cleanup_interval_seconds(config: &OtelConfig) -> u64 {
match config.log_cleanup_interval_seconds {
Some(0) => {
@@ -451,12 +504,7 @@ fn init_file_logging_internal(
_ => Rotation::Daily,
};
let max_single_file_size = config
.log_max_single_file_size_bytes
.unwrap_or(DEFAULT_OBS_LOG_MAX_SINGLE_FILE_SIZE_BYTES);
let file_appender =
RollingAppender::new(log_directory, log_filename.to_string(), rotation, max_single_file_size, match_mode)?;
let (file_appender, roll_signal) = open_rolling_appender(config, log_directory, log_filename, rotation, match_mode)?;
validate_stdout_sink(&file_appender)?;
let (non_blocking, guard) = tracing_appender::non_blocking(file_appender);
@@ -497,7 +545,7 @@ fn init_file_logging_internal(
set_observability_metric_enabled(false);
// ── 5. Start background cleanup task ─────────────────────────────────────
let cleanup_handle = spawn_cleanup_task(config, log_directory, log_filename, keep_files);
let cleanup_handle = spawn_cleanup_task(config, log_directory, log_filename, keep_files, roll_signal);
info!(
event = EVENT_LOCAL_LOGGING_STATE,
@@ -606,6 +654,10 @@ pub(super) fn emit_file_logging_fallback_warning(log_directory: &str, error: &Te
// ─── Cleanup task ─────────────────────────────────────────────────────────────
/// Shortest pause between two cleanup passes started by rolls. A pass that the
/// timer starts is never delayed by it.
const ROLL_TRIGGERED_CLEANUP_SPACING: Duration = Duration::from_secs(1);
/// Spawn a background task that periodically cleans up old log files.
///
/// All cleanup parameters are derived from [`OtelConfig`] fields, with
@@ -623,6 +675,9 @@ pub(super) fn emit_file_logging_fallback_warning(log_directory: &str, error: &Te
/// * `log_filename` - Base filename (used as the file prefix for matching).
/// * `keep_files` - Legacy keep-files count; used as fallback when the new
/// `log_keep_count` field is absent.
/// * `roll_signal` - Notified by the appender after every roll. The loop then
/// runs a pass right away, so the byte budget is enforced when a file is
/// rolled instead of up to one cleanup interval later.
///
/// # Returns
/// A [`tokio::task::JoinHandle`] for the spawned cleanup loop.
@@ -631,6 +686,7 @@ pub fn spawn_cleanup_task(
log_directory: &str,
log_filename: &str,
keep_files: usize,
roll_signal: Arc<tokio::sync::Notify>,
) -> tokio::task::JoinHandle<()> {
let log_dir = std::path::PathBuf::from(log_directory);
// Use suffix matching for log files like `2026-03-01-06-21.rustfs.log`
@@ -645,9 +701,7 @@ pub fn spawn_cleanup_task(
let max_total_size = config
.log_max_total_size_bytes
.unwrap_or(DEFAULT_OBS_LOG_MAX_TOTAL_SIZE_BYTES);
let max_single_file_size = config
.log_max_single_file_size_bytes
.unwrap_or(DEFAULT_OBS_LOG_MAX_SINGLE_FILE_SIZE_BYTES);
let max_single_file_size = resolve_cleanup_single_file_limit(config);
let compress = config.log_compress_old_files.unwrap_or(DEFAULT_OBS_LOG_COMPRESS_OLD_FILES);
let gzip_level = config
.log_gzip_compression_level
@@ -740,9 +794,16 @@ pub fn spawn_cleanup_task(
tokio::spawn(async move {
let mut interval = tokio::time::interval(Duration::from_secs(cleanup_interval));
loop {
// Wait for the next scheduled tick before dispatching another pass.
// The blocking filesystem work runs on a dedicated blocking thread.
interval.tick().await;
// Wait for the next scheduled tick, or for the appender to roll a
// file, before dispatching another pass. The blocking filesystem
// work runs on a dedicated blocking thread.
tokio::select! {
_ = interval.tick() => {}
_ = roll_signal.notified() => {
// Rolls can come in bursts; one pass covers all of them.
tokio::time::sleep(ROLL_TRIGGERED_CLEANUP_SPACING).await;
}
}
let cleaner_clone = cleaner.clone();
let result = tokio::task::spawn_blocking(move || cleaner_clone.cleanup()).await;
@@ -1253,6 +1314,153 @@ mod tests {
assert_eq!(resolve_log_cleanup_interval_seconds(&config), DEFAULT_OBS_LOG_CLEANUP_INTERVAL_SECONDS);
}
#[test]
fn active_file_rolls_by_size_unless_explicitly_disabled() {
let unset = OtelConfig {
log_max_single_file_size_bytes: None,
log_max_total_size_bytes: None,
..OtelConfig::default()
};
assert_eq!(resolve_active_file_roll_size(&unset), DEFAULT_OBS_LOG_ACTIVE_FILE_ROLL_SIZE_BYTES);
assert_eq!(DEFAULT_OBS_LOG_ACTIVE_FILE_ROLL_SIZE_BYTES, 256 * 1024 * 1024);
assert_eq!(
resolve_cleanup_single_file_limit(&unset),
0,
"the appender's default roll size is not a per-file trim for the cleaner"
);
let disabled = OtelConfig {
log_max_single_file_size_bytes: Some(0),
..OtelConfig::default()
};
assert_eq!(resolve_active_file_roll_size(&disabled), 0, "an explicit 0 keeps size rolling off");
assert_eq!(resolve_cleanup_single_file_limit(&disabled), 0);
let configured = OtelConfig {
log_max_single_file_size_bytes: Some(64 * 1024 * 1024),
log_max_total_size_bytes: Some(1024),
..OtelConfig::default()
};
assert_eq!(
resolve_active_file_roll_size(&configured),
64 * 1024 * 1024,
"an explicit limit is never derived"
);
assert_eq!(resolve_cleanup_single_file_limit(&configured), 64 * 1024 * 1024);
}
#[test]
fn default_roll_size_follows_a_small_byte_budget() {
let with_budget = |budget: u64| OtelConfig {
log_max_single_file_size_bytes: None,
log_max_total_size_bytes: Some(budget),
..OtelConfig::default()
};
const MIB: u64 = 1024 * 1024;
assert_eq!(
resolve_active_file_roll_size(&with_budget(2048 * MIB)),
256 * MIB,
"the default budget keeps 256 MiB"
);
assert_eq!(resolve_active_file_roll_size(&with_budget(1024 * MIB)), 256 * MIB);
assert_eq!(
resolve_active_file_roll_size(&with_budget(256 * MIB)),
64 * MIB,
"a quarter of a small budget"
);
assert_eq!(resolve_active_file_roll_size(&with_budget(MIB)), MIB, "never below 1 MiB");
assert_eq!(resolve_active_file_roll_size(&with_budget(0)), 256 * MIB, "no budget, no derivation");
}
/// Both logging setups open the appender through this function, so the
/// cleanup task they start is always the one the appender wakes.
#[test]
fn opened_appender_reports_its_rolls_on_the_returned_signal() {
use std::io::Write as _;
let tmp = tempfile::tempdir().expect("log directory");
let config = OtelConfig {
log_max_single_file_size_bytes: Some(8),
..OtelConfig::default()
};
let (mut appender, roll_signal) = open_rolling_appender(
&config,
tmp.path().to_str().expect("UTF-8 path"),
"rustfs.log",
Rotation::Daily,
FileMatchMode::Suffix,
)
.expect("appender");
let runtime = tokio::runtime::Builder::new_current_thread()
.enable_time()
.build()
.expect("runtime");
let reported = || runtime.block_on(async { tokio::time::timeout(Duration::ZERO, roll_signal.notified()).await.is_ok() });
appender.write_all(b"12345678").expect("first write");
assert!(!reported());
appender.write_all(b"9").expect("write that rolls at the configured size");
assert!(reported(), "the configured roll size is in effect and the roll reaches the signal");
}
/// The cleanup loop runs a pass when the appender reports a roll, long
/// before its interval elapses, and that pass enforces the byte budget on
/// files younger than the age gate.
#[tokio::test]
async fn cleanup_loop_enforces_the_byte_budget_when_a_roll_is_reported() {
let tmp = tempfile::tempdir().expect("log directory");
let dir = tmp.path();
let config = OtelConfig {
log_max_total_size_bytes: Some(2 * 1024),
log_max_single_file_size_bytes: Some(0),
log_compress_old_files: Some(false),
log_min_file_age_seconds: Some(3600),
log_cleanup_interval_seconds: Some(3600),
log_match_mode: Some("suffix".to_string()),
log_filename: Some("rustfs.log".to_string()),
..OtelConfig::default()
};
let roll_signal = Arc::new(tokio::sync::Notify::new());
let rolled_files = || {
fs::read_dir(dir)
.expect("log directory")
.filter_map(Result::ok)
.filter(|entry| entry.file_name() != "rustfs.log")
.count()
};
let write_rolled = |range: std::ops::Range<u32>| {
for index in range {
fs::write(dir.join(format!("2026-01-01-00-{index:02}.rustfs.log")), vec![b'x'; 1024]).expect("rolled file");
}
};
let wait_for_two_rolled_files = |what: &'static str| async move {
tokio::time::timeout(Duration::from_secs(10), async {
while rolled_files() > 2 {
tokio::time::sleep(Duration::from_millis(50)).await;
}
})
.await
.unwrap_or_else(|_| panic!("{what}"));
};
// The pass the interval starts at once is observed through its effect,
// so what follows cannot be that pass arriving late.
fs::write(dir.join("rustfs.log"), vec![b'a'; 1024]).expect("active file");
write_rolled(0..6);
let cleanup = spawn_cleanup_task(&config, dir.to_str().expect("UTF-8 path"), "rustfs.log", 30, roll_signal.clone());
wait_for_two_rolled_files("the startup pass enforces the budget").await;
write_rolled(10..14);
tokio::time::sleep(Duration::from_millis(300)).await;
assert_eq!(rolled_files(), 6, "without a roll report nothing runs before the next interval");
roll_signal.notify_one();
wait_for_two_rolled_files("the reported roll starts a pass that brings the rolled files back under the budget").await;
assert_eq!(rolled_files(), 2);
assert!(dir.join("rustfs.log").exists(), "the active file is never removed");
cleanup.abort();
}
#[test]
fn empty_cleanup_pattern_falls_back_to_runtime_log_filename() {
let config = OtelConfig {
+5 -10
View File
@@ -41,10 +41,10 @@ use crate::config::OtelConfig;
use crate::global::set_observability_metric_enabled;
use crate::telemetry::filter::{build_env_filter, pyroscope_log_filter, s3s_log_filter};
use crate::telemetry::guard::{OtelGuard, ProfilingAgent};
use crate::telemetry::local::{build_json_log_layer, spawn_cleanup_task};
use crate::telemetry::local::{build_json_log_layer, open_rolling_appender, spawn_cleanup_task};
use crate::telemetry::recorder::{Recorder, install_process_global_recorder};
use crate::telemetry::resource::build_resource;
use crate::telemetry::rolling::{RollingAppender, Rotation};
use crate::telemetry::rolling::Rotation;
// Import helper functions from local.rs (sibling module)
use crate::TelemetryError;
use metrics::counter;
@@ -58,7 +58,7 @@ use opentelemetry_sdk::{
trace::{RandomIdGenerator, Sampler, SdkTracerProvider},
};
use percent_encoding::percent_decode_str;
use rustfs_config::observability::{DEFAULT_OBS_LOG_MATCH_MODE, DEFAULT_OBS_LOG_MAX_SINGLE_FILE_SIZE_BYTES};
use rustfs_config::observability::DEFAULT_OBS_LOG_MATCH_MODE;
use rustfs_config::{
APP_NAME, DEFAULT_LOG_KEEP_FILES, DEFAULT_LOG_ROTATION_TIME, DEFAULT_OBS_LOGS_EXPORT_ENABLED,
DEFAULT_OBS_METRICS_EXPORT_ENABLED, DEFAULT_OBS_TRACES_EXPORT_ENABLED, METER_INTERVAL, SAMPLE_RATIO,
@@ -244,17 +244,12 @@ pub(super) fn init_observability_http(
"daily" => Rotation::Daily,
_ => Rotation::Daily,
};
let max_single_file_size = config
.log_max_single_file_size_bytes
.unwrap_or(DEFAULT_OBS_LOG_MAX_SINGLE_FILE_SIZE_BYTES);
let file_appender =
RollingAppender::new(log_directory, log_filename.to_string(), rotation, max_single_file_size, match_mode)?;
let (file_appender, roll_signal) = open_rolling_appender(config, log_directory, log_filename, rotation, match_mode)?;
crate::telemetry::local::validate_stdout_sink(&file_appender)?;
let (non_blocking, guard) = tracing_appender::non_blocking(file_appender);
let file_layer = build_json_log_layer(non_blocking, false, span_events.clone(), span_list);
let cleanup_handle = spawn_cleanup_task(config, log_directory, log_filename, keep_files);
let cleanup_handle = spawn_cleanup_task(config, log_directory, log_filename, keep_files, roll_signal);
Ok((file_layer, guard, cleanup_handle, rotation_str))
})();
+88
View File
@@ -29,6 +29,7 @@ use metrics::{counter, gauge, histogram};
use std::fs::{self, File};
use std::io::{self, Write};
use std::path::{Path, PathBuf};
use std::sync::Arc;
use std::sync::atomic::{AtomicU64, Ordering};
use std::thread;
use std::time::{Duration, Instant};
@@ -82,6 +83,9 @@ pub struct RollingAppender {
// Store as seconds since Unix epoch
last_roll_ts: i64,
rotation_retry_cooldown_until: Option<Instant>,
/// Woken after every successful roll so the cleaner can enforce the
/// directory budget on the file that was just rolled.
roll_signal: Option<Arc<tokio::sync::Notify>>,
}
impl RollingAppender {
@@ -129,6 +133,7 @@ impl RollingAppender {
size: 0,
last_roll_ts: Zoned::now().timestamp().as_second(),
rotation_retry_cooldown_until: None,
roll_signal: None,
};
// Eagerly open the file to validate the path and capture accurate
// initial size / last-roll timestamp.
@@ -136,6 +141,12 @@ impl RollingAppender {
Ok(appender)
}
/// Notify `signal` after every successful roll.
pub(crate) fn with_roll_signal(mut self, signal: Arc<tokio::sync::Notify>) -> Self {
self.roll_signal = Some(signal);
self
}
pub(crate) fn active_file_path(&self) -> PathBuf {
self.dir.join(&self.filename)
}
@@ -289,6 +300,9 @@ impl RollingAppender {
counter!(METRIC_LOG_CLEANER_ROTATION_TOTAL).increment(1);
histogram!(METRIC_LOG_CLEANER_ROTATION_DURATION_SECONDS).record(rotate_started.elapsed().as_secs_f64());
gauge!(METRIC_LOG_CLEANER_ACTIVE_FILE_SIZE_BYTES).set(self.size as f64);
if let Some(signal) = &self.roll_signal {
signal.notify_one();
}
return Ok(());
}
Err(e) => {
@@ -452,6 +466,80 @@ mod tests {
// ── Size-based rotation ────────────────────────────────────────────────────
/// A storm with the shipped roll behavior: the appender rolls by size and
/// wakes the cleaner, and the directory stays within the byte budget plus
/// the active file although every rolled file is younger than the age gate.
#[test]
fn log_directory_stays_bounded_under_a_storm() -> io::Result<()> {
use crate::cleaner::LogCleaner;
const ROLL_SIZE: u64 = 4 * 1024;
const BUDGET: u64 = 16 * 1024;
let tmp = TempDir::new()?;
let dir = tmp.path().to_path_buf();
let signal = Arc::new(tokio::sync::Notify::new());
let mut appender =
RollingAppender::new(&dir, "rustfs.log".to_string(), Rotation::Never, ROLL_SIZE, FileMatchMode::Suffix)?
.with_roll_signal(signal.clone());
let cleaner = LogCleaner::builder(dir.clone(), "rustfs.log".to_string(), "rustfs.log".to_string())
.match_mode(FileMatchMode::Suffix)
.keep_files(30)
.max_total_size_bytes(BUDGET)
.compress_old_files(false)
.min_file_age_seconds(3600)
.build();
let directory_bytes = |dir: &Path| -> io::Result<u64> {
let mut total = 0;
for entry in fs::read_dir(dir)? {
total += entry?.metadata()?.len();
}
Ok(total)
};
let runtime = tokio::runtime::Builder::new_current_thread().enable_time().build()?;
let line = [b'x'; 512];
let mut passes = 0;
let mut peak = 0;
// 512 KiB of log lines, 32 times the budget.
for _ in 0..1024 {
appender.write_all(&line)?;
// The cleanup loop runs one pass per reported roll.
let rolled = runtime.block_on(async { tokio::time::timeout(Duration::ZERO, signal.notified()).await.is_ok() });
if rolled {
cleaner.cleanup()?;
passes += 1;
}
peak = peak.max(directory_bytes(&dir)?);
}
appender.flush()?;
assert!(passes > 100, "size rolls woke the cleaner {passes} times");
// One freshly rolled file may sit above the budget until its pass runs.
assert!(
peak <= BUDGET + 2 * ROLL_SIZE,
"peak directory size {peak} exceeded the budget {BUDGET} plus the active and one rolled file"
);
assert!(directory_bytes(&dir)? <= BUDGET + ROLL_SIZE);
Ok(())
}
#[test]
fn every_successful_roll_wakes_the_roll_signal() {
let tmp = TempDir::new().unwrap();
let signal = Arc::new(tokio::sync::Notify::new());
let mut appender = RollingAppender::new(tmp.path(), "app.log".to_string(), Rotation::Never, 5, FileMatchMode::Suffix)
.unwrap()
.with_roll_signal(signal.clone());
let runtime = tokio::runtime::Builder::new_current_thread().enable_time().build().unwrap();
let woken = || runtime.block_on(async { tokio::time::timeout(Duration::ZERO, signal.notified()).await.is_ok() });
appender.write_all(b"12345").expect("write should succeed");
assert!(!woken(), "no roll yet");
appender.write_all(b"abcde").expect("write after rotation should succeed");
assert!(woken(), "the size roll is reported");
assert!(!woken(), "one roll is one wake-up");
}
#[test]
fn test_size_rotation_creates_archive() {
let tmp = TempDir::new().unwrap();
+1
View File
@@ -642,6 +642,7 @@ impl AdminAction {
self,
AdminAction::GetTableBucketAction
| AdminAction::SetTableBucketAction
| AdminAction::MigrateTableCatalogAction
| AdminAction::GetTableNamespaceAction
| AdminAction::SetTableNamespaceAction
| AdminAction::UpdateTableNamespacePropertiesAction
+140
View File
@@ -1795,6 +1795,146 @@ mod test {
Ok(())
}
async fn assert_catalog_migration_policy_matrix(data: &[u8], cases: &[(&str, AdminAction, bool)]) -> Result<()> {
let policy = Policy::parse_config(data)?;
let conditions = HashMap::new();
let claims = HashMap::new();
let groups = None;
for (bucket, action, expected) in cases {
let args = Args {
account: "catalog-migration-operator",
groups: &groups,
action: Action::AdminAction(*action),
bucket,
conditions: &conditions,
is_owner: false,
object: "",
claims: &claims,
deny_only: false,
};
assert_eq!(policy.is_allowed(&args).await, *expected, "{action:?} on warehouse {bucket:?}");
}
Ok(())
}
#[tokio::test]
async fn catalog_migration_allow_resource_is_limited_to_the_requested_warehouse() -> Result<()> {
assert_catalog_migration_policy_matrix(
br#"{
"Version": "2012-10-17",
"Statement": [{
"Effect": "Allow",
"Action": "admin:MigrateTableCatalog",
"Resource": "arn:aws:s3:::warehouse-a"
}]
}"#,
&[
("warehouse-a", AdminAction::MigrateTableCatalogAction, true),
("warehouse-b", AdminAction::MigrateTableCatalogAction, false),
("", AdminAction::MigrateTableCatalogAction, false),
],
)
.await
}
#[tokio::test]
async fn catalog_migration_allow_not_resource_excludes_only_the_named_warehouse() -> Result<()> {
assert_catalog_migration_policy_matrix(
br#"{
"Version": "2012-10-17",
"Statement": [{
"Effect": "Allow",
"Action": "admin:MigrateTableCatalog",
"NotResource": "arn:aws:s3:::warehouse-b"
}]
}"#,
&[
("warehouse-a", AdminAction::MigrateTableCatalogAction, true),
("warehouse-b", AdminAction::MigrateTableCatalogAction, false),
("warehouse-c", AdminAction::MigrateTableCatalogAction, true),
],
)
.await
}
#[tokio::test]
async fn catalog_migration_explicit_deny_resource_does_not_spread_to_other_warehouses() -> Result<()> {
assert_catalog_migration_policy_matrix(
br#"{
"Version": "2012-10-17",
"Statement": [
{"Effect": "Allow", "Action": "admin:*", "Resource": "*"},
{"Effect": "Deny", "Action": "admin:MigrateTableCatalog", "Resource": "arn:aws:s3:::warehouse-b"}
]
}"#,
&[
("warehouse-a", AdminAction::MigrateTableCatalogAction, true),
("warehouse-b", AdminAction::MigrateTableCatalogAction, false),
("warehouse-c", AdminAction::MigrateTableCatalogAction, true),
("warehouse-b", AdminAction::ServerInfoAdminAction, true),
],
)
.await
}
#[tokio::test]
async fn catalog_migration_explicit_deny_not_resource_preserves_its_exception() -> Result<()> {
assert_catalog_migration_policy_matrix(
br#"{
"Version": "2012-10-17",
"Statement": [
{"Effect": "Allow", "Action": "admin:*", "Resource": "*"},
{"Effect": "Deny", "Action": "admin:MigrateTableCatalog", "NotResource": "arn:aws:s3:::warehouse-a"}
]
}"#,
&[
("warehouse-a", AdminAction::MigrateTableCatalogAction, true),
("warehouse-b", AdminAction::MigrateTableCatalogAction, false),
("warehouse-c", AdminAction::MigrateTableCatalogAction, false),
("warehouse-b", AdminAction::ServerInfoAdminAction, true),
],
)
.await
}
#[tokio::test]
async fn catalog_migration_admin_wildcard_still_honors_the_requested_action_scope() -> Result<()> {
assert_catalog_migration_policy_matrix(
br#"{
"Version": "2012-10-17",
"Statement": [{
"Effect": "Allow",
"Action": "admin:*",
"Resource": "arn:aws:s3:::warehouse-a"
}]
}"#,
&[
("warehouse-a", AdminAction::MigrateTableCatalogAction, true),
("warehouse-b", AdminAction::MigrateTableCatalogAction, false),
("", AdminAction::MigrateTableCatalogAction, false),
("warehouse-b", AdminAction::ServerInfoAdminAction, true),
("", AdminAction::GetTableCatalogAction, true),
],
)
.await
}
#[tokio::test]
async fn catalog_migration_resource_free_admin_grants_keep_their_global_behavior() -> Result<()> {
assert_catalog_migration_policy_matrix(
br#"{
"Version": "2012-10-17",
"Statement": [{"Effect": "Allow", "Action": "admin:MigrateTableCatalog"}]
}"#,
&[
("warehouse-a", AdminAction::MigrateTableCatalogAction, true),
("warehouse-b", AdminAction::MigrateTableCatalogAction, true),
("", AdminAction::MigrateTableCatalogAction, true),
],
)
.await
}
#[tokio::test]
async fn test_table_admin_action_with_resource_is_limited_to_bucket() -> Result<()> {
use crate::policy::action::{Action, AdminAction};
+119
View File
@@ -1873,4 +1873,123 @@ mod tests {
assert_eq!(err.kind(), std::io::ErrorKind::InvalidData);
assert!(err.to_string().contains("invalid checksum value"), "unexpected error: {err}");
}
fn software_crc32_bitwise(reflected_poly: u32, data: &[u8]) -> [u8; 4] {
let mut crc = !0u32;
for &b in data {
crc ^= u32::from(b);
for _ in 0..8 {
let mask = (crc & 1).wrapping_neg();
crc = (crc >> 1) ^ (reflected_poly & mask);
}
}
(!crc).to_be_bytes()
}
fn software_crc64_nvme_bitwise(data: &[u8]) -> [u8; 8] {
let reflected_poly = super::CRC64_NVME_POLYNOMIAL.reverse_bits();
let mut crc = !0u64;
for &b in data {
crc ^= u64::from(b);
for _ in 0..8 {
let mask = (crc & 1).wrapping_neg();
crc = (crc >> 1) ^ (reflected_poly & mask);
}
}
(!crc).to_be_bytes()
}
#[test]
fn rio_crc_simd_matches_bitwise_scalar_across_tail_and_vector_boundaries() {
for &size in &[
0usize,
1,
7,
8,
15,
16,
17,
31,
32,
33,
63,
64,
65,
127,
128,
129,
255,
256,
257,
1024,
4096 + 17,
64 * 1024,
] {
let payload: Vec<u8> = (0..size).map(|i| ((i * 31 + 17) % 251) as u8).collect();
let crc32_simd = Checksum::new_from_data(ChecksumType::CRC32, &payload).expect("crc32");
let crc32c_simd = Checksum::new_from_data(ChecksumType::CRC32C, &payload).expect("crc32c");
let crc64_simd = Checksum::new_from_data(ChecksumType::CRC64_NVME, &payload).expect("crc64nvme");
let crc32_sw = software_crc32_bitwise(0xEDB88320, &payload);
let crc32c_sw = software_crc32_bitwise(0x82F63B78, &payload);
let crc64_sw = software_crc64_nvme_bitwise(&payload);
assert_eq!(crc32_simd.raw.as_slice(), &crc32_sw, "CRC32 SIMD vs scalar mismatch at {size}B");
assert_eq!(crc32c_simd.raw.as_slice(), &crc32c_sw, "CRC32C SIMD vs scalar mismatch at {size}B");
assert_eq!(crc64_simd.raw.as_slice(), &crc64_sw, "CRC64_NVME SIMD vs scalar mismatch at {size}B");
}
}
#[test]
#[ignore = "utility throughput microbenchmark; run via cargo test -p rustfs-rio --lib --release -- --ignored --nocapture"]
fn microbench_rio_checksum_simd_vs_scalar_and_all_base_algorithms() {
use std::hint::black_box;
use std::io::Write as _;
use std::time::Instant;
for &(size, iters) in &[(1024usize, 100usize), (64 * 1024, 20), (1024 * 1024, 4)] {
let payload: Vec<u8> = (0..size).map(|i| ((i * 31 + 17) % 251) as u8).collect();
let total_mib = (size * iters) as f64 / (1024.0 * 1024.0);
// Verify SIMD-accelerated CRC32 / CRC32C / CRC64_NVME against bitwise scalar reference at 1 KiB & 64 KiB.
if size <= 64 * 1024 {
let crc32_simd = Checksum::new_from_data(ChecksumType::CRC32, &payload).expect("crc32");
let crc32c_simd = Checksum::new_from_data(ChecksumType::CRC32C, &payload).expect("crc32c");
let crc64_simd = Checksum::new_from_data(ChecksumType::CRC64_NVME, &payload).expect("crc64nvme");
let t_sw = Instant::now();
let crc32_sw = software_crc32_bitwise(0xEDB88320, black_box(&payload));
let crc32c_sw = software_crc32_bitwise(0x82F63B78, black_box(&payload));
let crc64_sw = software_crc64_nvme_bitwise(black_box(&payload));
let sw_elapsed = t_sw.elapsed();
assert_eq!(crc32_simd.raw.as_slice(), &crc32_sw, "CRC32 SIMD vs scalar mismatch at {size}B");
assert_eq!(crc32c_simd.raw.as_slice(), &crc32c_sw, "CRC32C SIMD vs scalar mismatch at {size}B");
assert_eq!(crc64_simd.raw.as_slice(), &crc64_sw, "CRC64_NVME SIMD vs scalar mismatch at {size}B");
println!(
"[microbench][rio::checksum] size={size}B bitwise scalar reference check passed in {:?}",
sw_elapsed
);
}
for &checksum_type in super::BASE_CHECKSUM_TYPES {
let mut hasher = checksum_type.hasher().expect("base checksum type must have hasher");
let t0 = Instant::now();
let mut digest = Vec::new();
for _ in 0..iters {
hasher.reset();
hasher.write_all(black_box(&payload)).expect("hasher write_all");
digest = hasher.finalize();
black_box(&digest);
}
let elapsed = t0.elapsed();
assert_eq!(digest.len(), checksum_type.raw_byte_len());
let mibs = total_mib / elapsed.as_secs_f64().max(1e-9);
println!(
"[microbench][rio::checksum] size={size}B algo={} throughput={mibs:.1} MiB/s",
checksum_type
);
}
}
}
}

Some files were not shown because too many files have changed in this diff Show More