Compare commits

...

107 Commits

Author SHA1 Message Date
overtrue 23bffc4ca0 fix(obs): trim only what the log byte budget needs
Lifting the minimum file age for an over-budget pass merged fresh files into the regular selection, so the keep_files ceiling and the per-file limit applied to them too: one byte over the budget could process every fresh file but keep_files of them. The override is now a byte trim on the files that would remain, oldest first, until the budget holds.

An unset single-file limit now rolls at a quarter of a small byte budget instead of 256 MiB, so a freshly rolled file is not larger than the whole budget. Both logging setups open the appender through one function that returns the roll signal the cleanup task listens on. The README states that a log directory must not be shared by processes whose file names match the same pattern.
2026-11-11 16:25:59 +08:00
overtrue f2655911db fix(obs): bound the log directory with default settings
With the defaults the active log file had no size limit (size rolling was off), and the cleaner counted only rolled files older than the minimum file age against the byte budget, on a 30-minute interval. A log storm could therefore fill the disk for up to two hours without the budget applying.

The active file now rolls at 256 MiB when the single-file limit is unset; an explicit 0 still turns size rolling off, and the cleaner's per-file trim keeps its previous default. The appender wakes the cleaner after every roll, and once the rolled files exceed the byte budget the cleaner processes the oldest ones regardless of age. The README no longer calls the budget a hard cap on the directory and states the actual bound.
2026-11-11 15:30:11 +08:00
Chris An 0e402b2534 fix(scanner): retain child objects beneath object prefixes (#9277) 2026-11-11 14:36:53 +08:00
Chris An db39cd5634 fix(ecstore): let missing-key deletes complete on UUID-named folders (#9280)
* fix(ecstore): let missing-key deletes complete on UUID-named folders

Deleting a key that does not exist probes the key's directory for
interrupted delete transactions. The probe was bounded at 4096 entries
per drive and charged every entry inside a UUID-named child, so a key
naming a folder of roughly 1,400 or more UUID-named child keys answered
503 SlowDown on every retry.

Inside a UUID-named child only journal files and data-directory markers
can be evidence, so every other name (a child key's own xl.meta, data
directories or children) is now skipped unread at 1/64 like plain
children of the folder. The per-call allowance of the bounded protocol
rises to 262,144 entries per drive; the resumable background step keeps
4096 and the request budget keeps its sizing, so the per-request total
is unchanged. The probe runs on the durable-mutation lane instead of
the one-worker foreground lane, and stops incomplete when its caller
goes away or after a third of the drive timeout (at most 10 s), so a
slow walk answers SlowDown instead of timing out the drive.

* test(ecstore): keep discovery fixtures meaningful under 1/64 charging

Part files inside a data directory are inert to object-scoped discovery
now, so reclaim_orphan_data_dirs_drains_empty_discovery_pages fills its
first step with the referenced data directories of many versions, which
still cost a full entry each. The charging test drives the walk directly
instead of through the wall-clock-limited disk entry point, so a loaded
machine cannot turn it into an incomplete probe.
2026-11-10 23:24:50 -05:00
Ramakrishna Chilaka 89d01aef4f perf(ecstore): reduce syscalls and directory re-opens in PUT and DELETE (#9264)
Co-authored-by: RustFS <hello@rustfs.com>
Co-authored-by: Hauser Zhi <housemecn@gmail.com>
2026-11-11 00:06:11 +00:00
dependabot[bot] 86c3395d07 chore(deps): bump the dependencies group across 1 directory with 12 updates (#9251)
Bumps the dependencies group with 12 updates in the / directory:

| Package | From | To |
| --- | --- | --- |
| [async-nats](https://github.com/nats-io/nats.rs) | `0.51.0` | `0.52.0` |
| [tar-codec](https://github.com/astral-sh/astral-codec) | `0.0.15` | `0.0.16` |
| [tar-framing](https://github.com/astral-sh/astral-codec) | `0.0.15` | `0.0.16` |
| [flate2](https://github.com/rust-lang/flate2-rs) | `1.1.10` | `1.1.11` |
| [smallvec](https://github.com/servo/rust-smallvec) | `1.16.4` | `1.16.5` |
| [starshard](https://github.com/houseme/starshard) | `2.4.0` | `2.4.1` |
| [zip](https://github.com/zip-rs/zip2) | `9.3.0` | `9.4.1` |
| [dial9](https://github.com/dial9-rs/dial9) | `0.5.7` | `0.5.8` |
| [rustfs-mimalloc](https://github.com/houseme/rustfs-mimalloc) | `0.7.3` | `0.7.4` |
| [hotpath](https://github.com/hotpath-rs/hotpath) | `0.30.3` | `0.31.1` |
| [rustfs-mimalloc-sys](https://github.com/houseme/rustfs-mimalloc) | `0.7.3` | `0.7.4` |
| [toml](https://github.com/toml-rs/toml) | `1.2.0+spec-1.1.0` | `1.2.1+spec-1.1.0` |



Updates `async-nats` from 0.51.0 to 0.52.0
- [Release notes](https://github.com/nats-io/nats.rs/releases)
- [Commits](https://github.com/nats-io/nats.rs/compare/async-nats/v0.51.0...async-nats/v0.52.0)

Updates `tar-codec` from 0.0.15 to 0.0.16
- [Release notes](https://github.com/astral-sh/astral-codec/releases)
- [Commits](https://github.com/astral-sh/astral-codec/compare/v0.0.15...v0.0.16)

Updates `tar-framing` from 0.0.15 to 0.0.16
- [Release notes](https://github.com/astral-sh/astral-codec/releases)
- [Commits](https://github.com/astral-sh/astral-codec/compare/v0.0.15...v0.0.16)

Updates `flate2` from 1.1.10 to 1.1.11
- [Release notes](https://github.com/rust-lang/flate2-rs/releases)
- [Changelog](https://github.com/rust-lang/flate2-rs/blob/main/CHANGELOG.md)
- [Commits](https://github.com/rust-lang/flate2-rs/compare/1.1.10...v1.1.11)

Updates `smallvec` from 1.16.4 to 1.16.5
- [Release notes](https://github.com/servo/rust-smallvec/releases)
- [Commits](https://github.com/servo/rust-smallvec/compare/v1.16.4...v1.16.5)

Updates `starshard` from 2.4.0 to 2.4.1
- [Changelog](https://github.com/houseme/starshard/blob/main/CHANGELOG.md)
- [Commits](https://github.com/houseme/starshard/compare/2.4.0...2.4.1)

Updates `zip` from 9.3.0 to 9.4.1
- [Release notes](https://github.com/zip-rs/zip2/releases)
- [Changelog](https://github.com/zip-rs/zip2/blob/master/CHANGELOG.md)
- [Commits](https://github.com/zip-rs/zip2/compare/v9.3.0...v9.4.1)

Updates `dial9` from 0.5.7 to 0.5.8
- [Release notes](https://github.com/dial9-rs/dial9/releases)
- [Changelog](https://github.com/dial9-rs/dial9/blob/main/CHANGELOG.md)
- [Commits](https://github.com/dial9-rs/dial9/compare/dial9-v0.5.7...dial9-v0.5.8)

Updates `rustfs-mimalloc` from 0.7.3 to 0.7.4
- [Release notes](https://github.com/houseme/rustfs-mimalloc/releases)
- [Changelog](https://github.com/houseme/rustfs-mimalloc/blob/main/CHANGELOG.md)
- [Commits](https://github.com/houseme/rustfs-mimalloc/compare/v0.7.3...v0.7.4)

Updates `hotpath` from 0.30.3 to 0.31.1
- [Release notes](https://github.com/hotpath-rs/hotpath/releases)
- [Changelog](https://github.com/hotpath-rs/hotpath/blob/main/CHANGELOG.md)
- [Commits](https://github.com/hotpath-rs/hotpath/compare/v0.30.3...v0.31.1)

Updates `rustfs-mimalloc-sys` from 0.7.3 to 0.7.4
- [Release notes](https://github.com/houseme/rustfs-mimalloc/releases)
- [Changelog](https://github.com/houseme/rustfs-mimalloc/blob/main/CHANGELOG.md)
- [Commits](https://github.com/houseme/rustfs-mimalloc/compare/v0.7.3...v0.7.4)

Updates `toml` from 1.2.0+spec-1.1.0 to 1.2.1+spec-1.1.0
- [Commits](https://github.com/toml-rs/toml/compare/toml-v1.2.0...toml-v1.2.1)

---
updated-dependencies:
- dependency-name: async-nats
  dependency-version: 0.52.0
  dependency-type: direct:production
  update-type: version-update:semver-minor
  dependency-group: dependencies
- dependency-name: tar-codec
  dependency-version: 0.0.16
  dependency-type: direct:production
  update-type: version-update:semver-patch
  dependency-group: dependencies
- dependency-name: tar-framing
  dependency-version: 0.0.16
  dependency-type: direct:production
  update-type: version-update:semver-patch
  dependency-group: dependencies
- dependency-name: flate2
  dependency-version: 1.1.11
  dependency-type: direct:production
  update-type: version-update:semver-patch
  dependency-group: dependencies
- dependency-name: smallvec
  dependency-version: 1.16.5
  dependency-type: direct:production
  update-type: version-update:semver-patch
  dependency-group: dependencies
- dependency-name: starshard
  dependency-version: 2.4.1
  dependency-type: direct:production
  update-type: version-update:semver-patch
  dependency-group: dependencies
- dependency-name: zip
  dependency-version: 9.4.1
  dependency-type: direct:production
  update-type: version-update:semver-minor
  dependency-group: dependencies
- dependency-name: dial9
  dependency-version: 0.5.8
  dependency-type: direct:production
  update-type: version-update:semver-patch
  dependency-group: dependencies
- dependency-name: rustfs-mimalloc
  dependency-version: 0.7.4
  dependency-type: direct:production
  update-type: version-update:semver-patch
  dependency-group: dependencies
- dependency-name: hotpath
  dependency-version: 0.31.1
  dependency-type: direct:production
  update-type: version-update:semver-minor
  dependency-group: dependencies
- dependency-name: rustfs-mimalloc-sys
  dependency-version: 0.7.4
  dependency-type: direct:production
  update-type: version-update:semver-patch
  dependency-group: dependencies
- dependency-name: toml
  dependency-version: 1.2.1+spec-1.1.0
  dependency-type: direct:production
  update-type: version-update:semver-patch
  dependency-group: dependencies
...

Signed-off-by: dependabot[bot] <support@github.com>
Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
Co-authored-by: Hauser Zhi <housemecn@gmail.com>
2026-11-11 00:04:47 +00:00
cxymds c9743f166e fix(s3): normalize leading-slash keys in bulk deletes (#9258)
* fix(s3): normalize leading-slash keys in bulk deletes

* test(s3): align bulk delete reserved-prefix guard expectations
2026-11-10 17:56:00 -05:00
Boyi (Elliot) Li 50707a610c test(utils,rio,checksums): add checksum SIMD equivalence and bench UTs (#9217)
Add unaligned-boundary SIMD vs scalar equivalence unit tests and on-demand (#[ignore]) utility-level throughput microbenchmarks across rustfs-utils, rustfs-rio, and rustfs-checksums:
- crates/utils/src/hash.rs: verify HighwayHasher SIMD (AVX2/SSE4.1/NEON) against highway::PortableHash scalar reference and contiguous hash_encode vs odd-chunked hash_encode_slices across unaligned tail boundaries; provide on-demand throughput microbenchmark.
- crates/rio/src/checksum.rs: verify SIMD-accelerated CRC32, CRC32C, and CRC64_NVME against bitwise scalar reference implementations across vector tail boundaries and measure utility-level throughput across all 10 BASE_CHECKSUM_TYPES.
- crates/checksums/src/lib.rs: verify streaming chunked update + finalize + headers() against one-shot digests across all 9 ChecksumAlgorithm variants for unaligned chunk and payload sizes, plus on-demand throughput microbenchmark.
2026-11-10 17:54:11 -05:00
Patrick Wicki ba3e0e1546 fix(admin): keep measured bucket usage when the overlay is unconverged (#9243)
The single-process usage overlay clears usage_snapshot_complete when it
holds counts it cannot reconcile with the persisted snapshot, for
example writes into a bucket that has no verified baseline. The admin
datausageinfo path then judged the overlaid document as structurally
incomplete and replaced it with an empty one, hiding every bucket the
scanner had measured. Because each write restarts that bucket's scanner
fence, a continuously written bucket kept the whole response empty
indefinitely.

Decide whether the persisted snapshot is usable before the overlay is
applied, and report overlay-induced incompleteness as a non-converged
snapshot instead, as is already done for buckets the scan never reached
(rustfs#5806).
2026-11-10 17:52:06 -05:00
Chris An 95e128336d ci: narrow checks for stable installation metadata (#9269)
* ci: narrow checks for stable installation metadata

* fix: reject release metadata marker collisions
2026-11-10 16:10:04 -05:00
Chris An 8b7bbdb17d chore(release): align installation references for 1.0.5 (#9268) 2026-11-10 14:48:02 -05:00
Chris An 2780bba67b ci: stage release builds and promote verified artifacts (#9265)
* ci: stage release builds and promote verified artifacts

* fix: preserve release package and channel retry contracts

* fix: align installation chart versions with sealed candidates
2026-11-10 13:15:05 -05:00
hector 3a23122104 ci: run macOS release builds on macos-latest-xlarge (#9260)
Use the larger GitHub-hosted macOS runner for the macos-aarch64
release target. The target id, triple, and other platforms stay the same.

Co-authored-by: hector <majinghe@users.noreply.github.com>
2026-11-10 18:55:19 +08:00
Ramakrishna Chilaka 0b1ebeffec perf(ecstore): reduce syscalls in multipart copy, commit, and delete (#9247) 2026-11-10 12:27:10 +08:00
Chris An 0f9258594c fix: preserve scanner backlog authority after pool replacement (#9252)
* fix(ecstore): allow bucket creation after failed decommission

* fix(scanner): preserve backlog membership across pool reindexing

* test(scanner): cover partial publication after pool reindexing
2026-11-09 20:46:20 -05:00
Chris An 33fc735ee2 chore(release): prepare 1.0.5 (#9249) 2026-11-09 17:19:35 -05:00
Chris An 509869cfc5 fix(tier): recover orphaned prepared mutations (#9246) 2026-11-09 22:18:14 +08:00
Ramakrishna Chilaka 395876ccd0 fix(ecstore): remove hardlinked backup during delete rollback (#9239)
When a fused recoverable delete is interrupted after publishing the transaction directory and before updating xl.meta, xl.meta and <transaction>/xl.meta.bkp remain hardlinks to the same inode. Under POSIX rename semantics, renaming xl.meta.bkp over xl.meta during rollback succeeds as a no-op without unlinking xl.meta.bkp, which leaves xl.meta.bkp in the transaction directory after journal removal and strands the transaction in DeleteRecoveryState::LegacyAmbiguous.

Unlink xl.meta.bkp after syncing the restored object directory and before syncing the rollback directory and removing journal markers, and extend delete_recovery_fused_prepare_checkpoint_reopens_before_head_publication to preserve the hardlink and verify rollback completion and retry.
2026-11-09 06:41:04 +08:00
Chris An 6083fbbeea fix(catalog): enumerate table owners without skipping namespaces (#9242) 2026-11-09 06:38:58 +08:00
Chris An 42d3267d88 fix(ecstore): decouple pool metadata health from the authority gate (#9241)
* fix(ecstore): share immutable pool metadata authority reads

* fix(ecstore): read pool metadata authority without the save gate

Health inferred pool metadata write authority from acquiring the local
authority gate within 100 ms. Sharing the gate among read-only
admissions still left held writers, and writers queued behind a reader
parked on the pool.bin fence, turning into cluster-write 503s.

Each writer now publishes the authority it leaves before releasing the
gate, and health reads that published state without acquiring it.
Transaction latches stay shared with the published copy; recovery
clears the latch and its failure under one lock so inspection never
observes a half-cleared state. The inspection-timeout path that can no
longer occur (check_timed_out, the check_timeout admin state, the
pool_metadata_check_timeout reason, the node timeout cache and its
metric) is removed.
2026-11-09 06:38:30 +08:00
Chris An 1d35e381b7 fix(rpc): bound replay memory and atomically admit signed requests (#9238)
* fix(rpc): index nonce expiry by signed validity

* fix(rpc): size replay cache from allocated storage layout

* fix(rpc): make replay admission atomic and release expired tables

* fix(rpc): preserve replay admission refusal at the S3 boundary
2026-11-09 06:36:55 +08:00
Chris An 0b109b4885 fix(select): bound LZ4 scratch and Arrow schema expansion (#9237)
* fix(select): admit legacy and linked LZ4 working memory

* fix(select): bound embedded Arrow schema expansion
2026-11-09 06:36:44 +08:00
cxymds 1dea00ffa7 ci: repair e2e selection and verify tool downloads via API (#9240)
* ci: align full e2e selection with retired catalog test

* ci: verify pinned tools through GitHub release asset API

* ci: fetch pinned typos through the release asset API

* ci: download pinned actionlint through the release asset API

* ci: ensure GitHub CLI exists before pinned mc download

* ci: budget full quick checks after tool installation
2026-11-08 14:09:26 +00:00
cxymds addf7a6219 fix: reduce scanner backoff and classify downstream closures (#9235) 2026-11-08 02:18:17 -05:00
Ramakrishna Chilaka 420cf32b2a perf(ecstore): optimize PUT, DELETE, and multipart commit hotpaths (#9227) 2026-11-08 02:12:40 -05:00
Chris An 907410b2f5 fix(select): bound Parquet allocations before decoding (#9231) 2026-11-08 01:36:12 -05:00
Chris An f10079f1a3 fix(rpc): size and reserve replay cache for compact entries (#9233) 2026-11-08 01:35:58 -05:00
Chris An a6c3016493 fix(cache): respect process cgroup memory limits (#9232) 2026-11-08 01:35:49 -05:00
Hauser Zhi 68fb52b752 chore(deps): update flake.lock (#9236) 2026-11-08 01:34:01 -05:00
cxymds 16f618d41b fix: correct scanner pacing and unverified MRF outcomes (#9225)
Co-authored-by: Chris An <anzhengchao@gmail.com>
2026-11-08 06:48:25 +08:00
Henry Guo 565d262ede refactor(table-catalog): retire durable-strong backing (#9130)
* refactor(table-catalog): retire durable-strong backing

* fix(table-catalog): avoid backup locks during S3 authorization

* fix(table-catalog): preserve directory and maintenance behavior

* fix(table-catalog): back up retained v1 snapshot history

* test(table-catalog): cover retained v1 backup recovery

Exercise backup inventory sealing and restore after a v1-to-v2 upgrade, including missing or corrupt references, nonzero inline snapshot sequences, v2 manifests in retained v1 history, and unchanged commit admission rules.

Clarify when direct maintenance can replace reports and discard retry budgets, and require quiescing scheduler admission and workers before invocation.

Co-Authored-By: heihutu <heihutu@gmail.com>
Co-Authored-By: zhi22915 <qiuzgang@gmail.com>

---------

Co-authored-by: Chris An <anzhengchao@gmail.com>
Co-authored-by: Hauser Zhi <housemecn@gmail.com>
Co-authored-by: heihutu <heihutu@gmail.com>
Co-authored-by: zhi22915 <qiuzgang@gmail.com>
2026-11-08 05:03:55 +08:00
Chris An 7b010ddd70 fix(ci,ecstore): restore validation and defer capacity exhaustion (#9223) 2026-11-07 15:17:25 -05:00
Chris An 272b6587e0 fix(ci): bound workspace archive builds to two workers (#9222) 2026-11-07 06:32:41 -05:00
Chris An e3ed2cf2f4 fix(decommission): reserve owned source usage during activation (#9220) 2026-11-07 01:49:42 -05:00
Chris An bf03b1117a fix(swift): preserve bucket metadata with Swift tag values (#9218) 2026-11-07 01:49:12 -05:00
Boyi (Elliot) Li c3fe0f9649 test(rio,ecstore): add negative-branch durability unit tests (#9216)
Expand positive and negative branch unit test coverage across durability-critical modules:
- crates/rio/src/checksum.rs: Checksum::matches (unset hasher, multipart want_parts mismatch, digest mismatch), new_with_type, as_map, obj_type, from_header, add_part (CRC32/CRC32C/CRC64_NVME combine + error branches), and get_content_checksum header validation.
- crates/rio/src/hash_reader.rs: HashReader::new wrapping invariants (already-read inner reader, conflicting MD5 checksum, conflicting positive size, unbounded size propagation) and checksum helper error paths.
- crates/ecstore/src/set_disk/ops/bitrot_self_verify.rs: verify_written_bitrot_shards missing inline data, missing disk shard, and under-quorum rejection.
- crates/ecstore/src/erasure/coding/encode.rs: read_full_buf_or_eof immediate read error, InvalidData preservation after partial fill, and UnexpectedEof normalization.
- crates/ecstore/src/io_support/shard_integrity.rs: ShardVerifier out-of-bounds coding_index, empty proof sources fail-closed, and spilled (>1 MiB) index disk write + verify_index_file + restore_proof_replicas round-trip and all-corrupt fail-closed boundary.
- crates/ecstore/src/services/integrity/tests.rs: JobRequest::validate boundary/format checks with exact error messages and model::protection / model::unsupported classifications.
2026-11-07 02:38:43 +00:00
唐小鸭 6e31f6a183 fix(ecstore): complete and abort native multipart uploads on Windows (#9191) 2026-11-06 22:36:16 -05:00
Hauser Zhi d66d144309 feat(observability): record unary read RPC durations (#9202)
* feat(observability): record unary read RPC durations

Record complete server-side durations for ReadMultiple and BatchReadVersion, including error and cancellation paths.\n\nCo-Authored-By: heihutu <heihutu@gmail.com>\nCo-Authored-By: zhi22915 <qiuzgang@gmail.com>

* test(observability): cover unary duration handlers

Exercise ReadMultiple and BatchReadVersion missing-disk paths and assert operation duration histograms.\n\nCo-Authored-By: heihutu <heihutu@gmail.com>\nCo-Authored-By: zhi22915 <qiuzgang@gmail.com>

* fix(observability): cache unary read RPC duration handles

Reuse cached duration histograms for ReadMultiple and BatchReadVersion once
the stable server label is available. Preserve the existing ReadVersion
handle and the fallback for other operations and backends.

Add an integration regression for handle reuse, startup fallback, labels,
and recorded values using the production cache path.

Co-Authored-By: heihutu <heihutu@gmail.com>
Co-Authored-By: zhi22915 <qiuzgang@gmail.com>

* fix(observability): separate server duration from client RTT

Record unary ReadMultiple and BatchReadVersion handler lifetimes in a
separate server duration histogram while preserving existing client RTT
series and allocation-free cached handles for both measurements.

Cover client/server sample separation, cache reuse, startup and backend
fallbacks, and missing-disk handlers. Document the new server metric.

Co-Authored-By: heihutu <heihutu@gmail.com>
Co-Authored-By: zhi22915 <qiuzgang@gmail.com>

---------

Co-authored-by: heihutu <heihutu@gmail.com>
Co-authored-by: zhi22915 <qiuzgang@gmail.com>
Co-authored-by: Chris An <anzhengchao@gmail.com>
2026-11-07 09:16:18 +08:00
cxymds 150836d707 feat(tier): support native Azure Blob endpoints (#9200)
* feat(tier): support native Azure Blob endpoints

* fix(tier): guard Azure version cleanup and prefix limits

* fix(tier): preserve Azure quoted headers and document version gate

* fix(tier): handle Azure forbidden current-version deletion

* test(tier): control transition commit lock failure timing
2026-11-07 09:16:06 +08:00
Henry Guo e4057b6746 fix(table-catalog): reject storage credentials for catalog access (#9185)
* fix(table-catalog): reject storage credentials for catalog access

* fix(table-catalog): avoid growing the s3s error macro footprint

Use the existing S3Error constructor for the storage-credential rejection
while preserving its AccessDenied code and message. Keep the merged PR
within the current s3s footprint baseline without relaxing the guard.

Co-Authored-By: heihutu <heihutu@gmail.com>
Co-Authored-By: zhi22915 <qiuzgang@gmail.com>

---------

Co-authored-by: Hauser Zhi <housemecn@gmail.com>
Co-authored-by: heihutu <heihutu@gmail.com>
Co-authored-by: zhi22915 <qiuzgang@gmail.com>
Co-authored-by: Chris An <anzhengchao@gmail.com>
2026-11-07 08:07:58 +08:00
Hauser Zhi cfdd705b62 fix(cors): preserve readiness and complete cache selectors (#9207)
Use a direct response future for originless requests and forward through
the polled-ready service. Nominate every retained CORS header input and
transfer evaluated headers without rebuilding the map.

Co-authored-by: heihutu <heihutu@gmail.com>
Co-authored-by: zhi22915 <qiuzgang@gmail.com>
2026-11-07 08:07:04 +08:00
Hauser Zhi a1d0fa41d0 fix: preserve usage generations and unify request tracing (#9204)
* fix(ecstore): prevent cached usage snapshot rollback

Order refreshed snapshots by their authoritative lineage before replacing cached usage. Share the rule with early bucket refreshes while retaining quota baselines, invalidation fencing, legitimate deletions, and clock recovery.

Co-Authored-By: heihutu <heihutu@gmail.com>
Co-Authored-By: zhi22915 <qiuzgang@gmail.com>

* fix(server): skip trace propagation for disabled RPC spans

Avoid extracting and assigning a remote parent when RPC tracing is disabled after authentication and admission. Cover disabled logging and the production HTTP span propagation path.

Co-Authored-By: heihutu <heihutu@gmail.com>
Co-Authored-By: zhi22915 <qiuzgang@gmail.com>

* fix(ecstore): preserve usage ordering across refresh recovery

Retain the last good snapshot independently of negative-cache status. Order observations within the same authoritative baseline by their own coordinated epoch and cycle before wall time, including clock rollback and recovery. Capture persisted baseline identity without cloning its bucket maps.

Co-Authored-By: heihutu <heihutu@gmail.com>
Co-Authored-By: zhi22915 <qiuzgang@gmail.com>

* refactor(server): centralize request trace propagation

Let the HTTP span factory own parent extraction and assignment. Use actual span filtering, preserve span-only subscribers, and remove repeated tracing work from RPC authentication.

Co-Authored-By: heihutu <heihutu@gmail.com>
Co-Authored-By: zhi22915 <qiuzgang@gmail.com>

---------

Co-authored-by: heihutu <heihutu@gmail.com>
Co-authored-by: zhi22915 <qiuzgang@gmail.com>
2026-11-07 08:06:47 +08:00
Chris An 7d4982f206 fix(storage): fence bounded deletes during rolling upgrades (#9214) 2026-11-07 08:06:28 +08:00
Chris An 1a7fb2461d fix(webdav): join connection and reload tasks on shutdown (#9213) 2026-11-07 08:06:14 +08:00
Chris An e1b9d1cb94 fix(ecstore): preserve ownership across multi-pool recovery (#9212) 2026-11-07 08:06:04 +08:00
Chris An 85313fc06f fix(runtime): release retained replay and allocator memory (#9211) 2026-11-07 08:05:46 +08:00
cxymds b5f3207378 fix(lock): preserve acquisition progress during multipart uploads (#9209)
* fix(lock): retain slow responses within the quorum deadline

* fix(lock): retain partial quorum timeout diagnostics

* fix(lock): avoid guard registry scheduling convoys

* chore(ecstore): trace metadata transaction lock purposes

* fix(scanner): keep capability probes on read fences

* fix(lock): preserve renewal and contention progress
2026-11-07 08:05:26 +08:00
Hauser Zhi 5490e493ab fix(s3): retain protected multipart ETags in object listings (#9208)
* fix(s3): retain protected multipart ETags in object listings

Record encrypted part ETag provenance, aggregate it at read quorum, and attest completed multipart ETags for keyless listings. Preserve legacy fingerprint protection and test versioned SSE-KMS and SSE-C upload, listing, and replay paths.

Co-Authored-By: heihutu <heihutu@gmail.com>
Co-Authored-By: zhi22915 <qiuzgang@gmail.com>

* perf(s3): stream multipart ETag aggregation

Restore the preserved ETag fast path when selected parts lack protected provenance. Stream native part digests through a fixed stack buffer, preserve legacy hex-or-raw semantics, and strengthen exact-output and override regression coverage.

Co-Authored-By: heihutu <heihutu@gmail.com>
Co-Authored-By: zhi22915 <qiuzgang@gmail.com>

* fix(s3): preserve encrypted ETag proof during data movement

Capture quorum-backed source proof before multipart completion metadata updates, require exact preserved ETag and version identity, and retain proof only when publication keeps the source version. Cover real two-pool rebalance, null versions, legacy sources, and injected or rewritten identities.

Co-Authored-By: heihutu <heihutu@gmail.com>
Co-Authored-By: zhi22915 <qiuzgang@gmail.com>

* fix(s3): retain encrypted multipart ETag proof on restore

---------

Co-authored-by: heihutu <heihutu@gmail.com>
Co-authored-by: zhi22915 <qiuzgang@gmail.com>
Co-authored-by: overtrue <anzhengchao@gmail.com>
2026-11-07 08:04:54 +08:00
Hauser Zhi 134820c1e0 test(storage): reproduce HTTPS rejoin readiness in Docker (#9206)
* test(storage): reproduce HTTPS rejoin readiness in Docker

Compare cluster-only and node-ready/write gates on immutable native binaries. Preserve first-failure and latest-acknowledged-content evidence across recovery, capture lock diagnostics before restart, and limit cleanup to verified owned resources.

Co-Authored-By: heihutu <heihutu@gmail.com>
Co-Authored-By: zhi22915 <qiuzgang@gmail.com>

* fix(testing): unify Docker repro ownership and controls

Use typed resource absence and one verified container inventory for staging, partial setup and retained nodes. Reject inherited experiment overrides before creating resources and remove duplicated state.

Co-Authored-By: heihutu <heihutu@gmail.com>
Co-Authored-By: zhi22915 <qiuzgang@gmail.com>

* fix(testing): guard staging volumes and interrupted capture

Verify volume ownership before privileged staging and disable implicit image-content copying. Preserve failed evidence and continue owned finalization when capture or cleanup is interrupted.

Co-Authored-By: heihutu <heihutu@gmail.com>
Co-Authored-By: zhi22915 <qiuzgang@gmail.com>

* fix(testing): capture live rejoin logs and verify trace coverage

Read live tmpfs logs inside the verified container and use one explicit stdout sink for diagnostics. Record bounded capture provenance and fail diagnostic coverage when acquisition, guard-drop or physical-release records are missing, without masking HTTP failures.

Co-Authored-By: heihutu <heihutu@gmail.com>
Co-Authored-By: zhi22915 <qiuzgang@gmail.com>

* fix(testing): reject inherited aliases and finalize on SIGTERM

Reject inherited MINIO compatibility settings before Docker resource creation. Route SIGTERM through interrupt-aware finalization and restore the previous handler after each run. Cover real process signals and failure preservation.

Co-Authored-By: heihutu <heihutu@gmail.com>

Co-Authored-By: zhi22915 <qiuzgang@gmail.com>

* fix(testing): normalize legacy controls and termination

Reject inherited MINIO compatibility aliases in neutral runtime images. Bridge run-mode SIGTERM into the existing interruption finalization and restore the caller signal handler on return or exception.

Co-Authored-By: heihutu <heihutu@gmail.com>
Co-Authored-By: zhi22915 <qiuzgang@gmail.com>

---------

Co-authored-by: heihutu <heihutu@gmail.com>
Co-authored-by: zhi22915 <qiuzgang@gmail.com>
2026-11-07 08:04:28 +08:00
Chris An 2b9217b6f9 test(e2e): fix quota and conditional write fixtures (#9210) 2026-11-07 08:03:00 +08:00
Hauser Zhi 70a46d0e8c fix(webdav): support configured browser CORS access (#9205)
* fix(webdav): support configured browser CORS access

Co-Authored-By: heihutu <heihutu@gmail.com>
Co-Authored-By: zhi22915 <qiuzgang@gmail.com>
Signed-off-by: Hauser Zhi <housemecn@gmail.com>

* perf(webdav): streamline CORS policy and preflight parsing

Borrow connection-owned policy state and clone the backend only for actual requests. Validate borrowed header names and normalize HTTP lists when required.

Co-Authored-By: heihutu <heihutu@gmail.com>
Co-Authored-By: zhi22915 <qiuzgang@gmail.com>
Signed-off-by: Hauser Zhi <housemecn@gmail.com>

---------

Signed-off-by: Hauser Zhi <housemecn@gmail.com>
Co-authored-by: heihutu <heihutu@gmail.com>
Co-authored-by: zhi22915 <qiuzgang@gmail.com>
2026-11-06 14:01:06 -05:00
Hauser Zhi a910ef0dd0 fix(cors): isolate cache variants and streamline policy evaluation (#9203)
Apply cache variation at the shared response boundary, preserve existing
Vary selectors, and reuse parsed global policy and compact CORS inputs.

Co-authored-by: heihutu <heihutu@gmail.com>
Co-authored-by: zhi22915 <qiuzgang@gmail.com>
2026-11-06 12:40:11 -05:00
Chris An 430ade191f fix(scanner): bound wide bucket-root fanout memory (#9193)
* fix(scanner): bound wide bucket-root fanout memory (#8961)

Root prefixes were excluded from ordinary compaction and force_compact only
flattened the root above a hardcoded 250k child ceiling, so buckets with
tens of thousands of top-level prefixes kept one cache entry each after
#8971. Compact wide roots outside checkpointed forward sweeps, and honor
force_compact's limit for root fanout.

* fix(scanner): keep root fanout on interrupted force_compact

Interrupted sweeps seal a resume frontier after force_compact. Applying
the ordinary 10k root limit there flattened frontier-named children and
made the next cycle replay completed prefixes. Use the emergency FORCE
ceiling on the interrupted path; successful completions still honor the
ordinary limit. Drop clippy assertions_on_constants in the new tests.

Signed-off-by: Chris An <anzhengchao@gmail.com>

* test(scanner): seal reloaded frontier after interrupted compaction

Const assertions keep the wide-root preconditions without tripping
clippy::assertions_on_constants. The interrupted and finished sweep
paths share one limit helper, and a cache wider than the ordinary
10k limit still seals its frontier child after reload.

Co-authored-by: Chris An <overtrue@users.noreply.github.com>

* fix(scanner): isolate unreadable children during root compaction

* fix(scanner): isolate compacted namespace child errors

* fix(scanner): preserve compacted scan accounting and coverage

Commit isolated namespace child usage only after successful traversal, while retaining completed accounting on cancellation and resource deferral. Synchronize reduced root totals before the cancellation-prone flush boundary.

Treat skipped namespaces as incomplete coverage and invalidate legacy sampling baselines until a fresh verification pass succeeds. Keep provisional usage out of progress updates and retain the upstream symlink regression.

Co-Authored-By: heihutu <heihutu@gmail.com>
Co-Authored-By: zhi22915 <qiuzgang@gmail.com>

* fix(scanner): preserve legacy retry coverage

* test(scanner): regress cancel overwrite after wide-root reduce

houseme's 714f590 already refreshes the root accumulator from the
compacted cache entry before flush_pacing. Add a unit regression that
simulates the interrupt replace_hashed path so a future regression
cannot reintroduce dangling child links over the aggregate.

Signed-off-by: overtrue <anzhengchao@gmail.com>

---------

Signed-off-by: Chris An <anzhengchao@gmail.com>
Signed-off-by: overtrue <anzhengchao@gmail.com>
Co-authored-by: Cursor Agent <cursoragent@cursor.com>
Co-authored-by: Chris An <overtrue@users.noreply.github.com>
Co-authored-by: Hauser Zhi <housemecn@gmail.com>
Co-authored-by: heihutu <heihutu@gmail.com>
Co-authored-by: zhi22915 <qiuzgang@gmail.com>
2026-11-06 12:36:25 -05:00
Hauser Zhi c5b15b7279 fix(container): unify directory setup and startup diagnostics (#9201)
* fix(container): explain log directory initialization failures

Report failed log directory creation with ownership guidance and a Compose quote hint while preserving configured values. Add a corrected example and real entrypoint regression coverage.

Co-Authored-By: heihutu <heihutu@gmail.com>
Co-Authored-By: zhi22915 <qiuzgang@gmail.com>

* refactor(container): unify startup directory preparation

Share directory creation, diagnostics, and best-effort ownership across data and log paths. Skip delimiter and regex subprocesses for ordinary volume paths without introducing a second parser. Expand real-entrypoint coverage for ownership, failures, delimiters, distributed ranges, and setup command counts.

Co-Authored-By: heihutu <heihutu@gmail.com>
Co-Authored-By: zhi22915 <qiuzgang@gmail.com>

---------

Co-authored-by: heihutu <heihutu@gmail.com>
Co-authored-by: zhi22915 <qiuzgang@gmail.com>
2026-11-06 11:49:09 -05:00
cxymds 2a6da8208b fix(lock): converge cleanup after lost release acknowledgements (#9199)
* fix(lock): converge cleanup after lost release acknowledgements

* fix(log-analyzer): track updated unlock cleanup logs

* test(lock): align clocks in unlock cleanup regression
2026-11-06 10:40:28 -05:00
cxymds 93b704420f fix(ecstore): preserve multipart shards within write budgets (#9197) 2026-11-06 10:37:24 -05:00
Ramakrishna Chilaka dfd4d3e213 perf(ecstore): fuse overwrite commits and optimize delete recovery (#9194) 2026-11-06 03:32:34 -05:00
Hauser Zhi 1c1baa6b08 fix(observability): avoid duplicate process metric labels (#9182) 2026-11-05 23:07:02 -05:00
Chris An 0147b8671d ci: bump overtrue/cla-bot to v0.1.1 (#9189)
Co-authored-by: cursoragent <cursoragent@users.noreply.github.com>
2026-11-06 07:01:52 +08:00
Hauser Zhi 8cd751f9d1 feat(ecstore): record unary gRPC operation duration (#9187)
Record end-to-end durations for ReadAll, WriteAll, ReadMultiple, and BatchReadVersion alongside existing request, error, and payload metrics. This makes direct unary payload ABBA results observable through the internode operation duration family.

Co-authored-by: zhi22915 <qiuzgang@gmail.com>
2026-11-06 07:01:38 +08:00
Hauser Zhi 5ea2991850 fix(ecstore): retry failed internode channel prewarm (#9186)
Release the per-peer prewarm deduplication marker when the best-effort dial fails, allowing a later RemoteDisk construction to retry the connection warmup without changing lazy connect fallback behavior.

Co-authored-by: zhi22915 <qiuzgang@gmail.com>
2026-11-06 07:01:26 +08:00
Chris An 1556e05b8a fix(sftp): put directory objects as trailing-slash keys (#9180)
SFTP subdirectory mkdir and rmdir sent the on-disk __XLDIR__ marker
through the public S3 API. After the suffix was reserved, those calls
were rejected as invalid client keys, so ordinary directory operations
failed while bucket mkdir still succeeded.

Send the logical trailing-slash key and let storage encode the marker.

Co-authored-by: Cursor Agent <cursoragent@cursor.com>
Co-authored-by: Chris An <overtrue@users.noreply.github.com>
2026-11-06 07:00:58 +08:00
Hauser Zhi 32238a179c fix(server): bound rejoin waits and preserve recovery ownership (#9181)
* test(server): cover recovered-node admission boundaries

Document backend readiness and write-availability checks during rejoin.
Extend the existing regression through the production middleware order
without changing runtime admission or cluster-quorum semantics.

Co-Authored-By: heihutu <heihutu@gmail.com>
Co-Authored-By: zhi22915 <qiuzgang@gmail.com>

* fix(server): bound rejoin waits and preserve recovery ownership

Enforce startup and unreachable-grace bounds while consuming peer replies.
Release admission when every set has its required peer majority and cancel
surplus read-only probes without carrying votes between rounds.

Bound local readiness observations and polling sleeps. Retain initialized
IAM across preparation failures, install the context once, and retry only
publication after readiness timeouts. Share preparation with inline startup.

Co-Authored-By: heihutu <heihutu@gmail.com>
Co-Authored-By: zhi22915 <qiuzgang@gmail.com>

* perf(server): make peer admission folding incremental

Compile peer-to-set memberships once and retain compact per-edge votes.
Consume each reply once, discard its payload, and update only affected
set counters instead of rescanning all sets and retained responses.

Recheck the captured deadline after polling and folding a reply so late
work cannot admit the node or clear an expired unreachable grace period.

Co-Authored-By: heihutu <heihutu@gmail.com>
Co-Authored-By: zhi22915 <qiuzgang@gmail.com>

---------

Co-authored-by: heihutu <heihutu@gmail.com>
Co-authored-by: zhi22915 <qiuzgang@gmail.com>
2026-11-06 07:00:36 +08:00
Chris An 6b70584dde chore(ci): bump overtrue/cla-bot to v0.1.0 (#9188)
Pin CLA Check to f054da99 (v0.1.0) so built-in bot allowlist skips
cursoragent, Dependabot, GitHub Actions, Renovate, and Copilot without
requiring a CLA signature.

Co-authored-by: cursoragent <cursoragent@users.noreply.github.com>
2026-11-05 23:54:13 +08:00
Chris An d6210ac17b fix(s3,ecstore,table-catalog): bound work driven by client input (#9172)
* fix(ecstore,heal,select): bound storage-layer work driven by client input

Scan findings in the erasure store, heal, file metadata and S3 Select
layers, grouped because they share the disk walk, the multipart staging
area, delete recovery and the select snapshot lease:

- Multipart: upload ids are canonicalized before they name a staging path
  or a lock key (backlog#3027); trailing-slash keys are encoded like every
  other handler (backlog#3055); unlisted parts are removed at completion
  (backlog#3106); per-bucket uploads are found through a signed per-disk
  index instead of a scan of the shared staging area (backlog#3121);
  encrypted part ETags carry a per-upload secret so a write-only user
  cannot confirm plaintext guesses (backlog#3044).
- Delete and recovery: recovery walks are bounded per request
  (backlog#3038); a delete marker on a key whose xl.meta path is a
  directory, and sidecar names used as child keys, no longer wedge the
  journal (backlog#3035, backlog#3105); over-quota PUTs no longer list the
  bucket under the ledger lock (backlog#3128).
- Listing: scan_dir sends metadata without inline data (backlog#3052),
  counts delete markers against max-keys (backlog#3108), and rejects a
  marker that carries the cursor tag (backlog#3059).
- Heal: bucket metadata and heal records live in separate namespaces so
  DeleteBucket cannot erase a drive rebuild (backlog#3039), a deleted
  bucket is skipped instead of failing the rebuild (backlog#3040), and
  directory objects are encoded on readback (backlog#3061).
- Restore and tiering: RestoreObject preserves tags and the version's
  modification time (backlog#3073, backlog#3074); Snowball transport
  headers are not forwarded to the tier (backlog#3090); an epoch
  modification time round-trips through xl.meta (backlog#3063).
- Locks: PutObject takes its bucket locks after the body is staged
  (backlog#3131); SelectObjectContent releases its namespace locks once
  the snapshot is taken (backlog#3132).
- RPC and peers: the v3 signature check consumes the v2 nonce
  (backlog#3129); a bucket-metadata reload timeout no longer marks the
  peer offline (backlog#3124); replication target requests carry an
  operation deadline (backlog#3102); the scanner resumes its interrupted
  wait after a lifecycle write (backlog#3118).
- S3 Select: bounded schema inference, column cap and budgeted CSV
  batches, Parquet dictionary bounds and pre-planner projection checks
  (backlog#3075, backlog#3076, backlog#3077, backlog#3078).

* test(e2e): seed legacy duplicate bucket tags offline for the OPA contract

* fix(s3): authorize what handlers store and bound request-driven work

Scan findings in the S3 request layer (access hooks, object use cases,
HTTP server layers), grouped because they share the authorization
condition builder, the PutObject admission path and the response
layers. Builds on the storage-layer batch for the bounded recovery and
multipart APIs it calls.

Authorization and policy conditions:
- copy source is percent-decoded and version-canonical before the
  s3:x-amz-copy-source condition (backlog#3135); CopyObject authorizes
  the tag set it copies (backlog#3034); HeadObject with the replication
  check header still authorizes s3:GetObject (backlog#3047); the
  x-rustfs-source-version-id header feeds s3:versionid (backlog#3072);
  recursive force-delete with a REPLICA header authorizes every
  descendant as a delete (backlog#3070); include-deleted listings require
  s3:ListBucketVersions (backlog#3099); the metadata listing route keeps
  table isolation and uses the same prefix it authorizes (backlog#3053,
  backlog#3054); If-Match probing needs read permission (backlog#3097);
  Object Lambda checks s3:GetObjectVersion (backlog#3058); replication
  proxy fallback keeps ExistingObjectTag denies (backlog#3089); HeadObject
  returns lock and tag headers only with permission (backlog#3113);
  IPv4 clients on the dual-stack listener match IPv4 SourceIp conditions
  (backlog#3049); bucket policies are parsed once and cached
  (backlog#3032); keystone: access keys no longer verify with an empty
  secret (backlog#3043).
Admission and resource bounds:
- GetObject releases disk-read permits when the client stops reading
  (backlog#3023); PutObject takes buffer and foreground admission only
  after body bytes arrive and bounds the wait (backlog#3031,
  backlog#3084); xl.meta size is bounded on the write path
  (backlog#3100); XML compat layers stream instead of buffering whole
  bodies (backlog#3094); tier GETs stream the remote body (backlog#3046); on-demand migration pulls release permits on
  unread GETs (backlog#3110); SSE-KMS key policy covers UploadPart and
  completion (backlog#3130); MinIO admin decryption runs Argon2id off the
  async worker (backlog#3024).
Input validation and leaks:
- replication headers are validated (source ETag shape, original size)
  (backlog#3092, backlog#3093); control characters in keys and lifecycle
  rule IDs are rejected (backlog#3101, backlog#3117); Snowball member
  mtime needs s3:ReplicateObject (backlog#3083); DeleteObjects checks
  Content-MD5 and admin/STS reject SigV2 replays with altered query or
  body (backlog#3037, backlog#3026); request logs redact presigned
  queries and handler spans skip credentials (backlog#3068,
  backlog#3107); SSE-C listings and completions withhold plaintext
  metadata and checksums (backlog#3120, backlog#3050); CORS resolves the
  bucket like s3s does (backlog#3104); PutBucketTagging rejects duplicate
  keys (backlog#3126); notifications fire after a client disconnect
  (backlog#3127); ListenNotification streams re-check access
  (backlog#3119).

* fix(api): drop imports left unused by the base merge

* test(s3): move the KMS multipart denial test to the app layer

The test drives app-layer multipart use cases, so it belongs with them;
living in the SSE module it imported app code from the infra layer and
failed the layer dependency guard. The SSE test helpers it borrows are
exposed through the storage and app facades.

* fix(s3): keep the SSE-C key fingerprint on keyless multipart completion

The backlog#3050 fix withheld both the plaintext checksums and the
customer-key MD5 header from a CompleteMultipartUpload that did not
present the key. The checksums are a digest of the plaintext and stay
hidden; the key MD5 identifies the key only, and the S3 conformance suite
(test_copy_part_enc) expects it on a keyless completion, so it is echoed
again.

* fix(tier): stream GET bodies and document replication budgets

* fix(ecstore,s3): keep the multipart index and cold fill off disk lifetimes

Two lifecycle regressions surfaced once the storage and request batches
ran together against main:

- The multipart index owner cloned the disk's pinned publication root and
  bootstrapped on a task of its own. That kept the mount alive past the
  last LocalDisk owner (the disk lifecycle allows exactly three holders of
  the root FD) and left a stray task behind, which the local-disk and
  read-repair tests detect. The owner now keeps only the root path and
  reopens it for identity checks, and the bootstrap scan rides the disk's
  existing cleanup task, where stopping the disk abandons it and a
  reconnect resumes it.
- The disk-read admission wrapper defers releasing its permit until an
  in-flight read completes, so a cancelled client cannot widen disk
  concurrency. The cold-fill producer, however, drops a read it has
  already classified as stalled when its deadline fires, and that permit
  stayed pinned for the lifetime of the stuck read. The producer now
  abandons the admission explicitly on timeout.

* test(ecstore): pass the index owner to the Linux cleanup-loop test

* fix(s3): release cold-fill admission after watchdog handoff

* fix(s3,ecstore): close the round-1 adversarial review findings

Adversarial review of the combined request- and storage-layer batch found
fixes that fell short of their issues, two behaviour regressions for
legitimate clients and several unrequested costs:

- backlog#3038: a bulk DeleteObjects reserved the whole per-disk
  inspection allowance up front for every key in flight and exhausted the
  16 Mi request budget on any set wider than eight drives, failing healthy
  keys with SlowDown. Calls are now admitted while budget remains and
  charged for the entries they actually visited. Object-scoped discovery
  charges plain child keys at one entry per 64 names instead of one each,
  so deleting a missing key that names a folder prefix completes instead
  of answering 503 forever while a hostile prefix stays bounded. A peer
  still on the previous binary is reported
  as an unsupported participant, the repository's rolling-upgrade
  contract, instead of as a budget verdict.
- backlog#3037/#3026: a SigV2 configuration PUT without Content-MD5 was
  still replayable with another body; it is refused with MissingContentMD5
  while SigV4 keeps the optional-MD5 contract.
- backlog#3072: DeleteObject now resolves x-rustfs-source-version-id before
  authorization like PUT and COPY, so a replica delete cannot remove a
  version the policy never saw.
- backlog#3121: the multipart index no longer fsyncs a marker and four
  directories per multipart metadata write (it is rebuilt on every start),
  and ListMultipartUploads discovers roots concurrently.
- backlog#3077: Parquet SelectObjectContent decodes budget-sized batches
  instead of one row per batch.
- backlog#3031: a full upload buffer pool hands out an unpooled buffer
  instead of refusing the upload; the progress floor already bounds stalls.
- backlog#3023: the GET disk-read idle cut-off is configurable
  (RUSTFS_GET_DISK_READ_IDLE_TIMEOUT_SECS, default 120 s) so a paused
  client is not truncated after 30 s.
- backlog#3075: JSON Lines schema inference keeps the sample collected at
  the 4 MiB bound instead of failing the query.
- backlog#3024: compat admin decryption queues briefly for a slot instead
  of refusing the third concurrent request.
- backlog#3113: HEAD skips its permission probes for the bucket owner.
- backlog#3117: control characters in a lifecycle rule ID are rejected on
  PutBucketLifecycle as well.

The lock-crate publication FIFO serves backlog#3111 and moves to the
table-catalog PR. Also fixed: a lossy admin error message, an unused
s3-client function, two unused test-util features, a duplicated key
validation, the s3s footprint baseline, and a stale log event name in the
reverse-proxy runbook. The ODM multipart write-back test runs on the large
test stack like its sibling.

* fix(table-catalog): bound maintenance work and isolate table namespaces (#9174)

* fix(table-catalog): bound maintenance work and isolate table namespaces

Scan findings in the Iceberg table catalog:

- Metadata maintenance deleted every unreferenced *.metadata.json in a
  table's directory without checking whether a renamed table in another
  namespace still owned those files (backlog#3060); it now resolves the
  owning table through the directory index before deleting.
- Manifest lists and manifests were decoded with the embedded Avro
  writer schema and no recursion depth limit; a self-referencing record
  schema overflowed the stack (backlog#3080). Schemas are validated for
  depth before decoding.
- Compaction decoded every input Parquet file into Arrow batches held in
  memory; a decompression bomb OOM-killed the server (backlog#3085). Row
  groups are streamed through a byte budget and oversized inputs are
  refused.
- An S3 request for a key outside an indexed table prefix listed the
  bucket's whole namespaces/ catalog prefix and failed once it held more
  than 4096 objects (backlog#3086); a per-bucket directory index answers
  index misses without the listing.
- The maintenance planner read whole orphan files to obtain their
  modification time (backlog#3087); it reads metadata only.
- A client disconnect between persisting active_rename_id and clearing it
  left the bucket's catalog entry locked at 503 (backlog#3088); the rename
  runs detached from the request future and clears the flag on every exit.
- Table commits held the bucket publication fence until publication,
  starving every object write (backlog#3111); commits now hold it under a
  deadline and object writes are admitted between commits.

The directory index is managed through three new admin routes under
{warehouse}/catalog/directory, registered in the admin route policy.

The durable-strong store is left as is (its snapshot partitioning for
backlog#3109 and the strong-path deadline are not ported): that backing
is being retired separately, so the new tests cover the object backing.

* fix(table-catalog): close the review gaps in the scan-fix batch

Adversarial review of the table-catalog batch found the fixes narrower
than the issues they close and two cost regressions:

- backlog#3085: the compaction footer check bounded nothing a page header
  could not override; parquet sizes its page buffers from the header.
  Walk every column chunk's page headers (Thrift compact, no decompression)
  and hold each page and the chunk total to the budget before decoding.
  The stored-input budget is per rewrite group, so a multi-group job no
  longer fails after writing earlier outputs.
- backlog#3086: the default index-miss scan still counted maintenance
  reports toward its 4096-object limit; only table entries count now, and
  the listing itself is bounded to 64 pages. The directory index remains
  an optional accelerator.
- backlog#3111: a create wrote its metadata file under the fence as the
  first durable mutation, which disarmed the deadline before the unbounded
  registration inventory. The write precedes publication now. The deadline
  gets its own knob (RUSTFS_TABLE_CATALOG_PUBLICATION_PREPARATION_TIMEOUT_SECS,
  default 30 s, never below the lock-acquire timeout) instead of reusing
  the 5 s lock timeout, and the ownership inventory reads entries
  concurrently.
- Directory candidates are written only while a directory exists; the
  backfill covers earlier entries. A directory on the migration-fence
  path is reported as a conflict instead of a parse error.
- Dead test hooks, a stale comment and the support-matrix text that did
  not match the code are removed or corrected.

* fix(select): preserve complete samples and Parquet batch limits

* fix(s3,ecstore): close the round-2 adversarial review findings

DeleteObject resolved x-rustfs-source-version-id for authorization on
every request while storage consumes it only for a REPLICA request, so a
non-replica caller with s3:DeleteObjectVersion could trigger a plain
delete of the current version. The hook now resolves the header under
the same predicate as storage, and the regression test covers the
non-replica case.

HEAD no longer skips its permission probes for the bucket owner; an
explicit bucket-policy Deny on the metadata actions binds the owner too.

Multipart index pages verify each root against the staging area and
prune markers whose staging root is gone, under the marker's own lease
with a re-check, so the per-drive index stays proportional to live
uploads instead of every key ever multipart-initiated on the drive.

SigV2 configuration PUTs that declare an empty body need no Content-MD5:
there is no body to replay. The minimum-progress floor stays independent
of RUSTFS_HTTP_REQUEST_BODY_READ_TIMEOUT; the knob's documentation now
says so instead of claiming 0 disables it. The quota bootstrap window
(503 until the background scan publishes the ledger) is kept, because
ledger admission must not come from scanner reporting; the compatibility
matrix now documents it.

The S3 compatibility matrix gains a section for the client-visible
validation bounds, and the drive-timeout tuning guide tabulates
RUSTFS_GET_DISK_READ_IDLE_TIMEOUT_SECS.

* fix(lock): queue table-bucket publication writers fairly

Table commits take the bucket publication fence as a write lock; under a
stream of commits the lock's try-and-retry acquisition let a new commit
jump ahead of object writers that had been waiting longer, which is part
of what starved them (backlog#3111). Requests for the publication lock
path now take a FIFO ticket per shard: a waiter is admitted only when it
is at the head of the queue, cancelled waiters release their turn, and
batch acquisitions join the same queue. Other lock paths are unchanged.

This moved here from the request-layer batch, where no cited issue
covered it; it serves the table-catalog issue this PR closes.

* fix(table-catalog): close the round-2 adversarial review findings

Parquet page-header walker: read the size fields by field id as parquet
does, accepting I16/I32/I64 wire kinds and rejecting any other kind, so
a header encoding uncompressed_page_size as I64 can no longer bypass the
budget while parquet allocates it. Bound every list, set and map count by
the bytes remaining in the chunk, and reject bool collection elements
(parquet skips them with zero width, so either width would desynchronize
the two parsers); this removes the 2^63-iteration spin on LIST<BOOL>.
Start each column-chunk walk at the dictionary page offset whenever the
footer declares one, exactly as parquet's byte_range does, instead of
min(dictionary, data).

Footer pre-walk: before parquet decodes a footer, walk FileMetaData with
the hardened cursor and charge the declared row_groups count times the
RowGroupMetaData size to the decode budget, because parquet 59 reserves
that Vec before reading one element.

Preparation deadline: clamp the
RUSTFS_TABLE_CATALOG_PUBLICATION_PREPARATION_TIMEOUT_SECS knob to 86,400
seconds so Instant + Duration cannot overflow.

Index-miss scan: after each listing page, jump the cursor past the
maintenance subtree the page ended in, and allow one extra page per table
entry already listed, so job-report history scales with the catalog while
a report flood under one table stays bounded.

Migration fence: read the fence-path object once and decide from its
stored kind whether it is directory state or a fence.

Expired or failed create: remove the initial metadata file when the
registration fails before any catalog write may have begun; nothing else
references it. Correct the comment and support matrix that claimed
maintenance reclaims it.

Tests cover each walker divergence, the footer row-group reservation, the
deadline ceiling, the paging budget for honest catalogs and for a report
flood, and the removal of the unpublished metadata file.

* test(admin): register the catalog directory routes in the route matrix

The table-catalog batch added GET, POST and DELETE
{warehouse}/catalog/directory on both the Iceberg and the compat admin
prefixes without listing them in the admin route matrix, so
test_admin_route_matrix_matches_registered_routes failed on CI.

* fix(s3,ecstore): close the round-3 adversarial review findings

Multipart index pages now treat the cursor as the last examined marker
rather than the last root, so a page that pruned a stale marker still
validates and ListMultipartUploads keeps using the index. A marker is
written before its staging root is published, so pruning spares markers
younger than a five-minute grace period and an existing marker is
touched on every new upload; the prune re-checks both under the marker
lease. Bounded delete-recovery inspection rides the durable-mutation
lane like the unbounded path instead of the one-worker foreground lane.
The on-demand-migration inline body budget scales with the object size
(at least 64 KiB/s) instead of a flat 120 s. The control-body digest
service honors RUSTFS_HTTP_REQUEST_BODY_READ_TIMEOUT=0 as "no
inactivity timeout" like every other reader of that knob.

Quota reconciliation throttle objects are deleted together with a
retired generation's ledger, so bucket turnover leaves no residue. The PUT guard
test shim is gone; tests call the shared guard. Bounded DiskAPI defaults
answer MethodNotAllowed like their siblings. Stale
budget docs, an unreachable CSV arm, an always-Some limit parameter,
commented-out constants and a misplaced doc comment are cleaned up;
weak test assertions name the expected error; lifecycle rule-ID control
characters gain a test. The compatibility matrix records the CopyObject
tag-copy permission, the exact SigV2 empty-body predicate, replica and
migration writes of control-character keys, and the SigV2 DeleteObjects
digest rule.

* fix(table-catalog,heal): close the round-3 review findings

A. Pack compaction rewrite groups to min(max-rewrite-bytes-per-job,
   TABLE_COMPACTION_MAX_STORED_BYTES), reject an over-ceiling group before
   any output is written, and name the ceiling in the commit-path errors;
   document the effective cap in the support matrix and scheduler runbook.
B. Keep TaskCancelled/TaskTimeout classification on replacement passes:
   the namespace verification runs only for completed or storage-failed
   passes, in the erasure healer and the task bucket prepass.
C. Compare the replacement bucket pass per bucket: a recorded identity
   that is gone or replaced still errors or restarts the pass, buckets
   created mid-pass are appended to the identities, pending_buckets and a
   persisted positional order without rewinding progress or the
   checkpoint; verification ignores additions.
D. Read the migration-fence object once in the object-backed write permit
   and dispatch on its kind.
E. Drop the strong store's copy of maintenance_cleanup_roots.
F. Remove the unused test hooks fail_put_prefix, corrupt_put_prefix and
   pause_after_metadata; make compact_parquet_data_files and
   PublicationTicket.state private.
G. Document the body-revision identity, the legacy revalidation budget
   and its HTTP 503 in the support matrix.
H. Define HEAL_REPLACEMENT_RECOVERY_DIR once in ecstore and consume it
   from heal through the storage_api facade.

* fix(ecstore): cap the index pages one disk may answer

Pruned multipart index pages may carry few roots, so the page cursor no
longer bounds how many pages a peer can make a listing fetch. Cut a disk
off after 4096 pages (about one million examined markers) so it
contributes no coverage and the listing falls back to the staging scan.

* fix(ecstore,table-catalog): close the round-4 adversarial review findings

Multipart index markers dated in the future (a backwards clock step
between recording a marker and publishing its staging root) were judged
old and could be pruned while their upload was in flight; a future-dated
marker is now kept. The marker touch on EEXIST is best effort, so a
filesystem that refuses to set times no longer fails the upload commit.
Pruning runs on the durable-mutation lane instead of the one-worker
foreground lane.

Quota reconciliation keeps one throttle object per bucket incarnation (a
quota revision change replaces it instead of orphaning the previous
revision's object), the retirement cleanup deletes it ahead of the ledger
and also when the ledger is already gone, and a scan the retirement
already refused writes nothing back. Bounded DiskAPI defaults answer
UnsupportedDisk, the verdict the aggregate gives a peer without the
protocol. The compaction ceiling pre-check runs over every rewrite group
before any output is written, and the worker's ceiling error no longer
names a placeholder group.

Docs: the inline on-demand-migration client budget, the protocol-2
downgrade conflict of replacement passes, the exact footprint numbers.
2026-11-05 23:32:54 +08:00
Chris An 158aa276e9 fix(storage): bound heal retries and recover pool metadata (#9164)
* fix(ecstore): retry metadata writes rejected before staging

* fix(ecstore): reuse registered disks during startup

* fix(ecstore): retain remote health monitors during reconnect

* test(heal): release the commit barrier during graceful shutdown

* test(e2e): register graceful shutdown barrier regressions

* fix(ecstore): roll back interrupted pool metadata commits

* test(upgrade): honor the 1.0.4 delete recovery protocol

* fix(heal): bound admission during object rate pressure

* test(e2e): handle rebalance completion before stop

* test(e2e): observe rebalance activity before the start reply

* test(rebalance): hold the first run until stop cancels admission
2026-11-05 21:42:19 +08:00
Hauser Zhi 3362421240 fix(obs): add fine-grained object lock histograms (#9177)
Use explicit buckets that preserve millisecond lock acquisitions while retaining visibility into long-held delete and recovery locks. Add an exporter-backed regression test for both histogram instruments.

Co-authored-by: zhi22915 <qiuzgang@gmail.com>
2026-11-05 08:11:08 +00:00
GatewayJ 57df9181d0 fix(audit): preserve OIDC identity in S3 request audits (#9178) 2026-11-05 06:03:33 +00:00
Hauser Zhi 371f5e4ffc fix(ecstore): streamline lock RPC and listing ownership (#9176)
* fix(ecstore): preserve lock cleanup and listing trace context

Own remote lock admission and dispatched RPC cleanup across caller cancellation. Reuse bounded late-result cleanup and retain the listing request span in metacache tasks.

Co-Authored-By: heihutu <heihutu@gmail.com>
Co-Authored-By: zhi22915 <qiuzgang@gmail.com>

* perf(ecstore): streamline lock RPC and listing fanout

Poll RPCs in callers and transfer pending work before channel eviction. Share walk options across producers and borrow cached peer keys. Keep deadline classification deterministic.

Co-Authored-By: heihutu <heihutu@gmail.com>
Co-Authored-By: zhi22915 <qiuzgang@gmail.com>

---------

Co-authored-by: heihutu <heihutu@gmail.com>
Co-authored-by: zhi22915 <qiuzgang@gmail.com>
2026-11-05 05:59:05 +00:00
cxymds 8a5a2c9c93 fix(s3): treat duplicate CreateBucket as owned by the requester (#9167)
* fix(s3): treat duplicate CreateBucket as owned by the requester

The duplicate-bucket response was decided from the requester's root-owner
flag, so every IAM user recreating a bucket received 409
BucketAlreadyExists, including a user recreating a bucket it created.
BucketAlreadyExists means the bucket belongs to another account; a
deployment owns a single namespace, so the answer is purely region-based:
the default region returns the legacy 200 OK, any other region returns
BucketAlreadyOwnedByYou.

Both the existing-bucket fast path and the BucketExists path now share one
helper. Authorization still runs first, so a requester without
s3:CreateBucket gets AccessDenied and cannot probe bucket existence.

Refs #8493

* test(ci): refresh e2e selections for duplicate CreateBucket

The two new create_bucket_region cases join the e2e-smoke and e2e-full
selections, which main also extended with multipart_upload_id tests.

smoke is measured from a local nextest listing: removing the two new cases
reproduces main's committed digest exactly. e2e-full is derived the same way
from main's listing artifact (run 41933823399, babfeaed0) plus the two
multipart_upload_id cases, which reproduces main's committed linux digest
before the new cases are added; CI re-verifies both platforms.

test_bucket_create_exists_nonowner asserts BucketAlreadyExists for a second
identity, which a single-owner deployment cannot produce. Move it to
excluded_tests.txt with the deployment-ownership rationale; the IAM-user
duplicate paths stay covered by crates/e2e_test/src/create_bucket_region_test.rs.
2026-11-05 03:06:50 +00:00
Hauser Zhi babfeaed00 fix(io-core): preserve pooled capacity and accounting (#9171)
* fix(io-core): preserve pooled capacity and accounting

Co-Authored-By: heihutu <heihutu@gmail.com>
Co-Authored-By: zhi22915 <qiuzgang@gmail.com>

* fix(io-core): preserve pool allocation admission order

Co-Authored-By: heihutu <heihutu@gmail.com>
Co-Authored-By: zhi22915 <qiuzgang@gmail.com>

* docs(io-core): clarify pool return locking

Co-Authored-By: heihutu <heihutu@gmail.com>

Co-Authored-By: zhi22915 <qiuzgang@gmail.com>

---------

Co-authored-by: heihutu <heihutu@gmail.com>
Co-authored-by: zhi22915 <qiuzgang@gmail.com>
2026-11-04 22:07:58 +00:00
Jose Meira 736bf777b4 feat(container): add declarative bucket bootstrap (#9162)
* feat(container): add declarative bucket bootstrap

* fix(container): handle false bucket versioning setting

* Update Rust base image to version 1.99.1-trixie

Signed-off-by: Hauser Zhi <housemecn@gmail.com>

* Update Rust base image version to 1.99.1

Signed-off-by: Hauser Zhi <housemecn@gmail.com>

---------

Signed-off-by: Hauser Zhi <housemecn@gmail.com>
Co-authored-by: Hauser Zhi <housemecn@gmail.com>
2026-11-04 16:59:50 +00:00
KarlEmm 7babc7d621 fix(s3): report malformed multipart upload IDs as NoSuchUpload (#9147)
UploadPart, UploadPartCopy, ListParts and CompleteMultipartUpload returned 400 InvalidArgument for a malformed upload ID, while AbortMultipartUpload returned 404 NoSuchUpload. AWS S3 documents NoSuchUpload for an upload ID that might be invalid.

Report MalformedUploadID as NoSuchUpload through one helper used by all operations that address an existing upload, replacing the Abort-only special case. ListMultipartUploads keeps InvalidArgument for a malformed upload-id-marker.

Add the multipart_upload_id_test e2e module to the e2e-smoke profile. It asserts 404 NoSuchUpload from every operation that addresses an existing upload, for a malformed upload ID and for an aborted one, and updates the e2e-smoke and Linux e2e-full selection digests.

Co-authored-by: Karl Bastarache <bastarachek@google.com>
Co-authored-by: Hauser Zhi <housemecn@gmail.com>
2026-11-04 16:25:00 +00:00
Hauser Zhi 4d46bd25f8 perf(ecstore): parallelize bulk-delete lock and metadata fan-out (#9155)
* perf(ecstore): probe bulk-delete write locks in parallel waves

Bulk delete acquired one distributed write lock per key in serialized
order, paying a full quorum round-trip per key. Probe the unique keys in
bounded parallel waves with a short timeout instead; any contention drops
the whole wave and the canonical serial order resolves it as before, so
two overlapping batches can never deadlock.

* perf(ecstore): probe bulk-delete transaction locks in parallel

The set-layer delete-transaction locks were also acquired one quorum
round-trip per key after the raw write locks. Apply the same optimistic
all-or-nothing probe wave with a canonical serial fallback, and keep the
per-key error reporting the fallback already provided.

* perf(ecstore): pre-fetch bulk-delete source metadata in parallel

The admission loop ran one metadata quorum fan-out per key in sequence
on top of the per-key lock round-trips. Pre-fetch the source reads the
loop needs with bounded parallelism under the locks already held, and
derive the per-key read inputs from one shared helper so the pre-fetch
and the loop cannot disagree.

* test(ecstore): drop unused import in batch-delete probe test

* refactor(ecstore,lock): await wave releases and unify probe waves

Adversarial review of the bulk-delete lock waves found three real
issues. Distributed guard release is fire-and-forget on drop, so the
serial fallback could re-acquire keys while unlock RPCs were still in
flight and block on locks it already owned: guards are now released
with an awaited unlock before the fallback runs. The wave also probed
every key after the first failure, adding up to one probe timeout per
chunk of pure overhead under contention: probing now stops at the
first failing chunk. Finally the two copy-pasted wave implementations
are replaced by one shared probe_lock_wave, and the per-key delete
read inputs are resolved once instead of being recomputed.

Tests now assert the wave/fallback counters so they prove which path
ran, and an overlapping-batch test pins the no-deadlock property.

* refactor(ecstore,lock): batch bulk-delete lock rounds per key

Replace the optimistic probe waves with the bulk lock primitive the lock
layer already supports on the wire: one lock_batch RPC per lock client
for a whole DeleteObjects key set, per-key quorum guards with the usual
TTL heartbeat, and per-key outcomes. The batched round is deadlock-free
by construction (every lock server orders concurrent batches by the
same global (shard, key) order), so the probe timeout, the wave
release-and-wait dance, and the canonical serial fallback all go away.

Per-key lock outcomes are now reported per key end to end: a locked-out
key fails alone in the response while the rest of the batch proceeds,
matching the S3 DeleteObjects contract and the set layer's existing
behavior. The set layer's raw-write-lock and delete-transaction-lock
waves fold into the same primitive via one shared resource-folding
helper, and the fast-lock batch fast path now counts acquisitions so
shard metrics keep their meaning.

* perf(ecstore): move journal publish off the one-worker namespace lane

A bulk delete publishes one recovery transaction per key, and every
journal inspection and settle-plan closure ran on the per-disk
foreground namespace lane (one worker), serializing the whole publish
phase per drive behind every other mutation. Route those closures onto
the durable-mutation lane the fused delete prepare already uses (eight
workers per disk, fsyncs coalesce in the filesystem journal), and keep
32 journal publications in flight so a 16-drive set can feed those
lanes.

* test(ecstore): flatten lock-error iteration in overlap test

* fix(ecstore,lock): refresh duplicate keys after recovery, bound unlock

Review findings on the batched lock rounds. A batch entry whose recovery
rolls back an interrupted prepared delete can bring a retained version
back; a later duplicate of the same key still held its pre-fetched
source-missing read, skipped the object-lock check, and deleted the
restored version. Recovery now evicts the pre-fetched reads of every
other entry naming the same object, so those entries take the loop's
live re-read and see the restored version.

The batch rollback release awaited the full unlock retry schedule,
including the deferred background pass, parking the request behind an
unreachable peer. The synchronous phase is now a single release pass
with the deferred retries spawned in the background.

* fix(ecstore): release local batch locks before deferred recovery

A set-layer batch delete that runs deferred key recovery re-acquires a
write lock on the same keys through recover_delete_object, but the raw
local batch guards were only ever released at function exit — the
distributed path releases its batch ids before the deferred loop and the
local path missed the mirror release, so the recovery deadlocked against
the batch's own locks and surfaced as a spurious Lock(Timeout) for every
deferred key.

A regression test plants the interrupted prepared delete this needs and
pins the end-to-end contract: a batch naming a GOVERNANCE-retained
version twice must deny both entries after the first entry's recovery
rolls the prepared delete back, and the restored version must survive.

* test(lock): cover batch acquire partial-grant rollback

The distributed acquire_guard_batch rollback path — a key granted by
fewer than quorum clients must fail alone and release its partial grants
before the call returns — was only reachable in distributed mode and had
no coverage. A scripted per-object LockClient pins both halves: the
below-quorum key reports QuorumNotReached with its partial vote, and the
orphaned grant is released synchronously while the quorate key's guard
stays held.
2026-11-04 13:20:13 +00:00
Ramakrishna Chilaka ff5f491c5a perf(ecstore): skip unmatched FdCache scans and maintenance wakeups (#9165)
Track active descriptor volumes, exact paths, and component prefixes in FdCacheTracker so unmatched invalidate_under, invalidate_volume, and invalidate_exact calls return without scanning Moka or waking the background maintenance task.
2026-11-04 13:12:11 +00:00
Hauser Zhi a8be876fd9 fix(s3): honor presigned x-amz-meta-* query parameters (#9146)
* fix(s3): honor presigned x-amz-meta-* query parameters

The AWS SDK presigners hoist signable x-amz-* headers into the query
string (smithy moveHeadersToQuery), and AWS S3 accepts the hoisted form
as the header it came from. RustFS read user metadata from headers only,
so a presigned PutObject carrying ?x-amz-meta-user-id=u1 returned 200 and
stored an object with empty metadata (rustfs/rustfs#9142).

Add one shared extractor for user-metadata query parameters and wire it
into every metadata-accepting write path (PutObject including snowball
members, CreateMultipartUpload, CopyObject REPLACE) plus the snowball
extract trigger, instead of normalizing per operation the way the
presigned UploadPart checksum fix did. The stored key form is produced by
the same helper as the header path, so the hoisted and header forms land
on disk identically.

Only a signature-covered query is honored: SigV4 signs the canonical
query string, while SigV2 signs headers plus a fixed subresource list
only, so honoring its query would let a presigned-URL holder add object
properties the presigner never authorized. Keys and values that could not
have arrived as a header are rejected rather than stored, since the read
path turns them into header fields and a control character would fail
every Head/Get of that object.

* refactor(s3): hoist signed x-amz query params at the request boundary

Replace the per-write-path query metadata extractor with one boundary
mechanism: restore the `x-amz-*` request properties the SigV4 presigners
hoist into the query (`@smithy/signature-v4` `moveHeadersToQuery`) as
request headers before s3s routes and deserializes the request.

One hook now covers every reader at once — s3s input parsing,
`extract_metadata_from_mime`, the snowball extract trigger and its
transport-key filter, tagging, storage class, object lock and copy
source — so PutObject, CreateMultipartUpload and CopyObject need no
changes at all. It also replaces repeated query parsing along the write
path with a single pass at the boundary.

An existing header always wins, so the hoist is purely additive and the
unsigned-header guard keeps rejecting anything the signer did not sign;
that guard now also accepts a header whose single value equals a
same-named query parameter, which is what the hoist produces and what
the signature covers.

`query_string_is_signed` rules out SigV2 before it trusts any SigV4
marker. SigV2's string-to-sign covers headers plus a fixed subresource
list rather than arbitrary query parameters, so a SigV2 presigned URL
stays valid after `X-Amz-Algorithm`/`X-Amz-Signature` markers are
appended to it; without this rule a URL holder could turn on
`x-amz-meta-snowball-auto-extract` and have a plain upload untarred
(review of #9146).

* fix(s3): hoist signed query params after s3s signature verification

The request-boundary hoist ran before s3s, so the restored `x-amz-*`
headers reached s3s's own verifier, which rejects any `x-amz-*` header
outside `X-Amz-SignedHeaders` (s3s `ops/signature.rs`,
`reject_unsigned_amz_headers`). A valid presigned PUT with
`SignedHeaders=host` and `?x-amz-meta-user-id=u1` therefore came back
with 403 AccessDenied before any handler ran (review of #9146).

Move the hoist to `authorize_request`, RustFS's shared authorization
entry: s3s has already verified the signature there, and policy
evaluation plus the handler read request properties after it, so the
restored headers reach every operation without ever appearing unsigned
to the verifier. The mechanism is unchanged — one gated, validated pass
over the signature-covered query.

CopyObject's replacement metadata now also reads the headers, because
s3s parsed `input.metadata` before the hoist ran.

The hoist itself and its tests stay in `server::layer`; the tower layer
is gone.

* perf(s3): single-pass signature-aware query hoist in auth

Move the query hoist into `auth.rs`, next to the unsigned-header rules it
is part of, so `storage` no longer reaches into `server` for it. Its
documented contract — which request properties the signature covers, and
why SigV2 must never be widened — now lives with the rule it protects.

Make the hot path allocation-free and single-pass. The caller no longer
clones the query just to satisfy the borrow checker, the function no
longer clones it again, and one `form_urlencoded::parse` pass both
recognizes the presigned-SigV4 marker set and stages the hoistable
parameters. Nothing is validated or applied until the gate passes, so a
request whose query is not signature-covered is never rejected here. A
single ASCII scan skips the whole thing when the query names no `x-amz-`
parameter at all, which is every ordinary `?versionId=`/`?uploadId=`
request and every queryless one.

The duplicate-parameter and header-value checks stay strict; only when
they moved, not how they behave.

* fix(s3): keep verifying a query-carried payload hash after the hoist

The hoist restores `?x-amz-content-sha256=<digest>` as a header, but it
runs after s3s has already built the request body. s3s reads its
payload-hash input from the headers only, so it installs no hash
validator for a digest that arrived in the query, and the restored header
then made `get_content_sha256_with_query` conclude the auth layer was
enforcing it. Neither layer verified the body, so a presigned PUT whose
digest exists only in the query accepted a modified payload (review of
#9146).

The question is not whether a digest header is present but whether the
digest reached the request as a query parameter: that one is always the
storage layer's to verify. A digest sent as a header is still left to the
auth layer, which wraps the body in a hash-validating reader for it, so
no payload is hashed twice.

* fix(auth): limit late query hoisting to user metadata

* refactor(s3): drop the stale payload-hash compensation, scope the pre-filter

Limiting the hoist to `x-amz-meta-*` user metadata means it never
manufactures an `x-amz-content-sha256` header, so the payload-hash
predicate's original signal — a digest header present at handler time is
one s3s is enforcing — is exact again. Restore it and drop the
query-aware form: that change existed only to undo a header the hoist is
no longer allowed to create, and it would have hashed a payload twice
for a presigned request that declared a real digest both in the query and
as a header.

Match the pre-filter to the same scope so a query carrying only a typed
property (`x-amz-tagging`, `x-amz-storage-class`, `versionId`) skips the
parse entirely instead of scanning past it.

* test(access): use explicit imports in the signed-query lock regression

The architecture migration rule requires RustFS storage-owner modules to
name their imports instead of pulling the parent module in wholesale, and
`signed_query_lock_permission_regression` was still on `use super::*`.
Import the two symbols it actually uses and drop the unused one.

* test(access): reference FS through the parent import

The layer dependency guard tracks `use crate::...` edges, and naming
`crate::storage::ecfs::FS` from the signed-query regression looked like a
new `infra -> interface` reverse dependency. `access.rs` already reaches
the same type through `use super::ecfs::FS`, so reuse that import instead
of introducing a second, tracked spelling of an edge the module has
always had. No baseline entry, no new edge.

---------

Co-authored-by: overtrue <anzhengchao@gmail.com>
2026-11-04 10:41:52 +00:00
Hauser Zhi fe3ada0689 chore(deps): refresh Cargo dependencies (#9156)
* chore(deps): refresh Cargo dependencies

Co-Authored-By: heihutu <heihutu@gmail.com>
Co-Authored-By: zhi22915 <qiuzgang@gmail.com>

* chore(deps): update rustfs kafka to 1.4.1

Co-Authored-By: heihutu <heihutu@gmail.com>
Co-Authored-By: zhi22915 <qiuzgang@gmail.com>

---------

Co-authored-by: heihutu <heihutu@gmail.com>
Co-authored-by: zhi22915 <qiuzgang@gmail.com>
2026-11-04 09:03:37 +00:00
Hauser Zhi 926355ae1f fix(api): keep SlowDownRead off write-path quorum failures (#9149)
* fix(api): keep SlowDownRead off write-path quorum failures

* refactor(api): write-safe quorum default, GET opts into SlowDownRead

* fix(api): keep object-read mapping behind error.rs architecture boundary

* refactor(api): drop redundant object-read mapper wrapper
2026-11-04 08:08:23 +00:00
Chris An 7f5fa4b8f6 fix(protocols,kms,keystone): enforce client CA, protect the KMS probe (#9168)
Scan findings in the alternate-protocol front ends and the KMS and
Keystone integrations:

- WebDAV and FTPS read RUSTFS_WEBDAV_CA_FILE / RUSTFS_FTPS_CA_FILE,
  checked that the file existed, and then built their TLS listeners
  with with_no_client_auth(), so mTLS was never enforced. A shared
  protocols::common::tls::server_config builds the rustls config with a
  client-certificate verifier from the configured CA, startup fails on
  an invalid CA, and a CA cannot be combined with cleartext
  (backlog#3069).
- The KMS readiness probe used a fixed key id that the key-lifecycle
  admin routes accepted like any customer key, so kms:DisableKey on it
  failed readiness on every node. The probe key is reserved from key
  management and a disabled probe key no longer counts as a backend
  failure (backlog#3115).
- The Keystone middleware answered anonymous requests carrying a junk
  X-Auth-Token with the configured Keystone URL in the body while
  Keystone was unreachable; failures no longer disclose upstream
  locations, and the login cache is bypassed when a subscription token
  refresh is required (backlog#3116).
2026-11-04 20:25:55 +08:00
Chris An a97c65895c fix(proxies,targets,obs): bound untrusted input in leaf crates (#9166)
Scan findings in crates whose behavior is driven by untrusted input and
that stand alone from the storage core:

- trusted-proxies: the simple implementation takes the client address
  from the right-most untrusted X-Forwarded-For hop instead of the
  left-most (backlog#3082); the legacy validator never falls back to the
  proxy identity when the chain is rejected and reads X-Forwarded-For
  unless RFC 7239 is explicitly selected (backlog#3051).
- targets: webhook and similar outbound clients drain responses up to a
  byte limit instead of buffering arbitrary bodies (backlog#3062); the
  on-disk notification queue has a shared byte budget in addition to the
  entry count, configurable via RUSTFS_TARGET_STORE_MAX_BYTES
  (backlog#3122); AMQP and Redis manifests declare the broker url as a
  secret and bind credentials to their connection fields, and the
  plugin-instance PUT refuses to restore a redacted secret when the
  connection fields changed (backlog#3065, backlog#3066).
- obs: the debug-level s3s response dump is filtered so STS and
  service-account credentials never reach the log (backlog#3029).
2026-11-04 19:49:53 +08:00
Chris An 73e7afa01a fix(iam): close identity, session and admin-authorization gaps (#9163)
Scan findings in the IAM crate and the admin identity handlers, grouped
because they share the IamCache mutation lock, the STS session sweep and
the admin client-address plumbing:

- Admin and STS authorization used the socket peer as aws:SourceIp
  behind a trusted proxy; they now read the verified client address the
  S3 path already uses (backlog#3025), and the admin usage route passes
  the bucket named in the path to the s3:ListBucket check (backlog#3033).
- AssumeRoleWithWebIdentity accepted HS256 ID tokens MACed with the OIDC
  client secret; only provider-signed tokens verify (backlog#3028), and
  JWKS refresh is single-flight with a failure cool-down (backlog#3103).
- A disabled group emptied the member's whole policy list, dropping
  every Deny; direct and other-group policies are preserved (backlog#3042).
- Identity rewrites (secret change, add-user, status) ran without a lock
  and without re-reading the record, so a user's password change could
  undo an admin disable or reset (backlog#3048). add-user on an existing
  user and in-flight AssumeRole both left sessions minted under the old
  secret valid (backlog#3071, backlog#3079); rotation now revokes under
  the per-key mutation lock and derived-credential publication is ordered
  with it.
- aws:groups never carried a built-in user's IAM group membership and
  jwt:iss was always the local marker, so group- and issuer-scoped
  Denies never matched; both now come from live IAM state
  (backlog#3095, backlog#3096).
- OPA mode skipped the account-status gates for service accounts and STS
  sessions of a disabled user (backlog#3057).
- The periodic IAM reload published nothing when any cache write raced
  it and kept stale partitions when a prefix listed empty, so a missed
  revocation never applied (backlog#3064, backlog#3114).
- update-group-members never evaluated admin:RemoveUserFromGroup
  (backlog#3098); ExportIAM compared the site-replicator account against
  a name it never has (backlog#3045); add-user accepted the access key of
  a live STS session or service account (backlog#3134); a service account
  could create or extend a lifetime beyond its own (backlog#3112);
  sts:DurationSeconds and svc:DurationSeconds were never populated
  (backlog#3136); temporary-account-info returned service accounts
  (backlog#3137); an access key the object layer refuses no longer holds
  the service-account lock through the write retries (backlog#3133).
2026-11-04 18:44:50 +08:00
Chris An e0f26a540d fix(ecstore): gate null-version overwrites under Object Lock (#9157)
A put-like write without a version id to a key that is not
versioning-Enabled replaces the key's null version in place. The
commit-time Object Lock gate in put_object_with_old_current_size_inner
ran only when the write carried an explicit version id, and the
request-layer pre-check reads the key's latest version. When the
retained null version was not the latest entry (a UUID version or a
delete marker sat above it), a user holding only s3:PutObject could
overwrite it together with its COMPLIANCE retention or legal hold, and
CompleteMultipartUpload had no commit-time gate at all.

Name the version a commit replaces (the explicit id, or the nil id when
the write is not versioned and the bucket's Object Lock snapshot is not
ConfirmedAbsent) and run the existing WORM gate against it, tolerating
a null delete marker. complete_multipart_upload gets the same gate at
its commit point, and ECStore::handle_complete_multipart_upload attaches
the bucket's Object Lock snapshot the way a single PUT already does.

Refs rustfs/backlog#3019 (ANT-2026-EH17ZP59).
2026-11-04 17:26:54 +08:00
Chris An 3b3cc018aa fix(policy): close condition, action, principal and resource gaps (#9160)
Eight scan findings in the shared policy engine, each a case where a
Deny did not apply or an Allow was broader than written:

- A bare `*` Action was parsed as `s3:*`, so `Deny *` never matched
  admin:, kms: or sts: actions. `Action::All` now denies every family
  while the historical S3-only Allow is preserved (backlog#3041).
- A bare IPv6 address in aws:SourceIp got `/32` appended, matching a
  2^96-address network; a bare address now means one host, and IPv4
  clients on a dual-stack listener are matched through their mapped
  form (backlog#3030).
- Date conditions on aws:EpochTime parsed the epoch seconds as RFC 3339
  and never matched, so a time-bounded Deny failed open (backlog#3036).
- NumericNotEquals and DateNotEquals evaluated to false when the key was
  absent; negated operators now follow the AWS rule and match
  (backlog#3123).
- Every trailing IfExists on a condition operator added a level of
  recursion without bound; the chain is rejected before parsing
  (backlog#3091).
- Policy variable substitution re-expanded its own output for up to ten
  passes, so an access key holding k placeholders grew to k^10; values
  are now substituted once as literal data (backlog#3067).
- A bucket-policy Service principal was matched against the signing
  access key, so a service account named after the service received its
  grants; Service principals round-trip but never match (backlog#3081).
- Resource matching cleaned the trailing slash off a directory object,
  so a Deny on `prefix/*` missed `prefix/` itself (backlog#3125).

Also deduplicate large Resource sets through a HashSet instead of a
quadratic scan.
2026-11-04 17:16:17 +08:00
Chris An f4f5635aab fix(auth): build policy conditions from the value a write stores (#9159)
PutObjectLegalHold and PutObjectTagging apply the value in their XML
body, and a POST upload applies its form fields, while the
s3:object-lock-legal-hold, s3:RequestObjectTag/<key> and
s3:RequestObjectTagKeys condition keys (and every other header-derived
key for POST) were built from the HTTP headers. The policy engine
therefore judged a value the handler did not store: absent when the
client sent no header, and whatever the caller wrote when it did. A
user limited by a Deny on the requested legal hold or tag value could
release a hold and delete the held version, or retag an object so a
tag-based grant exposed it.

Record the body value in request extensions before authorization, the
way PutObjectRetention already does, and let authorization_conditions
replace the header-derived keys with it. For POST uploads project the
parsed form fields over the headers that feed the condition map, removing
a header whose field is absent.

Refs rustfs/backlog#3018 (ANT-2026-KONBE376).
2026-11-04 17:00:42 +08:00
Chris An 36ad5a6b87 fix(ecstore): judge the null version a versionless delete removes (#9151)
DeleteObjects ran its Object Lock check on one version of a key and then
removed a different one. For an entry without a version id on a key that
is not versioning-Enabled, delete_objects_with_accounting judged the
key's latest entry and then had the disks remove the null version. When
a UUID version or a delete marker sat above a null version under
COMPLIANCE retention or legal hold, that retention was never read and a
user holding only s3:DeleteObject could remove the retained version. The
single-object DELETE on a Suspended key had the same gap.

Add SetDisks::check_null_version_object_lock_delete, which looks up the
null version under the lock the caller already holds and runs the
existing check_object_lock_delete against it when the latest entry
carries a UUID version id. The batch path is not gated on
source_missing because a delete marker on top makes the latest-version
lookup report the key as missing. Plain unversioned keys and
versioning-Enabled keys keep a single lookup and unchanged behavior.

Refs rustfs/backlog#3017 (ANT-2026-6ED2F7II).
2026-11-04 15:59:48 +08:00
Chris An 6c14128072 fix(s3): reserve the __XLDIR__ directory-marker suffix in client keys (#9153)
Storage encodes a directory object `photos/` as `photos__XLDIR__/xl.meta`
and LocalDisk::scan_dir decodes every directory entry ending in the
suffix back into a marker. The S3 write path never refused the suffix in
client keys, so a 1-byte PutObject to the key `__XLDIR__` made the walk
schedule the directory being scanned as its own child and recurse until
the stack overflowed, aborting the server on every restart until the
directory was removed by hand. Keys below a `*__XLDIR__/` segment were
stored but hidden from every listing and from quota accounting.

validate_object_key now rejects any client key containing the suffix,
which covers PutObject, CopyObject, Snowball extraction and the admin
internal put; CreateMultipartUpload and CompleteMultipartUpload now run
the same validation so uploads created before the key was reserved
cannot publish it. scan_dir additionally ignores a marker whose decoded
name is empty, so malformed markers left by older writers or corrupt
disks can no longer recurse into their parent.

Refs rustfs/backlog#3020 (ANT-2026-SBJ1JLC2).
2026-11-04 15:39:48 +08:00
Chris An addc258a24 fix(admin): keep Vault local-file reads on the operator connection (#9152)
POST /rustfs/admin/v3/kms/reconfigure and configure accepted
server-local file paths in the Vault request body (Kubernetes jwt_path,
AppRole secret_id_file, TokenFile path, TLS ca/client cert and key
paths), read the named file with the server's own authority and sent
its content to the Vault address given in the same body. A user holding
only kms:Configure could read any UTF-8 file the process can read,
including /proc/self/environ with the root credentials, and exfiltrate
it to an address of their choosing.

normalize_configure_request_secrets now runs
validate_vault_local_file_access for both handlers: a Vault request
that names a server-local file, or inherits the stored credentials by
sending a blank token, must keep the operator-configured address and
namespace, the same file-based auth method (role, path, mount) and the
same TLS paths and verification setting. Requests with explicit inline
credentials and no file paths are unchanged.

Refs rustfs/backlog#3021 (ANT-2026-F8MRFKCC).
2026-11-04 15:39:43 +08:00
Chris An 73ea527f55 fix(auth): canonicalize s3:VersionId before policy evaluation (#9150)
The object handlers accept several spellings of one version id (upper
case, unhyphenated, braced, urn:uuid:) and the storage layer resolves
all of them to the same version, but the s3:VersionId condition value
was the client string verbatim. A Deny with StringEquals on one version
could therefore be bypassed by re-spelling the id in ?versionId=.

Canonicalize the id once where the condition value is produced so every
hook and policy type (IAM, bucket, session) sees the spelling the server
emits: hyphenated lower-case UUID, with null in any case and the nil UUID
mapped to null. Non-UUID strings pass through unchanged and are rejected
by the handlers as before.

Refs rustfs/backlog#3022 (ANT-2026-O5MHY8VJ).
2026-11-04 15:39:38 +08:00
hector 55da14ad82 ci(nightly): embed the console UI in the GNU nightly build (#9158) 2026-11-04 15:21:41 +08:00
唐小鸭 116a5299b2 fix(replication): replay the MRF journal periodically (#9139)
* fix(replication): replay the MRF journal periodically

The persisted MRF journal was replayed only at startup. Delete-marker
purges whose watch window drained with targets unreachable, and
replications that overflowed the worker queues, reach the journal while
the server runs, so they waited for a restart to converge.

Move the startup pass into replay_mrf_journal and rerun it every five
minutes from a background loop. The existing recovery leader lock keeps
one pass at a time across the cluster, including the startup pass.

* fix(replication): space periodic MRF replay across nodes

Every node runs the MRF replay loop and the recovery leader lock only
serializes passes, so an N-node cluster replayed the same retained
backlog N times per interval while a target stayed down.

Record the start time of a pass that finds entries in a shared replay
stamp and skip a periodic pass when any node started one within the
interval. The startup pass still always runs.
2026-11-03 17:06:14 +00:00
Hauser Zhi fbc54f7212 fix(heal): treat SlowDown as a rate signal and bound page churn (#9144)
* fix(heal): treat SlowDown as a rate signal and bound page churn

* refactor(heal): unify page rate control and drop checkpoint clones

* refactor(heal): one pass-wide rate controller, single SlowDown observe

* fix(heal): address SlowDown backoff and checkpoint baseline review
2026-11-03 17:05:05 +00:00
Ramakrishna Chilaka 3aa492f5ee perf(ecstore,rpc): shard hot-path delete, multipart, and RPC lock tables (#9143)
Replace the single-mutex PREPARED_DELETE_RECEIPTS and ACTIVE_MULTIPART_SOURCE_HEADS maps in rustfs-ecstore with a 64-shard ShardedPathCache backed by SWAR canonical byte keys and xxh3_64 hashing. Replace PUT_FILE_TARGET_LOCKS in rustfs internode RPC with a 64-shard ShardedPutFileTargetLockRegistry using stack-buffered binary keys, in-place dead-Weak slot reuse, and amortized garbage collection.
2026-11-03 15:11:58 +00:00
Hauser Zhi 00a81b4048 chore(deps): refresh crate requirements and lockfile (#9140)
Run cargo update and cargo upgrade twice to bump compatible requirements (jiff, zip, dial9) and refresh transitive lockfile entries.
2026-11-03 15:04:29 +00:00
唐小鸭 20ef9ebf15 fix(replication): skip site-replication rules in replica delete fan-out (#9138)
* fix(replication): skip site-replication rules in replica delete fan-out

A site that receives a replicated delete forwards it to its own targets
so chained replication keeps propagating deletes. The fan-out used the
whole bucket config, including derived site-replication rules, so in a
site-replication mesh a replicated purge was echoed back to its origin
and duplicated to every other peer. The echo could remove the origin's
pending-purge version before its own delivery state was written back.

Ignore derived site-replication rules when computing the replica delete
fan-out. The origin already delivers and retries every delete to each
peer itself. Operator rules still fan out, so chained replication is
unchanged.

* fix(replication): keep operator rules matching the site-repl shape

Rule ids and target ARN shapes are not reserved for site replication:
an ordinary bucket target ARN carries its target id, so an operator rule
named site-repl-<target id> matched the derived shape and was dropped
from replica delete fan-out, breaking an A -> B -> C chain.

Skip a rule only when its deployment id is a current remote
site-replication peer, reading the peer set only when a rule has the
derived shape. If the state cannot be read, skip nothing.
2026-11-03 12:08:08 +00:00
Boyi (Elliot) Li cc08e0d11b feat(fips): route internode RPC HMAC through aws-lc-rs FIPS (#9137)
* feat(fips): route internode RPC HMAC through aws-lc-rs FIPS

* Revise KMS cryptographic compliance details

Updated the KMS cryptographic compliance documentation to reflect changes in HMAC authentication and added details about the internode RPC HMAC authentication.

Co-authored-by: Copilot Autofix powered by AI <175728472+Copilot@users.noreply.github.com>
Signed-off-by: Hauser Zhi <housemecn@gmail.com>

---------

Signed-off-by: Hauser Zhi <housemecn@gmail.com>
Co-authored-by: Hauser Zhi <housemecn@gmail.com>
Co-authored-by: Copilot Autofix powered by AI <175728472+Copilot@users.noreply.github.com>
2026-11-03 06:56:56 +00:00
Hauser Zhi 69294523d5 fix(heal): preserve lock, format, and repair identity (#9132)
* fix(heal): preserve lock, format, and repair identity

Co-Authored-By: heihutu <heihutu@gmail.com>
Co-Authored-By: zhi22915 <qiuzgang@gmail.com>

* fix(ecstore): keep pool meta legacy error message stable for ratchet

Co-authored-by: houseme <4829346+houseme@users.noreply.github.com>

* fix(heal): drop redundant clone in MRF dedup test

Co-authored-by: houseme <4829346+houseme@users.noreply.github.com>

---------

Co-authored-by: heihutu <heihutu@gmail.com>
Co-authored-by: zhi22915 <qiuzgang@gmail.com>
Co-authored-by: Chris An <anzhengchao@gmail.com>
Co-authored-by: copilot-swe-agent[bot] <198982749+Copilot@users.noreply.github.com>
Co-authored-by: houseme <4829346+houseme@users.noreply.github.com>
2026-11-03 04:59:06 +00:00
Chris An cd4d86f28a fix(replication): finish writeback for already purged versions (#9135) 2026-11-03 02:08:32 +00:00
Hauser Zhi 2f55cf69e1 perf(filemeta): grow metacache read chunks geometrically (#9133)
Large metacache elements (an xl.meta grown past 16 MiB with its version
history) were accumulated in fixed 64 KiB reads, paying hundreds of
read rounds per entry on the listing path. Double the per-round chunk up
to a 4 MiB bound so each reservation stays incremental and bounded
(rustfs/rustfs#2715 corrupt-length-prefix safety is unchanged) while
wide listings and oversized entries decode in far fewer rounds.

Add regression coverage for rustfs/rustfs#9120: a listing whose
aggregate stream exceeds the per-value 16 MiB bound, and an oversized
entry listed among ordinary neighbors with peek/skip interleaving.
2026-11-03 06:07:41 +08:00
Hauser Zhi fff6e94378 fix(metrics): reconcile peer-health gauge on successful RPCs and probes (#9134)
The rustfs_cluster_servers_offline_total gauge was event-driven only:
record_peer_reachable fired solely after a fresh connector.connect()
succeeded. When post-recovery traffic was served by a cached channel
(tonic auto-reconnect, addr-form cache-key split, or delete-recovery
channel), no fresh dial occurred and the gauge stayed stale until the
process restarted.

Add record_peer_reachable_if_offline (fast-path read / slow-path write)
and wire it into every successful peer interaction:
- remote-disk recovery probe success (gRPC DiskInfo)
- periodic TCP connectivity check success (bounded-interval reconcile)
- data-path RPC success (execute_with_timeout_for_op_and_health_action)
- lock-path RPC success (record_rpc_success and detached completion)

Fixes rustfs/rustfs#9119
Tracking: rustfs/backlog#3138
2026-11-02 19:07:37 +00:00
Hauser Zhi 41b13aeb8b fix(tls): unify FIPS provider policy across clients (#9125)
* fix(tls): enforce active FIPS provider at startup

Co-Authored-By: heihutu <heihutu@gmail.com>
Co-Authored-By: zhi22915 <qiuzgang@gmail.com>

* ci: provision FIPS validation runner

Co-Authored-By: heihutu <heihutu@gmail.com>
Co-Authored-By: zhi22915 <qiuzgang@gmail.com>

* fix(tls): validate FIPS before observability startup

Co-Authored-By: heihutu <heihutu@gmail.com>
Co-Authored-By: zhi22915 <qiuzgang@gmail.com>

* perf(tls): reuse the installed crypto provider

Co-Authored-By: heihutu <heihutu@gmail.com>
Co-Authored-By: zhi22915 <qiuzgang@gmail.com>

* fix(ecstore): gate Linux-only commit helpers

Co-Authored-By: heihutu <heihutu@gmail.com>
Co-Authored-By: zhi22915 <qiuzgang@gmail.com>

* refactor(tls): centralize FIPS provider policy

Co-Authored-By: heihutu <heihutu@gmail.com>
Co-Authored-By: zhi22915 <qiuzgang@gmail.com>

---------

Co-authored-by: heihutu <heihutu@gmail.com>
Co-authored-by: zhi22915 <qiuzgang@gmail.com>
Co-authored-by: Chris An <anzhengchao@gmail.com>
2026-11-02 16:58:42 +00:00
weile321 88587a8f94 fix(s3select-api): clamp overflowing u64 GetRange values to i64::MAX (#9118) 2026-11-02 14:05:07 +00:00
hector 8c8b5da63f ci(nightly): dispatch nightly-candidate-ready after asset publish (#9131)
After the nightly packages are on the auto-testing nightly release and
the schema 2 R2 candidate exists, post repository_dispatch to
rustfs/auto-testing. Vault lanes stay parallel, and the workflow_run
functional chain is left in place.

Co-authored-by: hector <majinghe@users.noreply.github.com>
2026-11-02 23:05:07 +08:00
Henry Guo 00a238f838 fix(table-catalog): resolve superseded staged commits (#9124)
fix(table-catalog): diagnose superseded staged commits

Co-authored-by: Chris An <anzhengchao@gmail.com>
2026-11-02 23:04:22 +08:00
hector 965d805a4f ci: run distributed and upgrade e2e on schedule only (#9126)
Remove pull_request from the distributed and upgrade workflows. Distributed
stays on the nightly cron, and upgrade stays on the Monday cron plus
version tags. Both remain manual-dispatch lanes and stay outside the
required merge checks.

Co-authored-by: majinghe <majinghe@users.noreply.github.com>
2026-11-02 23:03:36 +08:00
Hauser Zhi 9d66e04ae3 test(e2e): honor namespace backpressure in mixed workload (#9122)
Give each mixed-workload request a bounded policy for the server's
explicit namespace lock timeout response and Retry-After advice. Keep
unrelated SDK failures and data mismatches immediately visible.

Preserve 24 workers, four rounds, cross-node checks and all retained
objects. Cover exhausted SDK retries, bodyless HEAD, fatal responses,
persistent contention and incorrect acknowledged bytes, and include the
new cases in the protected distributed profile inventory.

Co-authored-by: heihutu <heihutu@gmail.com>
Co-authored-by: zhi22915 <qiuzgang@gmail.com>
2026-11-02 23:02:22 +08:00
dependabot[bot] 12c58d25b5 chore(deps): bump the dependencies group across 1 directory with 5 updates (#9113)
* chore(deps): bump the dependencies group across 1 directory with 5 updates

Bumps the dependencies group with 5 updates in the / directory:

| Package | From | To |
| --- | --- | --- |
| [pulsar](https://github.com/streamnative/pulsar-rs) | `6.9.0` | `6.9.1` |
| [arrow-array](https://github.com/apache/arrow-rs) | `59.3.2` | `60.1.0` |
| [arrow-ipc](https://github.com/apache/arrow-rs) | `59.3.2` | `60.1.0` |
| [arrow-schema](https://github.com/apache/arrow-rs) | `59.3.2` | `60.1.0` |
| [parquet](https://github.com/apache/arrow-rs) | `59.3.2` | `60.1.0` |



Updates `pulsar` from 6.9.0 to 6.9.1
- [Release notes](https://github.com/streamnative/pulsar-rs/releases)
- [Commits](https://github.com/streamnative/pulsar-rs/compare/v6.9.0...v6.9.1)

Updates `arrow-array` from 59.3.2 to 60.1.0
- [Release notes](https://github.com/apache/arrow-rs/releases)
- [Changelog](https://github.com/apache/arrow-rs/blob/main/CHANGELOG.md)
- [Commits](https://github.com/apache/arrow-rs/compare/59.3.2...60.1.0)

Updates `arrow-ipc` from 59.3.2 to 60.1.0
- [Release notes](https://github.com/apache/arrow-rs/releases)
- [Changelog](https://github.com/apache/arrow-rs/blob/main/CHANGELOG.md)
- [Commits](https://github.com/apache/arrow-rs/compare/59.3.2...60.1.0)

Updates `arrow-schema` from 59.3.2 to 60.1.0
- [Release notes](https://github.com/apache/arrow-rs/releases)
- [Changelog](https://github.com/apache/arrow-rs/blob/main/CHANGELOG.md)
- [Commits](https://github.com/apache/arrow-rs/compare/59.3.2...60.1.0)

Updates `parquet` from 59.3.2 to 60.1.0
- [Release notes](https://github.com/apache/arrow-rs/releases)
- [Changelog](https://github.com/apache/arrow-rs/blob/main/CHANGELOG.md)
- [Commits](https://github.com/apache/arrow-rs/compare/59.3.2...60.1.0)

---
updated-dependencies:
- dependency-name: arrow-array
  dependency-version: 60.1.0
  dependency-type: direct:production
  update-type: version-update:semver-major
  dependency-group: dependencies
- dependency-name: arrow-ipc
  dependency-version: 60.1.0
  dependency-type: direct:production
  update-type: version-update:semver-major
  dependency-group: dependencies
- dependency-name: arrow-schema
  dependency-version: 60.1.0
  dependency-type: direct:production
  update-type: version-update:semver-major
  dependency-group: dependencies
- dependency-name: parquet
  dependency-version: 60.1.0
  dependency-type: direct:production
  update-type: version-update:semver-major
  dependency-group: dependencies
- dependency-name: pulsar
  dependency-version: 6.9.1
  dependency-type: direct:production
  update-type: version-update:semver-patch
  dependency-group: dependencies
...

Signed-off-by: dependabot[bot] <support@github.com>

* chore(deps): update dependencies and pin pulsar 6.9.0

Co-Authored-By: heihutu <heihutu@gmail.com>

Co-Authored-By: zhi22915 <qiuzgang@gmail.com>

* chore(deps): ignore pulsar 6.9 updates

Co-Authored-By: heihutu <heihutu@gmail.com>

Co-Authored-By: zhi22915 <qiuzgang@gmail.com>

---------

Signed-off-by: dependabot[bot] <support@github.com>
Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
Co-authored-by: Hauser <housemecn@gmail.com>
Co-authored-by: zhi22915 <qiuzgang@gmail.com>
2026-11-02 23:01:55 +08:00
Ramakrishna Chilaka 1ac1d883f4 perf(ecstore): shard disk path lock registries with SWAR byte keys (#9127) 2026-11-02 23:00:48 +08:00
594 changed files with 94035 additions and 34993 deletions
+21 -26
View File
@@ -1,43 +1,38 @@
---
name: rustfs-release-publish
description: Publish a RustFS version through Console, release-branch CI, preview acceptance, human confirmation, and final publication. Use only for an explicit release request.
description: Publish a RustFS version through frozen Linux candidate acceptance, desktop completion, manifest approval, byte-preserving publication and post-release follow-up. Use only for an explicit release request.
---
# RustFS Release
Require the exact final SemVer target before version edits or publication. If unspecified, inspect current releases/tags and ask with concrete candidates; do not guess the channel or bump. A named bump type resolves to an exact version that still needs confirmation. Report the confirmed target verbatim; resolve any later mismatch before proceeding. Independent read-only preflight may continue.
Require the exact final SemVer target before version edits or publication. If unspecified, inspect releases/tags and ask with concrete candidates; do not guess the channel or bump. Reuse the user's confirmed target and authorized delivery scope. Independent read-only preflight may continue.
## Invariants
- Cargo stores `<target>`, never `-preview.N`. Before publication, installation defaults retain the previous available deliverable. Final and preview binaries embed their own tag names, so version/asset names differ despite identical source.
- Use annotated tags without `v`. Preview names are exactly `<target>-preview.<digits>`; choose the next unused iteration after fetching tags. Preview Releases are prereleases, never Latest; publish only versioned assets/exact Docker tags, never latest assets, R2, Helm, moving Docker channels, or installation/banner updates.
- `release/<target>` owns the candidate. Retain its history, backport only reviewed fixes, and never merge main wholesale or force-push it. Pin `PREVIEW_HASH` to its exact commit with complete passing CI. Preview and final tags both point there; never tag a later main/branch HEAD or insert another bump.
- Phase failures block dependents. Keep preview Releases until final publication and notes generation finish; CI removes Releases while retaining tags. Never move an existing preview or final tag.
- Cargo and every candidate executable contain the final product version. `<target>-preview.<run_number>` is an immutable candidate identifier, not a public tag or displayed binary version.
- Freeze the exact source, reviewed workflow recipe, toolchain, Console, CLI, dependency lock, previous stable version and validation plan. Newer unrelated main or Console releases do not change those inputs.
- Retain `release/<target>` history when a release branch is used; backport only reviewed fixes. Pin `CANDIDATE_SHA` to its exact CI-validated commit. Never force-push a release branch or move a public tag.
- Build Linux first. Complete field acceptance before macOS/Windows and packaging. Full certification and final acceptance are required before publication; a partial Linux PASS grants no release authority.
- Promote the accepted bytes. Final publication never recompiles binaries or rebuilds packages/images. Infrastructure retries reuse successful artifacts; changed source or inputs require a new candidate and affected acceptance.
- Phase failures block dependents. No latest/default-channel update before versioned delivery. Use the [pipeline runbook](../../../docs/operations/release-pipeline.md) for executable entry points, locks and recovery.
## Read by Phase
Read only the current phase's procedure; retain completed evidence across turns. Follow root verification, Git/PR, and authorization rules.
## Read by phase
| Phase | Procedure |
|---|---|
| 0 | Clean tree, fetch main/tags, verify GitHub access, then complete the mandatory [Console release gate](references/console-gate.md). A green Console build alone is insufficient. |
| 1–2 | [Prepare/reuse the release candidate and run exact-SHA branch CI](references/release-candidate.md). Source version changes use `rustfs-release-version-bump` with authorized delivery scope. |
| 3 | [Verify the preview build, assets, channels, and notes](references/preview-release.md). |
| 4–5 | [Run the downloaded binary, Console CRUD, and latest-rc matrix](references/preview-acceptance.md). Every check must pass. |
| Confirmation | Complete the gate below and end the turn. |
| 6 | After fresh confirmation, [publish and verify the final tag](references/final-release.md) at `PREVIEW_HASH`. |
| 7 | [Reconcile Security Advisories, milestones, installation references, and the website; reconcile main and remove the completed release branch](references/post-release-updates.md) only after complete publication. |
| 0 | Clean tree, fetch main/tags, verify access, then complete the mandatory [Console gate](references/console-gate.md). |
| 1–2 | [Prepare the exact source and run branch CI](references/release-candidate.md). Version edits use `rustfs-release-version-bump`. |
| 3 | [Build and verify the Linux candidate](references/preview-release.md). |
| 4–5 | [Run artifact, field, Console and rc acceptance](references/preview-acceptance.md), then complete desktop/package/image/chart validation. |
| Confirmation | Report the complete manifest and obtain the confirmation below. |
| 6 | [Publish the accepted manifest](references/final-release.md). |
| 7 | [Reconcile advisories, milestones, installation references, website and release branch](references/post-release-updates.md). |
## Human Confirmation
## Human confirmation
After Phases 3–5 pass, recheck that the release branch still equals `PREVIEW_HASH` and that the current Console asset tag/digest match the preview's recorded build inputs. Report the target, branch, preview tag/hash, branch CI run, preview Release URL, accepted Console asset identity/result, and rc matrix. Explicitly ask whether to publish the final tag and end the turn without creating/pushing it.
After all acceptance passes, report the target, source SHA, source CI run, Linux and completion run URLs, final manifest SHA-256, accepted Console/CLI identities, compatibility/performance results and backlog report. Explicitly ask whether to publish this exact manifest and end the turn without dispatching publication.
Only a new affirmative user reply to that exact report authorizes Phase 6. The original release request, earlier confirmation, silence, or automated follow-up does not count. Confirmation binds the target, preview tag, hash, and accepted build inputs; a changed value or repeated/failed acceptance cycle invalidates it. Revalidate and obtain fresh confirmation before continuing.
Only a new affirmative reply to that exact report authorizes Phase 6. An earlier general release request, silence or automatic continuation does not count. Reuse confirmation for retries of that same manifest; do not ask again merely because a destination failed. A changed source, input, manifest or acceptance result invalidates it and requires renewed acceptance/confirmation.
## Recovery
## Recovery and reporting
- Before the final tag exists, land fixes on main, backport to the candidate, verify target versions, and restart branch CI/acceptance. Use a new preview iteration once a preview was tagged; main may advance independently.
- If only an external build input changed, record it, create a new preview iteration even when the source SHA is unchanged, and repeat the affected build and artifact acceptance before fresh confirmation. Reuse source checks whose inputs remain valid.
- Once the final tag exists, retry publication jobs for that exact tag. A source fix requires a newly confirmed target, not retagging or a newer preview. Phase 7 retries recheck artifact availability and superseding releases.
- If abandoned after the bump merged, report the unpublished version on main and record whether to revert or let the next release overwrite it. Deleting unpublished candidate branches or preview tags requires an explicit decision. Completed release branches follow Phase 7 cleanup; keep release tags.
Report completed phase results and incomplete work with evidence: Console previous/latest tags, release need, hash and run/Release URLs; target, branch CI/SHA, previews and hash, confirmation state; preview/final assets, latest-channel state, Console/rc results and cleanup; advisory reconciliation results and pending disclosure decisions; both milestone states, installation PR checks/merge state, and website text/link/PR/deployment state. Keep artifact publication separate from follow-up delivery; do not call an open PR a live update. Report deviations and their authorization.
Resume failed infrastructure jobs without rebuilding successful units. Fix product defects on main, backport when needed, and start a new candidate. After a final tag exists, retry the same bytes or use a new patch version; never move the tag. Do not conflate implemented, CI-passed, field-accepted, approved, published and post-release-complete states. Report incomplete work and deviations with evidence; a tag or green build alone is not a completed release.
@@ -4,7 +4,7 @@ Read during Phase 0, before changing RustFS version files or tags. Follow the pa
### Console release gate
Complete this gate before changing any RustFS version file or creating any RustFS tag. RustFS `build.yml` downloads the asset returned by `repos/rustfs/console/releases/latest`, so a successful Console build alone is insufficient.
Complete this gate before changing any RustFS version file or creating any RustFS tag. Candidate creation resolves `repos/rustfs/console/releases/latest` once and freezes the exact asset for every build, so a successful Console build alone is insufficient. After freezing, later Console releases do not silently change or invalidate the accepted inputs.
1. Read the latest published Console tag and compare it with Console `main`:
@@ -1,22 +1,13 @@
# Final Release Publication
# Final release publication
Read only after fresh user confirmation of the preview acceptance report. The [parent confirmation gate](../SKILL.md#human-confirmation) applies.
Read only after fresh confirmation of the complete candidate report. The [parent confirmation gate](../SKILL.md#human-confirmation) applies.
## Phase 6 — Publish the final tag on the validated commit
Dispatch `release-publish.yml` on main with the successful completion run ID and approved `release-manifest.json` SHA-256. Do not create the tag manually. The workflow revalidates source, bytes and trusted certification, then creates the annotated final tag at `CANDIDATE_SHA`. A recipe change fails closed and requires a new candidate; unrelated main changes do not require a rebuild.
No second source version bump. Before creating the final tag, recheck the remote `release/<target>` HEAD equals `PREVIEW_HASH` and the current Console release tag/asset digest still match the accepted preview inputs. If either changed after confirmation, the approval no longer applies; validate the changed inputs with a new preview and obtain fresh confirmation. The final tag goes on the exact commit the preview validated:
Require both versioned publication and channel publication jobs to succeed. Verify the final tag's commit and manifest identity; all six platform ZIPs and compatibility aliases; DEB/RPM, checksums, SBOM, provenance and evidence bundle; both multi-architecture image variants in Docker Hub, GHCR and Quay; and the accepted Helm chart in the chart repository. Download a host-compatible final artifact and confirm its digest is exactly the accepted candidate digest and its version/source output agrees.
```bash
git fetch origin --tags
git rev-parse "<preview-tag>^{commit}" # must equal PREVIEW_HASH — abort if not
git tag -a "<target>" -m "Release <target>" "$PREVIEW_HASH"
git push origin "<target>"
```
For stable releases verify GitHub Latest, R2 download aliases and `latest.json` unless a newer stable release already owns those channels. Prereleases must remain GitHub prereleases and leave stable pointers unchanged. Verify generated release notes compare against the frozen previous stable release, excluding internal previews.
- CI rebuilds from the same source with a different tag name, so the binary self-reports `<target>`. Other inputs can change: `build.yml` reads Console `releases/latest` during each build. Compare the actual Console tag, URL, and verified digest in every final build job with the accepted preview record, and inspect available build provenance for other relevant changes. A pre-tag check cannot prevent a later input change. On a mismatch or missing evidence, repeat affected acceptance against the final artifacts and report the limitation; failed acceptance leaves release validation incomplete even if publication already succeeded. Never move the final tag to conceal a failure.
- Verify the final tag's complete publication path: all matrix and release jobs green; `gh release view "<target>"` shows the full versioned and `-latest` asset set plus checksums, SBOM, and provenance; Docker and Helm workflows succeed; `latest.json` points to `<target>`. A stable target must have `isPrerelease=false` and `isLatest=true`. An alpha/beta/rc target must have `isPrerelease=true`; GitHub does not permit prereleases to be Latest, but the project `latest.json` still advances to the final non-preview target.
- Verify the final Release body contains `## What's Changed` and a Full Changelog link. When `PREVIOUS_DELIVERABLE` exists, the link MUST be `https://github.com/rustfs/rustfs/compare/<PREVIOUS_DELIVERABLE>...<target>` and the baseline MUST equal the preview Release baseline; for example, both `1.0.0-beta.12-preview.1` and `1.0.0-beta.12` compare from `1.0.0-beta.11`.
- Verify the preview cleanup: `cleanup-preview-releases` must succeed and `gh release view "<preview-tag>"` must then report `release not found` for every preview iteration of this target. Each surviving preview tag must still resolve to its own recorded commit; the accepted preview resolves to `PREVIEW_HASH`. If the job failed, delete the leftover Releases manually with `gh release delete "<preview-tag>" --yes` and report it.
- Verify a host-compatible final artifact's checksum and `./rustfs --version`; it must report `<target>` and the expected source SHA. This identity check does not replace affected acceptance when build inputs changed.
On infrastructure failure retry the same workflow/seal; matching destinations are reused. Never clobber an asset, force a tag, rebuild the image or silently accept a changed digest. A source fix after the final tag requires a new patch version. Cross-registry writes are not atomic: report the unfinished destination rather than claiming publication complete.
Never pass `--cleanup-tag` when removing preview Releases. Remove them manually only if `cleanup-preview-releases` failed, after final publication and notes generation. Record deleted Releases and surviving tags.
New candidates create no preview Releases to clean up. For a legacy preview release, preserve its tag and remove its Release only after final delivery and notes are verified. Do not pass `--cleanup-tag`. Complete the [post-release follow-up](post-release-updates.md) separately.
@@ -24,7 +24,7 @@ Reconcile advisories after every completed final release, including Console-only
1. Re-read the published deliverable state before edits or a retry. If a newer release has superseded this target, do not downgrade installation references or the website announcement. Report the superseding release and leave those defaults intact.
2. Invoke `rustfs-release-version-bump` with stage `post-release`, the exact target, and the authorized delivery scope. Prepare the follow-up on current main; do not edit or move either validated release tag. Cargo versions may already be preparing the next target and must remain untouched.
3. Verify the exact Docker image manifest, Release download links, RPM source archive, and Helm repository entry before publishing the installation PR. The chart workflow overrides chart/app versions from the final tag, pins the source checkout to that release, and waits for its default Docker image before packaging.
3. Verify the exact Docker image manifest, Release download links, RPM source archive, and Helm repository entry before publishing the installation PR. The chart is packaged and tested before sealing; publication uploads those same bytes only after versioned Docker images are available.
4. Report the installation PR and its merge state separately from the completed release. Reuse existing merge authorization and repository checks; do not infer new merge authority from this reference.
## rustfs.com top banner
@@ -54,4 +54,4 @@ Use the existing announcement configuration from [website PR #112](https://githu
1. Compare the exact published tag with current main. Audit every release-side commit since their merge base, including inherited backports. An ahead count is a history difference, not proof of missing changes. Use ancestry and patch equivalence, then inspect current code and regression evidence for squashed, adapted, superseded, or reverted changes. Record each commit as present, superseded, release-only, or missing; a positive `git cherry` result alone does not prove a missing fix.
2. Bring missing shared changes to main through a reviewed PR and the required checks before closing the follow-up. Preserve main's later changes; decide version bumps and pinned Console configuration separately. If a history merge is needed, use a merge commit and verify the release commit is an ancestor of main; squash does not establish that ancestry. Do not merge main into the published release branch or move its tags. Report unresolved changes instead of claiming reconciliation is complete.
3. After publication and authorized follow-ups are complete, delete the remote `release/<target>` branch. Immediately before deletion, verify its HEAD still equals the published tag's commit and no open PR targets or originates from it. Preserve a branch with newer work or open PRs and report the blocker. Use a deletion with an expected-old-SHA check to avoid removing a concurrently advanced branch.
4. Verify the remote branch is absent and the final and preview tags still resolve to their recorded commits. Remove a local release branch only if it is task-owned, unchecked out, and has no unpreserved work. Record cleanup separately from artifact publication; an explicitly authorized deletion before reconciliation does not make unresolved main changes complete.
4. Verify the remote branch is absent and the final tag and any legacy preview tags still resolve to their recorded commits. Remove a local release branch only if it is task-owned, unchecked out, and has no unpreserved work. Record cleanup separately from artifact publication; an explicitly authorized deletion before reconciliation does not make unresolved main changes complete.
@@ -1,10 +1,10 @@
# Preview Artifact Acceptance
# Candidate Artifact Acceptance
Read after Phase 3 succeeds. Complete every check below before the parent skill's manual confirmation gate. These checks cover the downloaded artifact, embedded Console, and latest rc client.
Read after Phase 3 succeeds. Complete every check below before the parent skill's manual confirmation gate. These checks cover the downloaded artifact, embedded Console, and the rc release frozen in the candidate inputs.
## Phase 4 — Run the artifact locally, verify the console
Use a unique task-owned scratch directory. Select the versioned archive matching the host OS, architecture, and runtime, download its checksum file, and verify the archive before extraction. Do not substitute a different build when a matching artifact is unavailable; report that acceptance gap. The shell example below assumes the verified executable is at `RUSTFS_BINARY` (`rustfs.exe` on Windows), `SCRATCH` is the task directory, and `S3_PORT` and `CONSOLE_PORT` are distinct available ports. Use equivalent native commands when the shell differs.
Use a unique task-owned scratch directory. Select the versioned archive matching the acceptance host, and verify it against the sealed manifest before extraction. Before desktop completion, use the Linux acceptance environment rather than rebuilding for the local desktop. Do not substitute a different build when a matching artifact is unavailable; report that acceptance gap. The shell example below assumes the verified executable is at `RUSTFS_BINARY` (`rustfs.exe` on Windows), `SCRATCH` is the task directory, and `S3_PORT` and `CONSOLE_PORT` are distinct available ports. Use equivalent native commands when the shell differs.
Bind both listeners to loopback. Choose task-only credentials, set loopback proxy exclusions for the server/client processes, and record the server PID. A port probe does not reserve a port; retry startup with fresh ports on a bind conflict rather than connecting to an existing service.
@@ -24,16 +24,16 @@ PREVIEW_SERVER_PID=$!
Checks (all must pass):
- `"$RUSTFS_BINARY" --version` reports the preview tag name and the short SHA of `PREVIEW_HASH`. Reporting `<target>` without the `-preview.N` suffix means the build did not embed the tag — treat as FAIL and investigate before proceeding.
- `"$RUSTFS_BINARY" --version` reports the exact final `<target>` and full `CANDIDATE_SHA`. A preview suffix, another version or another source is a failure.
- Wait with a bounded readiness deadline while checking the recorded server process remains alive; `curl --noproxy '*' -fsS "$S3_ENDPOINT/health/ready"` must return ready from this process's listener.
- Startup log shows the embedded console being served (this was the regression that `fix(release): require embedded console assets` guards).
- Open `CONSOLE_ENDPOINT` in the browser and log in with the task credentials. The dashboard must render without JS console errors; create a bucket, upload a file, download it back (byte-identical), and delete the object and bucket. Keep the server running for Phase 5.
## Phase 5 — Validate with the latest rc client
## Phase 5 — Validate with the pinned rc client
`rc` is the RustFS CLI client from <https://github.com/rustfs/cli>.
- Resolve the latest release with `gh api repos/rustfs/cli/releases/latest --jq .tag_name`; verify the local client's version or download and verify a host-compatible client into the task directory. Set `RC_BIN` to that executable and record its version/digest. Avoid changing the user's installed client for this check.
- Read the CLI release and asset digests from `inputs.json`; verify the local client's version or download a host-compatible asset from that exact release into the task directory. If the host is a desktop, resolve its asset within the same pinned CLI release and record the digest; do not independently follow `latest`. Set `RC_BIN` to that executable and record its version/digest. Avoid changing the user's installed client for this check.
- Inspect the client's current help for an isolated configuration option and use it when supported. Otherwise choose a unique `PREVIEW_ALIAS` and confirm it is absent before creating it; never overwrite an existing alias. Use task-owned bucket/user names and input/output files. The names below are safe only inside the newly started isolated server.
- Point the client at `S3_ENDPOINT` with loopback proxy exclusions and run the command matrix, recording PASS/FAIL per command:
@@ -60,3 +60,9 @@ cmp <local-file> "$SCRATCH/out"
```
- Any FAIL blocks acceptance. On success or failure, stop only the recorded task process and remove only task-created aliases/data after retaining the needed failure evidence. Do not erase an existing client configuration or unrelated server data.
## Field acceptance and desktop completion
Before accepting the Linux manifest, record Scanner/Heal, multi-pool, memory, major features, high-concurrency 503, performance and previous-version integrity/upgrade results in backlog. Preserve the release-specific agreed limits and baseline; a PASS label without artifact identities or test results does not satisfy acceptance.
Dispatch `release-complete.yml` only after that report passes, using the exact Linux manifest digest and backlog URL. Require native macOS/Windows checks, package extraction, image startup/scans, chart tests and complete certification to pass. Complete any desktop Console/client checks applicable to the release against these same archives. Record the completion run and final manifest SHA-256 before entering the parent confirmation gate. Partial Linux acceptance does not authorize final publication.
@@ -1,25 +1,9 @@
# Preview Release Verification
# Linux candidate verification
Read for Phase 3 after the candidate passed exact-SHA CI. The [parent release invariants](../SKILL.md) apply.
Read for Phase 3 after the source passed exact-SHA CI. The [parent invariants](../SKILL.md) apply; commands and recovery live in the [pipeline runbook](../../../../docs/operations/release-pipeline.md).
## Preview channels
Dispatch `release-candidate.yml` on main with the confirmed final version and `CANDIDATE_SHA`. Record the run ID, frozen `inputs.json`, every Linux archive digest and `linux-manifest.json` SHA-256. Require the four Linux builds, native startup/S3 checks, previous-release upgrade matrix and Linux seal to succeed. Missing, skipped, cancelled or failed required work is incomplete acceptance.
- Use `<target>-preview.N` for every target, e.g. `1.0.0-beta.10-preview.3` or `1.1.0-preview.1`.
- The canonical suffix is exactly `-preview.<digits>`. `build.yml` recognizes it before alpha/beta/rc classification and routes it to the preview-only path; any other tag containing `-preview` fails closed instead of being treated as a release.
- A preview Release MUST be published with `isPrerelease=true` and `isLatest=false`. Any `*-latest` preview asset or preview-triggered `latest.json`, R2, Helm, or moving Docker channel tag is a pipeline failure. Docker images may use the exact preview version tag, including its existing variant suffixes.
- Preview Releases are cleaned up by the `cleanup-preview-releases` job after `publish-release` succeeds for the deliverable tag. It deletes every Release whose tag is exactly `<target>-preview.<digits>` and never passes `--cleanup-tag`, so the tags survive.
Verify the frozen Console identity equals the one accepted in Phase 0. Every matrix job consumes that same digest and exact toolchain. The binary must report the final product version and source commit; its candidate ID is separate. No public tag, Release, registry tag, R2 release path, Helm repository or latest pointer should change during this stage.
## Phase 3 — CI and preview Release verification
- Find and watch the tag build: `gh run list --workflow build.yml --branch "<preview-tag>" --limit 1` then `gh run watch <run-id>`. Every build matrix target must succeed (linux x86_64/aarch64 × musl/gnu, macos-aarch64, windows-x86_64).
- Confirm the preview tag resolves to the Phase 2 CI-validated `PREVIEW_HASH`. If the release branch advances during acceptance, repeat Phases 2–5 with a new preview tag.
- Confirm the Release publication jobs (`create-release`, `upload-release-assets`, and `publish-release`) succeed while `update-latest-version` is skipped.
- Record the inputs actually consumed by each build job: source SHA and the embedded Console release tag, asset URL, and verified SHA256 from the download logs. The workflow reads Console `releases/latest` separately for each build; confirm the matrix used the Console asset accepted in Phase 0. A missing identity or differing asset blocks acceptance until resolved. Retain available toolchain and build provenance with the artifact checksums; the same source SHA alone does not establish equivalent builds.
- Verify `gh release view "<preview-tag>" --json isPrerelease,assets,url`: `isPrerelease` must be `true`, and the Release must contain all 6 versioned platform zips, checksums, SBOM, and provenance with no `-latest` assets. Confirm `gh api repos/{owner}/{repo}/releases/latest --jq .tag_name` does not return `<preview-tag>`.
- Record `PREVIOUS_DELIVERABLE`, selected from published Releases by `publishedAt` after excluding the current tag and every `-preview.N` tag. Verify `gh release view "<preview-tag>" --json body --jq .body` contains `## What's Changed` and, when `PREVIOUS_DELIVERABLE` exists, `**Full Changelog**: https://github.com/rustfs/rustfs/compare/<PREVIOUS_DELIVERABLE>...<preview-tag>`. For a repository with no previous deliverable, verify a Full Changelog link exists and record the GitHub baseline fallback.
- Confirm Helm is skipped. If preview Docker images are published, confirm they use only exact preview tags (and variant suffixes), with no `latest`, `alpha`, `beta`, or `rc` channel updates. Preview validation covers the built RustFS binaries, embedded console, and rc compatibility; it does not authorize advancing any installation default.
## Release notes
- When a previous deliverable exists, GitHub Release notes for the preview and final tags MUST use it as their shared comparison baseline: the most recently published non-preview Release before the target. Internal `-preview.N` Releases are explicitly excluded from that selection, even when they point at the same commit as the final tag — cleanup runs after the notes are generated, so the preview Release is still present and would otherwise be picked as the baseline. If no previous deliverable exists, omit `previous_tag_name` and record that GitHub's default baseline fallback was used.
- Generated Release notes carry a workflow-management marker so retries can repair them. Before manually curating a generated body, remove that marker; unmarked non-placeholder notes are preserved by later workflow runs.
A green Linux run is ready for field acceptance, not publication. The partial certification must have `publish_allowed: false`. Retain the exact Linux manifest for the next stage; a new source/build-input candidate must not inherit the old approval. A newer unrelated main commit does not invalidate this evidence.
@@ -16,22 +16,13 @@ Read for Phases 1–2. The [parent release invariants](../SKILL.md) and Console
- Only when neither the release branch, an unfinished preview tag, nor current main already has `<target>`: if main is at a later version, select and verify a historical commit containing `<target>` rather than downgrading main; block if none exists. Otherwise invoke the `rustfs-release-version-bump` skill with stage `prepare`, the final `<target>` (NOT a preview version), and full GitHub flow (commit/push/PR).
- If a bump PR was needed, get it merged into main and record its merge commit. For a new release branch when main already has `<target>`, identify the main commit that introduced those versions. Phase 2 chooses a release candidate containing that commit; an existing release branch keeps its own history and a newer main HEAD is not required.
Do not wait for the moving latest main HEAD to become green. The release-branch CI in Phase 2 validates the chosen commit before a preview tag is created.
Do not wait for the moving latest main HEAD to become green. The release-branch CI in Phase 2 validates the chosen commit before a candidate build is dispatched.
## Phase 2 — Pin and validate the release candidate, then publish the preview tag
## Phase 2 — Pin and validate the release source
- Use `release/<target>`. On the first attempt, create it from a selected main commit containing the version bump and intended changes; it need not be the latest main HEAD. On a restart, retain the existing branch and backport only reviewed fixes from main. Verify the branch's Cargo versions still equal `<target>`. If adopting an existing preview, run this branch CI before continuing its acceptance; reuse the tag only when it points to the validated SHA.
- The current `ci.yml` runs automatically on main pushes, not release-branch pushes. Dispatch it explicitly with `gh workflow run ci.yml --ref "release/<target>"`. Locate that dispatch run, verify its `head_sha` equals the release branch's remote HEAD, and require the full expected CI job set to pass, including `Quick Checks`, `Test and Lint`, and `End-to-End Tests (full merge gate)`. Treat a missing or skipped required lane as incomplete. A successful PR check or CI run for a different SHA does not qualify. If the branch advances or a run is cancelled, validate the new SHA again. Do not tag a candidate with failing or incomplete CI.
- Use `release/<target>`. On the first attempt, create it from a selected main commit containing the version bump and intended changes; it need not be the latest main HEAD. On a restart, retain the existing branch and backport only reviewed fixes from main. Verify the branch's Cargo versions still equal `<target>`. A legacy preview can anchor source selection, but its tag and old artifacts do not constitute a candidate in the new protocol; freeze and verify a new candidate after the release recipe is aligned.
- The current `ci.yml` runs automatically on main pushes, not release-branch pushes. Dispatch it explicitly with `gh workflow run ci.yml --ref "release/<target>"`. Locate that dispatch run, verify its `head_sha` equals the release branch's remote HEAD, and require the full expected CI job set to pass, including `Quick Checks`, `Test and Lint`, and `End-to-End Tests (full merge gate)`. Treat a missing or skipped required lane as incomplete. A successful PR check or CI run for a different SHA does not qualify. If the branch advances or a run is cancelled, validate the new SHA again. Do not dispatch a candidate with failing or incomplete CI.
- Prefer protecting `release/*` against force pushes and deletion and requiring reviewed backport PRs. Check live rulesets rather than assuming the main ruleset covers release branches; report a missing release-branch rule as a gap.
- Set `PREVIEW_HASH` to the CI-validated release-branch commit and recheck the remote branch HEAD immediately before tagging. Report the branch, commit, and CI run URL; do not derive `PREVIEW_HASH` from a later `origin/main` fetch. When adopting an existing preview tag at this SHA, skip tag creation and continue Phase 3.
```bash
git tag -a "<preview-tag>" -m "Release <preview-tag>" "$PREVIEW_HASH"
git push origin "<preview-tag>"
```
Pushing the tag triggers `.github/workflows/build.yml` ("Build and Release"); `docker.yml` chains off it via `workflow_run`.
The preview run builds versioned artifacts and publishes them in a GitHub prerelease. Docker may publish exact preview image tags. Latest-channel, R2, and Helm publication must be skipped; installation references and the website announcement stay unchanged.
On a restart after a backport, re-run release-branch CI before setting the new `PREVIEW_HASH`. Use the next unused preview iteration when the validated commit differs from an existing preview tag; never move an existing tag.
- Set `CANDIDATE_SHA` to the CI-validated release-branch commit. Record the branch, exact commit and CI run URL. The new candidate protocol starts from a commit whose release recipe matches the reviewed main dispatcher; backport the recipe before freezing an older release branch if needed.
- Dispatch the Linux candidate as described in [candidate verification](preview-release.md). Do not create a preview or final tag here. The final executable already contains `<target>`; publication later promotes those same bytes.
- After a source or build-input change, run the affected source checks and create a new candidate. Reuse the frozen candidate after an infrastructure failure. Do not restart acceptance for unrelated main advances.
@@ -4,18 +4,18 @@ description: Prepare an exact RustFS Cargo version or align installation referen
---
# Release Version Files
Require an exact target; ask if missing or ambiguous. Reject targets containing `-preview`: those are tag-only. Infer `post-release` only from an explicit request or parent workflow; otherwise use `prepare`. Delivery defaults to local edit/verify; commit, push, PR, and merge follow existing authorization and [Git rules](../../references/pull-requests.md).
Require an exact target; ask if missing or ambiguous. Reject targets containing `-preview`: those identify candidates, not product versions or new public tags. Infer `post-release` only from an explicit request or parent workflow; otherwise use `prepare`. Delivery defaults to local edit/verify; commit, push, PR, and merge follow existing authorization and [Git rules](../../references/pull-requests.md).
| Stage | Files and constraints |
|---|---|
| `prepare` | Only `Cargo.toml` and `Cargo.lock`: workspace package, internal dependency, and workspace member versions become `<target>`. External dependencies stay unchanged. |
| `post-release` | Only `README.md`, `README_ZH.md`, `flake.nix`, `helm/rustfs/Chart.yaml`, and `rustfs.spec`. Cargo may already target the next release; leave it untouched. |
Before installation edits, verify the exact non-preview Release, downloadable assets/source archive, pullable `rustfs/rustfs:<target>` image manifest, and published Helm chart. Block on missing artifacts. Do not downgrade defaults if a newer deliverable superseded the target. Preview preparation retains the previous published installation references; follow-up commits never replace either validated tag.
Before installation edits, verify the exact non-preview Release, downloadable assets/source archive, pullable `rustfs/rustfs:<target>` image manifest, and published Helm chart. Block on missing artifacts. Do not downgrade defaults if a newer deliverable superseded the target. Candidate preparation retains the previous published installation references; follow-up commits never replace validated tags or sealed artifacts.
Packaging policy:
- Docker image tags have no `v` prefix. Derive chart/app versions with `scripts/helm_chart_version.sh <target>` (`beta.N -> 0.N.0`; other versions unchanged). Helm CI derives versions from the final tag, so no early chart bump is required.
- Docker image tags have no `v` prefix. Derive chart/app versions with `scripts/helm_chart_version.sh <target>`; both equal the final product SemVer, including prerelease suffixes. Helm CI uses the frozen candidate version before sealing, so no early installation chart bump is required. Previously published chart versions remain unchanged.
- RPM `Version` is numeric; `Release` is the prerelease suffix or `1` for stable. Expanded `Source0` and the unpack directory must resolve to the exact target including its suffix. Add the changelog entry using current git identity, date, and target; never copy an example identity/date.
- Changing these mappings requires explicit confirmation.
+2 -2
View File
@@ -1,2 +1,2 @@
sha256-linux=686dcc075e41a75dde3d37e5bd0393d016469f02f83aa3508f92bfc17f26a4d4
sha256-darwin=686dcc075e41a75dde3d37e5bd0393d016469f02f83aa3508f92bfc17f26a4d4
sha256-linux=348fa5eed9ae58962c06b33283ff4f15122cff97bce9b4eb1e3ffcb227264b0b
sha256-darwin=348fa5eed9ae58962c06b33283ff4f15122cff97bce9b4eb1e3ffcb227264b0b
+2 -2
View File
@@ -1,2 +1,2 @@
sha256-darwin=7c705942d3a9e583ebaa85845daa21c4f81fdbf36def24db931aaef38636c53e
sha256-linux=5c33dc395233671cb77452f8fb12900ecdbeb368632222b96835d310bce1aab8
sha256-darwin=9901a4950099b94d3d2618524d540f1ed4bf6023044903886f98b6c4bf5f30fb
sha256-linux=09fd5f48f5556c9fc576357222dccd7ebb0b90b790798f1834db2ec5d67b4952
+2 -2
View File
@@ -1,2 +1,2 @@
sha256-darwin=3814cdde8cacf4187e18188d342ed70931cc32e268e42e5bb438d565bb4d0f81
sha256-linux=663025a80273579f95ae3f49af02c0afcbfce94df09701c360a13e607b7637db
sha256-darwin=e84e04208f19dc0596b10d4d1b799ce43d6cc431415902f9c5458aa8232bc2d8
sha256-linux=1f9c7add72c44d1c12ef4ac050bb7917431be657c0c90b29f70c3df56f8bbae3
+1 -1
View File
@@ -1 +1 @@
sha256=00f8a56b75990b487a594b97c36610b94476a6944ba00473e37187ea654d42bf
sha256=5bdd4c2eadbc41625ae3184fff34062e000d328108436259b2b7c4fe2531ecaf
+12
View File
@@ -22,12 +22,21 @@ TEST_THREADS ?= 1
# differ from CI and .config/nextest.toml test-groups will NOT apply.
.PHONY: script-tests
.PHONY: release-package-tests
release-package-tests: ## Run release package integration on Linux with fpm, dpkg and rpm installed
$(RUSTFS_PYTHON_BIN) ./scripts/test_release_packages.py
.PHONY: release-image-tests
release-image-tests: ## Run OCI integration with Docker, Go and skopeo; set RELEASE_IMAGE_VARIANT and RELEASE_IMAGE_OUTPUT
$(RUSTFS_PYTHON_BIN) ./scripts/test_release_images.py
script-tests: ## Run shell script tests
@echo "Running script tests..."
trap 'status=$$?; printf "ERROR: script-tests failed (exit %s): %s\n" "$$status" "$$BASH_COMMAND" >&2; exit "$$status"' ERR
./scripts/test_build_rustfs_options.sh
./scripts/test_docker_runtime_timezone.sh
./scripts/test_entrypoint_credentials.sh
./scripts/test_entrypoint_directories.sh
./scripts/test_internode_grpc_ab_bench.sh
./scripts/test_object_batch_bench_enhanced.sh
./scripts/test_package_service_scripts.sh
@@ -53,8 +62,10 @@ script-tests: ## Run shell script tests
./scripts/check_embedded_secrets.sh --self-test
$(RUSTFS_PYTHON_BIN) ./scripts/check_test_wiring.py --self-test
$(RUSTFS_PYTHON_BIN) ./scripts/test_e2e_binary.py
$(RUSTFS_PYTHON_BIN) ./scripts/test_download_release_asset.py
$(RUSTFS_PYTHON_BIN) ./scripts/test_inspect_btf_prototypes.py
$(RUSTFS_PYTHON_BIN) ./scripts/test_release_build_workflow.py
$(RUSTFS_PYTHON_BIN) -m unittest discover -s scripts/release -p "test_*.py"
$(RUSTFS_PYTHON_BIN) ./scripts/test_release_glibc.py
$(RUSTFS_PYTHON_BIN) ./scripts/test_upgrade_workflow.py
$(RUSTFS_PYTHON_BIN) ./scripts/test_migration_gate_evidence.py
@@ -70,6 +81,7 @@ script-tests: ## Run shell script tests
$(RUSTFS_PYTHON_BIN) ./scripts/test_nightly_candidate.py
$(RUSTFS_PYTHON_BIN) ./scripts/test_functional_chain.py
$(RUSTFS_PYTHON_BIN) ./scripts/test_functional_chain_health.py
$(RUSTFS_PYTHON_BIN) ./scripts/test_repro_rejoin_readiness_docker.py
$(RUSTFS_PYTHON_BIN) ./scripts/test_validation_registry.py
$(RUSTFS_PYTHON_BIN) ./scripts/test_inspect_validation_runner.py
$(RUSTFS_PYTHON_BIN) ./scripts/test_retain_validation_evidence.py
+4 -4
View File
@@ -110,7 +110,7 @@ threads-required = "num-test-threads"
# Reserve capacity so sibling disk-heavy tests cannot consume its 2s
# registration budget; retain the queued backup and deadlock assertions.
[[profile.default.overrides]]
filter = 'package(rustfs) & test(=admin::handlers::table_catalog::tests::cold_strong_registration_completes_while_other_bucket_backup_waits)'
filter = 'package(rustfs) & test(=admin::handlers::table_catalog::tests::cold_object_registration_completes_while_other_bucket_backup_waits)'
threads-required = "num-test-threads"
[[profile.default.overrides]]
@@ -410,7 +410,7 @@ threads-required = "num-test-threads"
# Reserve capacity so sibling disk-heavy tests cannot consume its 2s
# registration budget; retain the queued backup and deadlock assertions.
[[profile.ci.overrides]]
filter = 'package(rustfs) & test(=admin::handlers::table_catalog::tests::cold_strong_registration_completes_while_other_bucket_backup_waits)'
filter = 'package(rustfs) & test(=admin::handlers::table_catalog::tests::cold_object_registration_completes_while_other_bucket_backup_waits)'
threads-required = "num-test-threads"
[[profile.ci.overrides]]
@@ -701,7 +701,7 @@ test-group = 'ecstore-serial-flaky'
[profile.e2e-smoke]
default-filter = """
package(e2e_test) & (
test(/^(delete_marker_migration_semantics|version_id_regression|list_objects_v2_pagination|list_object_versions_regression|list_objects_duplicates|list_buckets_double_slash|list_buckets_auth|list_buckets_iam_filter|leading_slash_key|special_chars|create_bucket_region|delete_objects_versioning|head_object_consistency|head_object_range|copy_object_metadata|copy_object_tagging|copy_source_invalid_date|content_encoding|compression|multipart_storage_class|storage_class_capability|ssec_copy|anonymous_access|bucket_policy_check|presigned_negative|negative_sigv4|admin_auth|notification_webhook|tls_hot_reload|console_smoke|admin_iam_crud|admin_pools|sts_query_compat|data_usage)_test::|^fake_s3_target::/)
test(/^(delete_marker_migration_semantics|version_id_regression|list_objects_v2_pagination|list_object_versions_regression|list_objects_duplicates|list_buckets_double_slash|list_buckets_auth|list_buckets_iam_filter|leading_slash_key|special_chars|create_bucket_region|delete_objects_versioning|head_object_consistency|head_object_range|copy_object_metadata|copy_object_tagging|copy_source_invalid_date|content_encoding|compression|multipart_storage_class|multipart_upload_id|storage_class_capability|ssec_copy|anonymous_access|bucket_policy_check|presigned_negative|negative_sigv4|admin_auth|notification_webhook|tls_hot_reload|console_smoke|admin_iam_crud|admin_pools|sts_query_compat|data_usage)_test::|^fake_s3_target::/)
| test(/^replication_extension_test::(test_replication_check_succeeds_with_remote_target|test_replication_check_rejects_target_without_object_lock|test_set_remote_target_rejects_unversioned_source_bucket|test_replication_check_rejects_unversioned_source_bucket|test_replication_check_rejects_missing_replication_config|test_replication_check_rejects_invalid_bucket|test_set_remote_target_rejects_same_bucket_on_same_deployment|test_set_remote_target_rejects_unversioned_target_bucket|test_set_remote_target_update_requires_arn|test_set_remote_target_update_rejects_missing_target|test_set_remote_target_rejects_invalid_target_url|test_set_remote_target_rejects_self_signed_https_target_without_skip_tls_verify|test_set_remote_target_rejects_private_ca_https_target_without_ca_cert_pem|test_list_remote_targets_rejects_empty_bucket|test_list_remote_targets_rejects_invalid_bucket|test_remove_remote_target_rejects_missing_target|test_remove_remote_target_rejects_missing_arn|test_remove_remote_target_rejects_invalid_bucket|test_remove_remote_target_rejects_target_used_by_replication|test_delete_bucket_replication_removes_remote_target)$/)
| test(/^reliant::lifecycle::/)
| test(/^conditional_delete_objects_test::/)
@@ -827,7 +827,7 @@ slow-timeout = { period = "120s", terminate-after = 12, grace-period = "10s" }
# ---------------------------------------------------------------------------
# e2e-distributed profile — 4-node 4-disk Actions suite
# ---------------------------------------------------------------------------
# Storage-sensitive PR / nightly / dispatch lane owned by
# Nightly and manual-dispatch lane owned by
# .github/workflows/e2e-distributed.yml.
# Each case starts four rustfs processes (and for site replication, two
# clusters). Upgrade cases also require RUSTFS_UPGRADE_SOURCE_BINARY.
+82 -24
View File
@@ -1,44 +1,93 @@
{
"schema_version": 1,
"plan_id": "rustfs-release-validation",
"description": "Versioned certification plan for every public RustFS release exit (rustfs/backlog#2662, R01). preview, prerelease and stable all require this explicit plan. Scripts/release/release_gate.py fail-closes publication on any missing, duplicate, stale, substituted or non-PASS required item.",
"release_types": ["preview", "prerelease", "stable"],
"description": "Certification for immutable candidates. All shipped platforms require native identity and startup; stable and prerelease require the pinned upgrade matrix. Linux-only acceptance never authorizes publication.",
"release_types": [
"preview",
"prerelease",
"stable"
],
"targets": [
{"os": "linux", "arch": "x86_64", "abi": "musl"},
{"os": "linux", "arch": "aarch64", "abi": "musl"},
{"os": "linux", "arch": "x86_64", "abi": "gnu"},
{"os": "linux", "arch": "aarch64", "abi": "gnu"},
{"os": "macos", "arch": "aarch64", "abi": "none"},
{"os": "windows", "arch": "x86_64", "abi": "msvc"}
{
"os": "linux",
"arch": "x86_64",
"abi": "musl"
},
{
"os": "linux",
"arch": "aarch64",
"abi": "musl"
},
{
"os": "linux",
"arch": "x86_64",
"abi": "gnu"
},
{
"os": "linux",
"arch": "aarch64",
"abi": "gnu"
},
{
"os": "macos",
"arch": "aarch64",
"abi": "none"
},
{
"os": "windows",
"arch": "x86_64",
"abi": "msvc"
}
],
"targets_basis": "Platform matrix of .github/workflows/build.yml (prepare-platform-matrix). gnu and macOS targets build with the pyroscope feature; musl and Windows targets build with no extra features.",
"features": ["pyroscope"],
"trusted_workflows": [
{"repository": "rustfs/s3chaos", "workflow_id": "release-gate.yml"}
"features": [
"pyroscope"
],
"trusted_workflows_basis": "The shared evidence envelope producer lives in rustfs/s3chaos (G01, fault/release_gate.rs) and its plan/live split workflow is .github/workflows/release-gate.yml there. Only results from these identities may enter a release certification; PR-uploaded JSON never gains release authority. R02 must extend this list when additional verifier workflows land, never widen it to untrusted sources.",
"trusted_workflows": [
{
"repository": "rustfs/rustfs",
"workflow_id": "release-candidate.yml"
},
{
"repository": "rustfs/rustfs",
"workflow_id": "release-complete.yml"
}
],
"trusted_workflows_basis": "Receipts are produced by the reviewed main workflows and fetched by exact run ID with GitHub artifact digest verification. Candidate inputs freeze the harness commit separately from product source.",
"lanes": [
{
"lane": "identity",
"harness_repository": "rustfs/s3chaos",
"harness_sha": "9a48f55c1d6d58ce63a8fb1ea7317e024f32f11d",
"harness_basis": "Fixed G01 evidence-envelope producer revision (rustfs/s3chaos PR #102 head recorded in rustfs/backlog#2658). R02 re-pins this when the live verifier ships.",
"harness_repository": "rustfs/rustfs",
"harness_sha": "candidate.workflow_sha",
"harness_basis": "The exact workflow commit is frozen in candidate inputs; no mutable branch is accepted as a harness identity.",
"timeout_seconds": 1800,
"environment_capabilities": ["process-exec"],
"cases": ["artifact-identity", "startup"],
"environment_capabilities": [
"process-exec"
],
"cases": [
"artifact-identity",
"startup"
],
"applies_to": {
"release_types": ["preview", "prerelease", "stable"],
"release_types": [
"preview",
"prerelease",
"stable"
],
"platforms": "all"
},
"applies_basis": "Artifact identity and startup checks are mandatory for every release type, preview included, on every shipped target (rustfs/backlog#2662 frozen design item 5)."
},
{
"lane": "upgrade",
"harness_repository": "rustfs/s3chaos",
"harness_sha": "9a48f55c1d6d58ce63a8fb1ea7317e024f32f11d",
"harness_basis": "Same fixed envelope producer as the identity lane; the workflow SHA and the product build SHA are bound separately and need not be equal.",
"harness_repository": "rustfs/rustfs",
"harness_sha": "candidate.workflow_sha",
"harness_basis": "The exact workflow commit is frozen in candidate inputs; no mutable branch is accepted as a harness identity.",
"timeout_seconds": 5400,
"environment_capabilities": ["process-exec", "previous-release-artifact"],
"environment_capabilities": [
"process-exec",
"previous-release-artifact"
],
"cases": [
"direct-upgrade",
"mixed-version-upgrade",
@@ -49,8 +98,17 @@
],
"cases_basis": "Case IDs of the pinned previous-release matrix in .github/workflows/e2e-upgrade.yml (source.version/asset/sha256 and the case list mapped there).",
"applies_to": {
"release_types": ["prerelease", "stable"],
"platforms": [{"os": "linux", "arch": "x86_64", "abi": "gnu"}]
"release_types": [
"prerelease",
"stable"
],
"platforms": [
{
"os": "linux",
"arch": "x86_64",
"abi": "gnu"
}
]
},
"applies_basis": "preview releases use the pre-declared smaller set (identity lane only). The upgrade lane is only exercised on linux-x86_64-gnu, the platform of the pinned source assets."
}
+33 -3
View File
@@ -22,9 +22,9 @@ The following files in the project root are essential for Docker operations:
| File | Description | Usage |
| :--- | :--- | :--- |
| **`docker-buildx.sh`** | **Multi-Arch Build Script**<br>Automates building and pushing Docker images for `amd64` and `arm64`. Supports release and dev channels. | `./docker-buildx.sh --push` |
| **`Dockerfile`** | **Production Image (Alpine)**<br>Lightweight image using musl libc. Downloads pre-built binaries from GitHub Releases. | `docker build -t rustfs:latest .` |
| **`Dockerfile.glibc`** | **Production Image (Ubuntu)**<br>Standard image using glibc. Useful if you need specific dynamic libraries. | `docker build -f Dockerfile.glibc .` |
| **`Dockerfile.source`** | **Development Image**<br>Builds RustFS from source code. Includes build tools. Ideal for local development and CI. | `docker build -f Dockerfile.source .` |
| **`Dockerfile`** | **Production Image (Alpine)**<br>Lightweight image using musl libc. Downloads verified RustFS and `rc` binaries from GitHub Releases and bundles `/etc/rustfs/bootstrap`. | `docker build -t rustfs:latest .` |
| **`Dockerfile.glibc`** | **Production Image (Ubuntu)**<br>Standard image using glibc. Includes the bundled bootstrap configuration. | `docker build -f Dockerfile.glibc .` |
| **`Dockerfile.source`** | **Development Image**<br>Builds RustFS from source code and bundles `rc` and the default bootstrap configuration. | `docker build -f Dockerfile.source .` |
### Docker Compose Configurations
@@ -110,6 +110,36 @@ docker compose up -d
---
## Docker Compose environment values
Use map syntax when quoting environment values in YAML:
```yaml
services:
rustfs:
image: rustfs/rustfs:latest
ports:
- "9000:9000"
- "9001:9001"
volumes:
- ./data:/data
- ./logs:/var/log/rustfs
environment:
RUSTFS_ADDRESS: ":9000"
RUSTFS_CONSOLE_ADDRESS: ":9001"
RUSTFS_CONSOLE_ENABLE: "true"
RUSTFS_OBS_LOGGER_LEVEL: error
RUSTFS_OBS_LOG_DIRECTORY: "/var/log/rustfs/"
RUSTFS_ACCESS_KEY: "${RUSTFS_ACCESS_KEY:?Set RUSTFS_ACCESS_KEY}"
RUSTFS_SECRET_KEY: "${RUSTFS_SECRET_KEY:?Set RUSTFS_SECRET_KEY}"
```
Create the host directories and grant the container user write access before starting this example; official images run as UID/GID `10001:10001`. Set both credential variables to your own values, then run `docker compose config` to inspect the resolved configuration locally before `docker compose up -d`. The resolved output includes credentials, so do not share it publicly.
With list syntax, use `- RUSTFS_OBS_LOG_DIRECTORY=/var/log/rustfs/` or quote the whole item as `- "RUSTFS_OBS_LOG_DIRECTORY=/var/log/rustfs/"`. Writing `- RUSTFS_OBS_LOG_DIRECTORY="/var/log/rustfs/"` passes the inner quotes literally, making this a relative path beginning with `"`, which can fail with `mkdir: cannot create directory '"': Permission denied`. This also applies to addresses and credentials: fix every affected list item, rather than only the log directory. The entrypoint preserves configured values, including literal quotes in valid paths and secrets.
---
## 🔧 Common Operations
### Cleaning Up
@@ -28,6 +28,15 @@ processors:
check_interval: 1s
limit_mib: 1024
spike_limit_mib: 256
transform/metrics:
error_mode: ignore
metric_statements:
# RustFS process metrics already expose process_pid as a metric label.
# Remove only the duplicate datapoint label and keep the shared resource
# process.pid available to sibling metrics and lock histograms.
- context: datapoint
statements:
- delete_key(attributes, "process_pid") where IsMatch(metric.name, "^rustfs_system_process_")
transform/logs:
error_mode: ignore
log_statements:
@@ -112,7 +121,7 @@ service:
exporters: [ otlp/tempo, otlp/jaeger ]
metrics:
receivers: [ otlp ]
processors: [ batch ]
processors: [ transform/metrics, memory_limiter, batch ]
exporters: [ prometheus ]
logs:
receivers: [ otlp ]
+40
View File
@@ -0,0 +1,40 @@
# syntax=docker/dockerfile:1.6
# Copyright 2026 RustFS Team
# Licensed under the Apache License, Version 2.0.
# A compatible native runtime with Python 3.11+, getent and stat is required by
# the isolated fixture's staging/provenance checks. This is a test image.
ARG RUNTIME_BASE=python:3.13-slim-trixie
FROM ${RUNTIME_BASE}
ARG SOURCE_REVISION
ARG BINARY_SHA256
ARG BUILD_PROFILE=dev
COPY rustfs /usr/bin/rustfs
RUN python3 - "$SOURCE_REVISION" "$BINARY_SHA256" "$BUILD_PROFILE" <<'PY'
import hashlib
import os
import re
import shutil
import sys
source, digest, profile = sys.argv[1:]
if not re.fullmatch(r"[0-9a-f]{40}", source):
raise SystemExit("invalid source revision")
if not re.fullmatch(r"[0-9a-f]{64}", digest):
raise SystemExit("invalid binary digest")
if profile not in ("dev", "release"):
raise SystemExit("invalid build profile")
with open("/usr/bin/rustfs", "rb") as binary:
if hashlib.file_digest(binary, "sha256").hexdigest() != digest:
raise SystemExit("binary digest mismatch")
for tool in ("getent", "stat"):
if shutil.which(tool) is None:
raise SystemExit(f"missing runtime tool: {tool}")
os.chmod("/usr/bin/rustfs", 0o755)
PY
LABEL io.rustfs.repro.source_revision="${SOURCE_REVISION}" \
io.rustfs.repro.binary_sha256="${BINARY_SHA256}" \
io.rustfs.repro.build_profile="${BUILD_PROFILE}"
USER 10001:10001
ENTRYPOINT ["/usr/bin/rustfs"]
+4 -3
View File
@@ -28,11 +28,12 @@ runs:
- name: Install actionlint
shell: bash
env:
GH_TOKEN: ${{ github.token }}
ASSET_DOWNLOADER: ${{ github.action_path }}/../setup/download_release_asset.py
run: |
actionlint_dir="$(mktemp -d "${RUNNER_TEMP}/actionlint.XXXXXX")"
curl --fail --location --silent --show-error \
--output "$actionlint_dir/actionlint.tar.gz" \
https://github.com/rhysd/actionlint/releases/download/v1.7.12/actionlint_1.7.12_linux_amd64.tar.gz
python3 "$ASSET_DOWNLOADER" --repository rhysd/actionlint --tag v1.7.12 --asset actionlint_1.7.12_linux_amd64.tar.gz --output "$actionlint_dir/actionlint.tar.gz" --sha256 8aca8db96f1b94770f1b0d72b6dddcb1ebb8123cb3712530b08cc387b349a3d8
echo "8aca8db96f1b94770f1b0d72b6dddcb1ebb8123cb3712530b08cc387b349a3d8 $actionlint_dir/actionlint.tar.gz" | sha256sum --check --status
tar -xzf "$actionlint_dir/actionlint.tar.gz" -C "$actionlint_dir" actionlint
rm "$actionlint_dir/actionlint.tar.gz"
+23 -5
View File
@@ -66,10 +66,7 @@ inputs:
runs:
using: "composite"
steps:
# protobuf-compiler is deliberately absent: the setup-protoc step below
# installs 35.1 into the tool cache and prepends it to PATH, so the apt
# build (older, and never version-matched) was shadowed on every run and
# simply never used.
# Use the pinned code generators rather than Ubuntu's older packages.
- name: Install system dependencies (Ubuntu)
if: runner.os == 'Linux'
shell: bash
@@ -79,7 +76,9 @@ runs:
build-essential \
pkg-config \
libssl-dev \
ripgrep
ripgrep \
python3 \
gh
# musl-gcc is needed by the native musl release leg, and zip/unzip by the
# release packaging steps. No CI test lane touches any of them.
@@ -88,13 +87,32 @@ runs:
shell: bash
run: sudo apt-get install -y musl-tools zip unzip
- name: Install pinned code generators through GitHub API (Linux x64)
if: runner.os == 'Linux' && runner.arch == 'X64'
shell: bash
env:
GH_TOKEN: ${{ github.token }}
ASSET_DOWNLOADER: ${{ github.action_path }}/download_release_asset.py
run: |
python3 "$ASSET_DOWNLOADER" --repository protocolbuffers/protobuf --tag v35.1 --asset protoc-35.1-linux-x86_64.zip --output "$RUNNER_TEMP/protoc.zip"
python3 "$ASSET_DOWNLOADER" --repository google/flatbuffers --tag v25.12.19 --asset 'Linux.flatc.binary.g++-13.zip' --output "$RUNNER_TEMP/flatc.zip"
python3 -m zipfile -e "$RUNNER_TEMP/protoc.zip" "$RUNNER_TEMP/protoc"
python3 -m zipfile -e "$RUNNER_TEMP/flatc.zip" "$RUNNER_TEMP/flatc"
chmod +x "$RUNNER_TEMP/protoc/bin/protoc" "$RUNNER_TEMP/flatc/flatc"
"$RUNNER_TEMP/protoc/bin/protoc" --version
"$RUNNER_TEMP/flatc/flatc" --version
echo "$RUNNER_TEMP/protoc/bin" >> "$GITHUB_PATH"
echo "$RUNNER_TEMP/flatc" >> "$GITHUB_PATH"
- name: Install protoc
if: runner.os != 'Linux' || runner.arch != 'X64'
uses: rustfs/setup-protoc@a3705324d8f9bf5b6c3573fb6cf8ae421db55dd6 # v3.0.1
with:
version: "35.1"
repo-token: ${{ github.token }}
- name: Install flatc
if: runner.os != 'Linux' || runner.arch != 'X64'
uses: Nugine/setup-flatc@698800de72a96bfb22cf60431dc21a2ff9a7e07b # v1
with:
version: "25.12.19"
@@ -0,0 +1,84 @@
#!/usr/bin/env python3
"""Download a named GitHub release asset through its API and verify its digest."""
import argparse
import hashlib
import json
from pathlib import Path
import re
import subprocess
import sys
import tempfile
from urllib.parse import quote
def download(repository, tag, name, output, expected_sha256=None):
if not re.fullmatch(r"[A-Za-z0-9][A-Za-z0-9_.-]*/[A-Za-z0-9][A-Za-z0-9_.-]*", repository):
raise ValueError("invalid release repository")
if not tag or not name or "/" in name or "\\" in name:
raise ValueError("invalid release tag or asset name")
if expected_sha256 is not None and not re.fullmatch(r"[0-9a-f]{64}", expected_sha256):
raise ValueError("invalid pinned asset digest")
result = subprocess.run(
["gh", "api", "--hostname", "github.com", f"repos/{repository}/releases/tags/{quote(tag, safe='')}"],
check=True, capture_output=True, text=True, timeout=60,
)
release = json.loads(result.stdout)
if not isinstance(release, dict) or release.get("tag_name") != tag:
raise ValueError("release metadata does not match the requested tag")
assets = release.get("assets")
if not isinstance(assets, list) or any(not isinstance(asset, dict) for asset in assets):
raise ValueError("invalid release asset inventory")
matches = [asset for asset in assets if asset.get("name") == name]
if len(matches) != 1:
raise ValueError("release must contain exactly one named asset")
asset = matches[0]
asset_id, size, digest = asset.get("id"), asset.get("size"), asset.get("digest")
if type(asset_id) is not int or asset_id <= 0 or type(size) is not int or not 0 < size <= 512 * 1024 * 1024:
raise ValueError("invalid release asset identity or size")
if not isinstance(digest, str) or not re.fullmatch(r"sha256:[0-9a-f]{64}", digest):
raise ValueError("release asset has no valid SHA-256 digest")
sha256 = digest.removeprefix("sha256:")
if expected_sha256 is not None and expected_sha256 != sha256:
raise ValueError("release digest differs from the pinned digest")
output = Path(output)
output.parent.mkdir(parents=True, exist_ok=True)
temporary = None
try:
with tempfile.NamedTemporaryFile(dir=output.parent, prefix=".release-asset-", delete=False) as stream:
temporary = Path(stream.name)
# gh authenticates to the API; its HTTP client strips credentials on
# cross-host redirects. No browser URL or token is supplied in argv.
subprocess.run(
["gh", "api", "--hostname", "github.com", "--header", "Accept: application/octet-stream", f"repos/{repository}/releases/assets/{asset_id}"],
check=True, stdout=stream, timeout=300,
)
with temporary.open("rb") as stream:
actual = hashlib.file_digest(stream, "sha256").hexdigest()
if temporary.stat().st_size != size or actual != sha256:
raise ValueError("downloaded release asset failed size or SHA-256 verification")
temporary.replace(output)
print(f"Verified release asset: {repository} {tag} {name} sha256={actual}")
finally:
if temporary is not None:
temporary.unlink(missing_ok=True)
def main():
parser = argparse.ArgumentParser(description=__doc__)
parser.add_argument("--repository", required=True)
parser.add_argument("--tag", required=True)
parser.add_argument("--asset", required=True)
parser.add_argument("--output", required=True, type=Path)
parser.add_argument("--sha256")
args = parser.parse_args()
try:
download(args.repository, args.tag, args.asset, args.output, args.sha256)
except (ValueError, KeyError, TypeError, json.JSONDecodeError, subprocess.SubprocessError, OSError) as error:
print(f"Release asset download failed: {error}", file=sys.stderr)
return 1
return 0
if __name__ == "__main__":
raise SystemExit(main())
+2
View File
@@ -32,6 +32,8 @@ updates:
- "overtrue"
- "majinghe"
ignore:
- dependency-name: "pulsar"
versions: [ "6.9.x" ]
- dependency-name: "object_store"
versions: [ "0.13.x" ]
- dependency-name: "libunftp"
+83 -26
View File
@@ -350,10 +350,10 @@
"roles": [
"periodic",
"manual",
"release"
"reusable"
],
"events": [
"push",
"workflow_call",
"schedule",
"workflow_dispatch"
],
@@ -367,7 +367,7 @@
"monitor": "workflow",
"max_age_hours": 36,
"result_owner": "build.yml",
"disposition": "retained independent entry",
"disposition": "Builds artifacts only; public release writes belong to release-publish.yml.",
"failure_owner": "workflow",
"origin": "baseline",
"alert_contract": "scheduled-validation"
@@ -745,14 +745,13 @@
},
{
"workflow": ".github/workflows/docker.yml",
"name": "Docker Images",
"name": "Build candidate images",
"roles": [
"manual",
"release"
"release",
"reusable"
],
"events": [
"workflow_run",
"workflow_dispatch"
"workflow_call"
],
"cadence": "event",
"timezone": "UTC",
@@ -761,7 +760,7 @@
"policy": "active",
"monitor": "none",
"result_owner": "docker.yml",
"disposition": "retained independent entry",
"disposition": "Builds artifacts only; public release writes belong to release-publish.yml.",
"monitor_reason": "Event-driven or operator-requested work has no periodic freshness obligation.",
"origin": "baseline"
},
@@ -769,12 +768,10 @@
"workflow": ".github/workflows/e2e-distributed.yml",
"name": "e2e-distributed",
"roles": [
"pr",
"periodic",
"manual"
],
"events": [
"pull_request",
"workflow_dispatch",
"schedule"
],
@@ -849,12 +846,11 @@
"workflow": ".github/workflows/e2e-upgrade.yml",
"name": "Upgrade Compatibility",
"roles": [
"pr",
"periodic",
"manual"
"manual",
"release"
],
"events": [
"pull_request",
"push",
"schedule",
"workflow_dispatch"
@@ -957,14 +953,13 @@
},
{
"workflow": ".github/workflows/helm-package.yml",
"name": "Publish helm chart to artifacthub",
"name": "Build candidate Helm chart",
"roles": [
"manual",
"release"
"release",
"reusable"
],
"events": [
"workflow_run",
"workflow_dispatch"
"workflow_call"
],
"cadence": "event",
"timezone": "UTC",
@@ -973,7 +968,7 @@
"policy": "active",
"monitor": "none",
"result_owner": "helm-package.yml",
"disposition": "retained independent entry",
"disposition": "Builds artifacts only; public release writes belong to release-publish.yml.",
"monitor_reason": "Event-driven or operator-requested work has no periodic freshness obligation.",
"origin": "baseline"
},
@@ -1185,14 +1180,13 @@
},
{
"workflow": ".github/workflows/package.yml",
"name": "Package DEB/RPM",
"name": "Build candidate packages",
"roles": [
"manual",
"release"
"release",
"reusable"
],
"events": [
"workflow_run",
"workflow_dispatch"
"workflow_call"
],
"cadence": "event",
"timezone": "UTC",
@@ -1201,7 +1195,7 @@
"policy": "active",
"monitor": "none",
"result_owner": "package.yml",
"disposition": "retained independent entry",
"disposition": "Builds artifacts only; public release writes belong to release-publish.yml.",
"monitor_reason": "Event-driven or operator-requested work has no periodic freshness obligation.",
"origin": "baseline"
},
@@ -1730,6 +1724,69 @@
"disposition": "retained independent entry",
"monitor_reason": "Event-driven or operator-requested work has no periodic freshness obligation.",
"origin": "baseline"
},
{
"workflow": ".github/workflows/release-candidate.yml",
"name": "Release candidate (Linux)",
"roles": [
"manual",
"release"
],
"events": [
"workflow_dispatch"
],
"cadence": "event",
"timezone": "UTC",
"schedule": [],
"owner": "@overtrue",
"policy": "active",
"monitor": "none",
"result_owner": "release-candidate.yml",
"disposition": "Explicit immutable candidate stage",
"monitor_reason": "Operator-requested release stage has no periodic freshness obligation.",
"origin": "release-candidate-migration"
},
{
"workflow": ".github/workflows/release-complete.yml",
"name": "Release candidate (complete)",
"roles": [
"manual",
"release"
],
"events": [
"workflow_dispatch"
],
"cadence": "event",
"timezone": "UTC",
"schedule": [],
"owner": "@overtrue",
"policy": "active",
"monitor": "none",
"result_owner": "release-complete.yml",
"disposition": "Explicit immutable candidate stage",
"monitor_reason": "Operator-requested release stage has no periodic freshness obligation.",
"origin": "release-candidate-migration"
},
{
"workflow": ".github/workflows/release-publish.yml",
"name": "Publish accepted release",
"roles": [
"manual",
"release"
],
"events": [
"workflow_dispatch"
],
"cadence": "event",
"timezone": "UTC",
"schedule": [],
"owner": "@overtrue",
"policy": "active",
"monitor": "none",
"result_owner": "release-publish.yml",
"disposition": "Explicit immutable candidate stage",
"monitor_reason": "Operator-requested release stage has no periodic freshness obligation.",
"origin": "release-candidate-migration"
}
]
}
+11 -4
View File
@@ -5,7 +5,7 @@
## Billing facts
This repository is public. For a public repository, GitHub's standard hosted runners (`ubuntu-latest`, `windows-latest`, `macos-latest`) cost nothing and have no minute cap, whatever plan the organisation is on. GitHub bills only two things: larger hosted runners, which have custom labels and are not used here, and standard runners in private repositories.
This repository is public. For a public repository, GitHub's standard hosted runners (`ubuntu-latest`, `windows-latest`, `macos-latest`) cost nothing and have no minute cap, whatever plan the organisation is on. GitHub bills only two things: larger hosted runners, including the `macos-latest-xlarge` release lane used here, and standard runners in private repositories.
Every other label in this repository is capacity RustFS pays for. ARC pods and lab machines accrue cost while the machine exists. A Ubicloud label is metered for the minutes a job holds its VM:
@@ -13,6 +13,7 @@ Every other label in this repository is capacity RustFS pays for. ARC pods and l
|---|---|---|---|
| `ubuntu-latest` | nobody | GitHub-hosted, Ubuntu 24.04, 4 vCPU, 16 GB RAM, about 14 GB of free disk. Ships a C toolchain, `docker`, `gh`, `curl`, `wget`, `jq`, Python. Can `mount` tmpfs and loop devices; io_uring works | the default for every job |
| `windows-latest`, `macos-latest` | nobody | GitHub-hosted standard runners | Windows and macOS targets |
| `macos-latest-xlarge` | RustFS | GitHub-hosted larger Apple Silicon runner | macOS release builds, deferred until Linux acceptance for candidates |
| `sm-standard-2` | RustFS | ARC pod, Ubuntu 24.04, minimal image: no `gh`, no C toolchain; Docker only on the `dind-` label | only a job that a hosted runner cannot run (see the placement rule) |
| `sm-standard-4` | RustFS | ARC pod, Ubuntu 24.04, 14 CPUs and 28 GB (the label is not the core count), large disk | workspace-scale Cargo builds |
| `ubicloud-standard-16-arm` | RustFS | Ubicloud VM, alias of `ubicloud-standard-16-arm-ubuntu-2404`: Ubuntu 24.04 arm64, 16 vCPU, 48 GB, 300 GB disk. Reached through the `build.yml` matrix, so `actionlint.yaml` does not list it | native aarch64 release builds (`linux-aarch64-musl`, `linux-aarch64-gnu`) |
@@ -32,7 +33,7 @@ A job runs on `ubuntu-latest` unless it has one of these reasons, and the reason
Jobs that do only the following have no reason to be self-hosted: shell or Python scripts, `cargo fmt`, `cargo deny`, typos, actionlint, API calls (`gh`, dependency review, issue and stale bots, CLA checks), downloading an artifact and running a prebuilt binary, or a `cargo build -p rustfs` with its own cache key. `e2e-distributed.yml` and `e2e-upgrade.yml` build the server and the `e2e_test` harness on `ubuntu-latest` inside their budgets, so "it compiles something" alone is not a reason.
The functional chain's `prepare` and `complete-chain` jobs, the performance runner availability check, and runner hygiene use `ubuntu-latest`: they inspect APIs or artifacts, not the lab hosts. The actual functional and performance suites remain on their lab runners. Functional-chain health collection uses `ubuntu-latest` so a lab outage cannot also stop its independent monitor. Docker image packaging also uses `ubuntu-latest` for both musl and glibc variants; it downloads release binaries rather than compiling the workspace, and still sets up QEMU and Buildx explicitly.
The functional chain's `prepare` and `complete-chain` jobs, the performance runner availability check, and runner hygiene use `ubuntu-latest`: they inspect APIs or artifacts, not the lab hosts. The actual functional and performance suites remain on their lab runners. Functional-chain health collection uses `ubuntu-latest` so a lab outage cannot also stop its independent monitor. Docker image packaging also uses `ubuntu-latest` for both musl and glibc variants; it wraps verified candidate binaries rather than compiling the workspace, and still sets up QEMU and Buildx explicitly.
Moving these jobs off AKS reduces capacity demand, not necessarily the Azure bill immediately. VM costs fall only when nodes scale down or smaller capacity replaces them.
@@ -52,7 +53,7 @@ A hosted runner is also the only choice for a few things the pods cannot do: `mo
These are the patterns that have cost real money here without validating anything more:
- An `if: always()` summary or alert job on a self-hosted label inside a `workflow_run`-triggered workflow. The upstream workflow completes many times a day, the real jobs skip, and the summary still starts a pod each time. Gate it on the upstream job having run, and keep it on `ubuntu-latest`.
- Per-push workflows on `main`. A merge triggers the full `ci.yml` matrix, `cache-warm.yml` and the `build.yml` development build; with tens of merges a day that is as much self-hosted time as all pull requests together. Anything that does not verify the merged tree itself belongs on a schedule or behind a `paths` filter.
- Per-push workflows on `main`. A merge triggers the applicable `ci.yml` matrix and cache warming; `build.yml` development builds are scheduled or manual; with tens of merges a day that is as much self-hosted time as all pull requests together. Anything that does not verify the merged tree itself belongs on a schedule or behind a `paths` filter.
- `apt-get install` of tools the hosted image already ships. It is only needed on the minimal pod images.
- A throttled `CARGO_BUILD_JOBS` on a 14-CPU pod. The limit exists for link-storm reasons documented in `ci.yml`; measure before raising it, and measure before lowering it further.
- Tiny jobs on ARC pods. Each one pays roughly a minute of pod start-up before its first step; a compile-free job that takes ten seconds costs more to schedule than to run.
@@ -66,7 +67,7 @@ gh api "repos/rustfs/rustfs/actions/runs/<run_id>/jobs?per_page=100" \
--jq '.jobs[] | "\(.conclusion)\t\(.labels | join(","))\t\(.started_at)\t\(.completed_at)\t\(.name)"'
```
Only rows whose label starts with `sm-`, `dind-`, `ubicloud-`, `smoke-` or `pf-` cost anything. At the October 2026 review, a full pull-request run of `ci.yml` spent about 150 self-hosted minutes, a third of it in jobs that never compiled; treat that as the scale of what a placement mistake costs, not as a current figure.
Rows using `macos-latest-xlarge` or a label starting with `sm-`, `dind-`, `ubicloud-`, `smoke-` or `pf-` use paid capacity. At the October 2026 review, a full pull-request run of `ci.yml` spent about 150 self-hosted minutes, a third of it in jobs that never compiled; treat that as the scale of what a placement mistake costs, not as a current figure.
## Checklist for a new or moved job
@@ -76,3 +77,9 @@ Only rows whose label starts with `sm-`, `dind-`, `ubicloud-`, `smoke-` or `pf-`
- A self-hosted `runs-on` carries a comment naming its reason from the placement rule.
- Every `./.github/actions/setup` call states `cache-save-if`, and a key read from one pool is not written from the other.
- Moving a job does not change its check name, so the `main` ruleset and `docs/testing/ci-gates.md` stay as they are; renaming it does, and then that document's change checklist applies.
## Release stages
The [candidate release pipeline](../../docs/operations/release-pipeline.md) builds Linux first and defers paid desktop capacity until acceptance. Daily development builds also select Linux; use manual platform selection for desktop development artifacts. All six platforms remain required before publication. The pipeline records and promotes the exact tested bytes. Builder workflows have no public-release credentials. Keep periodic development builds, PR verification and field-test chains separate when their contracts differ; do not re-enable disabled workflows as part of a source-file migration.
Larger runners remain billed for public repositories; see [GitHub runner billing](https://docs.github.com/en/actions/concepts/runners/larger-runners#billing). Actual spending comes from organization billing, not runner labels or an estimated build duration.
+149 -8
View File
@@ -23,9 +23,16 @@ on:
- 'deny.toml'
- '.github/actions/**'
- '.github/workflows/**'
- 'scripts/release/create_or_update_release.sh'
- 'scripts/release/package_service_scripts.sh'
- 'scripts/release/package_versions.sh'
- 'scripts/release/**'
- 'scripts/security/generate_release_supply_chain_assets.py'
- '.config/release-validation-plan.json'
- '.github/scheduled-validations.json'
- 'scripts/test_helm_release_workflow.py'
- 'scripts/test_release_packages.py'
- 'scripts/test_release_images.py'
- 'Dockerfile*'
- 'entrypoint.sh'
- 'bootstrap/**'
- 'scripts/test_package_service_scripts.sh'
- 'scripts/test_package_versions.sh'
- 'scripts/test_docker_workflow.py'
@@ -41,9 +48,16 @@ on:
- 'deny.toml'
- '.github/actions/**'
- '.github/workflows/**'
- 'scripts/release/create_or_update_release.sh'
- 'scripts/release/package_service_scripts.sh'
- 'scripts/release/package_versions.sh'
- 'scripts/release/**'
- 'scripts/security/generate_release_supply_chain_assets.py'
- '.config/release-validation-plan.json'
- '.github/scheduled-validations.json'
- 'scripts/test_helm_release_workflow.py'
- 'scripts/test_release_packages.py'
- 'scripts/test_release_images.py'
- 'Dockerfile*'
- 'entrypoint.sh'
- 'bootstrap/**'
- 'scripts/test_package_service_scripts.sh'
- 'scripts/test_package_versions.sh'
- 'scripts/test_docker_workflow.py'
@@ -132,12 +146,38 @@ jobs:
if: github.repository == 'rustfs/rustfs' && (github.event_name != 'pull_request' || github.event.action != 'closed')
runs-on: ubuntu-latest
timeout-minutes: 5
permissions:
contents: read
actions: read
outputs:
release_changed: ${{ steps.scope.outputs.release_changed }}
steps:
- name: Checkout repository
uses: actions/checkout@f548e57e544e1ff5a4c46bf1e1b8685f8e4a348a # v7
with:
fetch-depth: 0
persist-credentials: false
- name: Scope native packaging checks to release changes
id: scope
env:
BASE_SHA: ${{ github.event.pull_request.base.sha || github.event.before }}
run: |
python3 - <<'PY'
import os, re, subprocess
event, base = os.environ['GITHUB_EVENT_NAME'], os.environ.get('BASE_SHA', '')
changed = event == 'workflow_dispatch'
if event in ('push', 'pull_request'):
if not re.fullmatch(r'[0-9a-f]{40}', base) or base == '0' * 40:
changed = True
else:
paths = subprocess.check_output(['git', 'diff', '--name-only', base, os.environ['GITHUB_SHA']], text=True).splitlines()
pattern = r'^(scripts/release/|scripts/test_release_|Dockerfile|entrypoint.sh$|bootstrap/|\.dockerignore$|\.config/release-validation-plan.json$|\.github/workflows/(audit|build|docker|package|helm-package|release-[^.]+)\.yml$)'
changed = any(re.match(pattern, path) for path in paths)
with open(os.environ['GITHUB_OUTPUT'], 'a') as output:
output.write(f'release_changed={str(changed).lower()}\n')
PY
- name: Report unpinned GitHub Actions
run: ./scripts/security/check_workflow_pins.sh --enforce
@@ -150,8 +190,35 @@ jobs:
- name: Check checkouts clear their credentials
run: ./scripts/security/check_persist_credentials.sh
- name: Check preview release workflow policy
run: ./scripts/security/check_preview_release_workflow.sh
- name: Check release candidate and promotion contracts
run: python3 -m unittest discover -s scripts/release -p 'test_*.py'
- name: Prepare artifact transport rehearsal
run: |
mkdir -p "$RUNNER_TEMP/release-transport"
printf '{"purpose":"transport contract, not release evidence"}\n' > "$RUNNER_TEMP/release-transport/transport.json"
- uses: actions/upload-artifact@043fb46d1a93c77aae656e7c1c64a875d1fc6a0a # v7.0.1
with:
name: release-transport-contract-${{ github.run_attempt }}
path: ${{ runner.temp }}/release-transport/transport.json
if-no-files-found: error
retention-days: 1
- name: Verify actual GitHub artifact metadata and downloaded bytes
env:
GH_TOKEN: ${{ github.token }}
run: |
python3 - <<'PY'
import os, sys
from pathlib import Path
sys.path.insert(0, 'scripts/release')
import candidate as c
run = c.github(f"repos/{c.REPOSITORY}/actions/runs/{os.environ['GITHUB_RUN_ID']}")
destination = Path(os.environ['RUNNER_TEMP']) / 'release-transport-downloaded'
c.download_artifact(run, f"release-transport-contract-{os.environ['GITHUB_RUN_ATTEMPT']}", destination)
c.require(c.read_json(destination / 'transport.json') == {'purpose': 'transport contract, not release evidence'}, 'artifact round trip changed')
PY
- name: Check tier evidence workflow isolation
run: ./scripts/security/check_tier_artifact_workflow.sh
@@ -162,6 +229,80 @@ jobs:
- name: Check package service lifecycle contract
run: ./scripts/test_package_service_scripts.sh
release-package-contract:
name: Release package contract
needs: workflow-pin-report
if: needs.workflow-pin-report.outputs.release_changed == 'true'
runs-on: ubuntu-latest
timeout-minutes: 15
steps:
- uses: actions/checkout@f548e57e544e1ff5a4c46bf1e1b8685f8e4a348a # v7
with:
persist-credentials: false
- name: Install real package managers
run: |
sudo apt-get update
sudo apt-get install -y ruby ruby-dev build-essential rpm fakeroot cpio
sudo gem install fpm --version 1.17.0 --no-document
- name: Exercise accepted and substituted payloads through both recipes
run: python3 scripts/test_release_packages.py
release-image-contract:
name: Release image contract (${{ matrix.variant }})
needs: workflow-pin-report
if: needs.workflow-pin-report.outputs.release_changed == 'true'
runs-on: ubuntu-latest
timeout-minutes: 35
strategy:
fail-fast: false
matrix:
variant: [musl, gnu]
env:
RELEASE_IMAGE_VARIANT: ${{ matrix.variant }}
NO_PROXY: 127.0.0.1,localhost,::1
steps:
- uses: actions/checkout@f548e57e544e1ff5a4c46bf1e1b8685f8e4a348a # v7
with:
persist-credentials: false
- name: Install OCI tools
run: |
echo "RELEASE_IMAGE_OUTPUT=$RUNNER_TEMP/image-contract" >> "$GITHUB_ENV"
echo 'no_proxy=127.0.0.1,localhost,::1' >> "$GITHUB_ENV"
sudo apt-get update
sudo apt-get install -y skopeo
go version
- uses: docker/setup-qemu-action@c7c53464625b32c7a7e944ae62b3e17d2b600130 # v3
- uses: docker/setup-buildx-action@8d2750c68a42422c14e847fe6c8ac0403b4cbd6f # v3
- name: Exercise real OCI packaging, startup, registry retry and conflicts
run: python3 scripts/test_release_images.py
- name: Exercise amd64 OCI archive scanning
uses: aquasecurity/trivy-action@ed142fd0673e97e23eac54620cfb913e5ce36c25
env:
TRIVY_PLATFORM: linux/amd64
with:
input: ${{ runner.temp }}/image-contract-scan/image-${{ matrix.variant }}-amd64.tar
format: sarif
output: ${{ runner.temp }}/image-contract/scan-${{ matrix.variant }}-amd64.sarif
exit-code: "0"
- name: Exercise arm64 OCI archive scanning
uses: aquasecurity/trivy-action@ed142fd0673e97e23eac54620cfb913e5ce36c25
env:
TRIVY_PLATFORM: linux/arm64
with:
input: ${{ runner.temp }}/image-contract-scan/image-${{ matrix.variant }}-arm64.tar
format: sarif
output: ${{ runner.temp }}/image-contract/scan-${{ matrix.variant }}-arm64.sarif
exit-code: "0"
- name: Verify scan receipts before recording the archive
run: |
python3 - <<'PY'
import os, sys
from pathlib import Path
sys.path.insert(0, 'scripts/release')
import deliver, test_candidate
deliver.image_record(test_candidate.inputs(), os.environ['RELEASE_IMAGE_VARIANT'], Path(os.environ['RELEASE_IMAGE_OUTPUT']))
PY
dependency-review:
name: Dependency Review
runs-on: ubuntu-latest
File diff suppressed because it is too large Load Diff
+78 -5
View File
@@ -113,7 +113,7 @@ jobs:
printf '%s\n' 'mode=full' >> "$GITHUB_OUTPUT"
echo "Base CI policy unavailable; running the full matrix."
fi
if ! grep -q '^plan=' "$GITHUB_OUTPUT" || ! grep -q '^plan_contract=2$' "$GITHUB_OUTPUT"; then
if ! grep -q '^plan=' "$GITHUB_OUTPUT" || ! grep -q '^plan_contract=3$' "$GITHUB_OUTPUT"; then
printf '%s\n' 'mode=full' 'connect=true' 'full_validation=true' 'rehearse_shards=false' 'packages=' 'plan=' >> "$GITHUB_OUTPUT"
echo "Base policy does not cover this workflow contract; retaining the full matrix."
fi
@@ -142,15 +142,23 @@ jobs:
- uses: actions/checkout@f548e57e544e1ff5a4c46bf1e1b8685f8e4a348a # v7
with:
persist-credentials: false
- name: Install pinned typos through GitHub API
env:
GH_TOKEN: ${{ github.token }}
TYPOS_VERSION: 1.50.2
run: |
python3 .github/actions/setup/download_release_asset.py --repository crate-ci/typos --tag "v${TYPOS_VERSION}" --asset "typos-v${TYPOS_VERSION}-x86_64-unknown-linux-musl.tar.gz" --output "$RUNNER_TEMP/typos.tar.gz"
mkdir -p "$RUNNER_TEMP/typos"
tar -xzf "$RUNNER_TEMP/typos.tar.gz" -C "$RUNNER_TEMP/typos"
- name: Typos check with custom config file
uses: crate-ci/typos@512fc24f32f44ab01972217aaaf3dc86ec234d53 # v1.50.2
run: '"$RUNNER_TEMP/typos/typos" .'
# Fail early with compile-free checks for every pull request.
quick-checks:
name: Quick Checks
if: github.event_name != 'pull_request' || github.event.action != 'closed'
runs-on: ubuntu-latest
timeout-minutes: 10
timeout-minutes: 20
steps:
- name: Checkout repository
uses: actions/checkout@f548e57e544e1ff5a4c46bf1e1b8685f8e4a348a # v7
@@ -215,7 +223,8 @@ jobs:
- name: Archive workspace test binaries
env:
CARGO_BUILD_JOBS: ${{ (github.event_name == 'push' || github.event_name == 'workflow_dispatch') && '3' || '2' }}
# Workspace test compilation has high per-process memory use.
CARGO_BUILD_JOBS: "2"
run: |
./scripts/ci/resource_sampler.sh start workspace-archive
trap './scripts/ci/resource_sampler.sh stop' EXIT
@@ -497,6 +506,63 @@ jobs:
- name: Check offline enrollment E2E root boundary
run: ./scripts/check_offline_enrollment_e2e.sh
fips-provider:
name: FIPS provider boundaries
if: needs.classify-changes.outputs.mode == 'full' && contains(fromJSON('["schedule", "workflow_dispatch"]'), github.event_name)
needs: [ quick-checks, classify-changes ]
# This builds the RustFS library test target, which exceeds hosted-runner disk capacity.
runs-on: sm-standard-4
timeout-minutes: 90
strategy:
fail-fast: false
matrix:
mode: [default, fips]
env:
FORCE_JAVASCRIPT_ACTIONS_TO_NODE24: "true"
CARGO_BUILD_JOBS: "2"
steps:
- name: Checkout repository
uses: actions/checkout@f548e57e544e1ff5a4c46bf1e1b8685f8e4a348a # v7
with:
persist-credentials: false
- name: Setup Rust environment
uses: ./.github/actions/setup
with:
rust-version: stable
cache-shared-key: ci-fips-provider
cache-save-if: 'false'
install-build-packaging-tools: 'false'
- name: Install FIPS build tools
if: matrix.mode == 'fips'
run: sudo apt-get install -y cmake golang clang libclang-dev
- name: Verify provider installation
shell: bash
run: |
args=()
if [[ '${{ matrix.mode }}' == 'fips' ]]; then
args+=(--features fips)
fi
cargo nextest run --locked -p rustfs-tls-runtime --lib "${args[@]}" --no-tests fail -E 'test(provider_policy_isolated)'
cargo nextest run --locked -p rustfs-s3-client --lib "${args[@]}" --no-tests fail -E 'test(overflowing_closed_range_is_rejected_before_network_io)'
cargo nextest run --locked -p rustfs-targets --lib "${args[@]}" --no-tests fail -E 'test(validate_pulsar)'
cargo nextest run --locked -p rustfs-targets --lib "${args[@]}" --no-tests fail -E 'test(webhook_provider_policy_isolated)'
if [[ '${{ matrix.mode }}' == 'fips' ]]; then
cargo nextest run --locked -p rustfs-protocols --lib --features ftps,webdav,fips --no-tests fail -E 'test(protocols_reject_incompatible_host_provider)'
cargo nextest run --locked -p rustfs-targets --lib --features fips --no-tests fail -E 'test(fips_postgres_rejects_incompatible_host_provider)'
cargo nextest run --locked -p rustfs-s3-client --lib --features fips --no-tests fail -E 'test(fips_transition_rejects_incompatible_host_provider)'
fi
tests=(provider_installation_isolated prepared_generation_shares_crypto_and_serves_sni_with_mtls)
if [[ '${{ matrix.mode }}' == 'fips' ]]; then
tests+=(fips_provider_builds_fips_tls_configs fips_provider_tls_handshake_accepts_and_rejects_algorithms)
fi
for test_name in "${tests[@]}"; do
cargo nextest run --locked -p rustfs --lib "${args[@]}" --no-tests fail -E "test($test_name)"
done
cargo clippy --locked -p rustfs --lib "${args[@]}" -- -D warnings
test-and-lint-rio-v2:
name: Test and Lint (rio-v2)
# Specialized boundary, hardware and alternate-feature lanes run only on
@@ -1079,11 +1145,16 @@ jobs:
- name: Install mc
env:
GH_TOKEN: ${{ github.token }}
MC_VERSION: RELEASE.2025-08-13T08-35-41Z
MC_SHA256: 01f866e9c5f9b87c2b09116fa5d7c06695b106242d829a8bb32990c00312e891
run: |
if ! command -v gh >/dev/null 2>&1; then
sudo apt-get update
sudo apt-get install -y gh
fi
MC_BINARY="mc.linux-amd64.${MC_VERSION}"
curl -fsSLo "$RUNNER_TEMP/mc" "https://github.com/minio/mc/releases/download/${MC_VERSION}/${MC_BINARY}"
python3 .github/actions/setup/download_release_asset.py --repository minio/mc --tag "$MC_VERSION" --asset "$MC_BINARY" --output "$RUNNER_TEMP/mc" --sha256 "$MC_SHA256"
echo "${MC_SHA256} $RUNNER_TEMP/mc" | sha256sum --check --status
chmod +x "$RUNNER_TEMP/mc"
echo "$RUNNER_TEMP" >> "$GITHUB_PATH"
@@ -1397,6 +1468,7 @@ jobs:
- workspace-shard-evidence
- core-contract
- connect-feature-boundaries
- fips-provider
- test-and-lint-rio-v2
- test-and-lint-protocols
- build-rustfs-debug-binary
@@ -1424,6 +1496,7 @@ jobs:
needs:
- classify-changes
- connect-feature-boundaries
- fips-provider
- required-checks
- typos
- quick-checks
+1 -1
View File
@@ -95,7 +95,7 @@ jobs:
- name: Run CLA Bot
if: github.event_name != 'merge_group'
uses: overtrue/cla-bot@5e5929846a60cfb42b16d0d38e135ef67d94fc88 # v0.0.9
uses: overtrue/cla-bot@4bd1387c19691168c42fcf5edfa17b15819b2ab3 # v0.1.1
with:
github-token: ${{ github.token }}
registry-token: ${{ steps.registry-token.outputs.token }}
+71 -751
View File
@@ -1,779 +1,99 @@
# Copyright 2024 RustFS Team
#
# Licensed under the Apache License, Version 2.0 (the "License");
# you may not use this file except in compliance with the License.
# You may obtain a copy of the License at
#
# http://www.apache.org/licenses/LICENSE-2.0
#
# Unless required by applicable law or agreed to in writing, software
# distributed under the License is distributed on an "AS IS" BASIS,
# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
# See the License for the specific language governing permissions and
# limitations under the License.
# Docker Images Workflow
#
# This workflow builds Docker images using pre-built binaries from the build workflow.
#
# Trigger Types:
# 1. workflow_run: Automatically triggered when "Build and Release" workflow completes
# 2. workflow_dispatch: Manual trigger for standalone Docker builds
#
# Key Features:
# - Only triggers when Linux builds (x86_64 + aarch64) are successful
# - Independent of macOS/Windows build status
# - Uses workflow_run event for precise control
# - Only builds Docker images for releases and prereleases (development builds are skipped)
name: Docker Images
# Permissions needed for workflow_run event and Docker registry access
# Copyright 2026 RustFS Team
# SPDX-License-Identifier: Apache-2.0
name: Build candidate images
on:
workflow_call:
inputs:
candidate_manifest:
type: string
required: true
permissions:
contents: read
packages: write
on:
# Automatically triggered when build workflow completes
workflow_run:
workflows: [ "Build and Release" ]
types: [ completed ]
# Manual trigger with same parameters for consistency
workflow_dispatch:
inputs:
push_images:
description: "Push images to registries"
required: false
default: true
type: boolean
version:
description: "Version to build (latest, v1.0.0, v1.0.0-alpha1, or v1.0.1-preview.1)"
required: false
default: "latest"
type: string
force_rebuild:
description: "Force rebuild even if binary exists (useful for testing)"
required: false
default: false
type: boolean
env:
CONCLUSION: ${{ github.event.workflow_run.conclusion }}
HEAD_BRANCH: ${{ github.event.workflow_run.head_branch }}
HEAD_SHA: ${{ github.event.workflow_run.head_sha }}
TRIGGERING_EVENT: ${{ github.event.workflow_run.event }}
DOCKERHUB_USERNAME: rustfs
CARGO_TERM_COLOR: always
REGISTRY_DOCKERHUB: rustfs/rustfs
REGISTRY_GHCR: ghcr.io/${{ github.repository }}
REGISTRY_QUAY: quay.io/rustfs/rustfs
DOCKER_PLATFORMS: linux/amd64,linux/arm64
actions: read
jobs:
# Check if we should build Docker images.
# Short-circuit at job level so per-merge development builds (workflow_run
# from a push to main) skip the whole run without spawning a checkout job:
# images are only published for tag builds — build.yml push triggers cover
# only main and tags, so a non-main head_branch is a tag name — and for
# manual dispatch.
build-check:
name: Docker Build Check
if: >-
github.event_name == 'workflow_dispatch' ||
(github.event.workflow_run.conclusion == 'success' &&
github.event.workflow_run.event == 'push' &&
github.event.workflow_run.head_branch != 'main')
runs-on: ubuntu-latest
timeout-minutes: 30
outputs:
should_build: ${{ steps.check.outputs.should_build }}
should_push: ${{ steps.check.outputs.should_push }}
build_type: ${{ steps.check.outputs.build_type }}
version: ${{ steps.check.outputs.version }}
short_sha: ${{ steps.check.outputs.short_sha }}
is_prerelease: ${{ steps.check.outputs.is_prerelease }}
create_latest: ${{ steps.check.outputs.create_latest }}
source_ref: ${{ steps.check.outputs.source_ref }}
steps:
- name: Checkout repository
uses: actions/checkout@f548e57e544e1ff5a4c46bf1e1b8685f8e4a348a # v7
with:
persist-credentials: false
# For workflow_run events, checkout the specific commit that triggered the workflow
ref: ${{ github.event.workflow_run.head_sha || github.sha }}
- name: Check build conditions
id: check
env:
# dispatch inputs via env, not `${{ }}` interpolation: they are
# free-form strings and would otherwise be evaluated by bash.
INPUT_VERSION: ${{ github.event.inputs.version }}
INPUT_PUSH_IMAGES: ${{ github.event.inputs.push_images }}
INPUT_FORCE_REBUILD: ${{ github.event.inputs.force_rebuild }}
run: |
should_build=false
should_push=false
build_type="none"
version=""
short_sha=""
is_prerelease=false
create_latest=false
source_ref="$GITHUB_SHA"
# Pre-GA policy: until the first stable (vX.Y.Z) tag exists, every
# prerelease (alpha/beta/rc) also moves `latest`, so users pulling
# `latest` get the newest test build. Once a stable tag is published
# this returns false and `latest` follows stable releases only.
prerelease_moves_latest() {
local stable_tags
stable_tags=$(git ls-remote --tags --refs origin 2>/dev/null \
| awk '{print $2}' \
| grep -E '^refs/tags/v?[0-9]+\.[0-9]+\.[0-9]+$' || true)
if [[ -z "$stable_tags" ]]; then
return 0
fi
echo "ℹ️ Stable release tag(s) already exist; prereleases no longer update latest"
return 1
}
if [[ "${{ github.event_name }}" == "workflow_run" ]]; then
# Triggered by build workflow completion
echo "🔗 Triggered by build workflow completion"
# Check if the triggering workflow was successful
# If the workflow succeeded, it means ALL builds (including Linux x86_64 and aarch64) succeeded
if [[ "$CONCLUSION" == "success" ]]; then
echo "✅ Build workflow succeeded, all builds including Linux are successful"
should_build=true
should_push=true
else
echo "❌ Build workflow failed (conclusion: $CONCLUSION), skipping Docker build"
should_build=false
fi
# Extract version info from commit message or use commit SHA
# Use Git to generate consistent short SHA (ensures uniqueness like build.yml)
short_sha=$(git rev-parse --short "$HEAD_SHA")
source_ref="$HEAD_SHA"
# Determine build type based on triggering workflow event and ref
triggering_event="$TRIGGERING_EVENT"
head_branch="$HEAD_BRANCH"
echo "🔍 Analyzing triggering workflow:"
echo " 📋 Event: $triggering_event"
echo " 🌿 Head branch: $head_branch"
echo " 📎 Head SHA: $HEAD_SHA"
# Check if this was triggered by a tag push
if [[ "$triggering_event" == "push" ]]; then
# For tag pushes, head_branch will be like "refs/tags/v1.0.0" or just "v1.0.0"
if [[ "$head_branch" == refs/tags/* ]]; then
# Extract tag name from refs/tags/TAG_NAME
tag_name="${head_branch#refs/tags/}"
version="$tag_name"
elif [[ "$head_branch" =~ ^v?[0-9]+\.[0-9]+\.[0-9]+ ]]; then
# Direct tag name like "v1.0.0" or "1.0.0-alpha.1"
version="$head_branch"
elif [[ "$head_branch" == "main" ]]; then
# Regular branch push to main
build_type="development"
version="dev-${short_sha}"
should_build=false
echo "⏭️ Skipping Docker build for development version (main branch push)"
else
# Other branch push
build_type="development"
version="dev-${short_sha}"
should_build=false
echo "⏭️ Skipping Docker build for development version (branch: $head_branch)"
fi
# If we extracted a version (tag), determine release type
if [[ -n "$version" ]] && [[ "$version" != "dev-${short_sha}" ]]; then
# Remove 'v' prefix if present for consistent version format
if [[ "$version" == v* ]]; then
version="${version#v}"
fi
if [[ "$version" =~ -preview\.[0-9]+$ ]]; then
build_type="preview"
is_prerelease=true
echo "🔍 Building Docker image for preview: $version (version tags only)"
elif [[ "$version" == *"-preview"* ]]; then
echo "❌ Invalid preview tag: $version (expected suffix: -preview.<number>)" >&2
exit 1
elif [[ "$version" == *"alpha"* ]] || [[ "$version" == *"beta"* ]] || [[ "$version" == *"rc"* ]]; then
build_type="prerelease"
is_prerelease=true
# Pre-GA policy: prereleases update latest until the first stable tag exists.
if prerelease_moves_latest; then
create_latest=true
echo "🧪 Building Docker image for prerelease: $version (creating latest tag)"
else
echo "🧪 Building Docker image for prerelease: $version"
fi
else
build_type="release"
create_latest=true
echo "🚀 Building Docker image for release: $version"
fi
fi
else
# Non-push events
build_type="development"
version="dev-${short_sha}"
should_build=false
echo "⏭️ Skipping Docker build for development version (event: $triggering_event)"
fi
echo "🔄 Build triggered by workflow_run:"
echo " 📋 Conclusion: $CONCLUSION"
echo " 🌿 Branch: $HEAD_BRANCH"
echo " 📎 SHA: $HEAD_SHA"
echo " 🎯 Event: $TRIGGERING_EVENT"
elif [[ "${{ github.event_name }}" == "workflow_dispatch" ]]; then
# Manual trigger
input_version="$INPUT_VERSION"
version="${input_version}"
should_push="$INPUT_PUSH_IMAGES"
should_build=true
# Get short SHA
short_sha=$(git rev-parse --short HEAD)
echo "🎯 Manual Docker build triggered:"
echo " 📋 Requested version: $input_version"
echo " 🔧 Force rebuild: $INPUT_FORCE_REBUILD"
echo " 🚀 Push images: $should_push"
case "$input_version" in
"latest")
build_type="release"
create_latest=true
echo "🚀 Building with latest stable release version"
;;
*-preview*)
if [[ ! "$input_version" =~ ^v?[0-9]+\.[0-9]+\.[0-9]+(-[0-9A-Za-z.-]+)?-preview\.[0-9]+$ ]]; then
echo "❌ Invalid preview version: $input_version" >&2
exit 1
fi
build_type="preview"
is_prerelease=true
version="${input_version#v}"
echo "🔍 Building with preview version: $version (version tags only)"
;;
# Prerelease versions (must match first, more specific)
v*alpha*|v*beta*|v*rc*|*alpha*|*beta*|*rc*)
build_type="prerelease"
is_prerelease=true
# Pre-GA policy: prereleases update latest until the first stable tag exists.
if prerelease_moves_latest; then
create_latest=true
echo "🧪 Building with prerelease version: $input_version (creating latest tag)"
else
echo "🧪 Building with prerelease version: $input_version"
fi
;;
# Release versions (match after prereleases, more general)
v[0-9]*|[0-9]*.*.*)
build_type="release"
create_latest=true
echo "📦 Building with specific release version: $input_version"
;;
*)
# Invalid version for Docker build
should_build=false
echo "❌ Invalid version for Docker build: $input_version"
echo "⚠️ Only release versions (latest, v1.0.0, 1.0.0) and prereleases (v1.0.0-alpha1, 1.0.0-beta2) are supported"
;;
esac
if [[ "$should_build" == true && "$input_version" != "latest" ]]; then
tag_ref="refs/tags/$input_version"
if ! git ls-remote --exit-code origin "$tag_ref" >/dev/null 2>&1; then
if [[ "$input_version" == v* ]]; then
tag_ref="refs/tags/${input_version#v}"
else
tag_ref="refs/tags/v$input_version"
fi
fi
if ! git ls-remote --exit-code origin "$tag_ref" >/dev/null 2>&1; then
echo "❌ Release tag not found for Docker build: $input_version"
exit 1
fi
source_ref="$tag_ref"
fi
fi
{
echo "should_build=$should_build"
echo "should_push=$should_push"
echo "build_type=$build_type"
echo "version=$version"
echo "short_sha=$short_sha"
echo "is_prerelease=$is_prerelease"
echo "create_latest=$create_latest"
echo "source_ref=$source_ref"
} >> "$GITHUB_OUTPUT"
echo "🐳 Docker Build Summary:"
echo " - Should build: $should_build"
echo " - Should push: $should_push"
echo " - Build type: $build_type"
echo " - Version: $version"
echo " - Short SHA: $short_sha"
echo " - Is prerelease: $is_prerelease"
echo " - Create latest: $create_latest"
echo " - Source ref: $source_ref"
# Consume the trusted release certification for the exact version this
# workflow wraps and verify the certified input digests. A manual dispatch
# without that certification fails here and pushes no official tag; only
# results from the trusted verification sources carry release authority.
release-certification:
name: Release Certification
needs: build-check
if: needs.build-check.outputs.should_build == 'true'
runs-on: ubuntu-latest
timeout-minutes: 30
permissions:
contents: read
actions: read
steps:
- name: Checkout repository
uses: actions/checkout@f548e57e544e1ff5a4c46bf1e1b8685f8e4a348a # v7
with:
persist-credentials: false
ref: ${{ needs.build-check.outputs.source_ref }}
- name: Download and verify release certification
env:
GH_TOKEN: ${{ secrets.RELEASE_VALIDATION_TOKEN || github.token }}
TRIGGERING_RUN_ID: ${{ github.event.workflow_run.id }}
VERSION: ${{ needs.build-check.outputs.version }}
shell: bash
run: |
set -euo pipefail
mkdir -p certification
run_id="$TRIGGERING_RUN_ID"
if [[ -z "$run_id" ]]; then
run_id="$(gh api --method GET \
"repos/${GITHUB_REPOSITORY}/actions/workflows/build.yml/runs" \
-f branch="$VERSION" -f status=success -F per_page=1 \
--jq '.workflow_runs[0].id // empty' || true)"
fi
if [[ -z "$run_id" ]]; then
echo "No certified Build and Release run found for ${VERSION}; failing closed" >&2
exit 1
fi
if ! gh run download "$run_id" -n release-certification -D certification; then
echo "Build run ${run_id} published no release-certification artifact; failing closed" >&2
exit 1
fi
python3 scripts/release/release_gate.py verify-promotion \
--plan .config/release-validation-plan.json \
--candidate certification/candidate.json \
--summary certification/release-gate-output \
--artifact-sha256 "$(jq -r '.deliverables[0].sha256' certification/candidate.json)" \
--artifact-name "$(jq -r '.deliverables[0].name' certification/candidate.json)"
# Input digest binding: every zip this image wraps must be the
# certified byte-for-byte content, never a mutable latest pointer.
while IFS=$'\t' read -r name location expected; do
echo "Verifying certified input ${name}"
actual="$(curl -fsSL "$location" | sha256sum | awk '{print $1}')"
if [[ "$actual" != "$expected" ]]; then
echo "Certified input ${name} does not match its recorded digest" >&2
exit 1
fi
done < <(jq -r '.deliverables[] | select(.name | contains("linux")) | [.name, .location, .sha256] | @tsv' certification/candidate.json)
# Build multi-arch Docker images
# Strategy: Build images using pre-built binaries from dl.rustfs.com
# Supports both release and dev channel binaries based on build context
# Only runs when should_build is true (which includes workflow success check)
build-docker:
name: Build Docker Images
needs: [ build-check, release-certification ]
if: needs.build-check.outputs.should_build == 'true'
runs-on: ubuntu-latest
timeout-minutes: 60
timeout-minutes: 90
strategy:
fail-fast: false
matrix:
include:
- variant: musl
file: Dockerfile
suffix: ""
- variant: glibc
- variant: gnu
file: Dockerfile.glibc
suffix: "-glibc"
env:
CANDIDATE_MANIFEST: ${{ inputs.candidate_manifest }}
VARIANT: ${{ matrix.variant }}
steps:
- name: Checkout repository
uses: actions/checkout@f548e57e544e1ff5a4c46bf1e1b8685f8e4a348a # v7
- uses: actions/checkout@f548e57e544e1ff5a4c46bf1e1b8685f8e4a348a # v7
with:
ref: ${{ fromJson(inputs.candidate_manifest).source_sha }}
persist-credentials: false
ref: ${{ needs.build-check.outputs.source_ref }}
- name: Login to Docker Hub
uses: docker/login-action@c94ce9fb468520275223c153574b00df6fe4bcc9 # v3
with:
username: ${{ env.DOCKERHUB_USERNAME }}
password: ${{ secrets.DOCKERHUB_TOKEN }}
- name: Login to GitHub Container Registry
uses: docker/login-action@c94ce9fb468520275223c153574b00df6fe4bcc9 # v3
with:
registry: ghcr.io
username: ${{ secrets.GHCR_USERNAME }}
password: ${{ secrets.GHCR_PASSWORD }}
- name: Login to Quay.io
uses: docker/login-action@c94ce9fb468520275223c153574b00df6fe4bcc9 # v3
with:
registry: quay.io
username: ${{ secrets.QUAY_USERNAME }}
password: ${{ secrets.QUAY_PASSWORD }}
- name: Set up QEMU
uses: docker/setup-qemu-action@c7c53464625b32c7a7e944ae62b3e17d2b600130 # v3
- name: Set up Docker Buildx
uses: docker/setup-buildx-action@8d2750c68a42422c14e847fe6c8ac0403b4cbd6f # v3
- name: Extract metadata and generate tags
id: meta
- name: Reuse completed image
id: reuse
env:
GH_TOKEN: ${{ github.token }}
run: python3 scripts/release/deliver.py reuse --kind image --target "$VARIANT"
- name: Install OCI tools
if: steps.reuse.outputs.reused != 'true'
run: |
BUILD_TYPE="${{ needs.build-check.outputs.build_type }}"
VERSION="${{ needs.build-check.outputs.version }}"
CREATE_LATEST="${{ needs.build-check.outputs.create_latest }}"
VARIANT_SUFFIX="${{ matrix.suffix }}"
# Convert version format for Dockerfile compatibility. The former
# DOCKER_CHANNEL was "release" down every branch and was passed as a
# build-arg no Dockerfile declares, so it is gone.
case "$VERSION" in
"latest")
DOCKER_RELEASE="latest"
;;
v*)
# For versioned releases (v1.0.0), remove 'v' prefix for Dockerfile
DOCKER_RELEASE="${VERSION#v}"
;;
*)
# For other versions, pass as-is
DOCKER_RELEASE="${VERSION}"
;;
esac
echo "docker_release=$DOCKER_RELEASE" >> "$GITHUB_OUTPUT"
echo "🐳 Docker build parameters:"
echo " - Original version: $VERSION"
echo " - Docker RELEASE: $DOCKER_RELEASE"
# Generate tags based on build type
# Only support release and prerelease builds (no development builds)
TAG_BASE="${VERSION}${VARIANT_SUFFIX}"
TAGS="${{ env.REGISTRY_DOCKERHUB }}:$TAG_BASE,${{ env.REGISTRY_GHCR }}:$TAG_BASE,${{ env.REGISTRY_QUAY }}:$TAG_BASE"
# Add latest when requested (stable releases, and prereleases before GA)
if [[ "$CREATE_LATEST" == "true" ]]; then
TAGS="$TAGS,${{ env.REGISTRY_DOCKERHUB }}:latest${VARIANT_SUFFIX},${{ env.REGISTRY_GHCR }}:latest${VARIANT_SUFFIX},${{ env.REGISTRY_QUAY }}:latest${VARIANT_SUFFIX}"
fi
# Always add the channel tag for prereleases, independent of latest
if [[ "$BUILD_TYPE" == "prerelease" ]]; then
# Prerelease channel tags (alpha, beta, rc)
if [[ "$VERSION" == *"alpha"* ]]; then
CHANNEL="alpha"
elif [[ "$VERSION" == *"beta"* ]]; then
CHANNEL="beta"
elif [[ "$VERSION" == *"rc"* ]]; then
CHANNEL="rc"
fi
if [[ -n "$CHANNEL" ]]; then
TAGS="$TAGS,${{ env.REGISTRY_DOCKERHUB }}:${CHANNEL}${VARIANT_SUFFIX},${{ env.REGISTRY_GHCR }}:${CHANNEL}${VARIANT_SUFFIX},${{ env.REGISTRY_QUAY }}:${CHANNEL}${VARIANT_SUFFIX}"
fi
fi
# Output tags
echo "tags=$TAGS" >> "$GITHUB_OUTPUT"
# Staging identity for this run's promotion: official tags are only
# created after the staged image verifies against the certification,
# and they point at this same digest (no rebuild-after-verify).
STAGING_TAG="staging-${GITHUB_RUN_ID}-${GITHUB_RUN_ATTEMPT}${VARIANT_SUFFIX}"
STAGING_TAGS="${{ env.REGISTRY_DOCKERHUB }}:${STAGING_TAG},${{ env.REGISTRY_GHCR }}:${STAGING_TAG},${{ env.REGISTRY_QUAY }}:${STAGING_TAG}"
echo "staging_tags=$STAGING_TAGS" >> "$GITHUB_OUTPUT"
echo "staging_tag=$STAGING_TAG" >> "$GITHUB_OUTPUT"
# Generate labels
LABELS="org.opencontainers.image.title=RustFS"
LABELS="$LABELS,org.opencontainers.image.description=RustFS distributed object storage system"
LABELS="$LABELS,org.opencontainers.image.version=$VERSION"
SOURCE_REVISION="$(git rev-parse HEAD)"
LABELS="$LABELS,org.opencontainers.image.revision=$SOURCE_REVISION"
LABELS="$LABELS,org.opencontainers.image.source=${{ github.server_url }}/${{ github.repository }}"
LABELS="$LABELS,org.opencontainers.image.created=$(date -u +'%Y-%m-%dT%H:%M:%SZ')"
LABELS="$LABELS,org.opencontainers.image.build-type=$BUILD_TYPE"
echo "labels=$LABELS" >> "$GITHUB_OUTPUT"
echo "🐳 Generated Docker tags:"
echo "$TAGS" | tr ',' '\n' | sed 's/^/ - /'
echo "📋 Build type: $BUILD_TYPE"
echo "🔖 Version: $VERSION"
- name: Build and push Docker image
sudo apt-get update
sudo apt-get install -y skopeo
- uses: docker/setup-qemu-action@c7c53464625b32c7a7e944ae62b3e17d2b600130 # v3
if: steps.reuse.outputs.reused != 'true'
- uses: docker/setup-buildx-action@8d2750c68a42422c14e847fe6c8ac0403b4cbd6f # v3
if: steps.reuse.outputs.reused != 'true'
- name: Supply accepted binaries and pinned CLI
if: steps.reuse.outputs.reused != 'true'
env:
GH_TOKEN: ${{ github.token }}
run: python3 scripts/release/deliver.py image-prepare --target "$VARIANT"
- name: Build OCI archive without registry writes
if: steps.reuse.outputs.reused != 'true'
uses: docker/build-push-action@10e90e3645eae34f1e60eeb005ba3a3d33f178e8 # v6
with:
context: .
file: ${{ matrix.file }}
platforms: ${{ env.DOCKER_PLATFORMS }}
push: ${{ needs.build-check.outputs.should_push == 'true' }}
# Push to staging tags only: official tags are created by the
# certification-gated promotion step below from this same digest.
tags: ${{ steps.meta.outputs.staging_tags }}
labels: ${{ steps.meta.outputs.labels }}
# No layer cache. This build compiles nothing — it downloads a
# release zip and runs apk/apt — so the cache could only save the
# minute or two those take, while creating a correctness problem: with
# RELEASE=latest the binary URL is resolved by curl *inside* a RUN
# layer, and the layer key does not include what that resolved to. A
# rebuild at the same RELEASE value (dispatch with version=latest, or
# a re-run of the same version) would hit the old layer and ship the
# previous release's binary. mode=max also consumed the same 10GB
# Actions cache quota the Rust lanes are fighting over.
#
# Only RELEASE is passed: it is the sole build-arg the Dockerfiles
# declare besides TARGETARCH. BUILDTIME, VERSION, BUILD_TYPE, REVISION
# and CHANNEL were never read by any stage (and BUILDTIME's $(date ...)
# was a literal here, not a shell substitution). BUILD_DATE and VCS_REF
# are declared by the Dockerfiles but deliberately left unset —
# supplying them would change the published image labels.
platforms: linux/amd64,linux/arm64
push: false
provenance: false
outputs: type=oci,dest=delivery/image-${{ matrix.variant }}.tar
build-args: |
RELEASE=${{ steps.meta.outputs.docker_release }}
provenance: true
sbom: true
# Add retry mechanism by splitting the build process
no-cache: false
pull: true
# A later-built wrapper: the staged image must complete this platform's
# own binary identity verification before any official tag is written.
# It may not inherit an unverified package's conclusion.
- name: Verify staged image against certification
if: needs.build-check.outputs.should_push == 'true'
env:
GH_TOKEN: ${{ secrets.RELEASE_VALIDATION_TOKEN || github.token }}
TRIGGERING_RUN_ID: ${{ github.event.workflow_run.id }}
VERSION: ${{ needs.build-check.outputs.version }}
shell: bash
run: |
set -euo pipefail
mkdir -p certification
run_id="$TRIGGERING_RUN_ID"
if [[ -z "$run_id" ]]; then
run_id="$(gh api --method GET \
"repos/${GITHUB_REPOSITORY}/actions/workflows/build.yml/runs" \
-f branch="$VERSION" -f status=success -F per_page=1 \
--jq '.workflow_runs[0].id // empty' || true)"
fi
gh run download "$run_id" -n release-certification -D certification
STAGING_REF="${{ env.REGISTRY_DOCKERHUB }}:staging-${GITHUB_RUN_ID}-${GITHUB_RUN_ATTEMPT}${{ matrix.suffix }}"
case "${{ matrix.variant }}" in
musl) libc="musl" ;;
glibc) libc="gnu" ;;
esac
for platform in linux/amd64 linux/arm64; do
case "$platform" in
linux/amd64) arch_sub="x86_64" ;;
linux/arm64) arch_sub="aarch64" ;;
esac
docker pull --platform "$platform" "$STAGING_REF"
container="$(docker create --platform "$platform" "$STAGING_REF")"
docker cp "$container:/usr/bin/rustfs" "$RUNNER_TEMP/staged-rustfs"
docker rm "$container" >/dev/null
actual="$(sha256sum "$RUNNER_TEMP/staged-rustfs" | awk '{print $1}')"
expected="$(jq -r --arg needle "linux-${arch_sub}-${libc}" \
'[.deliverables[] | select(.name | contains($needle)) | .binary_sha256][0] // empty' \
certification/candidate.json)"
if [[ -z "$expected" || "$actual" != "$expected" ]]; then
echo "Staged image binary for ${platform} does not match the certified binary digest" >&2
exit 1
fi
python3 scripts/release/release_gate.py verify-promotion \
--plan .config/release-validation-plan.json \
--candidate certification/candidate.json \
--summary certification/release-gate-output \
--artifact "$RUNNER_TEMP/staged-rustfs" \
--artifact-role binary
done
- name: Promote staged image to official tags
if: needs.build-check.outputs.should_push == 'true'
shell: bash
run: |
set -euo pipefail
STAGING_REF="${{ env.REGISTRY_DOCKERHUB }}:staging-${GITHUB_RUN_ID}-${GITHUB_RUN_ATTEMPT}${{ matrix.suffix }}"
IFS=',' read -r -a official_tags <<< "${{ steps.meta.outputs.tags }}"
for tag in "${official_tags[@]}"; do
docker buildx imagetools create -t "$tag" "$STAGING_REF"
echo "Promoted $tag -> $STAGING_REF"
done
# Downstream consumers (Helm) verify this record instead of inheriting
# an unverified wrapper's conclusion.
- name: Record image promotion
if: needs.build-check.outputs.should_push == 'true'
shell: bash
env:
GH_TOKEN: ${{ secrets.RELEASE_VALIDATION_TOKEN || github.token }}
TRIGGERING_RUN_ID: ${{ github.event.workflow_run.id }}
VERSION: ${{ needs.build-check.outputs.version }}
run: |
set -euo pipefail
mkdir -p certification
run_id="$TRIGGERING_RUN_ID"
if [[ -z "$run_id" ]]; then
run_id="$(gh api --method GET \
"repos/${GITHUB_REPOSITORY}/actions/workflows/build.yml/runs" \
-f branch="$VERSION" -f status=success -F per_page=1 \
--jq '.workflow_runs[0].id // empty' || true)"
fi
gh run download "$run_id" -n release-certification -D certification
STAGING_REF="${{ env.REGISTRY_DOCKERHUB }}:staging-${GITHUB_RUN_ID}-${GITHUB_RUN_ATTEMPT}${{ matrix.suffix }}"
digest="$(docker buildx imagetools inspect "$STAGING_REF" --format '{{json .Manifest}}' | jq -r '.digest')"
candidate_sha="$(python3 -c "import json,hashlib,sys; sys.path.insert(0,'scripts/release'); import release_gate; print(release_gate.canonical_sha256(json.load(open('certification/candidate.json'))))")"
jq -n \
--arg version "$VERSION" \
--arg digest "$digest" \
--arg staging_ref "$STAGING_REF" \
--arg candidate_sha256 "$candidate_sha" \
--arg tags "${{ steps.meta.outputs.tags }}" \
--slurpfile candidate certification/candidate.json \
'{schema_version: 1, version: $version, image_digest: $digest, staging_ref: $staging_ref,
promoted_tags: ($tags | split(",")), candidate_sha256: $candidate_sha256,
binary_sha256s: ($candidate[0].deliverables | map(.binary_sha256) | unique)}' \
> image-promotion-record.json
- name: Upload image promotion record
if: needs.build-check.outputs.should_push == 'true'
uses: actions/upload-artifact@043fb46d1a93c77aae656e7c1c64a875d1fc6a0a # v7.0.1
with:
name: image-promotion-record${{ matrix.suffix }}
path: image-promotion-record.json
if-no-files-found: error
retention-days: 30
# Note: Manifest creation is no longer needed as we only build one variant
# Multi-arch manifests are automatically created by docker/build-push-action
# Report-only container image vulnerability scan
scan-docker-image:
name: Scan Docker Images
needs: [ build-check, build-docker ]
if: needs.build-check.outputs.should_build == 'true' && needs.build-check.outputs.should_push == 'true'
runs-on: ubuntu-latest
timeout-minutes: 30
permissions:
contents: read
security-events: write
strategy:
fail-fast: false
matrix:
include:
- variant: musl
suffix: ""
- variant: glibc
suffix: "-glibc"
steps:
- name: Login to GitHub Container Registry
# docker/login-action v3
uses: docker/login-action@c94ce9fb468520275223c153574b00df6fe4bcc9
with:
registry: ghcr.io
username: ${{ secrets.GHCR_USERNAME }}
password: ${{ secrets.GHCR_PASSWORD }}
- name: Scan image with Trivy
# aquasecurity/trivy-action v0.36.0
BINARY_SOURCE=supplied
RELEASE=${{ fromJson(inputs.candidate_manifest).version }}
VCS_REF=${{ fromJson(inputs.candidate_manifest).source_sha }}
- name: Verify wrapped binaries and startup on both architectures
if: steps.reuse.outputs.reused != 'true'
run: python3 scripts/release/deliver.py image-smoke --target "$VARIANT"
- name: Scan amd64 image before sealing
if: steps.reuse.outputs.reused != 'true'
uses: aquasecurity/trivy-action@ed142fd0673e97e23eac54620cfb913e5ce36c25
env:
TRIVY_PLATFORM: linux/amd64
with:
image-ref: ${{ env.REGISTRY_GHCR }}:${{ needs.build-check.outputs.version }}${{ matrix.suffix }}
input: delivery-scan/image-${{ matrix.variant }}-amd64.tar
format: sarif
output: trivy-${{ matrix.variant }}.sarif
ignore-unfixed: true
vuln-type: os,library
severity: CRITICAL,HIGH
output: delivery/scan-${{ matrix.variant }}-amd64.sarif
exit-code: "0" # Keep the repository's report-only vulnerability policy.
- name: Scan arm64 image before sealing
if: steps.reuse.outputs.reused != 'true'
uses: aquasecurity/trivy-action@ed142fd0673e97e23eac54620cfb913e5ce36c25
env:
TRIVY_PLATFORM: linux/arm64
with:
input: delivery-scan/image-${{ matrix.variant }}-arm64.tar
format: sarif
output: delivery/scan-${{ matrix.variant }}-arm64.sarif
exit-code: "0"
# Surface findings in the Security tab; an artifact alone is write-only
# reporting nobody reviews.
- name: Upload scan results to code scanning
uses: github/codeql-action/upload-sarif@99df26d4f13ea111d4ec1a7dddef6063f76b97e9 # v4.37.0
- name: Record verified OCI identity
if: steps.reuse.outputs.reused != 'true'
run: python3 scripts/release/deliver.py image-record --target "$VARIANT"
- uses: actions/upload-artifact@043fb46d1a93c77aae656e7c1c64a875d1fc6a0a # v7.0.1
if: steps.reuse.outputs.reused != 'true'
with:
sarif_file: trivy-${{ matrix.variant }}.sarif
category: container-image-${{ matrix.variant }}
- name: Upload container scan report
uses: actions/upload-artifact@043fb46d1a93c77aae656e7c1c64a875d1fc6a0a # v7.0.1
with:
name: container-image-scan-${{ matrix.variant }}
path: trivy-${{ matrix.variant }}.sarif
name: image-${{ matrix.variant }}
path: delivery/*
if-no-files-found: error
retention-days: 30
# Docker build summary
docker-summary:
name: Docker Build Summary
needs: [ build-check, build-docker ]
if: always() && needs.build-check.outputs.should_build == 'true'
runs-on: ubuntu-latest
timeout-minutes: 10
steps:
- name: Docker build completion summary
run: |
BUILD_TYPE="${{ needs.build-check.outputs.build_type }}"
VERSION="${{ needs.build-check.outputs.version }}"
CREATE_LATEST="${{ needs.build-check.outputs.create_latest }}"
echo "🐳 Docker build completed successfully!"
echo "📦 Build type: $BUILD_TYPE"
echo "🔢 Version: $VERSION"
echo "🚀 Strategy: Images using pre-built binaries (release channel only)"
echo ""
case "$BUILD_TYPE" in
"preview")
echo "🔍 Preview Docker image has been built with ${VERSION} tags"
echo "⏭️ Preview images do not update latest or prerelease channels"
;;
"release")
echo "🚀 Release Docker image has been built with ${VERSION} tags"
echo "✅ This image is ready for production use"
if [[ "$CREATE_LATEST" == "true" ]]; then
echo "🏷️ Latest tag has been created for stable release"
fi
;;
"prerelease")
echo "🧪 Prerelease Docker image has been built with ${VERSION} tags"
echo "⚠️ This is a prerelease image - use with caution"
# Prereleases move latest until the first stable tag exists (pre-GA policy).
if [[ "$CREATE_LATEST" == "true" ]]; then
echo "🏷️ Latest tag has been created for prerelease: $VERSION"
else
echo "🚫 Latest tag NOT created for prerelease"
fi
;;
*)
echo "❌ Unexpected build type: $BUILD_TYPE"
;;
esac
compression-level: 0
+2 -28
View File
@@ -17,8 +17,8 @@
# Each selected test starts a real localhost cluster via
# `RustFSTestClusterEnvironment` (4 processes; 4 drives per node unless the
# case is a two-site 4-node 1-drive pair or a 4-node upgrade). Membership is
# `[profile.e2e-distributed]` in `.config/nextest.toml`. Storage-sensitive PRs,
# nightly runs, and manual dispatches all execute the same fail-closed suite.
# `[profile.e2e-distributed]` in `.config/nextest.toml`. The nightly schedule
# and manual dispatches execute the same fail-closed suite.
# Upgrade cases download the same pinned previous release as e2e-upgrade.yml.
#
# Isolated pool filesystems: expand/decommission/rebalance cases require
@@ -33,32 +33,6 @@
name: e2e-distributed
on:
pull_request:
paths:
- "Cargo.lock"
- "Cargo.toml"
- ".config/nextest.toml"
- ".config/e2e-distributed-selection.txt"
- ".github/workflows/e2e-distributed.yml"
- "crates/audit/**"
- "crates/common/**"
- "crates/config/**"
- "crates/e2e_test/**"
- "crates/ecstore/**"
- "crates/filemeta/**"
- "crates/heal/**"
- "crates/iam/**"
- "crates/lock/**"
- "crates/madmin/**"
- "crates/notify/**"
- "crates/replication/**"
- "crates/s3-client/**"
- "crates/s3-ops/**"
- "crates/s3-types/**"
- "crates/scanner/**"
- "crates/storage-api/**"
- "crates/utils/**"
- "rustfs/**"
workflow_dispatch:
inputs:
filter:
+7 -18
View File
@@ -12,30 +12,19 @@
# See the License for the specific language governing permissions and
# limitations under the License.
# Single-node upgrade and rollback lane.
#
# The weekly Monday schedule, version tags, and workflow_dispatch run this
# suite. Distributed upgrade cases stay on the nightly e2e-distributed lane.
name: Upgrade Compatibility
on:
pull_request:
paths:
- ".github/workflows/e2e-upgrade.yml"
- "crates/e2e_test/src/common.rs"
- "crates/e2e_test/src/distributed/**"
- "crates/e2e_test/src/fake_s3_target/**"
- "crates/e2e_test/src/lib.rs"
- "crates/e2e_test/src/replication_extension_test.rs"
- "crates/e2e_test/src/rpc_fault_proxy.rs"
- "crates/e2e_test/src/upgrade_compatibility_test.rs"
- "crates/ecstore/**"
- "crates/filemeta/**"
- "crates/heal/**"
- "crates/kms/**"
- "crates/storage-api/**"
- "rustfs/**"
- "Cargo.lock"
push:
tags:
- "[0-9]*.[0-9]*.[0-9]*"
schedule:
# 03:17 UTC every Monday.
- cron: "17 3 * * 1"
workflow_dispatch:
@@ -44,7 +33,7 @@ permissions:
concurrency:
group: ${{ github.workflow }}-${{ github.ref }}
cancel-in-progress: ${{ github.event_name == 'pull_request' }}
cancel-in-progress: false
env:
CARGO_TERM_COLOR: always
+51 -277
View File
@@ -1,229 +1,26 @@
# Copyright 2024 RustFS Team
#
# Licensed under the Apache License, Version 2.0 (the "License");
# you may not use this file except in compliance with the License.
# You may obtain a copy of the License at
#
# http://www.apache.org/licenses/LICENSE-2.0
#
# Unless required by applicable law or agreed to in writing, software
# distributed under the License is distributed on an "AS IS" BASIS,
# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
# See the License for the specific language governing permissions and
# limitations under the License.
name: Publish helm chart to artifacthub
# Copyright 2026 RustFS Team
# SPDX-License-Identifier: Apache-2.0
name: Build candidate Helm chart
on:
workflow_run:
workflows: [ "Build and Release" ]
types: [ completed ]
workflow_dispatch:
workflow_call:
inputs:
version:
description: "Release version to publish, e.g. 1.0.0-beta.1 or v1.0.0-beta.1"
required: true
default: "1.0.0-beta.1"
candidate_manifest:
type: string
required: true
permissions:
contents: read
actions: read
jobs:
# Consume the same trusted release certification as every other publication
# exit and verify the image promotion record of the Docker workflow. The
# chart wraps no binary of its own, so it may only reference an image whose
# platform verification already passed; it must not inherit an unverified
# package's conclusion. A run without that evidence fails closed here and
# never reaches the chart repository.
release-certification:
package:
runs-on: ubuntu-latest
timeout-minutes: 30
permissions:
contents: read
actions: read
outputs:
version: ${{ steps.version.outputs.raw_tag }}
timeout-minutes: 15
steps:
- name: Checkout repository
uses: actions/checkout@f548e57e544e1ff5a4c46bf1e1b8685f8e4a348a # v7
- uses: actions/checkout@f548e57e544e1ff5a4c46bf1e1b8685f8e4a348a # v7
with:
ref: ${{ fromJson(inputs.candidate_manifest).source_sha }}
persist-credentials: false
- name: Normalize release version
id: version
env:
RAW_INPUT: ${{ github.event.inputs.version }}
RAW_BRANCH: ${{ github.event.workflow_run.head_branch }}
run: |
set -euo pipefail
if [ "${{ github.event_name }}" = "workflow_dispatch" ]; then
RAW="$RAW_INPUT"
else
RAW="$RAW_BRANCH"
fi
case "$RAW" in
refs/tags/*) RAW_TAG="${RAW#refs/tags/}" ;;
*) RAW_TAG="$RAW" ;;
esac
echo "raw_tag=$RAW_TAG" >> "$GITHUB_OUTPUT"
- name: Download and verify release certification and image record
env:
GH_TOKEN: ${{ secrets.RELEASE_VALIDATION_TOKEN || github.token }}
TRIGGERING_RUN_ID: ${{ github.event.workflow_run.id }}
RAW_TAG: ${{ steps.version.outputs.raw_tag }}
shell: bash
run: |
set -euo pipefail
mkdir -p certification image-record
build_run="$TRIGGERING_RUN_ID"
if [[ -z "$build_run" ]]; then
build_run="$(gh api --method GET \
"repos/${GITHUB_REPOSITORY}/actions/workflows/build.yml/runs" \
-f branch="$RAW_TAG" -f status=success -F per_page=1 \
--jq '.workflow_runs[0].id // empty' || true)"
fi
if [[ -z "$build_run" ]]; then
echo "No certified Build and Release run found for ${RAW_TAG}; failing closed" >&2
exit 1
fi
if ! gh run download "$build_run" -n release-certification -D certification; then
echo "Build run ${build_run} published no release-certification artifact; failing closed" >&2
exit 1
fi
python3 scripts/release/release_gate.py verify-promotion \
--plan .config/release-validation-plan.json \
--candidate certification/candidate.json \
--summary certification/release-gate-output \
--artifact-sha256 "$(jq -r '.deliverables[0].sha256' certification/candidate.json)" \
--artifact-name "$(jq -r '.deliverables[0].name' certification/candidate.json)"
docker_run="$(gh api --method GET \
"repos/${GITHUB_REPOSITORY}/actions/workflows/docker.yml/runs" \
-f branch="$RAW_TAG" -f status=success -F per_page=1 \
--jq '.workflow_runs[0].id // empty' || true)"
if [[ -z "$docker_run" ]] || ! gh run download "$docker_run" -n image-promotion-record -D image-record; then
echo "No image promotion record for ${RAW_TAG}; a chart may not wrap an unverified image" >&2
exit 1
fi
candidate_sha="$(python3 -c "import json,hashlib,sys; sys.path.insert(0,'scripts/release'); import release_gate; print(release_gate.canonical_sha256(json.load(open('certification/candidate.json'))))")"
record_sha="$(jq -r '.candidate_sha256 // empty' image-record/image-promotion-record.json)"
if [[ "$record_sha" != "$candidate_sha" ]]; then
echo "Image promotion record does not belong to this certification" >&2
exit 1
fi
overlap="$(jq -r --slurpfile candidate certification/candidate.json \
'[.binary_sha256s[]] - ($candidate[0].deliverables | map(.binary_sha256)) | length' \
image-record/image-promotion-record.json)"
if [[ "$overlap" -ne 0 ]]; then
echo "Image promotion record carries binary digests outside the certification" >&2
exit 1
fi
build-helm-package:
runs-on: ubuntu-latest
timeout-minutes: 30
needs: [ release-certification ]
if: |
(github.event_name == 'workflow_dispatch' && !contains(github.event.inputs.version, '-preview')) ||
(
github.event.workflow_run.conclusion == 'success' &&
github.event.workflow_run.event == 'push' &&
contains(github.event.workflow_run.head_branch, '.') &&
!contains(github.event.workflow_run.head_branch, '-preview')
)
outputs:
raw_tag: ${{ steps.version.outputs.raw_tag }}
app_version: ${{ steps.version.outputs.app_version }}
chart_version: ${{ steps.version.outputs.chart_version }}
steps:
- name: Checkout release source
uses: actions/checkout@f548e57e544e1ff5a4c46bf1e1b8685f8e4a348a # v7
with:
ref: ${{ github.event.workflow_run.head_sha || (startsWith(inputs.version, 'refs/tags/') && inputs.version || format('refs/tags/{0}', inputs.version)) }}
persist-credentials: false
# Both inputs reach the shell through env rather than `${{ }}`
# interpolation. A git ref name may contain `$(...)` — anything without a
# space is a legal tag — and interpolation pastes it into the script
# verbatim, where bash would run it. Reading "$RAW_INPUT" instead makes it
# data.
- name: Normalize release version
id: version
env:
RAW_INPUT: ${{ github.event.inputs.version }}
RAW_BRANCH: ${{ github.event.workflow_run.head_branch }}
run: |
set -eux
if [ "${{ github.event_name }}" = "workflow_dispatch" ]; then
RAW="$RAW_INPUT"
else
RAW="$RAW_BRANCH"
fi
case "$RAW" in
refs/tags/*)
RAW_TAG="${RAW#refs/tags/}"
;;
*)
RAW_TAG="$RAW"
;;
esac
./scripts/helm_chart_version.sh "$RAW_TAG"
# Docker and Helm share the binary-build trigger. Do not publish a chart
# until the image selected by its default appVersion is pullable.
- name: Wait for release image
timeout-minutes: 20
env:
APP_VERSION: ${{ steps.version.outputs.app_version }}
run: |
set -euo pipefail
if [[ ! "$APP_VERSION" =~ ^[0-9]+\.[0-9]+\.[0-9]+(-[0-9A-Za-z.-]+)?$ ]] ||
[[ "$APP_VERSION" == *-preview* ]]; then
echo "Invalid deliverable image version: $APP_VERSION" >&2
exit 1
fi
for attempt in {1..60}; do
if token=$(curl --fail --silent --show-error --connect-timeout 10 --max-time 20 \
'https://auth.docker.io/token?service=registry.docker.io&scope=repository:rustfs/rustfs:pull' \
| jq -er '.token // empty') &&
curl --fail --silent --show-error --head --connect-timeout 10 --max-time 20 \
--header "Authorization: Bearer $token" \
--header 'Accept: application/vnd.oci.image.index.v1+json, application/vnd.docker.distribution.manifest.list.v2+json' \
"https://registry-1.docker.io/v2/rustfs/rustfs/manifests/$APP_VERSION" > /dev/null; then
echo "Release image is available: rustfs/rustfs:$APP_VERSION"
exit 0
fi
echo "Waiting for rustfs/rustfs:$APP_VERSION ($attempt/60)"
if (( attempt < 60 )); then
sleep 15
fi
done
echo "Release image is unavailable; refusing to publish Helm chart" >&2
exit 1
- name: Replace chart version and app version
env:
CHART_VERSION: ${{ steps.version.outputs.chart_version }}
APP_VERSION: ${{ steps.version.outputs.app_version }}
run: |
set -eux
sed -i -E "s/^version:.*/version: \"${CHART_VERSION}\"/" helm/rustfs/Chart.yaml
sed -i -E "s/^appVersion:.*/appVersion: \"${APP_VERSION}\"/" helm/rustfs/Chart.yaml
- name: Set up Helm
uses: azure/setup-helm@b9e51907a09c216f16ebe8536097933489208112 # v4.3.0
- name: Install yq
- uses: azure/setup-helm@b9e51907a09c216f16ebe8536097933489208112 # v4.3.0
- name: Install pinned yq
run: |
set -euo pipefail
yq_version="4.54.1"
@@ -238,66 +35,43 @@ jobs:
echo "$tools_dir" >> "$GITHUB_PATH"
"$tools_dir/yq" --version
- name: Test Helm Chart Templates
run: ./scripts/test_helm_templates.sh
- name: Package Helm Chart
run: |
set -eux
cp helm/README.md helm/rustfs/
helm package ./helm/rustfs \
--destination helm/rustfs/ \
--version "${{ steps.version.outputs.chart_version }}"
- name: Upload helm package as artifact
uses: actions/upload-artifact@043fb46d1a93c77aae656e7c1c64a875d1fc6a0a # v7.0.1
with:
name: helm-package
path: |
helm/rustfs/*.tgz
helm/rustfs/README.md
retention-days: 1
publish-helm-package:
runs-on: ubuntu-latest
timeout-minutes: 30
needs: [ build-helm-package ]
if: needs.build-helm-package.result == 'success'
steps:
- name: Checkout helm package repo
uses: actions/checkout@f548e57e544e1ff5a4c46bf1e1b8685f8e4a348a # v7
with:
# persist-credentials-exempt: this checkout's token IS the push credential —
# the job git-pushes to rustfs/helm below. Clearing it breaks chart publishing.
repository: rustfs/helm
token: ${{ secrets.RUSTFS_HELM_PACKAGE }}
- name: Download helm package
uses: actions/download-artifact@3e5f45b2cfb9172054b4087a40e8e0b5a5461e7c # v8.0.1
with:
name: helm-package
path: ./
- name: Set up helm
uses: azure/setup-helm@b9e51907a09c216f16ebe8536097933489208112 # v4.3.0
- name: Generate index
run: helm repo index . --url https://charts.rustfs.com
# app_version is derived from the triggering tag name, and this job holds
# the cross-repository push token with rustfs/helm already checked out —
# the worst place in the repo to paste an attacker-influenced string into
# a shell line. Passed through env so bash treats it as data.
- name: Push helm package and index file
- name: Test and package the exact application version
id: chart
env:
GIT_USERNAME: ${{ secrets.USERNAME }}
GIT_EMAIL: ${{ secrets.EMAIL_ADDRESS }}
APP_VERSION: ${{ needs.build-helm-package.outputs.app_version }}
GH_TOKEN: ${{ github.token }}
CANDIDATE_MANIFEST: ${{ inputs.candidate_manifest }}
run: |
set -eux
git config --global user.name "${GIT_USERNAME}"
git config --global user.email "${GIT_EMAIL}"
git add .
git commit -m "Update rustfs helm package with ${APP_VERSION}." || echo "No changes to commit"
git push origin main
python3 - <<'PYCHART'
import sys, subprocess, shutil
from pathlib import Path
sys.path.insert(0, 'scripts/release')
import candidate as c, pipeline
inputs = pipeline.manifest_env()
current = pipeline.current_run('release-complete.yml')
if any(a['name'] == 'helm-package' for a in c.artifact_list(current['id'])):
c.download_artifact(current, 'helm-package', 'chart')
record = c.read_json('chart/chart.json')
c.require(record['inputs_sha256'] == c.digest(inputs), 'chart belongs to another candidate')
c.emit_output('reused', 'true')
else:
chart = Path('helm/rustfs/Chart.yaml')
import re
value = chart.read_text()
for key in ('version', 'appVersion'):
value, count = re.subn(rf'(?m)^{key}:.*$', f'{key}: "{inputs["version"]}"', value)
c.require(count == 1, 'ambiguous chart metadata')
chart.write_text(value)
subprocess.run(['bash', 'scripts/test_helm_templates.sh'], check=True)
shutil.copyfile('helm/README.md', 'helm/rustfs/README.md')
subprocess.run(['helm', 'package', 'helm/rustfs', '--destination', 'chart'], check=True)
path = Path('chart') / f"rustfs-{inputs['version']}.tgz"
c.write_json('chart/chart.json', {'version': inputs['version'], 'inputs_sha256': c.digest(inputs),
'name': path.name, 'sha256': c.file_digest(path), 'size': path.stat().st_size})
PYCHART
- uses: actions/upload-artifact@043fb46d1a93c77aae656e7c1c64a875d1fc6a0a # v7.0.1
if: steps.chart.outputs.reused != 'true'
with:
name: helm-package
path: chart/*
if-no-files-found: error
retention-days: 30
+279 -7
View File
@@ -87,18 +87,206 @@ jobs:
install-build-packaging-tools: 'false'
install-test-tools: 'false'
- name: Download static console assets
shell: bash
env:
GITHUB_TOKEN: ${{ github.token }}
run: |
# Packaging tools are off on this lane, so setup does not install unzip.
if ! command -v unzip >/dev/null 2>&1; then
sudo apt-get update -qq
sudo apt-get install -y -qq unzip
fi
if ! command -v unzip >/dev/null 2>&1; then
echo "unzip is required to extract console assets" >&2
exit 1
fi
mkdir -p ./rustfs/static
verify_sha256() {
local expected="$1"
local file="$2"
local actual=""
local actual_line=""
if command -v sha256sum >/dev/null 2>&1; then
actual_line=$(sha256sum -- "$file") || return 1
elif command -v shasum >/dev/null 2>&1; then
actual_line=$(shasum -a 256 -- "$file") || return 1
else
echo "No SHA-256 verification tool found" >&2
return 1
fi
actual="${actual_line%%[[:space:]]*}"
if [ "$actual" = "$expected" ]; then
echo "SHA256 verified OK: $actual"
return 0
else
echo "SHA256 mismatch: expected=$expected actual=$actual" >&2
return 1
fi
}
download_console_assets() {
local curl_bin="curl"
local python_bin="python3"
local console_api="https://api.github.com/repos/rustfs/console/releases/latest"
local console_json="console-release.json"
local console_url
local console_sha256
local curl_auth_args=()
console_tag=""
chmod +w ./rustfs/static/LICENSE || true
if ! command -v "$python_bin" >/dev/null 2>&1; then
python_bin="python"
fi
if ! command -v "$python_bin" >/dev/null 2>&1; then
echo "No Python interpreter found for release metadata parsing" >&2
return 1
fi
if [[ -n "${GITHUB_TOKEN:-}" ]]; then
curl_auth_args=(-H "Authorization: Bearer ${GITHUB_TOKEN}" -H "X-GitHub-Api-Version: 2022-11-28")
fi
"$curl_bin" "${curl_auth_args[@]}" --fail -L "$console_api" \
-o "$console_json" --retry 3 --retry-delay 5 --max-time 300 || return 1
read -r console_tag console_url console_sha256 < <("$python_bin" - "$console_json" <<'PY'
import json
import re
import sys
with open(sys.argv[1], encoding="utf-8") as handle:
release = json.load(handle)
tag = release.get("tag_name", "") or "unknown"
for asset in release.get("assets", []):
name = asset.get("name", "")
digest = asset.get("digest", "")
url = asset.get("browser_download_url", "")
if name.startswith("rustfs-console-") and name.endswith(".zip") and digest.startswith("sha256:") and url:
sha256 = digest.split(":", 1)[1]
if not re.fullmatch(r"[0-9a-fA-F]{64}", sha256):
raise SystemExit(f"console zip asset has invalid sha256 digest: {sha256}")
sys.stdout.buffer.write(f"{tag} {url} {sha256}\n".encode("utf-8"))
break
else:
raise SystemExit("no console zip asset with sha256 digest found")
PY
) || return 1
if [[ -z "$console_url" || -z "$console_sha256" ]]; then
echo "Console release metadata is missing URL or SHA-256 digest" >&2
return 1
fi
echo "Console release: ${console_tag}"
echo "Downloading console asset: ${console_url}"
"$curl_bin" --fail -L "$console_url" -o console.zip --retry 3 --retry-delay 5 --max-time 300 || return 1
verify_sha256 "$console_sha256" console.zip || return 2
unzip -o console.zip -d ./rustfs/static || return 2
}
if download_console_assets; then
rm -f console.zip console-release.json
else
status=$?
rm -f console.zip console-release.json
if [[ "$status" -eq 2 ]]; then
echo "Console asset integrity verification failed" >&2
fi
echo "Failed to download verified console assets" >&2
exit 1
fi
if [[ ! -s ./rustfs/static/index.html ]]; then
echo "Console asset archive is missing static/index.html" >&2
exit 1
fi
asset_count=$(find ./rustfs/static -type f | wc -l | tr -d '[:space:]')
echo "Console assets ready: version=${console_tag:-unknown}, ${asset_count} files extracted to ./rustfs/static"
- name: Build RustFS
# Pin the glibc baseline (`<target>.<glibc>` for cargo-zigbuild) so
# nightlies run on older distros such as RHEL/Rocky 9 (glibc 2.34); #8062.
run: cargo zigbuild --release --locked --target x86_64-unknown-linux-gnu.2.31 -p rustfs --bins
shell: bash
run: |
# Force rebuild by touching build.rs so a restored cache cannot
# reuse a rustfs artifact compiled before these console assets existed.
touch rustfs/build.rs
cargo zigbuild --release --locked --target x86_64-unknown-linux-gnu.2.31 -p rustfs --bin rustfs
- name: Verify glibc symbol baseline
run: |
python3 scripts/release/check_glibc.py 2.31 \
target/x86_64-unknown-linux-gnu/release/rustfs \
target/x86_64-unknown-linux-gnu/release/rustfs-cli
target/x86_64-unknown-linux-gnu/release/rustfs
target/x86_64-unknown-linux-gnu/release/rustfs --version | grep -Fx 'build target : x86_64-unknown-linux-gnu'
- name: Verify console
shell: bash
run: |
set -euo pipefail
find_free_port() {
python3 - <<'PY'
import socket
with socket.socket() as sock:
sock.bind(("127.0.0.1", 0))
print(sock.getsockname()[1])
PY
}
binary="target/x86_64-unknown-linux-gnu/release/rustfs"
data_dir="$(mktemp -d)"
log_file="${RUNNER_TEMP}/rustfs-console-smoke.log"
server_pid=""
trap '
if [[ -n "${server_pid:-}" ]]; then
kill "$server_pid" 2>/dev/null || true
wait "$server_pid" 2>/dev/null || true
fi
rm -rf "$data_dir"
' EXIT
if [[ ! -x "$binary" ]]; then
echo "rustfs binary not found: ${binary}" >&2
exit 1
fi
api_port="$(find_free_port)"
console_port="$(find_free_port)"
console_url="http://127.0.0.1:${console_port}/rustfs/console/"
"$binary" server \
--address "127.0.0.1:${api_port}" \
--console-enable \
--console-address "127.0.0.1:${console_port}" \
--access-key console-smoke \
--secret-key console-smoke-secret \
"$data_dir" >"$log_file" 2>&1 &
server_pid=$!
response=""
for _ in {1..40}; do
response="$(curl --silent --output /dev/null --write-out '%{http_code} %{content_type}' "$console_url" || true)"
if [[ "$response" == 200\ text/html* ]]; then
exit 0
fi
sleep 0.25
done
echo "Console endpoint did not return 200 text/html: $response" >&2
cat "$log_file"
exit 1
- name: Build DEB package
id: deb
shell: bash
@@ -346,12 +534,12 @@ jobs:
path: ${{ steps.publish.outputs.candidate_file }}
if-no-files-found: error
# Publish the deb/rpm pair to the auto-testing repo's `assets` branch so
# engineers can download and install the nightly directly. The branch is
# a single-commit orphan rewritten on every build, which keeps the repo
# small while the latest files stay reachable at stable raw URLs.
# Publish the deb/rpm pair as assets of the rolling `nightly` release on
# rustfs/auto-testing (see scripts/release/publish_nightly_assets.sh).
# The dispatch below runs only after this upload and the immutable R2
# candidate both succeed. It does not replace rustfs-functional-chain.yml,
# which still starts from this workflow's workflow_run completion.
# kms-vault-lane and kms-vault-ha-failover stay parallel with this job.
- name: Publish packages to auto-testing release assets
env:
ASSETS_TOKEN: ${{ secrets.PF_TESTING_GH_TOKEN }}
@@ -360,6 +548,90 @@ jobs:
DEB_DATE: ${{ steps.deb.outputs.deb_date }}
BUILD_REF: ${{ env.NIGHTLY_BUILD_REF }}
run: bash scripts/release/publish_nightly_assets.sh
# Cross-repo repository_dispatch cannot use GITHUB_TOKEN. The secret is a
# PAT or GitHub App token for rustfs/auto-testing: classic `repo` scope,
# or fine-grained Contents write (the permission GitHub checks for
# POST /repos/{owner}/{repo}/dispatches). A dispatch failure fails this
# step, and therefore this job, so the scheduled failure alert fires.
# GitHub allows at most 10 top-level client_payload properties, so the
# schema 2 candidate object is nested under `candidate` and the release
# fields sit beside it.
- name: Dispatch nightly candidate to auto-testing
if: ${{ steps.publish.outputs.candidate_file != '' }}
env:
DISPATCH_TOKEN: ${{ secrets.AUTO_TESTING_DISPATCH_TOKEN }}
CANDIDATE_FILE: ${{ steps.publish.outputs.candidate_file }}
DEB_DATE: ${{ steps.deb.outputs.deb_date }}
SOURCE_REPO: ${{ github.repository }}
shell: bash
run: |
set -euo pipefail
: "${DISPATCH_TOKEN:?AUTO_TESTING_DISPATCH_TOKEN is required to dispatch rustfs/auto-testing}"
: "${CANDIDATE_FILE:?nightly candidate manifest is required}"
: "${DEB_DATE:?DEB_DATE is required}"
[[ "${DEB_DATE}" =~ ^[0-9]{4}-[0-9]{2}-[0-9]{2}$ ]]
[[ "${SOURCE_REPO}" == "rustfs/rustfs" ]]
payload="${RUNNER_TEMP}/nightly-candidate-dispatch-${GITHUB_RUN_ID}-${GITHUB_RUN_ATTEMPT}.json"
jq -n --slurpfile candidate "${CANDIDATE_FILE}" \
--arg event_type "nightly-candidate-ready" \
--arg source_repo "${SOURCE_REPO}" \
--arg latest_alias_url "https://dl.rustfs.com/artifacts/rustfs/packages/nightly/rustfs-nightly-latest.deb" \
--arg auto_testing_release "https://github.com/rustfs/auto-testing/releases/tag/nightly" \
--arg channel "nightly" \
--arg deb_date "${DEB_DATE}" \
--arg nightly_workflow ".github/workflows/nightly-gnu.yml" \
'{
event_type: $event_type,
source_repo: $source_repo,
latest_alias_url: $latest_alias_url,
auto_testing_release: $auto_testing_release,
channel: $channel,
deb_date: $deb_date,
nightly_workflow: $nightly_workflow,
candidate: $candidate[0]
}' > "${payload}"
if ! jq -e '
.event_type == "nightly-candidate-ready"
and .source_repo == "rustfs/rustfs"
and .channel == "nightly"
and .deb_date == env.DEB_DATE
and .nightly_workflow == ".github/workflows/nightly-gnu.yml"
and .latest_alias_url == "https://dl.rustfs.com/artifacts/rustfs/packages/nightly/rustfs-nightly-latest.deb"
and .auto_testing_release == "https://github.com/rustfs/auto-testing/releases/tag/nightly"
and .candidate.schema == 2
and ((.candidate | keys | sort) == ["build_run_attempt","build_run_id","package_sha256","package_url","schema","source_ref","source_sha","workflow_sha"])
and ((keys | sort) == ["auto_testing_release","candidate","channel","deb_date","event_type","latest_alias_url","nightly_workflow","source_repo"])
' "${payload}" >/dev/null; then
echo "nightly dispatch payload does not match the nightly-candidate-ready contract" >&2
exit 1
fi
body="${RUNNER_TEMP}/nightly-candidate-dispatch-body-${GITHUB_RUN_ID}-${GITHUB_RUN_ATTEMPT}.json"
jq -n --slurpfile client_payload "${payload}" \
'{event_type: "nightly-candidate-ready", client_payload: $client_payload[0]}' > "${body}"
response="${RUNNER_TEMP}/nightly-candidate-dispatch-response-${GITHUB_RUN_ID}-${GITHUB_RUN_ATTEMPT}.txt"
status="$(curl -sS --retry 3 --connect-timeout 15 --max-time 60 \
-o "${response}" -w '%{http_code}' \
-X POST \
-H "Authorization: Bearer ${DISPATCH_TOKEN}" \
-H "Accept: application/vnd.github+json" \
-H "X-GitHub-Api-Version: 2022-11-28" \
-H "Content-Type: application/json" \
--data-binary @"${body}" \
https://api.github.com/repos/rustfs/auto-testing/dispatches)" || {
echo "repository_dispatch request to rustfs/auto-testing failed before an HTTP status" >&2
exit 1
}
if [[ "${status}" != "204" ]]; then
echo "repository_dispatch to rustfs/auto-testing failed: HTTP ${status}" >&2
if [[ -s "${response}" ]]; then
cat "${response}" >&2
fi
exit 1
fi
echo "dispatched nightly-candidate-ready to rustfs/auto-testing (deb_date=${DEB_DATE}, run=${GITHUB_RUN_ID}, attempt=${GITHUB_RUN_ATTEMPT})"
# Live-Vault lane for the rustfs-kms suite (rustfs/backlog#1774).
#
# RUSTFS_KMS_VAULT_TOKEN is the single switch that adds the Vault KV2 and
+2 -2
View File
@@ -47,8 +47,8 @@ permissions:
contents: read
concurrency:
group: oidc-keycloak-live-${{ github.ref }}
cancel-in-progress: true
group: oidc-keycloak-live-${{ github.event_name }}-${{ github.ref }}
cancel-in-progress: ${{ github.event_name != 'schedule' }}
jobs:
oidc-keycloak-live:
+38 -825
View File
@@ -1,840 +1,53 @@
# Copyright 2024 RustFS Team
#
# Licensed under the Apache License, Version 2.0 (the "License");
# you may not use this file except in compliance with the License.
# You may obtain a copy of the License at
#
# http://www.apache.org/licenses/LICENSE-2.0
#
# Unless required by applicable law or agreed to in writing, software
# distributed under the License is distributed on an "AS IS" BASIS,
# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
# See the License for the specific language governing permissions and
# limitations under the License.
# Package Workflow - Build DEB/RPM packages
#
# This workflow builds DEB and RPM packages from pre-built Linux binaries
# and uploads them to Cloudflare R2 and the GitHub release.
#
# Trigger:
# - workflow_run: automatically package after "Build and Release" completes
# for a release tag (the mac/windows/linux binaries are already uploaded
# to the GitHub release before packaging starts)
# - workflow_dispatch: manual fallback with a release tag and/or exact build run ID
#
# Flow:
# 1. Resolve and validate the selected Build workflow run and source identity
# 2. Download Linux binaries (x86_64-gnu, aarch64-gnu, x86_64-musl,
# aarch64-musl) from build artifacts
# 3. Build DEB packages for amd64 and arm64, gnu and musl variants
# 4. Build RPM packages for x86_64 and aarch64, gnu and musl variants
# 5. Upload all packages to Cloudflare R2 and the GitHub release
name: Package DEB/RPM
permissions:
# contents: write is required to upload packages to the GitHub release
contents: write
actions: read
# Copyright 2026 RustFS Team
# SPDX-License-Identifier: Apache-2.0
name: Build candidate packages
on:
# Follows the same pattern as docker.yml: run after the release build
# workflow completes, so packaging is triggered only by release tags
# (e.g. 1.0.0-rc.2, 1.0.0-rc.3), never by development builds.
workflow_run:
workflows: [ "Build and Release" ]
types: [ completed ]
workflow_dispatch:
workflow_call:
inputs:
tag:
description: "Release tag to package (e.g. 1.0.0-beta.12). Leave empty for latest main build."
required: false
candidate_manifest:
type: string
build_run_id:
description: "Build workflow run ID (when combined with tag, both must identify the same release commit)"
required: false
type: string
concurrency:
group: ${{ github.workflow }}-${{ github.event.workflow_run.head_branch || github.event.inputs.tag || github.run_id }}
cancel-in-progress: true
env:
HEAD_BRANCH: ${{ github.event.workflow_run.head_branch }}
WORKFLOW_RUN_ID: ${{ github.event.workflow_run.id }}
required: true
permissions:
contents: read
actions: read
jobs:
# Resolve which build run to use and extract version info
resolve:
name: Resolve Build
# Auto-trigger only from successful tag builds of "Build and Release".
# Tag pushes arrive as event == push with head_branch != main (a
# non-main push head_branch is the release tag name). Manual dispatch
# stays available as a fallback for backfills and re-runs.
if: >-
github.event_name == 'workflow_dispatch' ||
(github.event.workflow_run.conclusion == 'success' &&
github.event.workflow_run.event == 'push' &&
github.event.workflow_run.head_branch != 'main')
runs-on: ubuntu-latest # gh CLI required; the sm-standard-2 fleet does not ship it
timeout-minutes: 10
outputs:
version: ${{ steps.resolve.outputs.version }}
build_type: ${{ steps.resolve.outputs.build_type }}
build_run_id: ${{ steps.resolve.outputs.build_run_id }}
build_run_number: ${{ steps.resolve.outputs.build_run_number }}
head_sha: ${{ steps.resolve.outputs.head_sha }}
dev_sequence: ${{ steps.resolve.outputs.dev_sequence }}
tag: ${{ steps.resolve.outputs.tag }}
steps:
- name: Resolve build run
id: resolve
shell: bash
env:
GH_TOKEN: ${{ github.token }}
EVENT_NAME: ${{ github.event_name }}
REPOSITORY: ${{ github.repository }}
INPUT_TAG: ${{ github.event.inputs.tag }}
INPUT_RUN_ID: ${{ github.event.inputs.build_run_id }}
run: |
set -euo pipefail
fail() {
echo "❌ $1" >&2
exit 1
}
TAG=""
BUILD_RUN_ID=""
case "$EVENT_NAME" in
workflow_run)
TAG="$HEAD_BRANCH"
BUILD_RUN_ID="$WORKFLOW_RUN_ID"
;;
workflow_dispatch)
TAG="$INPUT_TAG"
BUILD_RUN_ID="$INPUT_RUN_ID"
;;
*) fail "unsupported event: $EVENT_NAME" ;;
esac
# Validate and classify tags before using them in API paths or logs.
semver_core='(0|[1-9][0-9]*)\.(0|[1-9][0-9]*)\.(0|[1-9][0-9]*)'
prerelease_id='(alpha|beta|rc)\.(0|[1-9][0-9]*)'
if [[ -n "$TAG" ]]; then
if [[ "$TAG" =~ ^${semver_core}(-${prerelease_id})?-preview\.(0|[1-9][0-9]*)$ ]]; then
BUILD_TYPE=preview
elif [[ "$TAG" =~ ^${semver_core}-${prerelease_id}$ ]]; then
BUILD_TYPE=prerelease
elif [[ "$TAG" =~ ^${semver_core}$ ]]; then
BUILD_TYPE=release
else
fail "tag is not a supported strict package version"
fi
else
BUILD_TYPE=development
fi
if [[ -n "$BUILD_RUN_ID" ]]; then
[[ "$BUILD_RUN_ID" =~ ^[1-9][0-9]*$ ]] || fail "build run ID must be a positive decimal integer"
echo "Using selected build run: $BUILD_RUN_ID"
elif [[ -n "$TAG" ]]; then
echo "Looking for build run for tag: $TAG"
BUILD_RUN_ID=$(gh api --method GET \
"repos/${REPOSITORY}/actions/workflows/build.yml/runs" \
-f branch="$TAG" -f status=success -F per_page=1 \
--jq '.workflow_runs[0].id // empty' 2>/dev/null || true)
if [[ -z "$BUILD_RUN_ID" ]]; then
BUILD_RUN_ID=$(gh api --method GET \
"repos/${REPOSITORY}/actions/workflows/build.yml/runs" \
-f event=push -f status=success -F per_page=100 2>/dev/null |
jq -r --arg tag "$TAG" \
'[.workflow_runs[] | select(.head_branch == $tag)][0].id // empty' || true)
fi
[[ "$BUILD_RUN_ID" =~ ^[1-9][0-9]*$ ]] || fail "no successful build run found for tag"
echo "Found build run: $BUILD_RUN_ID"
else
echo "No tag specified, looking for latest main build"
BUILD_RUN_ID=$(gh api --method GET \
"repos/${REPOSITORY}/actions/workflows/build.yml/runs" \
-f branch=main -f status=success -F per_page=1 \
--jq '.workflow_runs[0].id // empty' 2>/dev/null || true)
[[ "$BUILD_RUN_ID" =~ ^[1-9][0-9]*$ ]] || fail "no successful main build found"
echo "Latest main build: $BUILD_RUN_ID"
fi
# Fetch once and use the same immutable run metadata for identity,
# ordering, workflow provenance, and release-channel validation.
RUN_JSON=$(gh api "repos/${REPOSITORY}/actions/runs/${BUILD_RUN_ID}") ||
fail "cannot read selected build run"
RUN_ID=$(jq -r '.id // empty' <<<"$RUN_JSON")
RUN_NUMBER=$(jq -r '.run_number // empty' <<<"$RUN_JSON")
RUN_STATUS=$(jq -r '.status // empty' <<<"$RUN_JSON")
RUN_CONCLUSION=$(jq -r '.conclusion // empty' <<<"$RUN_JSON")
RUN_PATH=$(jq -r '.path // empty' <<<"$RUN_JSON")
HEAD_SHA=$(jq -r '.head_sha // empty' <<<"$RUN_JSON")
RUN_HEAD_BRANCH=$(jq -r '.head_branch // empty' <<<"$RUN_JSON")
[[ "$RUN_ID" == "$BUILD_RUN_ID" ]] || fail "run metadata ID mismatch"
[[ "$RUN_NUMBER" =~ ^[1-9][0-9]*$ ]] || fail "build run number must be a positive decimal integer"
[[ "$RUN_STATUS" == completed && "$RUN_CONCLUSION" == success ]] || fail "selected build run is not successful"
[[ "$RUN_PATH" == .github/workflows/build.yml ]] || fail "selected run is not Build and Release"
[[ "$HEAD_SHA" =~ ^[0-9a-f]{40}$ ]] || fail "selected build run has an invalid head SHA"
[[ "$RUN_HEAD_BRANCH" != *$'\n'* && -n "$RUN_HEAD_BRANCH" ]] || fail "selected build run has an invalid head branch"
if [[ -n "$TAG" ]]; then
[[ "$RUN_HEAD_BRANCH" == "$TAG" ]] || fail "tag and build run head branch do not match"
TAG_REF_JSON=$(gh api "repos/${REPOSITORY}/git/ref/tags/${TAG}") ||
fail "cannot resolve release tag ref"
TAG_OBJECT_TYPE=$(jq -r '.object.type // empty' <<<"$TAG_REF_JSON")
TAG_OBJECT_SHA=$(jq -r '.object.sha // empty' <<<"$TAG_REF_JSON")
depth=0
while [[ "$TAG_OBJECT_TYPE" == tag && $depth -lt 5 ]]; do
TAG_OBJECT_JSON=$(gh api "repos/${REPOSITORY}/git/tags/${TAG_OBJECT_SHA}") ||
fail "cannot peel annotated release tag"
TAG_OBJECT_TYPE=$(jq -r '.object.type // empty' <<<"$TAG_OBJECT_JSON")
TAG_OBJECT_SHA=$(jq -r '.object.sha // empty' <<<"$TAG_OBJECT_JSON")
depth=$((depth + 1))
done
[[ "$TAG_OBJECT_TYPE" == commit && "$TAG_OBJECT_SHA" =~ ^[0-9a-f]{40}$ ]] ||
fail "release tag does not resolve to a commit"
[[ "$TAG_OBJECT_SHA" == "$HEAD_SHA" ]] || fail "release tag commit and build run head SHA do not match"
VERSION="$TAG"
DEV_SEQUENCE=""
else
VERSION="dev-${HEAD_SHA}"
DEV_SEQUENCE="$RUN_NUMBER"
fi
{
echo "version=$VERSION"
echo "build_type=$BUILD_TYPE"
echo "build_run_id=$BUILD_RUN_ID"
echo "build_run_number=$RUN_NUMBER"
echo "head_sha=$HEAD_SHA"
echo "dev_sequence=$DEV_SEQUENCE"
echo "tag=${TAG}"
} >> "$GITHUB_OUTPUT"
echo "📊 Resolved:"
echo " Version: $VERSION"
echo " Build type: $BUILD_TYPE"
echo " Build run ID: $BUILD_RUN_ID"
echo " Build run number: $RUN_NUMBER"
# Consume the trusted release certification of the resolved build run and
# verify the certified input digests before any package is wrapped. A run
# without that certification fails closed here; packages built later must
# complete their own verification and cannot inherit an unverified
# conclusion.
release-certification:
name: Release Certification
needs: resolve
if: needs.resolve.result == 'success'
runs-on: ubuntu-latest # gh CLI required; the sm-standard-2 fleet does not ship it
timeout-minutes: 30
permissions:
contents: read
actions: read
steps:
- name: Checkout repository
uses: actions/checkout@f548e57e544e1ff5a4c46bf1e1b8685f8e4a348a # v7
with:
persist-credentials: false
- name: Download and verify release certification
env:
GH_TOKEN: ${{ secrets.RELEASE_VALIDATION_TOKEN || github.token }}
BUILD_RUN_ID: ${{ needs.resolve.outputs.build_run_id }}
BUILD_TYPE: ${{ needs.resolve.outputs.build_type }}
shell: bash
run: |
set -euo pipefail
mkdir -p certification
if [[ "$BUILD_TYPE" == "development" ]]; then
# Development artifacts are explicitly uncertified and never touch
# official or latest channels; only the staging/dev paths remain.
jq -n '{schema_version: 1, certified: false, reason: "development artifacts are explicitly uncertified"}' \
> certification/development-uncertified.json
exit 0
fi
if ! gh run download "$BUILD_RUN_ID" -n release-certification -D certification; then
echo "Build run ${BUILD_RUN_ID} published no release-certification artifact; failing closed" >&2
exit 1
fi
python3 scripts/release/release_gate.py verify-promotion \
--plan .config/release-validation-plan.json \
--candidate certification/candidate.json \
--summary certification/release-gate-output \
--artifact-sha256 "$(jq -r '.deliverables[0].sha256' certification/candidate.json)" \
--artifact-name "$(jq -r '.deliverables[0].name' certification/candidate.json)"
# Build DEB and RPM packages for each architecture and libc variant
package:
name: Package (${{ matrix.arch }}-${{ matrix.libc }})
needs: [ resolve, release-certification ]
runs-on: ubuntu-latest # gh CLI required; the sm-standard-2 fleet does not ship it
timeout-minutes: 30
runs-on: ubuntu-latest
timeout-minutes: 45
strategy:
fail-fast: false
max-parallel: 1
matrix:
include:
- arch: x86_64
libc: gnu
deb_arch: amd64
rpm_arch: x86_64
artifact_name: "rustfs-linux-x86_64-gnu"
# gnu binaries dynamically link glibc; musl binaries are static
# and run without a libc dependency.
deb_depends: "libc6 (>= 2.31)"
rpm_depends: "glibc >= 2.31"
- arch: aarch64
libc: gnu
deb_arch: arm64
rpm_arch: aarch64
artifact_name: "rustfs-linux-aarch64-gnu"
deb_depends: "libc6 (>= 2.31)"
rpm_depends: "glibc >= 2.31"
- arch: x86_64
libc: musl
deb_arch: amd64
rpm_arch: x86_64
artifact_name: "rustfs-linux-x86_64-musl"
deb_depends: ""
rpm_depends: ""
- arch: aarch64
libc: musl
deb_arch: arm64
rpm_arch: aarch64
artifact_name: "rustfs-linux-aarch64-musl"
deb_depends: ""
rpm_depends: ""
target: [linux-x86_64-musl, linux-aarch64-musl, linux-x86_64-gnu, linux-aarch64-gnu]
steps:
- name: Checkout repository
uses: actions/checkout@f548e57e544e1ff5a4c46bf1e1b8685f8e4a348a # v7
- uses: actions/checkout@f548e57e544e1ff5a4c46bf1e1b8685f8e4a348a # v7
with:
ref: ${{ fromJson(inputs.candidate_manifest).source_sha }}
persist-credentials: false
- name: Normalize package metadata
id: versions
shell: bash
env:
BUILD_TYPE: ${{ needs.resolve.outputs.build_type }}
SOURCE_VERSION: ${{ needs.resolve.outputs.version }}
DEV_SEQUENCE: ${{ needs.resolve.outputs.dev_sequence }}
ARCH: ${{ matrix.rpm_arch }}
LIBC: ${{ matrix.libc }}
run: |
set -euo pipefail
normalized=$(./scripts/release/package_versions.sh \
"$BUILD_TYPE" "$SOURCE_VERSION" "$DEV_SEQUENCE" "$ARCH" "$LIBC")
printf '%s\n' "$normalized" >> "$GITHUB_OUTPUT"
- name: Download binary artifact from build run
uses: actions/download-artifact@3e5f45b2cfb9172054b4087a40e8e0b5a5461e7c # v8.0.1
with:
pattern: ${{ matrix.artifact_name }}*
path: ./binary-artifact
run-id: ${{ needs.resolve.outputs.build_run_id }}
github-token: ${{ github.token }}
merge-multiple: true
- name: Extract binary
id: binary
shell: bash
run: |
set -euo pipefail
ZIP_FILE=$(find ./binary-artifact -name "*.zip" -type f | head -1)
if [[ -z "$ZIP_FILE" ]]; then
echo "❌ No binary artifact found"
find ./binary-artifact -mindepth 1 -maxdepth 1 -print 2>/dev/null || true
exit 1
fi
echo "Found artifact: $ZIP_FILE"
mkdir -p ./bin
unzip -o "$ZIP_FILE" -d ./bin
if [[ ! -f ./bin/rustfs ]]; then
echo "❌ rustfs binary not found in archive"
exit 1
fi
chmod +x ./bin/rustfs
stat --printf='%n %s bytes\n' ./bin/rustfs
echo "✅ Binary extracted"
- name: Build DEB package
id: deb
shell: bash
env:
DEB_VERSION: ${{ steps.versions.outputs.deb_version }}
DEB_ARCH: ${{ matrix.deb_arch }}
DEB_FILE: ${{ steps.versions.outputs.deb_file }}
DEB_DEPENDS: ${{ matrix.deb_depends }}
LIBC: ${{ matrix.libc }}
run: |
set -euo pipefail
PKG_DIR="${DEB_FILE%.deb}"
echo "Building DEB: ${DEB_FILE}"
mkdir -p "${PKG_DIR}/DEBIAN"
mkdir -p "${PKG_DIR}/usr/bin"
mkdir -p "${PKG_DIR}/etc/default"
mkdir -p "${PKG_DIR}/lib/systemd/system"
mkdir -p "${PKG_DIR}/usr/share/doc/rustfs"
cp ./bin/rustfs "${PKG_DIR}/usr/bin/"
chmod 755 "${PKG_DIR}/usr/bin/rustfs"
cp deploy/build/rustfs.service "${PKG_DIR}/lib/systemd/system/"
cat > "${PKG_DIR}/etc/default/rustfs" << 'ENVEOF'
# RustFS Environment Configuration
# See https://docs.rustfs.com for more information
# RUSTFS_ACCESS_KEY=<changeme>
# RUSTFS_SECRET_KEY=<changeme>
# RUSTFS_VOLUMES=""
# RUSTFS_ADDRESS=":9000"
# RUSTFS_CONSOLE_ADDRESS=":9001"
# RUSTFS_CONSOLE_ENABLE=true
# RUSTFS_OBS_LOGGER_LEVEL=error
# RUSTFS_OBS_LOG_DIRECTORY="/var/log/rustfs/"
ENVEOF
cat > "${PKG_DIR}/DEBIAN/control" << EOF
Package: rustfs
Version: ${DEB_VERSION}
Section: utils
Priority: optional
Architecture: ${DEB_ARCH}
EOF
# gnu binaries dynamically link glibc and must declare it; musl
# binaries are statically linked and need no libc dependency.
if [[ -n "$DEB_DEPENDS" ]]; then
echo "Depends: ${DEB_DEPENDS}" >> "${PKG_DIR}/DEBIAN/control"
fi
cat >> "${PKG_DIR}/DEBIAN/control" << EOF
Maintainer: RustFS Team <support@rustfs.com>
Description: High-performance distributed object storage (${LIBC} build)
RustFS is a high-performance distributed object storage software
built using Rust. It is compatible with MinIO and S3 API.
Homepage: https://rustfs.com
EOF
# Declare /etc/default/rustfs as a conffile so dpkg preserves user
# modifications on upgrade instead of silently overwriting them.
cat > "${PKG_DIR}/DEBIAN/conffiles" << 'CONFFILES'
/etc/default/rustfs
CONFFILES
./scripts/release/package_service_scripts.sh after-install > "${PKG_DIR}/DEBIAN/postinst"
chmod 755 "${PKG_DIR}/DEBIAN/postinst"
./scripts/release/package_service_scripts.sh before-remove > "${PKG_DIR}/DEBIAN/prerm"
chmod 755 "${PKG_DIR}/DEBIAN/prerm"
./scripts/release/package_service_scripts.sh after-remove > "${PKG_DIR}/DEBIAN/postrm"
chmod 755 "${PKG_DIR}/DEBIAN/postrm"
cp LICENSE "${PKG_DIR}/usr/share/doc/rustfs/"
cp README.md "${PKG_DIR}/usr/share/doc/rustfs/"
fakeroot dpkg-deb --build "${PKG_DIR}" "$DEB_FILE"
[[ $(dpkg-deb -f "$DEB_FILE" Package) == rustfs ]]
[[ $(dpkg-deb -f "$DEB_FILE" Version) == "$DEB_VERSION" ]]
[[ $(dpkg-deb -f "$DEB_FILE" Architecture) == "$DEB_ARCH" ]]
dpkg-deb --fsys-tarfile "$DEB_FILE" | tar -tf - | grep -Fx './usr/bin/rustfs' >/dev/null
stat --printf='%n %s bytes\n' "$DEB_FILE"
echo "deb_file=$DEB_FILE" >> "$GITHUB_OUTPUT"
echo "✅ DEB built: $DEB_FILE"
- name: Build RPM package
id: rpm
shell: bash
env:
RPM_VERSION: ${{ steps.versions.outputs.rpm_version }}
RPM_RELEASE: ${{ steps.versions.outputs.rpm_release }}
RPM_ARCH: ${{ matrix.rpm_arch }}
RPM_FILE: ${{ steps.versions.outputs.rpm_file }}
RPM_DEPENDS: ${{ matrix.rpm_depends }}
LIBC: ${{ matrix.libc }}
run: |
set -euo pipefail
echo "Building RPM for ${RPM_ARCH} (${LIBC})"
sudo apt-get update && sudo apt-get install -y ruby ruby-dev build-essential rpm
sudo gem install fpm
./scripts/test_package_versions.sh --require-package-managers
# gnu binaries dynamically link glibc and must declare it; musl
# binaries are statically linked and need no libc dependency.
depends_args=()
if [[ -n "$RPM_DEPENDS" ]]; then
depends_args+=(--depends "$RPM_DEPENDS")
fi
# Create config file for fpm (DEB build creates it in its package dir structure,
# but fpm needs the file to exist before packaging)
mkdir -p ./tmp-pkg/etc/default
cat > ./tmp-pkg/etc/default/rustfs << 'ENVEOF'
# RustFS Environment Configuration
# See https://docs.rustfs.com for more information
# RUSTFS_ACCESS_KEY=<changeme>
# RUSTFS_SECRET_KEY=<changeme>
# RUSTFS_VOLUMES=""
# RUSTFS_ADDRESS=":9000"
# RUSTFS_CONSOLE_ADDRESS=":9001"
# RUSTFS_CONSOLE_ENABLE=true
# RUSTFS_OBS_LOGGER_LEVEL=error
# RUSTFS_OBS_LOG_DIRECTORY="/var/log/rustfs/"
ENVEOF
fpm -s dir -t rpm \
--name rustfs \
--version "$RPM_VERSION" \
--iteration "$RPM_RELEASE" \
--architecture "$RPM_ARCH" \
--package "$RPM_FILE" \
"${depends_args[@]}" \
--description "High-performance distributed object storage (${LIBC} build)" \
--maintainer "RustFS Team <support@rustfs.com>" \
--url "https://rustfs.com" \
--license "Apache-2.0" \
--before-install <(./scripts/release/package_service_scripts.sh rpm-before-install) \
--before-upgrade <(./scripts/release/package_service_scripts.sh rpm-before-upgrade) \
--after-install <(./scripts/release/package_service_scripts.sh after-install) \
--after-upgrade <(./scripts/release/package_service_scripts.sh after-install) \
--before-remove <(./scripts/release/package_service_scripts.sh before-remove) \
--after-remove <(./scripts/release/package_service_scripts.sh after-remove) \
--rpm-posttrans <(./scripts/release/package_service_scripts.sh rpm-posttrans) \
--config-files /etc/default/rustfs \
./bin/rustfs=/usr/bin/rustfs \
./tmp-pkg/etc/default/rustfs=/etc/default/rustfs \
deploy/build/rustfs.service=/lib/systemd/system/rustfs.service \
LICENSE=/usr/share/doc/rustfs/LICENSE \
README.md=/usr/share/doc/rustfs/README.md
if [[ ! -f "$RPM_FILE" ]]; then
echo "❌ RPM build failed"
exit 1
fi
RPM_METADATA=$(rpm -qp --qf '%{NAME}\n%{VERSION}\n%{RELEASE}\n%{ARCH}\n' "$RPM_FILE")
EXPECTED_METADATA=$(printf 'rustfs\n%s\n%s\n%s' "$RPM_VERSION" "$RPM_RELEASE" "$RPM_ARCH")
[[ "$RPM_METADATA" == "$EXPECTED_METADATA" ]]
rpm -qpl "$RPM_FILE" | grep -Fx '/usr/bin/rustfs' >/dev/null
stat --printf='%n %s bytes\n' "$RPM_FILE"
echo "rpm_file=$RPM_FILE" >> "$GITHUB_OUTPUT"
echo "✅ RPM built: $RPM_FILE"
- name: Download certification evidence
if: needs.resolve.outputs.build_type != 'development'
uses: actions/download-artifact@3e5f45b2cfb9172054b4087a40e8e0b5a5461e7c # v8.0.1
with:
name: release-certification
path: ./certification
# Later-built wrapper verification: the binary inside each package must
# match the certified binary digest for this platform before anything is
# uploaded, and the package itself stays on staging until promotion.
- name: Verify wrapped binary against certification
if: needs.resolve.outputs.build_type != 'development'
shell: bash
env:
DEB_FILE: ${{ steps.deb.outputs.deb_file }}
RPM_FILE: ${{ steps.rpm.outputs.rpm_file }}
ARTIFACT_NAME: ${{ matrix.artifact_name }}
run: |
set -euo pipefail
expected="$(jq -r --arg needle "$ARTIFACT_NAME" \
'[.deliverables[] | select(.name | contains($needle)) | .binary_sha256][0] // empty' \
certification/candidate.json)"
if [[ -z "$expected" ]]; then
echo "No certified binary digest recorded for ${ARTIFACT_NAME}" >&2
exit 1
fi
if ! command -v rpm2cpio >/dev/null 2>&1; then
sudo apt-get update && sudo apt-get install -y rpm2cpio
fi
if ! command -v cpio >/dev/null 2>&1; then
sudo apt-get update && sudo apt-get install -y cpio
fi
for package_file in "$DEB_FILE" "$RPM_FILE"; do
case "$package_file" in
*.deb)
dpkg-deb --fsys-tarfile "$package_file" | tar -xOf - ./usr/bin/rustfs > "$RUNNER_TEMP/wrapped-rustfs"
;;
*.rpm)
rpm2cpio "$package_file" | cpio -i --to-stdout ./usr/bin/rustfs > "$RUNNER_TEMP/wrapped-rustfs"
;;
esac
python3 scripts/release/release_gate.py verify-promotion \
--plan .config/release-validation-plan.json \
--candidate certification/candidate.json \
--summary certification/release-gate-output \
--artifact "$RUNNER_TEMP/wrapped-rustfs" \
--artifact-role binary
done
- name: Upload packages to artifacts
uses: actions/upload-artifact@043fb46d1a93c77aae656e7c1c64a875d1fc6a0a # v7.0.1
with:
name: packages-${{ matrix.arch }}-${{ matrix.libc }}
path: |
*.deb
*.rpm
retention-days: 30
- name: Upload packages to Cloudflare R2
if: needs.resolve.outputs.build_type != 'preview' && env.R2_ACCESS_KEY_ID != ''
env:
R2_ACCESS_KEY_ID: ${{ secrets.R2_ACCESS_KEY_ID }}
R2_SECRET_ACCESS_KEY: ${{ secrets.R2_SECRET_ACCESS_KEY }}
R2_ENDPOINT: ${{ secrets.R2_ENDPOINT }}
R2_BUCKET: ${{ secrets.R2_BUCKET }}
AWS_EC2_METADATA_DISABLED: true
BUILD_TYPE: ${{ needs.resolve.outputs.build_type }}
DEB_FILE: ${{ steps.deb.outputs.deb_file }}
RPM_FILE: ${{ steps.rpm.outputs.rpm_file }}
shell: bash
run: |
set -euo pipefail
if [[ -z "$R2_ACCESS_KEY_ID" || -z "$R2_SECRET_ACCESS_KEY" || -z "$R2_ENDPOINT" || -z "$R2_BUCKET" ]]; then
echo "⚠️ R2 credentials missing, skipping upload"
exit 0
fi
if ! command -v aws >/dev/null 2>&1; then
sudo apt-get update && sudo apt-get install -y awscli
fi
export AWS_ACCESS_KEY_ID="$R2_ACCESS_KEY_ID"
export AWS_SECRET_ACCESS_KEY="$R2_SECRET_ACCESS_KEY"
export AWS_DEFAULT_REGION="auto"
# Build-phase writes stay on staging for release types: the official
# packages/release and packages/latest paths are written only by the
# certification-gated promote-packages job, from these same staged
# objects. Development packages keep the uncertified dev channel.
if [[ "$BUILD_TYPE" == "development" ]]; then
R2_PREFIX="artifacts/rustfs/packages/dev"
else
R2_PREFIX="artifacts/rustfs/packages/staging/${GITHUB_RUN_ID}/${GITHUB_RUN_ATTEMPT}"
fi
R2_PATH="s3://${R2_BUCKET}/${R2_PREFIX}/"
echo "📤 Uploading to $R2_PATH"
for f in "$DEB_FILE" "$RPM_FILE"; do
if [[ -n "$f" && -f "$f" ]]; then
echo "Uploading: $f"
aws s3 cp "$f" "$R2_PATH" --endpoint-url "$R2_ENDPOINT" --only-show-errors
fi
done
echo "✅ Upload complete"
- name: Upload packages to GitHub Release
if: needs.resolve.outputs.tag != ''
- name: Reuse completed package set
id: reuse
env:
GH_TOKEN: ${{ github.token }}
TAG: ${{ needs.resolve.outputs.tag }}
DEB_FILE: ${{ steps.deb.outputs.deb_file }}
RPM_FILE: ${{ steps.rpm.outputs.rpm_file }}
shell: bash
CANDIDATE_MANIFEST: ${{ inputs.candidate_manifest }}
TARGET: ${{ matrix.target }}
run: python3 scripts/release/deliver.py reuse --kind packages --target "$TARGET"
- name: Install packaging tools
if: steps.reuse.outputs.reused != 'true'
run: |
set -euo pipefail
# Upload the packages, then refresh the release checksums so the new
# assets are covered, matching the binary release flow.
for f in "$DEB_FILE" "$RPM_FILE"; do
if [[ -n "$f" && -f "$f" ]]; then
echo "📤 Uploading $(basename "$f") to GitHub release ${TAG}..."
gh release upload "$TAG" "$f" --clobber
fi
done
CHECKSUM_DIR="$(mktemp -d)"
gh release download "$TAG" -p 'SHA256SUMS' -p 'SHA512SUMS' \
-D "$CHECKSUM_DIR" --clobber
for spec in "SHA256SUMS:sha256sum" "SHA512SUMS:sha512sum"; do
asset="${spec%%:*}"
checksum_cmd="${spec##*:}"
checksum_file="${CHECKSUM_DIR}/${asset}"
for f in "$DEB_FILE" "$RPM_FILE"; do
if [[ -n "$f" && -f "$f" ]]; then
base="$(basename "$f")"
# GitHub stores release asset names with '~' normalized to '.'.
# Current package file names embed the raw tag and contain no
# '~', but older releases may still ship such assets, so the
# checksum entries must reference the name as stored on the
# release.
github_base="${base//\~/.}"
# Remove any stale entry (both naming variants), then append
grep -Fv -- "$base" "$checksum_file" > "${checksum_file}.tmp" || true
grep -Fv -- "$github_base" "${checksum_file}.tmp" > "${checksum_file}.tmp2" || true
mv "${checksum_file}.tmp2" "$checksum_file"
digest=$("$checksum_cmd" -- "$f" | awk '{print $1}')
printf '%s %s\n' "$digest" "$github_base" >> "$checksum_file"
fi
done
echo "📤 Updating ${asset} for release ${TAG}..."
gh release upload "$TAG" "$checksum_file" --clobber
done
echo "✅ GitHub release assets updated"
# The only writer of the official packages/release and packages/latest R2
# paths. Promotion re-checks each package's wrapped binary against the
# passed certification list and copies the staged objects byte-for-byte:
# no rebuild-after-verify, no mutable latest resolution downstream.
promote-packages:
name: Promote Packages
needs: [ resolve, package, release-certification ]
# Target-channel lock for the official package paths.
concurrency:
group: release-channel-r2-packages
cancel-in-progress: false
if: needs.resolve.outputs.tag != '' && (needs.resolve.outputs.build_type == 'release' || needs.resolve.outputs.build_type == 'prerelease')
runs-on: ubuntu-latest # gh CLI required; the sm-standard-2 fleet does not ship it
timeout-minutes: 30
permissions:
contents: read
actions: read
env:
R2_ACCESS_KEY_ID: ${{ secrets.R2_ACCESS_KEY_ID }}
R2_SECRET_ACCESS_KEY: ${{ secrets.R2_SECRET_ACCESS_KEY }}
R2_ENDPOINT: ${{ secrets.R2_ENDPOINT }}
R2_BUCKET: ${{ secrets.R2_BUCKET }}
AWS_EC2_METADATA_DISABLED: true
steps:
- name: Checkout repository
uses: actions/checkout@f548e57e544e1ff5a4c46bf1e1b8685f8e4a348a # v7
with:
persist-credentials: false
- name: Download packages and certification
uses: actions/download-artifact@3e5f45b2cfb9172054b4087a40e8e0b5a5461e7c # v8.0.1
with:
path: ./packages
pattern: packages-*
merge-multiple: true
- name: Download certification evidence
uses: actions/download-artifact@3e5f45b2cfb9172054b4087a40e8e0b5a5461e7c # v8.0.1
with:
name: release-certification
path: ./certification
- name: Verify and promote staged packages
shell: bash
run: |
set -euo pipefail
if [[ -z "$R2_ACCESS_KEY_ID" || -z "$R2_SECRET_ACCESS_KEY" || -z "$R2_ENDPOINT" || -z "$R2_BUCKET" ]]; then
echo "❌ R2 credentials missing; refusing to promote to the official channel" >&2
exit 1
fi
STAGING_PREFIX="s3://${R2_BUCKET}/artifacts/rustfs/packages/staging/${GITHUB_RUN_ID}/${GITHUB_RUN_ATTEMPT}/"
RELEASE_PATH="s3://${R2_BUCKET}/artifacts/rustfs/packages/release/"
LATEST_PATH="s3://${R2_BUCKET}/artifacts/rustfs/packages/latest/"
if ! command -v aws >/dev/null 2>&1; then
sudo apt-get update && sudo apt-get install -y awscli
fi
export AWS_ACCESS_KEY_ID="$R2_ACCESS_KEY_ID"
export AWS_SECRET_ACCESS_KEY="$R2_SECRET_ACCESS_KEY"
export AWS_DEFAULT_REGION="auto"
if ! command -v rpm2cpio >/dev/null 2>&1; then
sudo apt-get install -y rpm2cpio
fi
if ! command -v cpio >/dev/null 2>&1; then
sudo apt-get install -y cpio
fi
promoted=0
for package_file in packages/*.deb packages/*.rpm; do
[[ -f "$package_file" ]] || continue
name="$(basename "$package_file")"
case "$package_file" in
*.deb)
dpkg-deb --fsys-tarfile "$package_file" | tar -xOf - ./usr/bin/rustfs > "$RUNNER_TEMP/wrapped-rustfs"
;;
*.rpm)
rpm2cpio "$package_file" | cpio -i --to-stdout ./usr/bin/rustfs > "$RUNNER_TEMP/wrapped-rustfs"
;;
esac
python3 scripts/release/release_gate.py verify-promotion \
--plan .config/release-validation-plan.json \
--candidate certification/candidate.json \
--summary certification/release-gate-output \
--artifact "$RUNNER_TEMP/wrapped-rustfs" \
--artifact-role binary
staged="$(mktemp)"
aws s3 cp "${STAGING_PREFIX}${name}" "$staged" --endpoint-url "$R2_ENDPOINT" --only-show-errors
staged_sha="$(sha256sum "$staged" | awk '{print $1}')"
built_sha="$(sha256sum "$package_file" | awk '{print $1}')"
if [[ "$staged_sha" != "$built_sha" ]]; then
echo "❌ Staged package ${name} does not match the verified bytes" >&2
exit 1
fi
aws s3 cp "$staged" "${RELEASE_PATH}${name}" --endpoint-url "$R2_ENDPOINT" --only-show-errors
aws s3 cp "$staged" "${LATEST_PATH}${name}" --endpoint-url "$R2_ENDPOINT" --only-show-errors
rm -f "$staged"
promoted=$((promoted + 1))
echo "✅ Promoted ${name} (${built_sha})"
done
if [[ "$promoted" -eq 0 ]]; then
echo "No staged packages found to promote" >&2
exit 1
fi
# Summary
summary:
name: Summary
needs: [ resolve, package ]
# Every completed Build and Release run triggers this workflow; when
# resolve declined it (not a release tag), there is nothing to summarise.
if: always() && needs.resolve.result != 'skipped'
runs-on: ubuntu-latest
timeout-minutes: 5
steps:
- name: Print summary
shell: bash
sudo apt-get update
sudo apt-get install -y ruby ruby-dev build-essential rpm fakeroot cpio
sudo gem install fpm --version 1.17.0 --no-document
- name: Package accepted executable and verify contents
if: steps.reuse.outputs.reused != 'true'
env:
SUMMARY_VERSION: ${{ needs.resolve.outputs.version }}
SUMMARY_BUILD_TYPE: ${{ needs.resolve.outputs.build_type }}
SUMMARY_BUILD_RUN_ID: ${{ needs.resolve.outputs.build_run_id }}
SUMMARY_PACKAGE_STATUS: ${{ needs.package.result }}
run: |
{
echo "## 📦 Package Summary"
echo ""
echo "| Item | Value |"
echo "|------|-------|"
echo "| Version | \`${SUMMARY_VERSION}\` |"
echo "| Build Type | ${SUMMARY_BUILD_TYPE} |"
echo "| Build Run | #${SUMMARY_BUILD_RUN_ID} |"
echo "| Package Status | ${SUMMARY_PACKAGE_STATUS} |"
} >> "$GITHUB_STEP_SUMMARY"
GH_TOKEN: ${{ github.token }}
CANDIDATE_MANIFEST: ${{ inputs.candidate_manifest }}
TARGET: ${{ matrix.target }}
run: python3 scripts/release/deliver.py package --target "$TARGET" --output delivery
- uses: actions/upload-artifact@043fb46d1a93c77aae656e7c1c64a875d1fc6a0a # v7.0.1
if: steps.reuse.outputs.reused != 'true'
with:
name: packages-${{ matrix.target }}
path: delivery/*
if-no-files-found: error
compression-level: 0
retention-days: 30
+133
View File
@@ -0,0 +1,133 @@
# Copyright 2026 RustFS Team
# SPDX-License-Identifier: Apache-2.0
name: Release candidate (Linux)
on:
workflow_dispatch:
inputs:
version:
description: Final product version (no preview suffix)
type: string
required: true
source_sha:
description: Full reviewed source commit
type: string
required: true
permissions:
contents: read
actions: read
concurrency:
group: release-candidate-${{ inputs.version }}
cancel-in-progress: false
jobs:
freeze:
if: github.repository == 'rustfs/rustfs' && github.ref == 'refs/heads/main'
runs-on: ubuntu-latest
timeout-minutes: 15
outputs:
manifest: ${{ steps.freeze.outputs.manifest }}
steps:
- uses: actions/checkout@f548e57e544e1ff5a4c46bf1e1b8685f8e4a348a # v7
with:
fetch-depth: 0
persist-credentials: false
- name: Freeze or recover original inputs
id: freeze
env:
GH_TOKEN: ${{ github.token }}
VERSION: ${{ inputs.version }}
SOURCE_SHA: ${{ inputs.source_sha }}
run: python3 scripts/release/pipeline.py prepare --version "$VERSION" --source "$SOURCE_SHA" --directory frozen
- uses: actions/upload-artifact@043fb46d1a93c77aae656e7c1c64a875d1fc6a0a # v7.0.1
if: steps.freeze.outputs.reused != 'true'
with:
name: release-inputs
path: frozen/inputs.json
if-no-files-found: error
retention-days: 30
linux:
needs: freeze
uses: ./.github/workflows/build.yml
with:
candidate_manifest: ${{ needs.freeze.outputs.manifest }}
platforms: linux
upgrade:
needs: [freeze, linux]
runs-on: ubuntu-latest
timeout-minutes: 150
env:
RUSTUP_TOOLCHAIN: ${{ fromJson(needs.freeze.outputs.manifest).toolchain }}
steps:
- uses: actions/checkout@f548e57e544e1ff5a4c46bf1e1b8685f8e4a348a # v7
with:
ref: ${{ fromJson(needs.freeze.outputs.manifest).workflow_sha }}
persist-credentials: false
- uses: ./.github/actions/setup
with:
rust-version: ${{ fromJson(needs.freeze.outputs.manifest).toolchain }}
cache-shared-key: release-upgrade-harness-hosted
cache-save-if: ${{ github.ref == 'refs/heads/main' }}
install-build-packaging-tools: "false"
install-test-tools: "false"
- name: Verify archived server against previous releases
env:
GH_TOKEN: ${{ github.token }}
CANDIDATE_MANIFEST: ${{ needs.freeze.outputs.manifest }}
run: |
python3 - <<'PYVERIFY'
import sys
from pathlib import Path
sys.path.insert(0, 'scripts/release')
import candidate as c, pipeline, verify
inputs = pipeline.manifest_env()
run = pipeline.current_run('release-candidate.yml')
if any(a['name'] == 'release-upgrade' for a in c.artifact_list(run['id'])):
c.download_artifact(run, 'release-upgrade', 'receipts')
pipeline.verify_evidence(inputs, Path('receipts'), {run['id']: run})
c.emit_output('reused', 'true')
else:
target = 'linux-x86_64-gnu'
c.download_artifact(run, f"candidate-{inputs['candidate_id']}-{target}", 'binary')
verify.upgrade(inputs, Path('binary') / f"rustfs-{target}-v{inputs['version']}.zip", Path('receipts'))
PYVERIFY
id: verify
- uses: actions/upload-artifact@043fb46d1a93c77aae656e7c1c64a875d1fc6a0a # v7.0.1
if: steps.verify.outputs.reused != 'true'
with:
name: release-upgrade
path: receipts/*
if-no-files-found: error
retention-days: 30
- uses: actions/upload-artifact@043fb46d1a93c77aae656e7c1c64a875d1fc6a0a # v7.0.1
if: failure()
with:
name: release-upgrade-failure-${{ github.run_attempt }}
path: receipts/*
retention-days: 7
seal-linux:
needs: [freeze, linux, upgrade]
runs-on: ubuntu-latest
timeout-minutes: 30
steps:
- uses: actions/checkout@f548e57e544e1ff5a4c46bf1e1b8685f8e4a348a # v7
with:
persist-credentials: false
- name: Collect trusted receipts and seal Linux bytes
id: seal
env:
GH_TOKEN: ${{ github.token }}
run: |
python3 scripts/release/pipeline.py linux-seal --directory sealed
{
echo '## Linux acceptance'
echo 'Use this run ID and manifest SHA-256 when starting Release candidate (complete).'
sha256sum sealed/linux-manifest.json
echo 'Record cluster, performance, integrity and upgrade acceptance in backlog before continuing.'
} >> "$GITHUB_STEP_SUMMARY"
- uses: actions/upload-artifact@043fb46d1a93c77aae656e7c1c64a875d1fc6a0a # v7.0.1
if: steps.seal.outputs.reused != 'true'
with:
name: release-linux
path: sealed/*
if-no-files-found: error
compression-level: 0
retention-days: 30
+97
View File
@@ -0,0 +1,97 @@
# Copyright 2026 RustFS Team
# SPDX-License-Identifier: Apache-2.0
name: Release candidate (complete)
on:
workflow_dispatch:
inputs:
candidate_run_id:
description: Successful Linux candidate run ID
type: string
required: true
linux_manifest_sha256:
description: Exact accepted Linux manifest SHA-256
type: string
required: true
acceptance_report:
description: Backlog report accepting cluster, performance, integrity and upgrade results
type: string
required: true
permissions:
contents: read
actions: read
concurrency:
group: release-complete-${{ inputs.candidate_run_id }}
cancel-in-progress: false
jobs:
acceptance:
if: github.repository == 'rustfs/rustfs' && github.ref == 'refs/heads/main'
runs-on: ubuntu-latest
timeout-minutes: 30
environment: release-linux-acceptance
outputs:
manifest: ${{ steps.accept.outputs.manifest }}
steps:
- uses: actions/checkout@f548e57e544e1ff5a4c46bf1e1b8685f8e4a348a # v7
with:
fetch-depth: 0
persist-credentials: false
- name: Bind acceptance to the exact Linux manifest
id: accept
env:
GH_TOKEN: ${{ github.token }}
CANDIDATE_RUN: ${{ inputs.candidate_run_id }}
MANIFEST_SHA: ${{ inputs.linux_manifest_sha256 }}
REPORT: ${{ inputs.acceptance_report }}
run: python3 scripts/release/pipeline.py complete-prepare --run-id "$CANDIDATE_RUN" --digest "$MANIFEST_SHA" --report "$REPORT" --directory linux
desktop:
needs: acceptance
uses: ./.github/workflows/build.yml
with:
candidate_manifest: ${{ needs.acceptance.outputs.manifest }}
platforms: desktop
packages:
needs: acceptance
uses: ./.github/workflows/package.yml
with:
candidate_manifest: ${{ needs.acceptance.outputs.manifest }}
images:
needs: acceptance
uses: ./.github/workflows/docker.yml
with:
candidate_manifest: ${{ needs.acceptance.outputs.manifest }}
chart:
needs: acceptance
uses: ./.github/workflows/helm-package.yml
with:
candidate_manifest: ${{ needs.acceptance.outputs.manifest }}
seal:
needs: [acceptance, desktop, packages, images, chart]
runs-on: ubuntu-latest
timeout-minutes: 45
steps:
- uses: actions/checkout@f548e57e544e1ff5a4c46bf1e1b8685f8e4a348a # v7
with:
fetch-depth: 0
persist-credentials: false
- name: Verify every deliverable and seal the candidate
id: seal
env:
GH_TOKEN: ${{ github.token }}
CANDIDATE_RUN: ${{ inputs.candidate_run_id }}
MANIFEST_SHA: ${{ inputs.linux_manifest_sha256 }}
REPORT: ${{ inputs.acceptance_report }}
run: |
python3 scripts/release/pipeline.py complete-seal --run-id "$CANDIDATE_RUN" --digest "$MANIFEST_SHA" --report "$REPORT" --directory sealed
{
echo '## Final publication approval'
echo 'Review this manifest, then explicitly dispatch Publish accepted release with its SHA-256.'
sha256sum sealed/release-manifest.json
} >> "$GITHUB_STEP_SUMMARY"
- uses: actions/upload-artifact@043fb46d1a93c77aae656e7c1c64a875d1fc6a0a # v7.0.1
if: steps.seal.outputs.reused != 'true'
with:
name: release-seal
path: sealed/*
if-no-files-found: error
compression-level: 0
retention-days: 30
+193
View File
@@ -0,0 +1,193 @@
# Copyright 2026 RustFS Team
# SPDX-License-Identifier: Apache-2.0
name: Publish accepted release
on:
workflow_dispatch:
inputs:
complete_run_id:
description: Successful completed candidate run ID
type: string
required: true
manifest_sha256:
description: Exact approved final manifest SHA-256 (authorizes publication)
type: string
required: true
permissions:
contents: read
actions: read
concurrency:
group: release-publish-${{ inputs.complete_run_id }}
cancel-in-progress: false
jobs:
verify:
if: github.repository == 'rustfs/rustfs' && github.ref == 'refs/heads/main'
runs-on: ubuntu-latest
timeout-minutes: 45
outputs:
version: ${{ steps.verify.outputs.version }}
steps:
- uses: actions/checkout@f548e57e544e1ff5a4c46bf1e1b8685f8e4a348a # v7
with:
fetch-depth: 0
persist-credentials: false
- name: Recheck approved bytes and trusted certification
id: verify
env:
GH_TOKEN: ${{ github.token }}
COMPLETE_RUN: ${{ inputs.complete_run_id }}
MANIFEST_SHA: ${{ inputs.manifest_sha256 }}
run: python3 scripts/release/publish.py verify --run-id "$COMPLETE_RUN" --digest "$MANIFEST_SHA" --directory sealed
version:
needs: verify
runs-on: ubuntu-latest
timeout-minutes: 90
concurrency:
group: release-version-${{ needs.verify.outputs.version }}
cancel-in-progress: false
environment: release-publication
permissions:
contents: write
actions: read
packages: write
security-events: write
env:
GH_TOKEN: ${{ github.token }}
COMPLETE_RUN: ${{ inputs.complete_run_id }}
MANIFEST_SHA: ${{ inputs.manifest_sha256 }}
AWS_ACCESS_KEY_ID: ${{ secrets.R2_ACCESS_KEY_ID }}
AWS_SECRET_ACCESS_KEY: ${{ secrets.R2_SECRET_ACCESS_KEY }}
AWS_DEFAULT_REGION: auto
AWS_REQUEST_CHECKSUM_CALCULATION: when_required
AWS_RESPONSE_CHECKSUM_VALIDATION: when_required
AWS_EC2_METADATA_DISABLED: "true"
R2_ENDPOINT: ${{ secrets.R2_ENDPOINT }}
R2_BUCKET: ${{ secrets.R2_BUCKET }}
steps:
- uses: actions/checkout@f548e57e544e1ff5a4c46bf1e1b8685f8e4a348a # v7
with:
fetch-depth: 0
persist-credentials: false
- name: Install promotion tools
run: |
sudo apt-get update
sudo apt-get install -y skopeo
aws --version
- uses: docker/login-action@c94ce9fb468520275223c153574b00df6fe4bcc9 # v3
with:
username: rustfs
password: ${{ secrets.DOCKERHUB_TOKEN }}
- uses: docker/login-action@c94ce9fb468520275223c153574b00df6fe4bcc9 # v3
with:
registry: ghcr.io
username: ${{ secrets.GHCR_USERNAME }}
password: ${{ secrets.GHCR_PASSWORD }}
- uses: docker/login-action@c94ce9fb468520275223c153574b00df6fe4bcc9 # v3
with:
registry: quay.io
username: ${{ secrets.QUAY_USERNAME }}
password: ${{ secrets.QUAY_PASSWORD }}
- name: Share registry credentials with the OCI copier
run: |
install -m 600 "$HOME/.docker/config.json" "$RUNNER_TEMP/registry-auth.json"
echo "REGISTRY_AUTH_FILE=$RUNNER_TEMP/registry-auth.json" >> "$GITHUB_ENV"
- name: Promote verified version
run: python3 scripts/release/publish.py version --run-id "$COMPLETE_RUN" --digest "$MANIFEST_SHA" --directory sealed
- name: Publish accepted musl amd64 scan
uses: github/codeql-action/upload-sarif@99df26d4f13ea111d4ec1a7dddef6063f76b97e9 # v4
with:
sarif_file: sealed/scan-musl-amd64.sarif
category: container-image-musl-amd64
- name: Publish accepted musl arm64 scan
uses: github/codeql-action/upload-sarif@99df26d4f13ea111d4ec1a7dddef6063f76b97e9 # v4
with:
sarif_file: sealed/scan-musl-arm64.sarif
category: container-image-musl-arm64
- name: Publish accepted GNU amd64 scan
uses: github/codeql-action/upload-sarif@99df26d4f13ea111d4ec1a7dddef6063f76b97e9 # v4
with:
sarif_file: sealed/scan-gnu-amd64.sarif
category: container-image-gnu-amd64
- name: Publish accepted GNU arm64 scan
uses: github/codeql-action/upload-sarif@99df26d4f13ea111d4ec1a7dddef6063f76b97e9 # v4
with:
sarif_file: sealed/scan-gnu-arm64.sarif
category: container-image-gnu-arm64
- uses: actions/upload-artifact@043fb46d1a93c77aae656e7c1c64a875d1fc6a0a # v7.0.1
with:
name: release-version-receipt-${{ github.run_attempt }}
path: publication.json
if-no-files-found: error
retention-days: 90
channels:
needs: [verify, version]
runs-on: ubuntu-latest
timeout-minutes: 90
concurrency:
group: rustfs-release-channels
cancel-in-progress: false
permissions:
contents: write
actions: read
packages: write
env:
GH_TOKEN: ${{ github.token }}
COMPLETE_RUN: ${{ inputs.complete_run_id }}
MANIFEST_SHA: ${{ inputs.manifest_sha256 }}
AWS_ACCESS_KEY_ID: ${{ secrets.R2_ACCESS_KEY_ID }}
AWS_SECRET_ACCESS_KEY: ${{ secrets.R2_SECRET_ACCESS_KEY }}
AWS_DEFAULT_REGION: auto
AWS_REQUEST_CHECKSUM_CALCULATION: when_required
AWS_RESPONSE_CHECKSUM_VALIDATION: when_required
AWS_EC2_METADATA_DISABLED: "true"
R2_ENDPOINT: ${{ secrets.R2_ENDPOINT }}
R2_BUCKET: ${{ secrets.R2_BUCKET }}
steps:
- uses: actions/checkout@f548e57e544e1ff5a4c46bf1e1b8685f8e4a348a # v7
with:
fetch-depth: 0
persist-credentials: false
- name: Install promotion tools
run: |
sudo apt-get update
sudo apt-get install -y skopeo
aws --version
- uses: docker/login-action@c94ce9fb468520275223c153574b00df6fe4bcc9 # v3
with:
username: rustfs
password: ${{ secrets.DOCKERHUB_TOKEN }}
- uses: docker/login-action@c94ce9fb468520275223c153574b00df6fe4bcc9 # v3
with:
registry: ghcr.io
username: ${{ secrets.GHCR_USERNAME }}
password: ${{ secrets.GHCR_PASSWORD }}
- uses: docker/login-action@c94ce9fb468520275223c153574b00df6fe4bcc9 # v3
with:
registry: quay.io
username: ${{ secrets.QUAY_USERNAME }}
password: ${{ secrets.QUAY_PASSWORD }}
- name: Share registry credentials with the OCI copier
run: |
install -m 600 "$HOME/.docker/config.json" "$RUNNER_TEMP/registry-auth.json"
echo "REGISTRY_AUTH_FILE=$RUNNER_TEMP/registry-auth.json" >> "$GITHUB_ENV"
- uses: azure/setup-helm@b9e51907a09c216f16ebe8536097933489208112 # v4.3.0
- name: Install pinned OSS client
run: |
curl --fail --location --retry 3 --output "$RUNNER_TEMP/ossutil.zip" https://gosspublic.alicdn.com/ossutil/v2/2.1.1/ossutil-2.1.1-linux-amd64.zip
printf '%s %s\n' 60f3a808cbbdfd4b5269b8f967c6a66f564c9dacff52d93a5d21a588976ab5a2 "$RUNNER_TEMP/ossutil.zip" | sha256sum --check -
unzip -q "$RUNNER_TEMP/ossutil.zip" -d "$RUNNER_TEMP/ossutil-extract"
install -m 755 "$RUNNER_TEMP/ossutil-extract/ossutil-2.1.1-linux-amd64/ossutil" "$RUNNER_TEMP/ossutil"
echo "OSSUTIL_BIN=$RUNNER_TEMP/ossutil" >> "$GITHUB_ENV"
- name: Promote verified channels
env:
HELM_TOKEN: ${{ secrets.RUSTFS_HELM_PACKAGE }}
OSS_ACCESS_KEY_ID: ${{ secrets.ALICLOUDOSS_KEY_ID }}
OSS_ACCESS_KEY_SECRET: ${{ secrets.ALICLOUDOSS_KEY_SECRET }}
OSS_REGION: cn-beijing
OSS_ENDPOINT: https://oss-cn-beijing.aliyuncs.com
run: python3 scripts/release/publish.py channels --run-id "$COMPLETE_RUN" --digest "$MANIFEST_SHA" --directory sealed
- uses: actions/upload-artifact@043fb46d1a93c77aae656e7c1c64a875d1fc6a0a # v7.0.1
with:
name: release-channels-receipt-${{ github.run_attempt }}
path: publication.json
if-no-files-found: error
retention-days: 90
@@ -26,7 +26,13 @@ permissions:
jobs:
alert-on-incomplete-run:
name: Alert on incomplete scheduled run
if: github.repository == 'rustfs/rustfs'
# The registry still owns responsibility and cadence selection. Successful
# and PR runs can never produce an incomplete scheduled-run alert.
if: >-
github.repository == 'rustfs/rustfs' &&
github.event.workflow_run.conclusion != 'success' &&
(github.event.workflow_run.event == 'schedule' ||
github.event.workflow_run.event == 'workflow_run')
# Stay on GitHub-hosted runners: this monitor must keep running when the
# self-hosted sm-standard-* pool is down, which is exactly when the
# scheduled validations it watches stall.
+26
View File
@@ -107,6 +107,32 @@ jobs:
shell: pwsh
run: cargo test -p rustfs-ecstore --lib multipart_owned_record_parent_barriers_ -- --nocapture
# Native multipart completion publishes its payload from a different
# parent than its metadata, and abort renames under a precondition; both
# need their own Windows handle discipline.
- name: Test native multipart complete and abort publication
shell: pwsh
run: >-
cargo test -p rustfs-ecstore --lib --
native_abort_missing_state_with_actual_owned_records_never_becomes_legacy_or_missing
native_abort_pending_intent_is_not_a_delete_certificate_and_raw_cleanup_preserves_exact_evidence
--nocapture
# Owned-undo reclaim removes a directory it pinned, copy heal persists
# fault bytes through a handle, and metadata reads meet a directory.
- name: Test native multipart reclaim, copy heal, and directory reads
shell: pwsh
run: >-
cargo test -p rustfs-ecstore --lib --
native_multipart_owned_undo
native_multipart_copy_heal
orphan_multipart_cleanup_rejects_metadata_directory_on_one_disk
--nocapture
- name: Test directory removal fallback
shell: pwsh
run: "cargo test -p rustfs-ecstore --lib disk::fs::tests::test_remove -- --nocapture"
- name: Test Windows path conversion
shell: pwsh
run: "cargo test -p rustfs-utils --no-default-features --features path --lib dunce:: -- --nocapture"
+1
View File
@@ -26,6 +26,7 @@ and this project adheres to [Semantic Versioning](https://semver.org/spec/v2.0.0
- IAM snapshot/deletion replay, target-assigned delete-marker purges, timestamp ordering, and best-effort peer broadcast now close the control-plane gaps found by the R6 review ([#7195](https://github.com/rustfs/rustfs/pull/7195)).
- A `set-remote-target?update=true&creds=true` request on a site-replication peer's bucket target is now rejected with `400 InvalidRequest`, naming the target and why it counts as peer-owned. It used to return success while keeping the stored endpoint, credentials, and TLS settings, so an endpoint edit from the Console's bucket replication form appeared to apply but did not. Change a peer's address with a site-replication edit instead; see "Changing a site's address" in `docs/operations/site-replication-operations.md`. The `sync`, `bandwidth`, `proxy`, and `path` update groups are unchanged.
- Site-replication edits that omit the site name, such as `mc admin replicate update --deployment-id <ID> --endpoint <URL>`, now keep the stored name on the editing site and on every receiver instead of renaming the site after the new endpoint host. No persisted format changes.
- The persisted replication MRF journal is now replayed every 5 minutes while the server runs, not only at startup. Delete-marker purges whose targets were unreachable, and replications that overflowed the worker queues, now converge within one replay interval after the target returns instead of waiting for a restart. A target that stays down is re-attempted once per retained entry per interval across the cluster: the start time of the last pass is shared through a small `config/replication/mrf.bin.replay` object, so nodes do not each replay once per interval. Older versions ignore that object; existing persisted formats are unchanged.
- Upgrade and rollback: upgrade every node in one site consecutively and verify reconciliation before moving to the next site; do not intentionally run a site mixed-version. Target-version ledger keys are harmless on rollback, although old code cannot use their routing. Before rolling back past [#7307](https://github.com/rustfs/rustfs/pull/7307), drain or repair every pending version purge: older code can free a retained version's data directory before its remote purge is acknowledged. See `docs/operations/site-replication-operations.md`.
### Security
Generated
+361 -237
View File
File diff suppressed because it is too large Load Diff
+66 -63
View File
@@ -75,7 +75,7 @@ edition = "2024"
license = "Apache-2.0"
repository = "https://github.com/rustfs/rustfs"
rust-version = "1.99.0"
version = "1.0.2"
version = "1.0.5"
homepage = "https://rustfs.com"
description = "RustFS is a high-performance distributed object storage software built using Rust, one of the most popular languages worldwide. "
keywords = ["RustFS", "Minio", "object-storage", "filesystem", "s3"]
@@ -92,64 +92,64 @@ redundant_clone = "warn"
[workspace.dependencies]
# RustFS Internal Crates
rustfs = { path = "./rustfs", version = "1.0.2" }
rustfs-heal = { path = "crates/heal", version = "1.0.2" }
rustfs-heal-contracts = { path = "crates/heal-contracts", version = "1.0.2" }
rustfs-scanner-metrics = { path = "crates/scanner-metrics", version = "1.0.2" }
rustfs-audit = { path = "crates/audit", version = "1.0.2" }
rustfs-checksums = { path = "crates/checksums", version = "1.0.2" }
rustfs-common = { path = "crates/common", version = "1.0.2" }
rustfs-data-usage = { path = "crates/data-usage", version = "1.0.2" }
rustfs-config = { path = "./crates/config", version = "1.0.2" }
rustfs-concurrency = { path = "./crates/concurrency", version = "1.0.2" }
rustfs-credentials = { path = "crates/credentials", version = "1.0.2" }
rustfs-crypto = { path = "crates/crypto", version = "1.0.2" }
rustfs-ecstore = { path = "crates/ecstore", version = "1.0.2" }
rustfs-filemeta = { path = "crates/filemeta", version = "1.0.2" }
rustfs-iam = { path = "crates/iam", version = "1.0.2" }
rustfs-keystone = { path = "crates/keystone", version = "1.0.2" }
rustfs-license = { path = "crates/license", version = "1.0.2" }
rustfs-lifecycle = { path = "crates/lifecycle", version = "1.0.2" }
rustfs-kms = { path = "crates/kms", version = "1.0.2" }
rustfs-lock = { path = "crates/lock", version = "1.0.2" }
rustfs-madmin = { path = "crates/madmin", version = "1.0.2" }
rustfs-notify = { path = "crates/notify", version = "1.0.2" }
rustfs-io-metrics = { path = "crates/io-metrics", version = "1.0.2" }
rustfs-io-core = { path = "crates/io-core", version = "1.0.2" }
rustfs-object-capacity = { path = "crates/object-capacity", version = "1.0.2" }
rustfs-object-data-cache = { path = "crates/object-data-cache", version = "1.0.2", default-features = false }
rustfs-log-analyzer = { path = "crates/log-analyzer", version = "1.0.2" }
rustfs-obs = { path = "crates/obs", version = "1.0.2" }
rustfs-policy = { path = "crates/policy", version = "1.0.2" }
rustfs-protos = { path = "crates/protos", version = "1.0.2" }
rustfs-protocols = { path = "crates/protocols", version = "1.0.2" }
rustfs-replication = { path = "crates/replication", version = "1.0.2" }
rustfs-rio = { path = "crates/rio", version = "1.0.2" }
rustfs-rio-v2 = { path = "crates/rio-v2", version = "1.0.2" }
rustfs-s3-client = { path = "crates/s3-client", version = "1.0.2" }
rustfs-s3-types = { path = "crates/s3-types", version = "1.0.2" }
rustfs-s3-ops = { path = "crates/s3-ops", version = "1.0.2" }
rustfs-s3select-api = { path = "crates/s3select-api", version = "1.0.2" }
rustfs-s3select-query = { path = "crates/s3select-query", version = "1.0.2" }
rustfs-scanner = { path = "crates/scanner", version = "1.0.2" }
rustfs-security-governance = { path = "crates/security-governance", version = "1.0.2" }
rustfs-extension-schema = { path = "crates/extension-schema", version = "1.0.2" }
rustfs-signer = { path = "crates/signer", version = "1.0.2" }
rustfs-storage-api = { path = "crates/storage-api", version = "1.0.2" }
rustfs-trusted-proxies = { path = "crates/trusted-proxies", version = "1.0.2" }
rustfs-targets = { path = "crates/targets", version = "1.0.2" }
rustfs-test-utils = { path = "crates/test-utils", version = "1.0.2" }
rustfs-tls-runtime = { path = "crates/tls-runtime", version = "1.0.2" }
rustfs-utils = { path = "crates/utils", version = "1.0.2" }
rustfs-zip = { path = "./crates/zip", version = "1.0.2" }
rustfs = { path = "./rustfs", version = "1.0.5" }
rustfs-heal = { path = "crates/heal", version = "1.0.5" }
rustfs-heal-contracts = { path = "crates/heal-contracts", version = "1.0.5" }
rustfs-scanner-metrics = { path = "crates/scanner-metrics", version = "1.0.5" }
rustfs-audit = { path = "crates/audit", version = "1.0.5" }
rustfs-checksums = { path = "crates/checksums", version = "1.0.5" }
rustfs-common = { path = "crates/common", version = "1.0.5" }
rustfs-data-usage = { path = "crates/data-usage", version = "1.0.5" }
rustfs-config = { path = "./crates/config", version = "1.0.5" }
rustfs-concurrency = { path = "./crates/concurrency", version = "1.0.5" }
rustfs-credentials = { path = "crates/credentials", version = "1.0.5" }
rustfs-crypto = { path = "crates/crypto", version = "1.0.5" }
rustfs-ecstore = { path = "crates/ecstore", version = "1.0.5" }
rustfs-filemeta = { path = "crates/filemeta", version = "1.0.5" }
rustfs-iam = { path = "crates/iam", version = "1.0.5" }
rustfs-keystone = { path = "crates/keystone", version = "1.0.5" }
rustfs-license = { path = "crates/license", version = "1.0.5" }
rustfs-lifecycle = { path = "crates/lifecycle", version = "1.0.5" }
rustfs-kms = { path = "crates/kms", version = "1.0.5" }
rustfs-lock = { path = "crates/lock", version = "1.0.5" }
rustfs-madmin = { path = "crates/madmin", version = "1.0.5" }
rustfs-notify = { path = "crates/notify", version = "1.0.5" }
rustfs-io-metrics = { path = "crates/io-metrics", version = "1.0.5" }
rustfs-io-core = { path = "crates/io-core", version = "1.0.5" }
rustfs-object-capacity = { path = "crates/object-capacity", version = "1.0.5" }
rustfs-object-data-cache = { path = "crates/object-data-cache", version = "1.0.5", default-features = false }
rustfs-log-analyzer = { path = "crates/log-analyzer", version = "1.0.5" }
rustfs-obs = { path = "crates/obs", version = "1.0.5" }
rustfs-policy = { path = "crates/policy", version = "1.0.5" }
rustfs-protos = { path = "crates/protos", version = "1.0.5" }
rustfs-protocols = { path = "crates/protocols", version = "1.0.5" }
rustfs-replication = { path = "crates/replication", version = "1.0.5" }
rustfs-rio = { path = "crates/rio", version = "1.0.5" }
rustfs-rio-v2 = { path = "crates/rio-v2", version = "1.0.5" }
rustfs-s3-client = { path = "crates/s3-client", version = "1.0.5" }
rustfs-s3-types = { path = "crates/s3-types", version = "1.0.5" }
rustfs-s3-ops = { path = "crates/s3-ops", version = "1.0.5" }
rustfs-s3select-api = { path = "crates/s3select-api", version = "1.0.5" }
rustfs-s3select-query = { path = "crates/s3select-query", version = "1.0.5" }
rustfs-scanner = { path = "crates/scanner", version = "1.0.5" }
rustfs-security-governance = { path = "crates/security-governance", version = "1.0.5" }
rustfs-extension-schema = { path = "crates/extension-schema", version = "1.0.5" }
rustfs-signer = { path = "crates/signer", version = "1.0.5" }
rustfs-storage-api = { path = "crates/storage-api", version = "1.0.5" }
rustfs-trusted-proxies = { path = "crates/trusted-proxies", version = "1.0.5" }
rustfs-targets = { path = "crates/targets", version = "1.0.5" }
rustfs-test-utils = { path = "crates/test-utils", version = "1.0.5" }
rustfs-tls-runtime = { path = "crates/tls-runtime", version = "1.0.5" }
rustfs-utils = { path = "crates/utils", version = "1.0.5" }
rustfs-zip = { path = "./crates/zip", version = "1.0.5" }
# Async Runtime and Networking
async-channel = "2.5.0"
async_zip = { default-features = false, version = "0.0.19" }
mysql_async = { default-features = false, version = "0.37.1" }
async-compression = { version = "0.4.51" }
async-compression = { version = "0.4.52" }
async-trait = "0.1.92"
async-nats = { version = "0.51.0", default-features = false }
async-nats = { version = "0.52.0", default-features = false }
axum = "0.8.9"
futures = "0.3.34"
futures-core = "0.3.34"
@@ -168,7 +168,7 @@ http-body-util = "0.1.5"
memchr = "2.8.3"
minlz = "1.2.3"
reqwest = "0.13.5"
rustfs-kafka-async = { version = "1.4.0" }
rustfs-kafka-async = { version = "1.4.1" }
socket2 = { version = "0.6.5" }
tokio = { version = "1.54.1" }
tokio-rustls = { default-features = false, version = "0.26.6" }
@@ -191,6 +191,7 @@ quinn = { version = "0.11.12", default-features = false, features = [
# Serialization and Data Formats
apache-avro = { version = "0.22.0", features = ["snappy", "zstandard"] }
bytes = { version = "1.12.1" }
csv-core = "0.1.13"
bytesize = "2.7.0"
byteorder = "1.5.0"
flatbuffers = "25.12.19"
@@ -212,6 +213,7 @@ serde_urlencoded = "0.7.1"
# releases.
aes-gcm = { version = "0.11.1" }
argon2 = { version = "0.6.0" }
aws-lc-rs = { version = "1.19.0", default-features = false }
blake2 = "0.11.0"
chacha20poly1305 = { version = "0.11.0" }
crc-fast = "1.10.0"
@@ -234,7 +236,7 @@ proptest = "1"
# Time and Date
chrono = { version = "0.4.45" }
humantime = "2.4.0"
jiff = { version = "0.2.38" }
jiff = { version = "0.2.39" }
time = { version = "0.3.55" }
# Database
@@ -247,8 +249,8 @@ anyhow = "1.0.104"
arc-swap = "1.9.3"
rustfs-tokio-tar = { version = "0.7.1" }
# Candidate Snowball parser versions exercised by rustfs-zip compatibility fixtures.
tar-codec = "0.0.15"
tar-framing = "0.0.15"
tar-codec = "0.0.16"
tar-framing = "0.0.16"
atoi = "3.1.1"
atomic_enum = "0.3.0"
aws-config = { version = "1.13.0" }
@@ -294,6 +296,7 @@ local-ip-address = "0.6.13"
log = "0.4"
memmap2 = "0.9.12"
lz4 = "1.28.1"
lz4_flex = { version = "=0.14.0", default-features = false, features = ["std", "frame"] }
matchit = "0.9.2"
md-5 = "0.11.0"
mime_guess = "2.0.5"
@@ -337,7 +340,7 @@ strum = { version = "0.28.0" }
sysinfo = "0.40.1"
temp-env = "0.3.6"
tempfile = "3.27.0"
test-case = "3.4.0"
test-case = "3.4.1"
thiserror = "2.0.21"
tracing = { version = "0.1.44" }
tracing-appender = "0.2.5"
@@ -356,13 +359,13 @@ winapi-util = "0.1.11"
windows = { version = "0.62.2" }
windows-sys = "0.61.2"
xxhash-rust = { version = "0.8.19" }
zip = "9.2.1"
zip = "9.3.0"
zstd = "0.14.3"
# Observability and Metrics
metrics = "0.24.6"
metrics-util = "0.20"
dial9 = { version = "0.5.6", default-features = false }
dial9 = { version = "0.5.7", default-features = false }
opentelemetry = { version = "0.33.1" }
opentelemetry-appender-tracing = { version = "0.33.1" }
opentelemetry-otlp = { version = "0.33.1" }
@@ -377,16 +380,16 @@ libunftp = { version = "0.24.0" }
unftp-core = "0.1.0"
suppaftp = { version = "12.2.1" }
rcgen = { version = "0.14.10", default-features = false, features = ["aws_lc_rs", "crypto", "pem"] }
russh = { version = "0.65.0" }
russh = { version = "0.65.1" }
russh-sftp = "3.0.1"
# WebDAV
dav-server = "0.11.0"
# Performance Analysis and Memory Profiling
rustfs-mimalloc = { version = "0.7.2" }
rustfs-mimalloc = { version = "0.7.3" }
# Preserve Unicode focus filters until rustfs/backlog#2302 is resolved.
hotpath = { version = "0.30.3", default-features = false }
hotpath = { version = "0.31.1", default-features = false }
# Snapshot testing for output format regression detection
insta = { version = "1.49" }
+22 -5
View File
@@ -12,10 +12,13 @@
# See the License for the specific language governing permissions and
# limitations under the License.
FROM alpine:3.24.2 AS build
ARG BINARY_SOURCE=download
FROM alpine:3.24.2 AS download
ARG TARGETARCH
ARG RELEASE=latest
ARG RC_RELEASE=latest
RUN apk add --no-cache ca-certificates curl jq unzip
WORKDIR /build
@@ -69,6 +72,17 @@ RUN set -eux; \
chmod +x /build/rustfs; \
rm -rf rustfs.zip /build/.tmp || true
COPY scripts/download_rc.sh /usr/local/bin/download_rc.sh
RUN sh /usr/local/bin/download_rc.sh "$TARGETARCH" musl "$RC_RELEASE" /build/rc
# Candidate packaging supplies the accepted executables from verified archives.
# The default download stage remains available for local image builds.
FROM scratch AS supplied
ARG TARGETARCH
COPY .release-inputs/${TARGETARCH}/ /build/
FROM ${BINARY_SOURCE} AS build
FROM alpine:3.24.2
@@ -95,14 +109,17 @@ RUN apk upgrade --no-cache && \
ca-certificates \
coreutils \
curl \
jq \
tzdata \
&& test "$(TZ=Asia/Kolkata date +%z)" = "+0530"
COPY --from=build /etc/ssl/certs/ca-certificates.crt /etc/ssl/certs/
COPY --from=build /build/rustfs /usr/bin/rustfs
COPY entrypoint.sh /entrypoint.sh
COPY --from=build /build/rc /usr/bin/rc
COPY entrypoint.sh /etc/rustfs/entrypoint.sh
COPY bootstrap/ /etc/rustfs/bootstrap/
RUN chmod +x /usr/bin/rustfs /entrypoint.sh
RUN chmod +x /usr/bin/rustfs /usr/bin/rc /etc/rustfs/entrypoint.sh && \
ln -s /etc/rustfs/entrypoint.sh /entrypoint.sh
RUN addgroup -g 10001 -S rustfs && \
adduser -u 10001 -G rustfs -S rustfs -D && \
@@ -124,6 +141,6 @@ VOLUME ["/data"]
USER rustfs
ENTRYPOINT ["/entrypoint.sh"]
ENTRYPOINT ["/etc/rustfs/entrypoint.sh"]
CMD ["rustfs"]
+13 -4
View File
@@ -12,7 +12,10 @@
# See the License for the specific language governing permissions and
# limitations under the License.
FROM rust:1.97.1-trixie
FROM rust:1.99.1-trixie
ARG TARGETARCH
ARG RC_RELEASE=latest
RUN set -eux; \
export DEBIAN_FRONTEND=noninteractive; \
@@ -21,6 +24,7 @@ RUN set -eux; \
build-essential \
ca-certificates \
curl \
jq \
git \
pkg-config \
libssl-dev \
@@ -36,6 +40,7 @@ COPY . .
RUN ./scripts/static.sh
RUN cargo run --bin gproto
RUN cargo build --release --locked --bin rustfs
RUN sh /usr/src/rustfs/scripts/download_rc.sh "$TARGETARCH" musl "$RC_RELEASE" /usr/local/bin/rc
RUN set -eux; \
groupadd -g 10001 rustfs; \
@@ -50,8 +55,12 @@ RUN set -eux; \
chown -R rustfs:rustfs /data /logs /app; \
chmod 0750 /data /logs
COPY entrypoint.sh /entrypoint.sh
RUN install -m 0755 /usr/src/rustfs/target/release/rustfs /usr/bin/rustfs && chmod +x /entrypoint.sh
COPY entrypoint.sh /etc/rustfs/entrypoint.sh
COPY bootstrap/ /etc/rustfs/bootstrap/
RUN install -m 0755 /usr/src/rustfs/target/release/rustfs /usr/bin/rustfs && \
install -m 0755 /usr/local/bin/rc /usr/bin/rc && \
chmod +x /etc/rustfs/entrypoint.sh && \
ln -s /etc/rustfs/entrypoint.sh /entrypoint.sh
ENV RUSTFS_VOLUMES="/data" \
RUST_LOG="warn" \
@@ -63,5 +72,5 @@ ENV RUSTFS_VOLUMES="/data" \
EXPOSE 9000 9001
ENTRYPOINT ["/entrypoint.sh"]
ENTRYPOINT ["/etc/rustfs/entrypoint.sh"]
CMD ["/usr/bin/rustfs"]
+22 -4
View File
@@ -12,10 +12,13 @@
# See the License for the specific language governing permissions and
# limitations under the License.
FROM ubuntu:26.04 AS build
ARG BINARY_SOURCE=download
FROM ubuntu:26.04 AS download
ARG TARGETARCH
ARG RELEASE=latest
ARG RC_RELEASE=latest
RUN apt-get update && apt-get install -y --no-install-recommends \
ca-certificates \
@@ -76,6 +79,17 @@ RUN set -eux; \
chmod +x /build/rustfs; \
rm -rf rustfs.zip /build/.tmp || true
COPY scripts/download_rc.sh /usr/local/bin/download_rc.sh
RUN sh /usr/local/bin/download_rc.sh "$TARGETARCH" gnu "$RC_RELEASE" /build/rc
# Candidate packaging supplies the accepted executables from verified archives.
# The default download stage remains available for local image builds.
FROM scratch AS supplied
ARG TARGETARCH
COPY .release-inputs/${TARGETARCH}/ /build/
FROM ${BINARY_SOURCE} AS build
FROM ubuntu:26.04
ARG RELEASE=latest
@@ -99,14 +113,18 @@ RUN apt-get update && apt-get upgrade -y \
&& DEBIAN_FRONTEND=noninteractive apt-get install -y --no-install-recommends \
ca-certificates \
curl \
jq \
tzdata \
&& test "$(TZ=Asia/Kolkata date +%z)" = "+0530" \
&& rm -rf /var/lib/apt/lists/*
COPY --from=build /build/rustfs /usr/bin/rustfs
COPY entrypoint.sh /entrypoint.sh
COPY --from=build /build/rc /usr/bin/rc
COPY entrypoint.sh /etc/rustfs/entrypoint.sh
COPY bootstrap/ /etc/rustfs/bootstrap/
RUN chmod +x /usr/bin/rustfs /entrypoint.sh
RUN chmod +x /usr/bin/rustfs /usr/bin/rc /etc/rustfs/entrypoint.sh && \
ln -s /etc/rustfs/entrypoint.sh /entrypoint.sh
RUN groupadd -g 10001 rustfs && \
useradd -u 10001 -g rustfs -m -s /sbin/nologin rustfs && \
@@ -131,6 +149,6 @@ VOLUME ["/data"]
USER rustfs
ENTRYPOINT ["/entrypoint.sh"]
ENTRYPOINT ["/etc/rustfs/entrypoint.sh"]
CMD ["rustfs"]
+19 -7
View File
@@ -28,17 +28,19 @@ ARG TARGETPLATFORM
ARG BUILDPLATFORM
ARG TARGETARCH
ARG RUSTFS_BUILD_FEATURES=""
ARG RC_RELEASE=latest
# -----------------------------
# Build stage
# -----------------------------
FROM rust:1.97.1-trixie AS builder
FROM rust:1.99.1-trixie AS builder
# Re-declare args after FROM
ARG TARGETPLATFORM
ARG BUILDPLATFORM
ARG TARGETARCH
ARG RUSTFS_BUILD_FEATURES
ARG RC_RELEASE
# Debug: print platforms
RUN echo "Build info -> BUILDPLATFORM=${BUILDPLATFORM}, TARGETPLATFORM=${TARGETPLATFORM}"
@@ -52,6 +54,7 @@ RUN set -eux; \
build-essential \
ca-certificates \
curl \
jq \
git \
pkg-config \
libssl-dev \
@@ -166,6 +169,8 @@ RUN --mount=type=cache,target=/usr/local/cargo/registry \
;; \
esac
RUN sh /usr/src/rustfs/scripts/download_rc.sh "$TARGETARCH" musl "$RC_RELEASE" /usr/local/bin/rc
# -----------------------------
# Development stage (keeps toolchain)
# -----------------------------
@@ -199,10 +204,13 @@ ENV RUSTFS_VOLUMES="/data" \
# We rely on cargo run to build and run
EXPOSE 9000 9001
COPY entrypoint.sh /entrypoint.sh
RUN chmod +x /entrypoint.sh
COPY entrypoint.sh /etc/rustfs/entrypoint.sh
COPY bootstrap/ /etc/rustfs/bootstrap/
RUN install -m 0755 /usr/local/bin/rc /usr/bin/rc && \
chmod +x /etc/rustfs/entrypoint.sh && \
ln -s /etc/rustfs/entrypoint.sh /entrypoint.sh
ENTRYPOINT ["/entrypoint.sh"]
ENTRYPOINT ["/etc/rustfs/entrypoint.sh"]
CMD ["cargo", "run", "--bin", "rustfs", "--"]
# -----------------------------
@@ -227,6 +235,7 @@ RUN set -eux; \
apt-get install -y --no-install-recommends \
ca-certificates \
curl \
jq \
tzdata \
coreutils; \
rm -rf /var/lib/apt/lists/*
@@ -248,8 +257,11 @@ RUN set -eux; \
# Copy the freshly built binary and the entrypoint
COPY --from=builder /usr/local/bin/rustfs /usr/bin/rustfs
COPY entrypoint.sh /entrypoint.sh
RUN chmod +x /usr/bin/rustfs /entrypoint.sh
COPY --from=builder /usr/local/bin/rc /usr/bin/rc
COPY entrypoint.sh /etc/rustfs/entrypoint.sh
COPY bootstrap/ /etc/rustfs/bootstrap/
RUN chmod +x /usr/bin/rustfs /usr/bin/rc /etc/rustfs/entrypoint.sh && \
ln -s /etc/rustfs/entrypoint.sh /entrypoint.sh
# Default environment (override in docker run/compose as needed)
ENV RUSTFS_VOLUMES="/data" \
@@ -263,5 +275,5 @@ EXPOSE 9000
VOLUME ["/data"]
# Keep root here; entrypoint will drop privileges using chroot --userspec
ENTRYPOINT ["/entrypoint.sh"]
ENTRYPOINT ["/etc/rustfs/entrypoint.sh"]
CMD ["/usr/bin/rustfs"]
+3 -3
View File
@@ -145,7 +145,7 @@ chown -R 10001:10001 data logs
docker run -d -p 9000:9000 -p 9001:9001 -v $(pwd)/data:/data -v $(pwd)/logs:/logs rustfs/rustfs:latest
# Using specific version
docker run -d -p 9000:9000 -p 9001:9001 -v $(pwd)/data:/data -v $(pwd)/logs:/logs rustfs/rustfs:1.0.4
docker run -d -p 9000:9000 -p 9001:9001 -v $(pwd)/data:/data -v $(pwd)/logs:/logs rustfs/rustfs:1.0.5
```
If you use [podman](https://github.com/containers/podman) instead of docker, you can install the RustFS with the below command
@@ -224,7 +224,7 @@ For developers who want to build RustFS Docker images from source with multi-arc
./docker-buildx.sh --push
# Build specific version
./docker-buildx.sh --release 1.0.4 --push
./docker-buildx.sh --release 1.0.5 --push
# Build for custom registry
./docker-buildx.sh --registry your-registry.com --namespace yourname --push
@@ -242,7 +242,7 @@ You can also use Make targets for convenience:
```bash
make docker-buildx # Build locally
make docker-buildx-push # Build and push
make docker-buildx-version VERSION=1.0.4 # Build specific version
make docker-buildx-version VERSION=1.0.5 # Build specific version
make help-docker # Show all Docker-related commands
```
+3 -3
View File
@@ -138,7 +138,7 @@ chown -R 10001:10001 data logs
docker run -d -p 9000:9000 -p 9001:9001 -v $(pwd)/data:/data -v $(pwd)/logs:/logs rustfs/rustfs:latest
# 使用指定版本运行
docker run -d -p 9000:9000 -p 9001:9001 -v $(pwd)/data:/data -v $(pwd)/logs:/logs rustfs/rustfs:1.0.4
docker run -d -p 9000:9000 -p 9001:9001 -v $(pwd)/data:/data -v $(pwd)/logs:/logs rustfs/rustfs:1.0.5
```
如果您通过绑定挂载启用 TLS 证书目录,也请用同样方式准备该目录:
@@ -176,7 +176,7 @@ docker compose -f docker-compose-simple.yml up -d
./docker-buildx.sh --push
# 构建指定版本
./docker-buildx.sh --release 1.0.4 --push
./docker-buildx.sh --release 1.0.5 --push
# 构建并推送到自定义仓库
./docker-buildx.sh --registry your-registry.com --namespace yourname --push
@@ -193,7 +193,7 @@ docker compose -f docker-compose-simple.yml up -d
```bash
make docker-buildx # 本地构建
make docker-buildx-push # 构建并推送
make docker-buildx-version VERSION=1.0.4 # 构建指定版本
make docker-buildx-version VERSION=1.0.5 # 构建指定版本
make help-docker # 显示所有 Docker 相关命令
```
+21
View File
@@ -0,0 +1,21 @@
# RustFS image bootstrap
The official image bundles this directory at `/etc/rustfs/bootstrap`. The default `buckets.json` is an empty JSON array, so the image starts RustFS directly and does not require a bootstrap endpoint or bootstrap credentials.
To create buckets, replace `buckets.json` in a derived image with a JSON array of bucket names. The entrypoint creates each bucket if it does not exist and applies `public-access-block.json`, whose default blocks all four public access paths. Edit that file in the derived image only when public access is an explicit requirement.
Optional per-bucket files use the bucket name as the filename without an extension:
| Directory | File format | Applied operation |
| --- | --- | --- |
| `policies/` | S3 bucket policy JSON | `rc bucket anonymous set-json` |
| `versioning/` | JSON boolean (`true` or `false`) | Enable or suspend bucket versioning |
| `lifecycle/` | `rc bucket lifecycle rule import` JSON | Replace lifecycle rules |
| `cors/` | JSON or XML accepted by `rc bucket cors set` | Replace CORS rules |
| `notifications/` | JSON array of `{ "arn": "...", "events": ["..."] }` | Add or replace event rules by ARN (prefix/suffix filters are not currently accepted) |
Every per-bucket file must correspond to a bucket listed in `buckets.json`. Applying these files is repeatable: bucket creation ignores existing buckets, while each configuration operation replaces or reconciles that bucket's requested state.
When `buckets.json` is non-empty, set `RUSTFS_BOOTSTRAP_ENDPOINT` to the RustFS S3 endpoint reachable from the container, for example `http://rustfs:9000` for a Compose service or the internal service DNS name in Kubernetes. The entrypoint waits on that endpoint's `/health/ready`; it never assumes `localhost`. `RUSTFS_BOOTSTRAP_TIMEOUT` controls the readiness wait in seconds (default `120`, maximum `3600`). `RUSTFS_BOOTSTRAP_DIR` can point to a different in-image configuration directory; it defaults to `/etc/rustfs/bootstrap`. Set `RUSTFS_ACCESS_KEY` and `RUSTFS_SECRET_KEY`, or their `_FILE` variants, for bootstrap authentication. The endpoint and credentials are read at runtime and are not baked into the image.
RustFS built-in IAM policies remain unchanged; the default bootstrap installs no custom bucket policies, lifecycle rules, versioning, CORS rules, or notification rules. The bundled RustFS CLI is resolved from a published release and its release checksum is verified. Set the Docker build argument `RC_RELEASE` to a version to pin it. Build the RustFS image after a CLI release containing the `bucket public-access-block` commands is available. For Kubernetes, the configured service must route traffic to the starting pod before its readiness probe succeeds (for example, with `publishNotReadyAddresses: true`); otherwise the service endpoint cannot reach the pod while the entrypoint waits for readiness.
+1
View File
@@ -0,0 +1 @@
[]
+1
View File
@@ -0,0 +1 @@
+1
View File
@@ -0,0 +1 @@
+1
View File
@@ -0,0 +1 @@
+1
View File
@@ -0,0 +1 @@
+6
View File
@@ -0,0 +1,6 @@
{
"block_public_acls": true,
"ignore_public_acls": true,
"block_public_policy": true,
"restrict_public_buckets": true
}
+1
View File
@@ -0,0 +1 @@
+100
View File
@@ -902,4 +902,104 @@ mod tests {
assert_eq!(raw.len(), 8);
assert_eq!(&raw[..], reference.digest().to_be_bytes().as_slice());
}
#[test]
fn checksums_streaming_chunks_and_header_encoding_match_oneshot_across_unaligned_boundaries() {
use crate::http::HttpChecksum;
let all = [
ChecksumAlgorithm::Crc32,
ChecksumAlgorithm::Crc32c,
ChecksumAlgorithm::Crc64Nvme,
ChecksumAlgorithm::Sha1,
ChecksumAlgorithm::Sha256,
ChecksumAlgorithm::Sha512,
ChecksumAlgorithm::Xxhash3,
ChecksumAlgorithm::Xxhash64,
ChecksumAlgorithm::Xxhash128,
];
for &payload_len in &[0usize, 1, 31, 33, 1024, 4096 + 37, 64 * 1024 + 19] {
let payload: Vec<u8> = (0..payload_len).map(|i| ((i * 19 + 5) % 253) as u8).collect();
for algorithm in all {
let mut one_shot = algorithm.into_impl();
one_shot.update(&payload);
let expected_raw = one_shot.finalize();
assert_eq!(expected_raw.len(), algorithm.raw_len());
let mut one_shot_hdr = algorithm.into_impl();
one_shot_hdr.update(&payload);
let expected_headers = one_shot_hdr.headers();
for &chunk_size in &[1usize, 31, 1021, 4096] {
if payload_len > 4096 && chunk_size == 1 {
continue;
}
let mut streaming: Box<dyn HttpChecksum> = algorithm.into_impl();
for chunk in payload.chunks(chunk_size) {
streaming.update(chunk);
}
let actual_headers = streaming.headers();
assert_eq!(
actual_headers.get(algorithm.http_header_name()),
expected_headers.get(algorithm.http_header_name()),
"{algorithm:?} streaming mismatch at payload_len={payload_len}, chunk_size={chunk_size}"
);
}
}
}
}
#[test]
#[ignore = "utility throughput microbenchmark; run via cargo test -p rustfs-checksums --lib --release -- --ignored --nocapture"]
fn microbench_checksums_streaming_chunks_and_header_encoding() {
use crate::http::HttpChecksum;
use std::hint::black_box;
use std::time::Instant;
let all = [
ChecksumAlgorithm::Crc32,
ChecksumAlgorithm::Crc32c,
ChecksumAlgorithm::Crc64Nvme,
ChecksumAlgorithm::Sha1,
ChecksumAlgorithm::Sha256,
ChecksumAlgorithm::Sha512,
ChecksumAlgorithm::Xxhash3,
ChecksumAlgorithm::Xxhash64,
ChecksumAlgorithm::Xxhash128,
];
let payload: Vec<u8> = (0..(256 * 1024)).map(|i| ((i * 19 + 5) % 253) as u8).collect();
for &chunk_size in &[4 * 1024usize, 64 * 1024, 256 * 1024] {
for algorithm in all {
let mut one_shot = algorithm.into_impl();
one_shot.update(&payload);
let expected_headers = one_shot.headers();
let iters = 8usize;
let t0 = Instant::now();
let mut last_headers = http::HeaderMap::new();
for _ in 0..iters {
let mut streaming: Box<dyn HttpChecksum> = algorithm.into_impl();
for chunk in payload.chunks(chunk_size) {
streaming.update(black_box(chunk));
}
last_headers = streaming.headers();
black_box(&last_headers);
}
let elapsed = t0.elapsed();
assert_eq!(
last_headers.get(algorithm.http_header_name()),
expected_headers.get(algorithm.http_header_name()),
"{algorithm:?} streaming chunk_size={chunk_size} header mismatch"
);
let total_mib = (payload.len() * iters) as f64 / (1024.0 * 1024.0);
let mibs = total_mib / elapsed.as_secs_f64().max(1e-9);
println!(
"[microbench][checksums] algo={algorithm:?} chunk_size={chunk_size}B throughput={mibs:.1} MiB/s (size={}B)",
algorithm.raw_len()
);
}
}
}
}
+21 -9
View File
@@ -184,15 +184,27 @@ const _: () = assert!(!DEFAULT_INTERNODE_RPC_REPLAY_SCOPE_STRICT);
/// Capacity (distinct nonces) of the process-local internode RPC replay cache that enforces
/// one-time consumption of authenticated RPC signatures.
///
/// The cache retains each nonce for the ~10-minute signature freshness envelope. Once peers use
/// replay-scoped v3 authentication, every authenticated RPC consumes one entry, so the steady
/// state holds roughly `authenticated RPC RPS x 601s` entries. This default is the minimum floor:
/// explicit operator values and resource-aware auto sizing both clamp upward to at least this
/// value. Overflow fails closed — legitimate signed traffic is the only thing that can fill the
/// cache (replays are rejected before insertion, and an attacker cannot mint valid nonces without
/// the shared secret) — and increments
/// `rustfs_system_network_internode_replay_cache_overflow_total`, so a sustained non-zero overflow
/// counter means this capacity is undersized for the node's peak authenticated RPC rate.
/// Nonces remain protected until their signed timestamp is outside the observed wall-clock
/// validity window, including after clock rollback. Current-time signatures live about five
/// minutes; the maximum accepted forward clock skew can extend that to ten minutes. A body-bound
/// v3 request consumes both its v3 and compatibility-v2 nonces, so size for receiver nonce intake,
/// not S3 request rate or CPU count.
///
/// Automatic admission budgets 13% of effective process memory for signed-time
/// hash tables, allocator overhead and simultaneous old/new tables during growth.
/// The reported count is an absolute UUID payload ceiling, not guaranteed throughput:
/// actual byte admission can reject earlier depending on timestamp distribution.
/// Tables grow with demand and are released together when their signed time expires.
/// CPU count, a historical minimum, and a fixed 32M ceiling do not constrain this budget.
/// If memory detection fails, the historical default remains a bounded count fallback.
///
/// Explicit values remain count ceilings with the historical minimum of 1,048,576
/// for configuration compatibility, and bypass automatic byte admission. That
/// compatibility floor does not apply to the automatically detected memory budget.
/// Zero never disables replay protection.
/// Overflow rejects the request without evicting a still-valid identity, increments
/// `rustfs_system_network_internode_replay_cache_overflow_total`, and returns resource
/// exhaustion rather than invalid credentials. Measure headroom before overriding.
pub const ENV_INTERNODE_RPC_REPLAY_CACHE_CAPACITY: &str = "RUSTFS_INTERNODE_RPC_REPLAY_CACHE_CAPACITY";
pub const DEFAULT_INTERNODE_RPC_REPLAY_CACHE_CAPACITY: usize = 1_048_576;
+3 -2
View File
@@ -32,8 +32,9 @@ pub const ENV_OBJECT_MEDIUM_CONCURRENCY_THRESHOLD: &str = "RUSTFS_OBJECT_MEDIUM_
/// Environment variable name for maximum concurrent disk reads for object operations.
/// - Purpose: Limit the number of concurrent disk read operations for object reads to prevent I/O saturation.
/// - Unit: request count (usize).
/// - Semantics: Throttling disk reads helps maintain overall system responsiveness under load.
/// - Unit: request count (usize) for GET; estimated drive fanout for native LIST.
/// - Semantics: GET bodies and native LIST walks use separate per-instance budgets of this size.
/// A LIST consumes up to one budget, including its disk-producer cleanup. `0` disables both caps.
/// - Example: `export RUSTFS_OBJECT_MAX_CONCURRENT_DISK_READS=16`
/// - Note: This setting may interact with OS-level I/O scheduling and should be tuned based on hardware capabilities.
pub const ENV_OBJECT_MAX_CONCURRENT_DISK_READS: &str = "RUSTFS_OBJECT_MAX_CONCURRENT_DISK_READS";
+3 -3
View File
@@ -30,10 +30,10 @@ pub const ENV_TRUSTED_PROXY_VALIDATION_MODE: &str = "RUSTFS_TRUSTED_PROXY_VALIDA
/// Default validation mode is "hop_by_hop".
pub const DEFAULT_TRUSTED_PROXY_VALIDATION_MODE: &str = "hop_by_hop";
/// Environment variable to enable RFC 7239 "Forwarded" header support.
/// Select RFC 7239 "Forwarded" instead of "X-Forwarded-For" in legacy mode.
pub const ENV_TRUSTED_PROXY_ENABLE_RFC7239: &str = "RUSTFS_TRUSTED_PROXY_ENABLE_RFC7239";
/// RFC 7239 support is enabled by default.
pub const DEFAULT_TRUSTED_PROXY_ENABLE_RFC7239: bool = true;
/// Legacy mode uses X-Forwarded-For unless RFC 7239 is explicitly selected.
pub const DEFAULT_TRUSTED_PROXY_ENABLE_RFC7239: bool = false;
/// Environment variable for the maximum allowed proxy hops.
pub const ENV_TRUSTED_PROXY_MAX_HOPS: &str = "RUSTFS_TRUSTED_PROXY_MAX_HOPS";
+6
View File
@@ -146,3 +146,9 @@ pub const ENV_TARGET_STORE_COMPRESS: &str = "RUSTFS_TARGET_STORE_COMPRESS";
/// Queue-store compression is enabled by default to reduce disk footprint.
pub const DEFAULT_TARGET_STORE_COMPRESS: bool = true;
/// Maximum on-disk bytes in each target queue, including failed entries and pending writes.
pub const ENV_TARGET_STORE_MAX_BYTES: &str = "RUSTFS_TARGET_STORE_MAX_BYTES";
/// Queue writes fail at one GiB unless the operator sets another positive byte budget.
pub const DEFAULT_TARGET_STORE_MAX_BYTES: u64 = 1024 * 1024 * 1024;
+6 -1
View File
@@ -175,12 +175,17 @@ pub const ENV_HTTP_SOCKET_RECV_BUFFER_BYTES: &str = "RUSTFS_HTTP_SOCKET_RECV_BUF
pub const DEFAULT_HTTP_SOCKET_RECV_BUFFER_BYTES: usize = 0;
/// Environment variable for the S3 request-body inter-chunk read timeout
/// (seconds). Default: 300. Set to 0 to disable.
/// (seconds). Default: 300. Set to 0 to disable the inactivity timeout.
///
/// This bounds how long a `PutObject`/upload may wait for the *next* body chunk
/// while more bytes are still expected. It resets on every chunk, so it does not
/// penalize slow-but-progressing uploads — it only fires when a peer sends a
/// partial body and then goes silent *without* closing the connection (no EOF).
/// Independently of this knob, `PutObject` and `UploadPart` require at least
/// 64 KiB of body progress per 60 seconds of server wait (the minimum-progress
/// floor, logged as `upload_body_progress_stalled`); it bounds trickle uploads
/// that would otherwise pin upload admission indefinitely, and it stays in
/// force when this timeout is set to 0.
/// A reverse proxy or CDN that forwards a truncated body this way would
/// otherwise make RustFS wait forever for bytes that never arrive; with this
/// timeout the stalled read is aborted and logged with the received/expected
+6 -1
View File
@@ -91,7 +91,12 @@ pub const ENV_OBS_LOG_MATCH_MODE: &str = "RUSTFS_OBS_LOG_MATCH_MODE";
/// Default values for log cleanup
pub const DEFAULT_OBS_LOG_MAX_TOTAL_SIZE_BYTES: u64 = 2 * 1024 * 1024 * 1024; // 2 GiB
pub const DEFAULT_OBS_LOG_MAX_SINGLE_FILE_SIZE_BYTES: u64 = 0; // No single file limit
pub const DEFAULT_OBS_LOG_MAX_SINGLE_FILE_SIZE_BYTES: u64 = 0; // No per-file trim by the cleaner
/// Size at which the active log file is rolled when
/// `RUSTFS_OBS_LOG_MAX_SINGLE_FILE_SIZE_BYTES` is unset. Without a size roll
/// the active file is bounded only by the rotation period, and the cleaner
/// never touches the active file. An explicit `0` keeps size rolling off.
pub const DEFAULT_OBS_LOG_ACTIVE_FILE_ROLL_SIZE_BYTES: u64 = 256 * 1024 * 1024; // 256 MiB
pub const DEFAULT_OBS_LOG_COMPRESS_OLD_FILES: bool = true;
pub const DEFAULT_OBS_LOG_GZIP_COMPRESSION_LEVEL: u32 = 6;
pub const DEFAULT_OBS_LOG_COMPRESSION_ALGORITHM_GZIP: &str = "gzip";
+55 -2
View File
@@ -675,6 +675,10 @@ impl DataUsageInfo {
/// cycle. The timestamp fallback preserves ordering for legacy snapshots that
/// predate those fields.
pub fn data_usage_snapshot_is_newer(candidate: &DataUsageInfo, baseline: &DataUsageInfo) -> bool {
data_usage_identity_is_newer(candidate.snapshot_identity(), baseline.snapshot_identity())
}
fn data_usage_identity_is_newer(candidate: DataUsageSnapshotIdentity, baseline: DataUsageSnapshotIdentity) -> bool {
match (
candidate.scanner_epoch.zip(candidate.scanner_cycle),
baseline.scanner_epoch.zip(baseline.scanner_cycle),
@@ -697,10 +701,18 @@ pub fn data_usage_snapshot_is_newer(candidate: &DataUsageInfo, baseline: &DataUs
/// already advance the authoritative timestamp when deleting a bucket, so a
/// rollback delete/recreate fences the previous bucket incarnation too.
pub fn observed_data_usage_is_newer(observed: &DataUsageInfo, authoritative: &DataUsageInfo) -> bool {
observed_data_usage_is_newer_than_identity(observed, authoritative.snapshot_identity())
}
/// Return whether a nonconverged observation may supersede `authoritative`.
///
/// This applies the structural, exact-baseline, and generation checks of
/// [`observed_data_usage_is_newer`] without retaining the baseline's bucket maps.
pub fn observed_data_usage_is_newer_than_identity(observed: &DataUsageInfo, authoritative: DataUsageSnapshotIdentity) -> bool {
observed.usage_snapshot_converged == Some(false)
&& (observed.is_complete_bucket_usage_snapshot() || observed.is_valid_partial_snapshot())
&& observed.usage_snapshot_authoritative_baseline.as_ref() == Some(&authoritative.snapshot_identity())
&& data_usage_snapshot_is_newer(observed, authoritative)
&& observed.usage_snapshot_authoritative_baseline == Some(authoritative)
&& data_usage_identity_is_newer(observed.snapshot_identity(), authoritative)
}
/// Metadata describing the status of a disk-level data usage snapshot.
@@ -3140,12 +3152,17 @@ mod tests {
};
assert!(observed_data_usage_is_newer(&observed, &authoritative));
assert!(observed_data_usage_is_newer_than_identity(&observed, authoritative.snapshot_identity()));
authoritative.last_update = Some(SystemTime::UNIX_EPOCH + Duration::from_secs(601));
assert!(
!observed_data_usage_is_newer(&observed, &authoritative),
"an old-binary namespace mutation must fence the prior bucket incarnation regardless of clock skew"
);
assert!(
!observed_data_usage_is_newer_than_identity(&observed, authoritative.snapshot_identity()),
"a captured identity must preserve the exact namespace fence"
);
}
#[test]
@@ -3172,6 +3189,18 @@ mod tests {
assert!(!observed_data_usage_is_newer(&candidate(2, 9, Some(false), true), &authoritative));
assert!(!observed_data_usage_is_newer(&candidate(2, 11, Some(true), true), &authoritative));
assert!(!observed_data_usage_is_newer(&candidate(2, 11, Some(false), false), &authoritative));
for (observed, expected) in [
(candidate(2, 11, Some(false), true), true),
(candidate(2, 9, Some(false), true), false),
(candidate(2, 11, Some(true), true), false),
(candidate(2, 11, Some(false), false), false),
] {
assert_eq!(
observed_data_usage_is_newer_than_identity(&observed, authoritative.snapshot_identity()),
expected,
"observation identity selection must preserve convergence, completeness, and generation gates"
);
}
let mut partial = candidate(2, 11, Some(false), false);
partial.usage_snapshot_partial = true;
@@ -3186,6 +3215,30 @@ mod tests {
segment_invalidation_proof: None,
}];
assert!(observed_data_usage_is_newer(&partial, &authoritative));
assert!(observed_data_usage_is_newer_than_identity(&partial, authoritative.snapshot_identity()));
}
#[test]
fn observation_identity_selection_keeps_legacy_timestamp_ordering() {
let baseline = DataUsageSnapshotIdentity {
last_update: Some(SystemTime::UNIX_EPOCH + Duration::from_secs(10)),
// An epoch without a cycle is not a coordinated position.
scanner_epoch: Some(2),
scanner_cycle: None,
};
let mut observed = DataUsageInfo {
last_update: Some(SystemTime::UNIX_EPOCH + Duration::from_secs(11)),
scanner_epoch: Some(1),
usage_snapshot_complete: true,
usage_snapshot_converged: Some(false),
usage_snapshot_authoritative_baseline: Some(baseline),
..Default::default()
};
assert!(observed_data_usage_is_newer_than_identity(&observed, baseline));
observed.last_update = baseline.last_update;
assert!(!observed_data_usage_is_newer_than_identity(&observed, baseline));
observed.last_update = None;
assert!(!observed_data_usage_is_newer_than_identity(&observed, baseline));
}
#[test]
+3 -3
View File
@@ -26,7 +26,7 @@ Registered in [`src/lib.rs`](src/lib.rs). Grouped by concern:
| **protocols** | [`src/protocols/`](src/protocols) | FTPS, WebDAV, SFTP compliance. Fixed ports, own guide: [`src/protocols/README.md`](src/protocols/README.md) |
| **reliant** | [`src/reliant/`](src/reliant) | Tests that reuse an **externally started** server (SQL/select, conditional writes, lifecycle, deleted-object reads, node-interact). Run via [`scripts/run_e2e_tests.sh`](../../scripts/run_e2e_tests.sh); see [`src/reliant/README.md`](src/reliant/README.md) |
| **cluster** | `cluster_concurrency_test`, `stale_multipart_cleanup_cluster_test`, `namespace_lock_quorum_test`, `admin_timeout_regression_test`, `object_lambda_test`, `replication_extension_test`, `tier_stats_cluster_test` | Multi-node scenarios via `RustFSTestClusterEnvironment` |
| **distributed 4×4** | [`src/distributed/`](src/distributed) | Storage-sensitive PR and nightly `e2e-distributed` lane: S3, object lock/WORM, versioning, bucket/site replication, quota, expand/decommission/rebalance, concurrency, chaos, 4-node upgrade of historical data and IAM AK/SK. Map: [`docs/testing/distributed-e2e.md`](../../docs/testing/distributed-e2e.md) |
| **distributed 4×4** | [`src/distributed/`](src/distributed) | Nightly `e2e-distributed` lane: S3, object lock/WORM, versioning, bucket/site replication, quota, expand/decommission/rebalance, concurrency, chaos, 4-node upgrade of historical data and IAM AK/SK. Map: [`docs/testing/distributed-e2e.md`](../../docs/testing/distributed-e2e.md) |
| **chaos / reliability** | [`src/chaos.rs`](src/chaos.rs), `reliability_disk_fault_test`, `heal_erasure_disk_rebuild_test`, `server_startup_failfast_test` | Disk offline/replace/corrupt, EC rebuild, heal, fail-fast startup |
| **upgrade compatibility** | `upgrade_compatibility_test` | Pinned previous-release writes followed by current-build reads on the same data directory |
@@ -194,9 +194,9 @@ the same profile for membership and execution with one nightly worker.
| `s3s-e2e` black-box | `e2e-tests` + `e2e-tests-rio-v2` jobs | **Active** (external conformance tool) |
| ILM / lifecycle (ignored) | `test-ilm-integration-serial` lane, `-j1` | **Active** (backlog#1148 ilm-1) |
| KMS suite | `e2e-full` job, merge queue + main | **Active** |
| Direct and mixed-version rolling upgrades from pinned previous release | `e2e-upgrade.yml`, storage-sensitive PRs + release tags + weekly | **Active** |
| Direct and mixed-version rolling upgrades from pinned previous release | `e2e-upgrade.yml`, release tags + weekly schedule + manual dispatch | **Active** |
| Cluster faults (`e2e-nightly` profile) | consolidated nightly workflow | **Active** (backlog#1149 ci-7) |
| Distributed 4-node 4-disk (`e2e-distributed` profile) | `.github/workflows/e2e-distributed.yml` | **Active** (storage-sensitive PR / nightly / dispatch) |
| Distributed 4-node 4-disk (`e2e-distributed` profile) | `.github/workflows/e2e-distributed.yml` | **Active** (nightly / dispatch) |
| Protocols (FTPS/WebDAV/SFTP) | consolidated nightly workflow, serial | **Active** (backlog#1149 ci-7) |
| Replication (fast subset) | `e2e-smoke` profile, `e2e-tests` job, every PR | **Active** (backlog#1147 repl-1) |
| Replication (slow + multi-node) | `e2e-repl-nightly` profile, consolidated nightly workflow | **Active** (backlog#1147 repl-1) |
@@ -260,7 +260,7 @@ async fn iam_policy_requires_if_match(env: &RustFSTestEnvironment, admin: &Clien
let key = "guarded.txt";
admin_create_user(env, &user, user_secret).await?;
let policy = serde_json::json!({
let mut policy = serde_json::json!({
"Version": "2012-10-17",
"Statement": [{
"Effect": "Allow",
@@ -268,9 +268,8 @@ async fn iam_policy_requires_if_match(env: &RustFSTestEnvironment, admin: &Clien
"Resource": [format!("arn:aws:s3:::{bucket}/*")],
"Condition": { "Null": { "s3:if-match": "false" } }
}]
})
.to_string();
admin_add_and_attach_policy(env, &policy_name, &policy, &user).await?;
});
admin_add_and_attach_policy(env, &policy_name, &policy.to_string(), &user).await?;
admin.create_bucket().bucket(&bucket).send().await?;
let original = admin
@@ -293,6 +292,46 @@ async fn iam_policy_requires_if_match(env: &RustFSTestEnvironment, admin: &Clien
.expect_err("PutObject without If-Match must be denied");
assert_error(&denied, "AccessDenied", 403, "unconditional PutObject");
let denied = client
.put_object()
.bucket(&bucket)
.key(key)
.if_match(&etag)
.body(ByteStream::from_static(b"missing-read-permission"))
.send()
.await
.expect_err("If-Match without GetObject must be denied");
assert_error(&denied, "AccessDenied", 403, "conditional PutObject without GetObject");
// S3 requires GetObject as well as PutObject for an ETag-based write.
policy["Statement"]
.as_array_mut()
.expect("policy statements")
.push(serde_json::json!({
"Effect": "Allow",
"Action": ["s3:GetObject"],
"Resource": [format!("arn:aws:s3:::{bucket}/{key}")]
}));
admin_add_canned_policy_via(
AdminTransport::Signed,
&env.url,
&env.access_key,
&env.secret_key,
&policy_name,
&policy.to_string(),
)
.await?;
let denied = client
.put_object()
.bucket(&bucket)
.key(key)
.body(ByteStream::from_static(b"blind-overwrite"))
.send()
.await
.expect_err("GetObject permission must not allow an unconditional PutObject");
assert_error(&denied, "AccessDenied", 403, "unconditional PutObject with GetObject");
client
.put_object()
.bucket(&bucket)
@@ -301,6 +340,8 @@ async fn iam_policy_requires_if_match(env: &RustFSTestEnvironment, admin: &Clien
.body(ByteStream::from_static(b"v2"))
.send()
.await?;
let written = client.get_object().bucket(&bucket).key(key).send().await?;
assert_eq!(written.body.collect().await?.into_bytes().as_ref(), b"v2");
let _ = admin.delete_object().bucket(&bucket).key(key).send().await;
let _ = admin.delete_bucket().bucket(&bucket).send().await;
@@ -502,10 +502,163 @@ mod tests {
assert!(response.contains(if expected == reqwest::StatusCode::FORBIDDEN {
"<Code>AccessDenied</Code>"
} else {
"<Code>BucketAlreadyExists</Code>"
"<Code>BucketAlreadyOwnedByYou</Code>"
}));
}
}
Ok(())
}
/// A deployment owns a single bucket namespace, so an IAM user that recreates
/// an existing bucket is the owner, not another account. In the default region
/// every such request returns the legacy idempotent `200 OK` (AWS CreateBucket
/// semantics), including buckets created by root and by another IAM user.
#[tokio::test]
async fn test_create_bucket_duplicate_in_default_region_is_idempotent_for_iam_user()
-> Result<(), Box<dyn Error + Send + Sync>> {
init_logging();
let mut env = RustFSTestEnvironment::new().await?;
env.start_rustfs_server(vec!["--region", "us-east-1"]).await?;
let root = env.create_s3_client();
admin_create_user(&env, "duplicate-iam", "duplicate-iam-secret").await?;
admin_attach_user_policy_via(
AdminTransport::Signed,
&env.url,
&env.access_key,
&env.secret_key,
"readwrite",
"duplicate-iam",
)
.await?;
let iam = Client::from_conf(build_test_s3_config(
&env.url,
"duplicate-iam",
"duplicate-iam-secret",
None,
"duplicate-create",
));
root.create_bucket()
.bucket("duplicate-root-owned")
.send()
.await
.expect("root creates the first bucket");
iam.create_bucket()
.bucket("duplicate-root-owned")
.send()
.await
.expect("an IAM user recreating a root-owned bucket in us-east-1 gets 200 OK");
iam.create_bucket()
.bucket("duplicate-iam-owned")
.send()
.await
.expect("the IAM user creates its own bucket");
iam.create_bucket()
.bucket("duplicate-iam-owned")
.send()
.await
.expect("the same IAM user recreating its bucket gets 200 OK");
root.create_bucket()
.bucket("duplicate-iam-owned")
.send()
.await
.expect("root recreating an IAM-created bucket gets 200 OK");
Ok(())
}
/// Outside the default region the same ownership answer becomes
/// `409 BucketAlreadyOwnedByYou` for every authorized requester. Requests that
/// lack `s3:CreateBucket` still fail authorization before the duplicate check.
#[tokio::test]
async fn test_create_bucket_duplicate_outside_default_region_reports_owned_by_you() -> Result<(), Box<dyn Error + Send + Sync>>
{
init_logging();
let mut env = RustFSTestEnvironment::new().await?;
env.start_rustfs_server(vec!["--region", "eu-west-1"]).await?;
let region = "eu-west-1";
let root = Client::from_conf(
env.create_s3_client()
.config()
.to_builder()
.region(Region::new(region))
.build(),
);
admin_create_user(&env, "duplicate-eu-iam", "duplicate-eu-iam-secret").await?;
admin_attach_user_policy_via(
AdminTransport::Signed,
&env.url,
&env.access_key,
&env.secret_key,
"readwrite",
"duplicate-eu-iam",
)
.await?;
let iam = Client::from_conf(
build_test_s3_config(&env.url, "duplicate-eu-iam", "duplicate-eu-iam-secret", None, "duplicate-create")
.to_builder()
.region(Region::new(region))
.build(),
);
admin_create_user(&env, "duplicate-eu-readonly", "duplicate-eu-readonly-secret").await?;
admin_attach_user_policy_via(
AdminTransport::Signed,
&env.url,
&env.access_key,
&env.secret_key,
"readonly",
"duplicate-eu-readonly",
)
.await?;
let readonly = Client::from_conf(
build_test_s3_config(
&env.url,
"duplicate-eu-readonly",
"duplicate-eu-readonly-secret",
None,
"duplicate-create",
)
.to_builder()
.region(Region::new(region))
.build(),
);
for (client, bucket) in [(&root, "duplicate-eu-root"), (&iam, "duplicate-eu-iam")] {
client
.create_bucket()
.bucket(bucket)
.send()
.await
.unwrap_or_else(|error| panic!("{bucket} must be created: {error:?}"));
// Both the creator and the other authenticated identity own the namespace.
for creator in [&root, &iam] {
let error = creator
.create_bucket()
.bucket(bucket)
.send()
.await
.expect_err("a duplicate create outside us-east-1 must report ownership");
assert_eq!(
error.as_service_error().and_then(ProvideErrorMetadata::code),
Some("BucketAlreadyOwnedByYou"),
"{bucket}"
);
assert_eq!(error.raw_response().map(|response| response.status().as_u16()), Some(409), "{bucket}");
}
}
let denied = readonly
.create_bucket()
.bucket("duplicate-eu-root")
.send()
.await
.expect_err("a requester without s3:CreateBucket must be denied before the duplicate check");
assert_eq!(denied.as_service_error().and_then(ProvideErrorMetadata::code), Some("AccessDenied"));
Ok(())
}
}
@@ -12,11 +12,249 @@
// See the License for the specific language governing permissions and
// limitations under the License.
use super::harness::{DistCluster, DistLayout, TestResult, assert_object_bytes, payload_for, put_object, unique_bucket};
use super::harness::{DistCluster, DistLayout, TestResult, payload_for, s3_probe_client, sha256_hex, unique_bucket};
use crate::common::init_logging;
use aws_sdk_s3::{
Client,
error::{ProvideErrorMetadata, SdkError},
primitives::ByteStream,
};
use std::any::TypeId;
use std::collections::BTreeSet;
use std::fmt::Debug;
use std::future::Future;
use std::sync::Arc;
use std::time::Duration;
use tokio::sync::Barrier;
use tokio::time::Instant;
const LOCK_ADMISSION_BUDGET: Duration = Duration::from_secs(45);
async fn after_lock_contention<T, E, F, Fut>(operation: &str, mut call: F) -> TestResult<Result<T, SdkError<E>>>
where
E: Debug + ProvideErrorMetadata + 'static,
F: FnMut() -> Fut,
Fut: Future<Output = Result<T, SdkError<E>>>,
{
let started = Instant::now();
let mut attempts = 0;
// The workload uses unique unversioned keys. Repeat only explicitly
// advertised lock backpressure; every step still needs its real result.
let outcome = tokio::time::timeout(LOCK_ADMISSION_BUDGET, async {
loop {
attempts += 1;
let result = call().await;
let retry = result.as_ref().err().is_some_and(|error| {
let Some(response) = error.raw_response() else {
return false;
};
let Some(service_error) = error.as_service_error() else {
return false;
};
// HEAD has no XML error body, so its SDK service error can
// lack a code even when the server supplies retry headers.
let namespace_code = service_error.code() == Some("ServiceUnavailable")
|| (service_error.code().is_none()
&& TypeId::of::<E>() == TypeId::of::<aws_sdk_s3::operation::head_object::HeadObjectError>());
response.status().as_u16() == 503
&& namespace_code
&& response.headers().get("x-rustfs-error-cause") == Some("namespace_lock_timeout")
&& response.headers().get("retry-after") == Some("1")
});
if !retry {
return result;
}
tokio::time::sleep(Duration::from_secs(1)).await;
}
})
.await;
if attempts > 1 || !matches!(&outcome, Ok(Ok(_))) {
eprintln!(
"mixed workload request: operation={operation}, attempts={attempts}, elapsed={:?}",
started.elapsed()
);
}
outcome.map_err(|_| {
format!("{operation} exceeded its {LOCK_ADMISSION_BUDGET:?} request budget after {attempts} attempts").into()
})
}
async fn assert_workload_bytes(client: &Client, bucket: &str, key: &str, expected: &[u8]) -> TestResult {
let output = after_lock_contention(&format!("GET {key}"), || client.get_object().bucket(bucket).key(key).send()).await??;
let actual = output.body.collect().await?.into_bytes();
if actual.as_ref() != expected {
return Err(format!(
"object {key} bytes mismatch: expected {} bytes sha256={} got {} bytes sha256={}",
expected.len(),
sha256_hex(expected),
actual.len(),
sha256_hex(&actual)
)
.into());
}
Ok(())
}
#[cfg(test)]
mod tests {
use super::*;
use aws_smithy_http_client::test_util::infallible_client_fn;
use aws_smithy_types::body::SdkBody;
use std::sync::atomic::{AtomicUsize, Ordering};
type Reply = (u16, &'static str, Option<&'static str>, Option<&'static str>, &'static str);
fn client(respond: impl Fn(usize) -> Reply + Send + Sync + 'static) -> (Client, Arc<AtomicUsize>) {
let requests = Arc::new(AtomicUsize::new(0));
let seen = Arc::clone(&requests);
let transport = infallible_client_fn(move |_| {
let (status, code, cause, retry_after, body) = respond(seen.fetch_add(1, Ordering::SeqCst));
let body = if code.is_empty() {
body.to_string()
} else {
format!("<Error><Code>{code}</Code></Error>")
};
let mut response = http::Response::builder()
.status(status)
.header("content-type", "application/xml")
.header("content-length", body.len());
if let Some(cause) = cause {
response = response.header("x-rustfs-error-cause", cause);
}
if let Some(retry_after) = retry_after {
response = response.header("retry-after", retry_after);
}
response.body(SdkBody::from(body)).expect("scripted S3 response")
});
let config = crate::common::build_test_s3_config(
"http://contention.example.invalid",
"synthetic-access",
"synthetic-secret",
None,
"contention-test",
);
let client = Client::from_conf(config.to_builder().http_client(transport).build());
(s3_probe_client(&client), requests)
}
fn contention() -> Reply {
(503, "ServiceUnavailable", Some("namespace_lock_timeout"), Some("1"), "")
}
#[tokio::test(start_paused = true)]
async fn advertised_contention_recovers_after_standard_sdk_attempts_are_exhausted() -> TestResult {
let (client, requests) = client(|attempt| {
if attempt < 3 {
contention()
} else {
(200, "", None, None, "")
}
});
let standard = Client::from_conf(
client
.config()
.to_builder()
.retry_config(aws_sdk_s3::config::retry::RetryConfig::standard().with_max_attempts(3))
.build(),
);
standard
.put_object()
.bucket("bucket")
.key("key")
.body(ByteStream::from_static(b"body"))
.send()
.await
.expect_err("three advertised lock responses exhaust the standard SDK policy");
assert_eq!(requests.load(Ordering::SeqCst), 3);
requests.store(0, Ordering::SeqCst);
let started = Instant::now();
after_lock_contention("PUT recovery", || {
client
.put_object()
.bucket("bucket")
.key("key")
.body(ByteStream::from_static(b"body"))
.send()
})
.await??;
assert_eq!(requests.load(Ordering::SeqCst), 4);
assert!(started.elapsed() >= Duration::from_secs(3), "honor each Retry-After response");
Ok(())
}
#[tokio::test(start_paused = true)]
async fn bodyless_head_contention_uses_explicit_retry_headers() -> TestResult {
let (client, requests) = client(|attempt| {
if attempt == 0 {
(503, "", Some("namespace_lock_timeout"), Some("1"), "")
} else {
(200, "", None, None, "")
}
});
after_lock_contention("HEAD recovery", || client.head_object().bucket("bucket").key("key").send()).await??;
assert_eq!(requests.load(Ordering::SeqCst), 2);
Ok(())
}
#[tokio::test(start_paused = true)]
async fn unrelated_or_malformed_responses_are_not_retried() -> TestResult {
for reply in [
(500, "InternalError", Some("namespace_lock_timeout"), Some("1"), ""),
(403, "AccessDenied", None, None, ""),
(404, "NoSuchKey", None, None, ""),
(503, "ServiceUnavailable", Some("storage_write_quorum"), None, ""),
(503, "ServiceUnavailable", Some("namespace_lock_unavailable"), Some("1"), ""),
(503, "InternalError", Some("namespace_lock_timeout"), Some("1"), ""),
(503, "", Some("namespace_lock_timeout"), Some("1"), ""),
(503, "ServiceUnavailable", None, Some("1"), ""),
(503, "ServiceUnavailable", Some("namespace_lock_timeout"), None, ""),
(503, "ServiceUnavailable", Some("namespace_lock_timeout"), Some("2"), ""),
] {
let (client, requests) = client(move |_| reply);
let result = after_lock_contention("PUT fatal response", || {
client
.put_object()
.bucket("bucket")
.key("key")
.body(ByteStream::from_static(b"body"))
.send()
})
.await?;
assert!(result.is_err(), "fatal response cannot become success: {reply:?}");
assert_eq!(requests.load(Ordering::SeqCst), 1, "fatal response must remain visible: {reply:?}");
}
Ok(())
}
#[tokio::test(start_paused = true)]
async fn persistent_lock_contention_exhausts_the_request_budget() {
let (client, requests) = client(|_| contention());
let started = Instant::now();
let result = after_lock_contention("PUT unavailable", || {
client
.put_object()
.bucket("bucket")
.key("key")
.body(ByteStream::from_static(b"body"))
.send()
})
.await;
assert!(result.is_err(), "persistent contention cannot acknowledge an operation");
assert!(started.elapsed() >= LOCK_ADMISSION_BUDGET);
assert!(requests.load(Ordering::SeqCst) > 1);
assert!(requests.load(Ordering::SeqCst) <= LOCK_ADMISSION_BUDGET.as_secs() as usize + 1);
}
#[tokio::test(start_paused = true)]
async fn acknowledged_wrong_bytes_fail_without_retry() {
let (client, requests) = client(|_| (200, "", None, None, "wrong bytes"));
let error = assert_workload_bytes(&client, "bucket", "key", b"expected bytes")
.await
.expect_err("a successful but incorrect read is a consistency failure");
assert!(error.to_string().contains("bytes mismatch"));
assert_eq!(requests.load(Ordering::SeqCst), 1);
}
}
#[tokio::test]
async fn four_node_high_concurrency_mixed_workload_is_consistent_on_every_node() -> TestResult {
@@ -24,7 +262,9 @@ async fn four_node_high_concurrency_mixed_workload_is_consistent_on_every_node()
let dist = DistCluster::start(DistLayout::FourByFour).await?;
let bucket = unique_bucket("conc");
dist.create_bucket(&bucket).await?;
let clients = Arc::new(dist.clients()?);
// This consistency probe owns its admission policy; unrelated responses
// must not be hidden by the SDK's general-purpose retries.
let clients = Arc::new(dist.clients()?.iter().map(s3_probe_client).collect::<Vec<_>>());
let worker_count = 24;
let rounds = 4;
let barrier = Arc::new(Barrier::new(worker_count));
@@ -44,31 +284,46 @@ async fn four_node_high_concurrency_mixed_workload_is_consistent_on_every_node()
let key = format!("source/worker-{idx:02}-round-{round}.bin");
let copy_key = format!("retained/worker-{idx:02}-round-{round}.bin");
let body = payload_for(&key, 64 * 1024);
put_object(writer, &bucket, &key, body.clone()).await?;
let context = format!("worker={idx} round={round}");
after_lock_contention(&format!("{context} PUT source"), || {
writer
.put_object()
.bucket(&bucket)
.key(&key)
.body(ByteStream::from(body.clone()))
.send()
})
.await??;
let head = reader.head_object().bucket(&bucket).key(&key).send().await?;
let head = after_lock_contention(&format!("{context} HEAD source"), || {
reader.head_object().bucket(&bucket).key(&key).send()
})
.await??;
if head.content_length() != Some(body.len() as i64) {
return Err(format!("HEAD returned the wrong size for {key}: {head:?}").into());
}
assert_object_bytes(reader, &bucket, &key, &body).await?;
assert_workload_bytes(reader, &bucket, &key, &body).await?;
copier
.copy_object()
.bucket(&bucket)
.key(&copy_key)
.copy_source(format!("{bucket}/{key}"))
.send()
.await?;
assert_object_bytes(writer, &bucket, &copy_key, &body).await?;
after_lock_contention(&format!("{context} COPY retained"), || {
copier
.copy_object()
.bucket(&bucket)
.key(&copy_key)
.copy_source(format!("{bucket}/{key}"))
.send()
})
.await??;
assert_workload_bytes(writer, &bucket, &copy_key, &body).await?;
writer.delete_object().bucket(&bucket).key(&key).send().await?;
let missing = reader
.head_object()
.bucket(&bucket)
.key(&key)
.send()
.await
.expect_err("deleted source key must not remain visible");
after_lock_contention(&format!("{context} DELETE source"), || {
writer.delete_object().bucket(&bucket).key(&key).send()
})
.await??;
let missing = after_lock_contention(&format!("{context} HEAD deleted source"), || {
reader.head_object().bucket(&bucket).key(&key).send()
})
.await?
.expect_err("deleted source key must not remain visible");
if missing.raw_response().map(|response| response.status().as_u16()) != Some(404) {
return Err(format!("deleted source {key} returned an unexpected result: {missing:?}").into());
}
@@ -82,14 +337,22 @@ async fn four_node_high_concurrency_mixed_workload_is_consistent_on_every_node()
for handle in handles {
inventory.extend(handle.await??);
}
assert_eq!(
inventory.len(),
worker_count * rounds,
"every worker round must retain its acknowledged copy"
);
let expected_keys: BTreeSet<_> = inventory.iter().map(|(key, _)| key.as_str()).collect();
for (node_idx, client) in clients.iter().enumerate() {
let listed = client.list_objects_v2().bucket(&bucket).prefix("retained/").send().await?;
let listed = after_lock_contention(&format!("node={node_idx} LIST retained"), || {
client.list_objects_v2().bucket(&bucket).prefix("retained/").send()
})
.await??;
let listed_keys: BTreeSet<_> = listed.contents().iter().filter_map(|object| object.key()).collect();
assert_eq!(listed_keys, expected_keys, "node {node_idx} returned a divergent retained-key listing");
for (key, body) in &inventory {
assert_object_bytes(client, &bucket, key, body)
assert_workload_bytes(client, &bucket, key, body)
.await
.map_err(|error| format!("node {node_idx} failed to read {key}: {error}"))?;
}
@@ -25,7 +25,8 @@
use super::harness::{
DistCluster, DistLayout, TestResult, cluster_admin, cluster_admin_ok, decommission_status_json, get_object_bytes,
payload_for, put_object, s3_probe_client, s3_probe_error_code, sha256_hex, unique_bucket, wait_until,
payload_for, put_object, rebalance_complete, rebalance_status_json, s3_probe_client, s3_probe_error_code, sha256_hex,
unique_bucket, wait_until,
};
use crate::chaos::{VersionShardCensus, census_object_version_on_disk};
use crate::common::{FAST_DATA_USAGE_SCANNER_ENV, RustFSTestClusterEnvironment};
@@ -47,6 +48,7 @@ use tokio::time::{Instant, sleep};
use uuid::Uuid;
pub(crate) const MOVEMENT_BARRIER_DIR_ENV: &str = "RUSTFS_E2E_DATA_MOVEMENT_BARRIER_DIR";
pub(crate) const STAGE_AFTER_ENTRY_ADMISSION: &str = "after_entry_admission";
pub(crate) const STAGE_BEFORE_TARGET_COMMIT: &str = "before_target_commit";
pub(crate) const STAGE_BEFORE_SOURCE_CLEANUP: &str = "before_source_cleanup";
pub(crate) const STAGE_AFTER_RESPONSIBILITY_PERSISTED: &str = "after_responsibility_persisted";
@@ -1166,8 +1168,21 @@ pub(crate) async fn clear_decommission(cluster: &RustFSTestClusterEnvironment, p
.await
}
pub(crate) async fn stop_rebalance(cluster: &RustFSTestClusterEnvironment) -> TestResult<String> {
cluster_admin_ok(cluster, Method::POST, "/rustfs/admin/v3/rebalance/stop", None).await
pub(crate) async fn stop_rebalance(cluster: &RustFSTestClusterEnvironment, expected_id: &str) -> TestResult<String> {
let path = "/rustfs/admin/v3/rebalance/stop";
let (status, response) = cluster_admin(cluster, Method::POST, path, None).await?;
if status.is_success() {
return Ok(response);
}
// The run can finish between the active observation and the stop request.
// Only completion of that same run makes a not-started response acceptable.
if status == StatusCode::NOT_FOUND
&& response.contains("<Code>NoSuchResource</Code>")
&& rebalance_complete(&rebalance_status_json(cluster).await?, expected_id)?
{
return Ok(response);
}
Err(format!("POST {path} failed: {status} {response}").into())
}
pub(crate) fn decommission_pool_entry(status: &Value, pool_id: usize) -> TestResult<Value> {
@@ -1585,6 +1600,129 @@ pub(crate) async fn assert_every_node_serves(dist: &DistCluster, bucket: &str, l
mod tests {
use super::*;
const REBALANCE_NOT_STARTED: &str =
"<Error><Code>NoSuchResource</Code><Message>pool rebalance is not started</Message></Error>";
async fn mock_rebalance_stop(
stop_status: StatusCode,
stop_body: &str,
run_status: Value,
) -> TestResult<(TestResult<String>, Vec<String>)> {
use bytes::Bytes;
use http_body_util::Full;
use hyper::service::service_fn;
use hyper_util::rt::TokioIo;
use tokio::net::TcpListener;
use tokio::sync::mpsc;
use tokio::task::JoinSet;
let listener = TcpListener::bind("127.0.0.1:0").await?;
let mut cluster = RustFSTestClusterEnvironment::new(1).await?;
cluster.nodes[0].url = format!("http://{}", listener.local_addr()?);
let stop_body = stop_body.to_string();
let (requests_tx, mut requests_rx) = mpsc::unbounded_channel();
let mut server = JoinSet::new();
server.spawn(async move {
loop {
let (stream, _) = match listener.accept().await {
Ok(connection) => connection,
Err(error) => return Err::<(), Box<dyn std::error::Error + Send + Sync>>(error.into()),
};
let stop_body = stop_body.clone();
let run_status = run_status.clone();
let requests_tx = requests_tx.clone();
let handler = service_fn(move |request: hyper::Request<hyper::body::Incoming>| {
let route = format!("{} {}", request.method(), request.uri().path());
requests_tx.send(route.clone()).expect("record mock admin request");
let (status, body) = match route.as_str() {
"POST /rustfs/admin/v3/rebalance/stop" => (stop_status, stop_body.clone()),
"GET /rustfs/admin/v3/rebalance/status" => (StatusCode::OK, run_status.to_string()),
_ => (StatusCode::BAD_REQUEST, "unexpected admin request".to_string()),
};
async move { http::Response::builder().status(status).body(Full::new(Bytes::from(body))) }
});
hyper::server::conn::http1::Builder::new()
.keep_alive(false)
.serve_connection(TokioIo::new(stream), handler)
.await?;
}
});
let outcome = tokio::time::timeout(Duration::from_secs(5), stop_rebalance(&cluster, "run-1")).await?;
server.abort_all();
let requests = std::iter::from_fn(|| requests_rx.try_recv().ok()).collect();
Ok((outcome, requests))
}
fn completed_rebalance() -> Value {
serde_json::json!({
"id": "run-1",
"pools": [
{ "id": 0, "status": "Completed", "progress": { "versions": 113 } },
{ "id": 1, "status": "None" }
]
})
}
#[tokio::test]
async fn rebalance_stop_success_does_not_query_completion() -> TestResult {
let (outcome, requests) = mock_rebalance_stop(StatusCode::OK, "", Value::Null).await?;
assert_eq!(outcome?, "");
assert_eq!(requests, ["POST /rustfs/admin/v3/rebalance/stop"]);
Ok(())
}
#[tokio::test]
async fn rebalance_stop_accepts_completed_run_after_not_started_response() -> TestResult {
let (outcome, requests) =
mock_rebalance_stop(StatusCode::NOT_FOUND, REBALANCE_NOT_STARTED, completed_rebalance()).await?;
outcome?;
assert_eq!(
requests,
[
"POST /rustfs/admin/v3/rebalance/stop",
"GET /rustfs/admin/v3/rebalance/status"
]
);
Ok(())
}
#[tokio::test]
async fn rebalance_stop_rejects_unproven_completion_and_other_errors() -> TestResult {
let complete = completed_rebalance();
let mut invalid = vec![Value::Null];
for (pointer, value) in [
("/id", serde_json::json!("run-2")),
("/pools/0/status", serde_json::json!("Started")),
("/pools/0/status", serde_json::json!("Failed")),
("/pools/0/status", serde_json::json!("Stopped")),
("/pools/0/progress/versions", serde_json::json!(0)),
] {
let mut status = complete.clone();
*status.pointer_mut(pointer).expect("completion fixture field") = value;
invalid.push(status);
}
let mut pending = complete.clone();
pending["stopPropagation"] = serde_json::json!({ "pendingTerminalReload": true });
invalid.push(pending);
let mut failed = complete.clone();
failed["pools"][0]["lastError"] = serde_json::json!("migration failed");
invalid.push(failed);
for status in invalid {
let (outcome, _) = mock_rebalance_stop(StatusCode::NOT_FOUND, REBALANCE_NOT_STARTED, status.clone()).await?;
assert!(outcome.is_err(), "unproven completion accepted: {status}");
}
for (code, body) in [
(StatusCode::FORBIDDEN, REBALANCE_NOT_STARTED),
(StatusCode::INTERNAL_SERVER_ERROR, REBALANCE_NOT_STARTED),
(StatusCode::NOT_FOUND, "<Error><Code>NoSuchBucket</Code></Error>"),
] {
let (outcome, requests) = mock_rebalance_stop(code, body, complete.clone()).await?;
assert!(outcome.is_err(), "unrelated error accepted: {code} {body}");
assert_eq!(requests, ["POST /rustfs/admin/v3/rebalance/stop"]);
}
Ok(())
}
#[test]
fn version_ids_normalize_null_spellings() {
for spelling in [None, Some(""), Some("null"), Some("00000000-0000-0000-0000-000000000000")] {
@@ -28,7 +28,7 @@ use super::harness::{
wait_for_rebalance_active, wait_for_rebalance_complete,
};
use super::scanner_heal_topology_fixture::{
LiveWorkload, MOVEMENT_ACTIVE_TIMEOUT, MOVEMENT_COMPLETE_TIMEOUT, MovementOperation, SEED_POOL,
LiveWorkload, MOVEMENT_ACTIVE_TIMEOUT, MOVEMENT_COMPLETE_TIMEOUT, MovementOperation, SEED_POOL, STAGE_AFTER_ENTRY_ADMISSION,
STAGE_AFTER_RESPONSIBILITY_PERSISTED, STAGE_BEFORE_SOURCE_CLEANUP, STAGE_BEFORE_TARGET_COMMIT, TopologyManifest, TopologyRun,
assert_bucket_durability_strict, assert_every_node_serves, assert_probe_objects, assert_process_durability_strict,
assert_seed_pool_census_unchanged, assert_single_pool_placement, cancel_decommission, capture_seed_pool_census,
@@ -274,6 +274,7 @@ async fn wait_for_rebalance_stopped_or_completed(run: &TopologyRun, rebalance_id
#[tokio::test]
async fn scanner_heal_topology_rebalance_survives_live_writes_stop_and_restart() -> TestResult {
init_logging();
require_test_hooks();
let mut run = TopologyRun::start("rebalance").await?;
let manifest = seed_manifest(&run).await?;
assert_bucket_durability_strict(&run.dist.cluster, &manifest.buckets()).await?;
@@ -288,23 +289,52 @@ async fn scanner_heal_topology_rebalance_survives_live_writes_stop_and_restart()
.map(|entry| (entry.key.clone(), entry.sha256.clone()))
.collect();
let first_id = start_rebalance(&run.dist.cluster).await?;
wait_for_rebalance_active(&run.dist.cluster, &first_id, MOVEMENT_ACTIVE_TIMEOUT).await?;
let during_rebalance = put_probe_objects(&run.dist, &scratch, "during-rebalance", 4).await?;
let workload = tokio::spawn(run_live_workload(LiveWorkload {
clients: run.dist.clients()?,
bucket: scratch.clone(),
prefix: "live".to_string(),
writers: LIVE_WRITERS,
rounds: LIVE_ROUNDS,
baseline_bucket: manifest.plain_bucket.clone(),
baseline: baseline.clone(),
pause: Duration::from_millis(100),
}));
sleep(Duration::from_secs(2)).await;
stop_rebalance(&run.dist.cluster).await?;
let (first_outcome, first_status) = wait_for_rebalance_stopped_or_completed(&run, &first_id).await?;
let ops = workload.await??;
// Keep an admitted entry on the source until stop cancels admission. Start
// propagation may await that entry, so observe and stop before awaiting ACK.
let first_key = manifest.plain_key("external", 0)?;
run.movement_barrier
.arm_movement("rebalance", &manifest.plain_bucket, &first_key, STAGE_AFTER_ENTRY_ADMISSION)?;
let observe_first = async {
let (observed_id, reached) = tokio::time::timeout(MOVEMENT_ACTIVE_TIMEOUT, async {
let reached = run.movement_barrier.wait_reached(MOVEMENT_ACTIVE_TIMEOUT).await?;
let status = rebalance_status_json(&run.dist.cluster).await?;
let observed_id = status
.get("id")
.and_then(Value::as_str)
.ok_or("rebalance status omitted id")?;
wait_for_rebalance_active(&run.dist.cluster, observed_id, MOVEMENT_ACTIVE_TIMEOUT).await?;
Ok::<_, Box<dyn std::error::Error + Send + Sync>>((observed_id.to_string(), reached))
})
.await??;
let probes = put_probe_objects(&run.dist, &scratch, "during-rebalance", 4).await?;
let workload = run_live_workload(LiveWorkload {
clients: run.dist.clients()?,
bucket: scratch.clone(),
prefix: "live".to_string(),
writers: LIVE_WRITERS,
rounds: LIVE_ROUNDS,
baseline_bucket: manifest.plain_bucket.clone(),
baseline: baseline.clone(),
pause: Duration::from_millis(100),
});
let stop = async {
sleep(Duration::from_secs(2)).await;
stop_rebalance(&run.dist.cluster, &observed_id).await?;
let (outcome, status) = wait_for_rebalance_stopped_or_completed(&run, &observed_id).await?;
if outcome != "stopped" {
return Err("rebalance completed before stop; mid-run stop was not exercised".into());
}
Ok::<_, Box<dyn std::error::Error + Send + Sync>>((outcome, status))
};
let (ops, (outcome, status)) = tokio::try_join!(workload, stop)?;
Ok::<_, Box<dyn std::error::Error + Send + Sync>>((observed_id, reached, probes, ops, outcome, status))
};
let (first_id, (observed_id, first_reached, during_rebalance, ops, first_outcome, first_status)) =
tokio::try_join!(start_rebalance(&run.dist.cluster), observe_first)?;
if first_id != observed_id {
return Err(format!("rebalance start acknowledged {first_id}, but the stopped run was {observed_id}").into());
}
run.movement_barrier.disarm()?;
let live_state = expected_live_state(&ops);
for node_idx in 0..run.dist.cluster.nodes.len() {
@@ -333,8 +363,40 @@ async fn scanner_heal_topology_rebalance_survives_live_writes_stop_and_restart()
verify_from_every_node(&run.dist, &manifest, "after restart following the stop").await?;
verify_live_state(&run.dist.client(0)?, &scratch, &live_state, "after restart following the stop").await?;
let second_id = start_rebalance(&run.dist.cluster).await?;
wait_for_rebalance_active(&run.dist.cluster, &second_id, MOVEMENT_ACTIVE_TIMEOUT).await?;
// A short second run can finish before the start response reaches us.
// Hold one remaining source entry until its active state is observed.
let mut second_key = None;
for entry in manifest.plain_entries() {
if pools_holding(&run.dist, &manifest.plain_bucket, &entry.key, None)?.contains(&SEED_POOL) {
second_key = Some(entry.key.clone());
break;
}
}
let second_key = second_key.ok_or("no seed-pool object remains for the second rebalance")?;
let second_barrier = run.movement_barrier.next_phase();
second_barrier.arm_movement("rebalance", &manifest.plain_bucket, &second_key, STAGE_BEFORE_SOURCE_CLEANUP)?;
let observe_second = async {
let reached = second_barrier.wait_reached(MOVEMENT_ACTIVE_TIMEOUT).await?;
let status = rebalance_status_json(&run.dist.cluster).await?;
let observed_id = status
.get("id")
.and_then(Value::as_str)
.ok_or("rebalance status omitted id")?;
if observed_id == first_id {
return Err("the second rebalance reused the stopped run id".into());
}
wait_for_rebalance_active(&run.dist.cluster, observed_id, MOVEMENT_ACTIVE_TIMEOUT).await?;
second_barrier.disarm()?;
second_barrier.release_movement()?;
Ok::<_, Box<dyn std::error::Error + Send + Sync>>((observed_id.to_string(), reached))
};
// Worker propagation can hold the start response until movement drains.
let (second_id, (observed_id, second_reached)) = tokio::try_join!(start_rebalance(&run.dist.cluster), async {
tokio::time::timeout(MOVEMENT_ACTIVE_TIMEOUT, observe_second).await?
})?;
if second_id != observed_id {
return Err(format!("rebalance start acknowledged {second_id}, but the active run was {observed_id}").into());
}
wait_for_rebalance_complete(&run.dist.cluster, &second_id, MOVEMENT_COMPLETE_TIMEOUT).await?;
verify_from_every_node(&run.dist, &manifest, "after the second rebalance completed").await?;
verify_live_state(&run.dist.client(3)?, &scratch, &live_state, "after the second rebalance completed").await?;
@@ -351,8 +413,8 @@ async fn scanner_heal_topology_rebalance_survives_live_writes_stop_and_restart()
&json!({
"gate": "G-REB",
"manifest": manifest_summary(&manifest),
"first_run": { "id": first_id, "outcome": first_outcome, "status": first_status, "versions_held_by_two_pools_after_stop": retained_after_stop },
"second_run": { "id": second_id, "versions_off_seed_pool": moved },
"first_run": { "id": first_id, "outcome": first_outcome, "status": first_status, "versions_held_by_two_pools_after_stop": retained_after_stop, "barrier_object": first_key, "barrier": first_reached },
"second_run": { "id": second_id, "versions_off_seed_pool": moved, "barrier_object": second_key, "barrier": second_reached },
"live_ops": ops,
"probe_placement_during_rebalance": during_placement,
"process_durability_strict_nodes": strict_nodes,
@@ -33,7 +33,7 @@ mod tests {
use sha2::{Digest, Sha256};
use std::collections::HashSet;
use std::error::Error;
use std::io::{Read, Write};
use std::io::{Read, Seek, SeekFrom, Write};
use std::net::SocketAddr;
use std::path::{Path, PathBuf};
use std::process::Command;
@@ -1654,6 +1654,89 @@ mod tests {
TargetEndpointBlackhole,
}
async fn release_heal_commit_barrier_after_shutdown(
log: &Path,
log_offset: u64,
barrier: &Path,
) -> Result<serde_json::Value, Box<dyn Error + Send + Sync>> {
// Observe within the existing graceful process-stop budget.
timeout(Duration::from_secs(45), async {
loop {
let mut file = std::fs::File::open(log)?;
file.seek(SeekFrom::Start(log_offset))?;
let mut tail = String::new();
file.read_to_string(&mut tail)?;
if let Some(event) = tail
.lines()
.filter_map(|line| serde_json::from_str::<serde_json::Value>(line).ok())
.find(|event| {
event["event"] == "server_shutdown_state"
&& event["component"] == "main"
&& event["subsystem"] == "startup"
&& event["state"] == "stopping"
})
{
// Local heal has stopped and HTTP shutdown has been signalled.
// Let admitted peer RPCs finish; keeping this synthetic barrier
// until process exit would make the fixture block its own drain.
std::fs::remove_file(barrier)?;
return Ok(event);
}
sleep(Duration::from_millis(10)).await;
}
})
.await?
}
#[tokio::test(start_paused = true)]
async fn heal_commit_barrier_waits_for_new_shutdown_event() -> Result<(), Box<dyn Error + Send + Sync>> {
let directory = tempfile::tempdir()?;
let log = directory.path().join("server.log");
let barrier = directory.path().join("barrier");
let event = serde_json::json!({
"event": "server_shutdown_state", "component": "main", "subsystem": "startup", "state": "stopping"
});
std::fs::write(&log, format!("{event}\n"))?;
std::fs::write(&barrier, "bucket/prefix/")?;
let log_offset = log.metadata()?.len();
let mut writer = std::fs::OpenOptions::new().append(true).open(&log)?;
writeln!(writer, "{}", serde_json::json!({"event": "shutdown_signal_received"}))?;
writeln!(writer, "{}", serde_json::json!({"event": "server_shutdown_state", "state": "stopping"}))?;
let release = release_heal_commit_barrier_after_shutdown(&log, log_offset, &barrier);
tokio::pin!(release);
assert!(
futures::poll!(release.as_mut()).is_pending(),
"old or unrelated events must not release the barrier"
);
assert!(barrier.is_file(), "the partial-rebuild barrier must remain armed before shutdown");
tokio::time::advance(Duration::from_secs(6)).await;
assert!(
futures::poll!(release.as_mut()).is_pending(),
"background shutdown may take longer than five seconds within the process-stop budget"
);
writeln!(writer, "{event}")?;
tokio::time::advance(Duration::from_millis(10)).await;
assert_eq!(release.await?, event, "retain the actual new shutdown witness");
assert!(!barrier.exists(), "shutdown must release the synthetic hold");
Ok(())
}
#[tokio::test(start_paused = true)]
async fn heal_commit_barrier_missing_shutdown_fails_without_releasing() -> Result<(), Box<dyn Error + Send + Sync>> {
let directory = tempfile::tempdir()?;
let log = directory.path().join("server.log");
let barrier = directory.path().join("barrier");
std::fs::write(&log, "")?;
std::fs::write(&barrier, "bucket/prefix/")?;
let error = release_heal_commit_barrier_after_shutdown(&log, 0, &barrier)
.await
.expect_err("missing shutdown evidence must fail the fixture");
assert!(error.is::<tokio::time::error::Elapsed>());
assert!(barrier.is_file(), "a missing shutdown witness must not release repairs");
Ok(())
}
async fn run_cluster_root_heal_interruption(scenario: InterruptionScenario) -> Result<(), Box<dyn Error + Send + Sync>> {
let server_binary = rustfs_binary_path();
let evidence_case = match scenario {
@@ -1791,6 +1874,7 @@ mod tests {
let mut created_online_objects = Vec::with_capacity(online_object_count);
let mut expected_manifests = Vec::with_capacity(online_object_count);
let mut unclean_shutdown_marker_observed = None;
let mut shutdown_barrier_release = None;
let mut attempt_count = 0usize;
let max_online_attempts = online_object_count.saturating_mul(topology.total_drives().max(1));
while expected_manifests.len() < online_object_count {
@@ -2373,7 +2457,21 @@ mod tests {
| InterruptionScenario::BackgroundTargetRestartEc84MultiSet
| InterruptionScenario::BackgroundCoordinatorRestart
) {
cluster.stop_node_gracefully(interruption_node).await?;
if interruption_node == 1
&& let Some(barrier) = &commit_barrier
{
let target_log = PathBuf::from(format!("{log_dir}/node1.log"));
let log_offset = target_log.metadata()?.len();
let (stopped, released) = tokio::join!(
cluster.stop_node_gracefully(interruption_node),
release_heal_commit_barrier_after_shutdown(&target_log, log_offset, barrier),
);
// Await both branches even on error so the child is reaped.
stopped?;
shutdown_barrier_release = Some(released?);
} else {
cluster.stop_node_gracefully(interruption_node).await?;
}
} else {
cluster.stop_node(interruption_node)?;
}
@@ -2415,7 +2513,9 @@ mod tests {
}
}
}
if let Some(barrier) = &commit_barrier {
if shutdown_barrier_release.is_none()
&& let Some(barrier) = &commit_barrier
{
std::fs::remove_file(barrier)?;
}
cluster.start_node_from_binary(interruption_node, &server_binary).await?;
@@ -2746,6 +2846,7 @@ mod tests {
"all_peers_bound_to_generation_window": true,
"pid_before": target_pid, "pid_after": restarted_pid,
"unclean_shutdown_marker": unclean_shutdown_marker_observed.unwrap_or(false),
"graceful_barrier_release": shutdown_barrier_release,
"objects": evidence_objects, "node_listings": node_listings,
});
if let Some(outage_write) = outage_write_diagnostic {
+255 -2
View File
@@ -21,12 +21,15 @@
#[cfg(test)]
mod tests {
use crate::common::{RustFSTestEnvironment, init_logging};
use crate::common::{RustFSTestEnvironment, admin_create_user, init_logging};
use aws_sdk_s3::Client;
use aws_sdk_s3::primitives::ByteStream;
use aws_sdk_s3::types::{BucketVersioningStatus, Delete, ObjectIdentifier, VersioningConfiguration};
use std::error::Error;
use tracing::info;
type TestResult = Result<(), Box<dyn Error + Send + Sync>>;
async fn create_bucket(client: &Client, bucket: &str) -> Result<(), Box<dyn Error + Send + Sync>> {
client.create_bucket().bucket(bucket).send().await?;
Ok(())
@@ -90,7 +93,7 @@ mod tests {
Ok(())
}
/// Duplicate and repeated slashes after a leading slash collapse MinIO-style.
/// Duplicate and repeated slashes after a leading slash collapse MinIO-style across single and bulk operations.
#[tokio::test]
async fn test_put_object_with_duplicate_slashes_normalized() -> Result<(), Box<dyn Error + Send + Sync>> {
init_logging();
@@ -144,8 +147,258 @@ mod tests {
"GET after DELETE with raw key must return HTTP 404, got {error:?}"
);
assert_bulk_delete_normalizes_keys(&client).await?;
assert_bulk_delete_authorizes_normalized_keys(&env, &client).await?;
assert_bulk_delete_preserves_version_identity(&client).await?;
env.stop_server();
info!("Test completed successfully");
Ok(())
}
async fn assert_bulk_delete_normalizes_keys(client: &Client) -> TestResult {
let bucket = "test-bulk-leading-slash";
create_bucket(client, bucket).await?;
let cases = [
("/foo/bar", "foo/bar"),
("//keyname", "keyname"),
("/dir///sub//file", "dir/sub/file"),
("/folder///", "folder/"),
("/literal%2F+名", "literal%2F+名"),
("plain", "plain"),
];
let invalid = ["///", "/dir/../keep", "dir//keep"];
client
.put_object()
.bucket(bucket)
.key("keep")
.body(ByteStream::from_static(b"keep"))
.send()
.await?;
client
.put_object()
.bucket(bucket)
.key("dir/keep")
.body(ByteStream::from_static(b"keep"))
.send()
.await?;
for quiet in [false, true] {
for (raw, _) in cases {
client
.put_object()
.bucket(bucket)
.key(raw)
.body(ByteStream::from_static(b""))
.send()
.await?;
}
let listing = client.list_objects_v2().bucket(bucket).send().await?;
let stored: Vec<_> = listing.contents().iter().filter_map(|object| object.key()).collect();
let mut expected: Vec<_> = cases
.iter()
.map(|(_, normalized)| *normalized)
.chain(["dir/keep", "keep"])
.collect();
expected.sort_unstable();
assert_eq!(stored, expected, "PUT must store the exact normalized targets before bulk deletion");
let objects = cases
.iter()
.map(|(raw, _)| *raw)
.chain(invalid)
.map(|key| ObjectIdentifier::builder().key(key).build())
.collect::<Result<Vec<_>, _>>()?;
let delete = Delete::builder().set_objects(Some(objects)).quiet(quiet).build()?;
// A second identical batch also pins idempotent deletion of absent normalized keys.
for _ in 0..2 {
let output = client.delete_objects().bucket(bucket).delete(delete.clone()).send().await?;
let errors: Vec<_> = output.errors().iter().map(|error| (error.key(), error.code())).collect();
assert_eq!(
errors,
invalid.map(|key| (Some(key), Some("InvalidArgument"))),
"unexpected per-key errors: {output:?}"
);
let deleted: Vec<_> = output.deleted().iter().filter_map(|object| object.key()).collect();
let expected = if quiet {
vec![]
} else {
cases.iter().map(|(raw, _)| *raw).collect()
};
assert_eq!(deleted, expected, "responses must retain request keys, including directory slashes");
}
let listing = client.list_objects_v2().bucket(bucket).send().await?;
let remaining: Vec<_> = listing.contents().iter().filter_map(|object| object.key()).collect();
assert_eq!(
remaining,
["dir/keep", "keep"],
"bulk delete must remove normalized objects without resolving dot segments"
);
}
Ok(())
}
async fn assert_bulk_delete_authorizes_normalized_keys(env: &RustFSTestEnvironment, admin: &Client) -> TestResult {
let bucket = "test-bulk-slash-policy";
let user = "bulk-slash-user";
admin_create_user(env, user, "bulkSlashTestSecret123!").await?;
create_bucket(admin, bucket).await?;
for key in ["allowed/item", "protected/item"] {
admin
.put_object()
.bucket(bucket)
.key(key)
.body(ByteStream::from_static(b"keep"))
.send()
.await?;
}
let policy = serde_json::json!({
"Version": "2012-10-17",
"Statement": [
{"Effect": "Allow", "Principal": {"AWS": [user]}, "Action": ["s3:DeleteObject"],
"Resource": [format!("arn:aws:s3:::{bucket}/*")]},
{"Effect": "Deny", "Principal": {"AWS": [user]}, "Action": ["s3:DeleteObject"],
"Resource": [format!("arn:aws:s3:::{bucket}/protected/*")]}
]
});
admin
.put_bucket_policy()
.bucket(bucket)
.policy(policy.to_string())
.send()
.await?;
let client = env.create_s3_client_with_credentials(user, "bulkSlashTestSecret123!");
let output = client
.delete_objects()
.bucket(bucket)
.delete(
Delete::builder()
.objects(ObjectIdentifier::builder().key("/allowed///item").build()?)
.objects(ObjectIdentifier::builder().key("/protected///item").build()?)
.build()?,
)
.send()
.await?;
assert_eq!(output.deleted().len(), 1, "allowed normalized key must be deleted: {output:?}");
assert_eq!(output.deleted()[0].key(), Some("/allowed///item"));
assert_eq!(output.errors().len(), 1, "normalized resource denial must apply: {output:?}");
assert_eq!(output.errors()[0].key(), Some("/protected///item"));
assert_eq!(output.errors()[0].code(), Some("AccessDenied"));
let listing = admin.list_objects_v2().bucket(bucket).send().await?;
let remaining: Vec<_> = listing.contents().iter().filter_map(|object| object.key()).collect();
assert_eq!(remaining, ["protected/item"], "only the authorized object may be removed");
let protected = admin.get_object().bucket(bucket).key("protected/item").send().await?;
assert_eq!(protected.body.collect().await?.into_bytes().as_ref(), b"keep");
Ok(())
}
async fn assert_bulk_delete_preserves_version_identity(client: &Client) -> TestResult {
let bucket = "test-bulk-slash-versions";
let raw = "/history///file";
let normalized = "history/file";
create_bucket(client, bucket).await?;
client
.put_bucket_versioning()
.bucket(bucket)
.versioning_configuration(
VersioningConfiguration::builder()
.status(BucketVersioningStatus::Enabled)
.build(),
)
.send()
.await?;
let first = client
.put_object()
.bucket(bucket)
.key(raw)
.body(ByteStream::from_static(b"old"))
.send()
.await?;
let first_id = first.version_id().ok_or("first PUT omitted version ID")?;
let latest = client
.put_object()
.bucket(bucket)
.key(raw)
.body(ByteStream::from_static(b"latest"))
.send()
.await?;
let latest_id = latest.version_id().ok_or("latest PUT omitted version ID")?;
let invalid = client
.delete_objects()
.bucket(bucket)
.delete(
Delete::builder()
.objects(ObjectIdentifier::builder().key("///").build()?)
.build()?,
)
.send()
.await?;
assert!(invalid.deleted().is_empty(), "slash-only keys must not delete a versioned object");
assert_eq!(
invalid.errors().len(),
1,
"slash-only keys must remain invalid in versioned buckets: {invalid:?}"
);
assert_eq!(invalid.errors()[0].key(), Some("///"));
assert_eq!(invalid.errors()[0].code(), Some("InvalidArgument"));
let deleted = client
.delete_objects()
.bucket(bucket)
.delete(
Delete::builder()
.objects(ObjectIdentifier::builder().key(raw).version_id(first_id).build()?)
.build()?,
)
.send()
.await?;
assert!(deleted.errors().is_empty(), "historical delete failed: {deleted:?}");
assert_eq!(deleted.deleted().len(), 1);
assert_eq!(deleted.deleted()[0].key(), Some(raw));
assert_eq!(deleted.deleted()[0].version_id(), Some(first_id));
let listing = client.list_object_versions().bucket(bucket).send().await?;
assert_eq!(listing.versions().len(), 1, "only the latest data version should remain: {listing:?}");
assert_eq!(listing.versions()[0].version_id(), Some(latest_id));
let deleted = client
.delete_objects()
.bucket(bucket)
.delete(
Delete::builder()
.objects(ObjectIdentifier::builder().key(raw).build()?)
.build()?,
)
.send()
.await?;
assert!(deleted.errors().is_empty(), "delete marker creation failed: {deleted:?}");
assert_eq!(deleted.deleted().len(), 1);
assert_eq!(deleted.deleted()[0].key(), Some(raw));
assert_eq!(deleted.deleted()[0].delete_marker(), Some(true));
let marker = deleted.deleted()[0]
.delete_marker_version_id()
.ok_or("missing delete marker ID")?;
let listing = client.list_object_versions().bucket(bucket).send().await?;
assert_eq!(listing.delete_markers().len(), 1);
assert_eq!(listing.delete_markers()[0].key(), Some(normalized));
assert_eq!(listing.delete_markers()[0].version_id(), Some(marker));
let deleted = client
.delete_objects()
.bucket(bucket)
.delete(
Delete::builder()
.objects(ObjectIdentifier::builder().key(raw).version_id(marker).build()?)
.build()?,
)
.send()
.await?;
assert!(deleted.errors().is_empty(), "explicit marker removal failed: {deleted:?}");
assert_eq!(deleted.deleted().len(), 1);
assert_eq!(deleted.deleted()[0].key(), Some(raw));
assert_eq!(deleted.deleted()[0].version_id(), Some(marker));
let listing = client.list_object_versions().bucket(bucket).send().await?;
assert!(listing.delete_markers().is_empty());
let restored = client.get_object().bucket(bucket).key(normalized).send().await?;
assert_eq!(restored.version_id(), Some(latest_id));
assert_eq!(restored.body.collect().await?.into_bytes().as_ref(), b"latest");
Ok(())
}
}
+3
View File
@@ -283,6 +283,9 @@ mod ssec_copy_test;
#[cfg(test)]
mod multipart_storage_class_test;
#[cfg(test)]
mod multipart_upload_id_test;
#[cfg(test)]
mod storage_class_capability_test;
@@ -0,0 +1,175 @@
// Copyright 2026 RustFS Team
//
// Licensed under the Apache License, Version 2.0 (the "License");
// you may not use this file except in compliance with the License.
// You may obtain a copy of the License at
//
// http://www.apache.org/licenses/LICENSE-2.0
//
// Unless required by applicable law or agreed to in writing, software
// distributed under the License is distributed on an "AS IS" BASIS,
// WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
// See the License for the specific language governing permissions and
// limitations under the License.
//! Multipart upload ID regression tests.
//!
//! Every operation that addresses an existing multipart upload must report an
//! upload ID that does not name a live upload as `404 NoSuchUpload`, whether
//! the ID is malformed or is a well-formed ID of an aborted upload.
#[cfg(test)]
mod tests {
use crate::common::{RustFSTestEnvironment, init_logging};
use aws_sdk_s3::Client;
use aws_sdk_s3::error::{ProvideErrorMetadata, SdkError};
use aws_sdk_s3::primitives::ByteStream;
use aws_sdk_s3::types::{CompletedMultipartUpload, CompletedPart};
type TestResult = Result<(), Box<dyn std::error::Error + Send + Sync>>;
const BUCKET: &str = "multipart-upload-id";
const KEY: &str = "target.bin";
const COPY_SOURCE_KEY: &str = "copy-source.bin";
// RustFS upload IDs are URL-safe base64; '.' is outside that alphabet, so
// the object layer rejects this ID as malformed instead of looking it up.
const MALFORMED_UPLOAD_ID: &str = "malformed.upload.id";
const PART_NUMBER: i32 = 1;
const PART_BODY: &[u8] = b"multipart upload id regression part";
const PART_ETAG: &str = "\"00000000000000000000000000000000\"";
const NO_SUCH_UPLOAD_STATUS: u16 = 404;
const NO_SUCH_UPLOAD_CODE: &str = "NoSuchUpload";
const OBJECT_NOT_FOUND_STATUS: u16 = 404;
/// Describes how `result` differs from `404 NoSuchUpload`, or returns `None` when it matches.
fn no_such_upload_mismatch<T, E>(operation: &str, result: Result<T, SdkError<E>>) -> Option<String>
where
E: ProvideErrorMetadata + std::fmt::Debug,
{
let Err(error) = result else {
return Some(format!("{operation}: succeeded"));
};
let status = error.raw_response().map(|response| response.status().as_u16());
let code = error.as_service_error().and_then(ProvideErrorMetadata::code);
(status != Some(NO_SUCH_UPLOAD_STATUS) || code != Some(NO_SUCH_UPLOAD_CODE))
.then(|| format!("{operation}: status {status:?}, code {code:?}: {error:?}"))
}
async fn start_server_with_copy_source() -> Result<(RustFSTestEnvironment, Client), Box<dyn std::error::Error + Send + Sync>>
{
init_logging();
let mut env = RustFSTestEnvironment::new().await?;
env.start_rustfs_server(Vec::new()).await?;
env.create_test_bucket(BUCKET).await?;
let client = env.create_s3_client();
client
.put_object()
.bucket(BUCKET)
.key(COPY_SOURCE_KEY)
.body(ByteStream::from_static(PART_BODY))
.send()
.await?;
Ok((env, client))
}
/// Sends every operation that addresses an existing upload and asserts that all of them report `404 NoSuchUpload`.
async fn assert_every_upload_operation_reports_no_such_upload(client: &Client, upload_id: &str) {
let upload_part = client
.upload_part()
.bucket(BUCKET)
.key(KEY)
.upload_id(upload_id)
.part_number(PART_NUMBER)
.body(ByteStream::from_static(PART_BODY))
.send()
.await;
let upload_part_copy = client
.upload_part_copy()
.bucket(BUCKET)
.key(KEY)
.upload_id(upload_id)
.part_number(PART_NUMBER)
.copy_source(format!("/{BUCKET}/{COPY_SOURCE_KEY}"))
.send()
.await;
let list_parts = client.list_parts().bucket(BUCKET).key(KEY).upload_id(upload_id).send().await;
let complete = client
.complete_multipart_upload()
.bucket(BUCKET)
.key(KEY)
.upload_id(upload_id)
.multipart_upload(
CompletedMultipartUpload::builder()
.parts(CompletedPart::builder().part_number(PART_NUMBER).e_tag(PART_ETAG).build())
.build(),
)
.send()
.await;
let abort = client
.abort_multipart_upload()
.bucket(BUCKET)
.key(KEY)
.upload_id(upload_id)
.send()
.await;
let mismatches: Vec<String> = [
no_such_upload_mismatch("UploadPart", upload_part),
no_such_upload_mismatch("UploadPartCopy", upload_part_copy),
no_such_upload_mismatch("ListParts", list_parts),
no_such_upload_mismatch("CompleteMultipartUpload", complete),
no_such_upload_mismatch("AbortMultipartUpload", abort),
]
.into_iter()
.flatten()
.collect();
assert!(
mismatches.is_empty(),
"upload ID {upload_id:?}: operations not reporting {NO_SUCH_UPLOAD_STATUS} {NO_SUCH_UPLOAD_CODE}:\n{}",
mismatches.join("\n")
);
}
/// Asserts that the rejected operations left neither a multipart upload nor an object under `KEY`.
async fn assert_no_upload_or_object_published(client: &Client) -> TestResult {
let uploads = client.list_multipart_uploads().bucket(BUCKET).prefix(KEY).send().await?;
assert!(uploads.uploads().is_empty(), "rejected operations must not leave an upload: {uploads:?}");
let head = client.head_object().bucket(BUCKET).key(KEY).send().await;
let status = head
.as_ref()
.err()
.and_then(|error| error.raw_response())
.map(|response| response.status().as_u16());
assert_eq!(
status,
Some(OBJECT_NOT_FOUND_STATUS),
"rejected operations must not publish {KEY}: {head:?}"
);
Ok(())
}
#[tokio::test]
async fn malformed_upload_id_reports_no_such_upload_on_every_upload_operation() -> TestResult {
let (_env, client) = start_server_with_copy_source().await?;
assert_every_upload_operation_reports_no_such_upload(&client, MALFORMED_UPLOAD_ID).await;
assert_no_upload_or_object_published(&client).await
}
#[tokio::test]
async fn aborted_upload_id_reports_no_such_upload_on_every_upload_operation() -> TestResult {
let (_env, client) = start_server_with_copy_source().await?;
let created = client.create_multipart_upload().bucket(BUCKET).key(KEY).send().await?;
let upload_id = created.upload_id().ok_or("CreateMultipartUpload returned no upload ID")?;
client
.abort_multipart_upload()
.bucket(BUCKET)
.key(KEY)
.upload_id(upload_id)
.send()
.await?;
assert_every_upload_operation_reports_no_such_upload(&client, upload_id).await;
assert_no_upload_or_object_published(&client).await
}
}
+1 -3
View File
@@ -77,9 +77,7 @@ core protocol surface plus cross-protocol consistency:
existing path returns an error, bad-password authentication is rejected
- Cross-protocol via aws-sdk-s3: SFTP write then S3 read with SHA256 match,
S3 write then SFTP read with SHA256 match
- Cross-API directory visibility: SFTP-created sub-directory visible via S3
ListObjectsV2, S3-created `__XLDIR__` marker visible via SFTP readdir as a
directory entry
- Cross-API directory visibility: SFTP-created sub-directory visible via S3 ListObjectsV2, and an S3 trailing-slash directory object visible via SFTP readdir as a directory entry
#### sftp_compliance (`test_sftp_compliance_suite`)
+14 -13
View File
@@ -394,9 +394,9 @@ pub async fn test_sftp_core_operations() -> Result<()> {
info!("PASS: S3-written object matches via S3 GetObject and SFTP READ");
// --- Cross-API directory visibility: SFTP mkdir, S3 ListObjectsV2 ---
// SFTP mkdir writes a __XLDIR__ marker. The rustfs S3 listing path
// decodes that marker back to a trailing-slash key, so the asserted
// pattern is "subdir_sftp/".
// SFTP mkdir writes a zero-byte object at the trailing-slash key.
// Storage encodes that key to the __XLDIR__ marker, and the S3
// listing path decodes the marker back to a trailing-slash key.
info!("Testing SFTP: SFTP-created sub-directory visible via S3 ListObjectsV2");
let sftp_subdir_name = "subdir_sftp";
let sftp_subdir_path = format!("/{bucket}/{sftp_subdir_name}");
@@ -422,20 +422,21 @@ pub async fn test_sftp_core_operations() -> Result<()> {
);
info!("PASS: SFTP mkdir visible to S3 ListObjectsV2");
// --- Cross-API directory visibility: S3 marker, SFTP readdir ---
// aws-sdk-s3 PutObject writes a zero-byte marker keyed with the
// __XLDIR__ suffix. SFTP readdir must decode the marker back to a
// bare directory entry whose file_type reports as a directory.
info!("Testing SFTP: S3-created __XLDIR__ marker visible via SFTP readdir");
// --- Cross-API directory visibility: S3 directory object, SFTP readdir ---
// aws-sdk-s3 PutObject writes a zero-byte object at the trailing-slash
// key. The __XLDIR__ suffix is reserved on the client-key boundary, so
// a direct put of that suffix is rejected. Storage encodes the slash
// key, and SFTP readdir must show a directory entry.
info!("Testing SFTP: S3-created directory object visible via SFTP readdir");
let s3_subdir_name = "subdir_s3";
let s3_subdir_marker_key = format!("{s3_subdir_name}{XLDIR_SUFFIX}");
let s3_subdir_key = format!("{s3_subdir_name}/");
s3.put_object()
.bucket(bucket)
.key(&s3_subdir_marker_key)
.key(&s3_subdir_key)
.body(ByteStream::from_static(b""))
.send()
.await
.map_err(|e| anyhow!("S3 PutObject {}/{} failed: {:?}", bucket, s3_subdir_marker_key, e))?;
.map_err(|e| anyhow!("S3 PutObject {}/{} failed: {:?}", bucket, s3_subdir_key, e))?;
let bucket_entries: Vec<(String, bool)> = sftp
.read_dir(&bucket_path)
.await?
@@ -444,9 +445,9 @@ pub async fn test_sftp_core_operations() -> Result<()> {
let visible_via_sftp = bucket_entries.iter().any(|(name, is_dir)| name == s3_subdir_name && *is_dir);
assert!(
visible_via_sftp,
"S3-created marker must appear as a directory in SFTP readdir: entries were {bucket_entries:?}"
"S3-created directory object must appear as a directory in SFTP readdir: entries were {bucket_entries:?}"
);
info!("PASS: S3 marker visible to SFTP readdir as a directory");
info!("PASS: S3 directory object visible to SFTP readdir as a directory");
// --- Pre-cleanup of cross-protocol fixtures ---
// Removes the new files and sub-directories so the existing rmdir
+34 -4
View File
@@ -412,6 +412,38 @@ mod integration_tests {
);
}
async fn upload_after_quota_reconciliation(
env: &QuotaTestEnv,
key: &str,
size_bytes: usize,
) -> Result<(), Box<dyn std::error::Error + Send + Sync>> {
let upload = async {
loop {
let result = env
.client
.put_object()
.bucket(&env.bucket_name)
.key(key)
.body(aws_sdk_s3::primitives::ByteStream::from(vec![0u8; size_bytes]))
.send()
.await;
match result {
Ok(_) => break,
Err(error) => assert_quota_rejection(
error.raw_response().map(|response| response.status().as_u16()),
error.as_service_error(),
&error,
),
}
sleep(Duration::from_secs(1)).await;
}
};
// The durable V1 ledger can retain the old charge for its 30-second reconciliation cooldown.
timeout(Duration::from_secs(45), upload)
.await
.map_err(|_| format!("PUT {key} did not succeed after quota reconciliation within 45 seconds").into())
}
async fn assert_put_rejected_by_quota(env: &QuotaTestEnv, key: &str, size_bytes: usize) {
let error = env
.client
@@ -573,8 +605,7 @@ mod integration_tests {
assert!(!env.object_exists("file1.txt").await?);
// Now we should be able to upload again (quota freed up)
env.upload_object("file3.txt", 256 * 1024).await?;
upload_after_quota_reconciliation(&env, "file3.txt", 256 * 1024).await?;
assert!(env.object_exists("file3.txt").await?);
env.cleanup_bucket().await?;
@@ -981,8 +1012,7 @@ mod integration_tests {
);
assert_eq!(delete_result.deleted().len(), 2);
// Now should be able to upload again (quota freed up)
env.upload_object("file3.txt", 256 * 1024).await?;
upload_after_quota_reconciliation(&env, "file3.txt", 256 * 1024).await?;
assert!(env.object_exists("file3.txt").await?);
env.cleanup_bucket().await?;
@@ -42,6 +42,7 @@ fn lock_result_from_response(response: rustfs_lock::LockResponse) -> GenerallyLo
success: response.success,
error_info: response.error,
lock_info: response.lock_info.and_then(|info| serde_json::to_string(&info).ok()),
..Default::default()
}
}
@@ -50,6 +51,7 @@ fn lock_result_from_error(error: impl Into<String>) -> GenerallyLockResult {
success: false,
error_info: Some(error.into()),
lock_info: None,
..Default::default()
}
}
@@ -59,6 +61,7 @@ fn lock_result_from_release(lock_id: &rustfs_lock::LockId, success: bool) -> Gen
success: true,
error_info: None,
lock_info: None,
..Default::default()
}
} else {
lock_result_from_error(format!("lock not found for release: {lock_id}"))
@@ -164,6 +167,7 @@ impl NodeService for MinimalLockNodeService {
success: false,
error_info: Some(format!("can not decode args, err: {err}")),
lock_info: None,
..Default::default()
}));
}
};
@@ -175,6 +179,7 @@ impl NodeService for MinimalLockNodeService {
success: result.success,
error_info: result.error,
lock_info: lock_info_json,
..Default::default()
}))
}
Err(err) => Ok(Response::new(GenerallyLockResponse {
@@ -184,6 +189,7 @@ impl NodeService for MinimalLockNodeService {
args.resource, args.owner, err
)),
lock_info: None,
..Default::default()
})),
}
}
@@ -197,6 +203,7 @@ impl NodeService for MinimalLockNodeService {
success: false,
error_info: Some(format!("can not decode args, err: {err}")),
lock_info: None,
..Default::default()
}));
}
};
@@ -208,6 +215,7 @@ impl NodeService for MinimalLockNodeService {
success: result.success,
error_info: result.error_info,
lock_info: None,
..Default::default()
}))
}
Err(err) => Ok(Response::new(GenerallyLockResponse {
@@ -217,6 +225,7 @@ impl NodeService for MinimalLockNodeService {
args.resource, args.owner, err
)),
lock_info: None,
..Default::default()
})),
}
}
@@ -230,6 +239,7 @@ impl NodeService for MinimalLockNodeService {
success: false,
error_info: Some(format!("can not decode args, err: {err}")),
lock_info: None,
..Default::default()
}));
}
};
@@ -241,6 +251,7 @@ impl NodeService for MinimalLockNodeService {
success: result.success,
error_info: result.error_info,
lock_info: None,
..Default::default()
}))
}
Err(err) => Ok(Response::new(GenerallyLockResponse {
@@ -250,6 +261,7 @@ impl NodeService for MinimalLockNodeService {
args.resource, args.owner, err
)),
lock_info: None,
..Default::default()
})),
}
}
@@ -263,6 +275,7 @@ impl NodeService for MinimalLockNodeService {
success: false,
error_info: Some(format!("can not decode args, err: {err}")),
lock_info: None,
..Default::default()
}));
}
};
@@ -272,11 +285,13 @@ impl NodeService for MinimalLockNodeService {
success,
error_info: None,
lock_info: None,
..Default::default()
})),
Err(err) => Ok(Response::new(GenerallyLockResponse {
success: false,
error_info: Some(format!("can not refresh, err: {err}")),
lock_info: None,
..Default::default()
})),
}
}
@@ -31,8 +31,8 @@ use aws_sdk_s3::error::ProvideErrorMetadata;
use aws_sdk_s3::operation::list_object_versions::ListObjectVersionsOutput;
use aws_sdk_s3::primitives::ByteStream;
use aws_sdk_s3::types::{
BucketVersioningStatus, CompletedMultipartUpload, CompletedPart, DeleteMarkerEntry, ObjectVersion, ServerSideEncryption,
VersioningConfiguration,
BucketVersioningStatus, CompletedMultipartUpload, CompletedPart, Delete, DeleteMarkerEntry, ObjectIdentifier, ObjectVersion,
ServerSideEncryption, VersioningConfiguration,
};
use base64_simd::STANDARD as BASE64_STANDARD;
use bytes::Bytes;
@@ -3729,6 +3729,8 @@ async fn test_bucket_replication_converges_delete_marker_and_version_purge() ->
init_logging();
let mut source_env = RustFSTestEnvironment::new().await?;
let source_log = format!("{}/source.log", source_env.temp_dir);
source_env.capture_log_path = Some(source_log.clone());
let mut source_env_vars = replication_fast_env();
source_env_vars.extend_from_slice(LOOPBACK_REPLICATION_TARGET_ENV);
source_env.start_rustfs_server_with_env(vec![], &source_env_vars).await?;
@@ -3791,19 +3793,66 @@ async fn test_bucket_replication_converges_delete_marker_and_version_purge() ->
"target did not preserve the source delete-marker version ID"
);
source_client
.delete_object()
let wait_for_source_writeback = || async {
let metrics_url = format!("{}/{source_bucket}?replication-metrics", source_env.url);
timeout(Duration::from_secs(30), async {
loop {
let response = signed_request(
http::Method::GET,
&metrics_url,
&source_env.access_key,
&source_env.secret_key,
None,
None,
)
.await?;
let metrics: serde_json::Value = response.error_for_status()?.json().await?;
let queued = metrics["queued"]["curr"]["count"]
.as_f64()
.ok_or("missing replication queue count")?;
if metrics["provider_available"] == true && queued == 0.0 {
return Ok::<(), Box<dyn Error + Send + Sync>>(());
}
sleep(Duration::from_millis(50)).await;
}
})
.await?
};
wait_for_source_writeback().await?;
let purged = source_client
.delete_objects()
.bucket(source_bucket)
.key(object_key)
.version_id(&purged_version_id)
.delete(
Delete::builder()
.objects(
ObjectIdentifier::builder()
.key(object_key)
.version_id(&purged_version_id)
.build()?,
)
.objects(
ObjectIdentifier::builder()
.key(object_key)
.version_id(delete_marker_version_id)
.build()?,
)
.build()?,
)
.send()
.await?;
assert!(purged.errors().is_empty(), "batch version purge failed: {:?}", purged.errors());
assert_eq!(purged.deleted().len(), 2, "both the data version and marker must be purged");
assert_replication_converged(&source_client, source_bucket, &target_client, target_bucket).await?;
let target_state = list_replication_state(&target_client, target_bucket).await?;
assert!(
target_state.iter().all(|entry| entry.version_id != purged_version_id),
"target retained the explicitly purged object version"
);
assert!(
target_state.iter().all(|entry| entry.version_id != delete_marker_version_id),
"target retained the explicitly purged delete marker"
);
assert!(
target_state
.iter()
@@ -3819,6 +3868,19 @@ async fn test_bucket_replication_converges_delete_marker_and_version_purge() ->
.await?;
assert_eq!(retained.body.collect().await?.into_bytes().as_ref(), b"versioned replication payload v2");
// Wait for the source workers to finish their final writeback, which runs
// after the remote purge has already become visible (rustfs#9121).
wait_for_source_writeback().await?;
let logs = fs::read_to_string(source_log).await?;
let writeback_failures: Vec<_> = logs
.lines()
.filter(|line| line.contains("apply_replication_delete_state"))
.collect();
assert!(
writeback_failures.is_empty(),
"a completed batch purge must not report a source writeback failure: {writeback_failures:?}"
);
Ok(())
}
+9
View File
@@ -94,3 +94,12 @@ pub(crate) mod scanner_peer_bucket {
pub(crate) use ecstore_erasure::{ReedSolomonEncoder, calc_shard_size};
pub(crate) use ecstore_rpc::PeerRestClient;
}
/// Storage handles used to seed a pre-validation bucket record while the fixture server is stopped.
#[cfg(test)]
pub(crate) mod bucket_tag_fixture {
pub(crate) use rustfs_ecstore::api::bucket::metadata::load_bucket_metadata;
pub(crate) use rustfs_ecstore::api::layout::{DisksLayout, EndpointServerPools};
pub(crate) use rustfs_ecstore::api::runtime::InstanceContext;
pub(crate) use rustfs_ecstore::api::storage::{ECStore, init_local_disks_with_instance_ctx};
}
+147 -29
View File
@@ -360,12 +360,26 @@ impl OpaMock {
pub(crate) async fn expect_bucket_tags(&mut self, action: &str, bucket: &str, department: Option<&str>) -> TestResult {
let payload = self.next_request().await?;
Self::assert_bucket_tags(&payload, action, bucket, department);
Ok(())
}
async fn expect_versioned_tag_permission(&mut self, bucket: &str, object: &str, version: &str) -> TestResult {
let payload = self.next_request().await?;
Self::assert_bucket_tags(&payload, "s3:GetObjectVersionTagging", bucket, Some("finance"));
assert_eq!(payload["input"]["resource"]["object"], object);
assert_eq!(payload["input"]["context"]["conditions"]["versionid"], serde_json::json!([version]));
Ok(())
}
fn assert_bucket_tags(payload: &Value, action: &str, bucket: &str, department: Option<&str>) {
let input = &payload["input"];
assert_eq!(input["identity"]["account"], "opabuckettags");
assert_eq!(input["identity"]["is_owner"], false);
assert_eq!(input["action"], action);
assert_eq!(input["resource"]["bucket"], bucket);
let expected = department.map(|value| serde_json::json!([value]));
assert_eq!(input["context"]["conditions"].get("ExistingBucketTag/department"), expected.as_ref());
Ok(())
}
async fn wait_for_validation(&mut self) -> TestResult {
@@ -678,22 +692,61 @@ async fn test_list_buckets_opa_contract() -> TestResult {
Ok(())
}
/// Stop and reap the directory's owning server before opening the fixture store.
async fn seed_legacy_duplicate_bucket_tags(env: &mut RustFSTestEnvironment, bucket: &str) -> TestResult {
use crate::storage_api::bucket_tag_fixture::{
DisksLayout, ECStore, EndpointServerPools, InstanceContext, init_local_disks_with_instance_ctx, load_bucket_metadata,
};
let server = env
.process
.as_mut()
.ok_or("missing server process before offline fixture write")?;
server.kill()?;
server.wait()?;
env.process = None;
let layout = DisksLayout::from_volumes(std::slice::from_ref(&env.temp_dir))?;
let (pools, _) = EndpointServerPools::create_server_endpoints(&env.address, &layout).await?;
let context = std::sync::Arc::new(InstanceContext::new());
init_local_disks_with_instance_ctx(&context, pools.clone()).await?;
let store = ECStore::new_with_instance_ctx(env.address.parse()?, pools, Default::default(), context.clone()).await?;
let _cancel = context
.background_cancel_token()
.expect("store binds its shutdown token")
.drop_guard();
let mut metadata = load_bucket_metadata(store.clone(), bucket).await?;
let incarnation = metadata.bucket_incarnation_id;
let policy = metadata.policy_config_json.clone();
assert!(!incarnation.is_nil());
assert!(!policy.is_empty(), "the bucket-policy Allow must survive legacy seeding");
// Encode the duplicate tag set with the same bucket record and erasure writer used by the store.
let xml = b"<Tagging><TagSet><Tag><Key>department</Key><Value>finance</Value></Tag><Tag><Key>department</Key><Value>engineering</Value></Tag></TagSet></Tagging>";
metadata.tagging_config_xml = xml.to_vec();
metadata.save_with_store(store.clone()).await?;
let persisted = load_bucket_metadata(store, bucket).await?;
assert_eq!(persisted.bucket_incarnation_id, incarnation);
assert_eq!(persisted.policy_config_json, policy);
assert_eq!(persisted.tagging_config_xml.as_slice(), xml);
Ok(())
}
#[tokio::test]
async fn test_bucket_tags_opa_contract() -> TestResult {
use aws_sdk_s3::primitives::ByteStream;
use aws_sdk_s3::types::{
BucketVersioningStatus, CompletedMultipartUpload, CompletedPart, Tag, Tagging, VersioningConfiguration,
BucketVersioningStatus, CompletedMultipartUpload, CompletedPart, Tag, Tagging, TaggingDirective, VersioningConfiguration,
};
init_logging();
let mut opa = OpaMock::start().await?;
let mut env = RustFSTestEnvironment::new().await?;
env.start_rustfs_server_without_cleanup_with_env(&[
("RUSTFS_POLICY_PLUGIN_URL", opa.url.as_str()),
let opa_url = opa.url.clone();
let server_env = [
("RUSTFS_POLICY_PLUGIN_URL", opa_url.as_str()),
("RUSTFS_POLICY_PLUGIN_AUTH_TOKEN", OPA_AUTH_TOKEN),
("NO_PROXY", "127.0.0.1,localhost"),
])
.await?;
];
env.start_rustfs_server_without_cleanup_with_env(&server_env).await?;
let secret = uuid::Uuid::new_v4().to_string();
create_user(&env, "opabuckettags", &secret).await?;
let client = aws_sdk_s3::Client::from_conf(build_test_s3_config(&env.url, "opabuckettags", &secret, None, "e2e-bucket-tags"));
@@ -733,7 +786,7 @@ async fn test_bucket_tags_opa_contract() -> TestResult {
.await?;
assert_eq!(read.body.collect().await?.into_bytes().as_ref(), b"report");
opa.expect_bucket_tags("s3:GetObjectVersion", source, Some("finance")).await?;
client
let head = client
.head_object()
.bucket(source)
.key("report")
@@ -741,6 +794,8 @@ async fn test_bucket_tags_opa_contract() -> TestResult {
.send()
.await?;
opa.expect_bucket_tags("s3:GetObjectVersion", source, Some("finance")).await?;
assert_eq!(head.tag_count(), Some(1));
opa.expect_versioned_tag_permission(source, "report", version).await?;
client.list_objects_v2().bucket(source).send().await?;
opa.expect_bucket_tags("s3:ListBucket", source, Some("finance")).await?;
client.list_object_versions().bucket(source).send().await?;
@@ -750,28 +805,65 @@ async fn test_bucket_tags_opa_contract() -> TestResult {
admin.create_bucket().bucket(destination).send().await?;
set_department(&admin, destination, "engineering").await?;
let copy_source = format!("{source}/report?versionId={version}");
let error = client
.copy_object()
.bucket(destination)
.key("copied")
.copy_source(&copy_source)
.send()
.await
.expect_err("source tags must not authorize a different destination");
assert_eq!(error.as_service_error().and_then(ProvideErrorMetadata::code), Some("AccessDenied"));
opa.expect_bucket_tags("s3:GetObjectVersion", source, Some("finance")).await?;
opa.expect_bucket_tags("s3:PutObject", destination, Some("engineering"))
.await?;
for directive in [TaggingDirective::Copy, TaggingDirective::Replace] {
let error = client
.copy_object()
.bucket(destination)
.key("copied")
.copy_source(&copy_source)
.set_tagging_directive((directive == TaggingDirective::Replace).then_some(TaggingDirective::Replace))
.set_tagging((directive == TaggingDirective::Replace).then(|| "department=legal".to_owned()))
.send()
.await
.expect_err("source tags must not authorize a different destination");
assert_eq!(error.as_service_error().and_then(ProvideErrorMetadata::code), Some("AccessDenied"));
opa.expect_bucket_tags("s3:GetObjectVersion", source, Some("finance")).await?;
if directive == TaggingDirective::Copy {
opa.expect_versioned_tag_permission(source, "report", version).await?;
}
opa.expect_bucket_tags("s3:PutObject", destination, Some("engineering"))
.await?;
let missing = admin
.head_object()
.bucket(destination)
.key("copied")
.send()
.await
.expect_err("denied COPY and REPLACE must leave no destination object");
assert_eq!(missing.raw_response().map(|response| response.status().as_u16()), Some(404));
}
set_department(&admin, destination, "finance").await?;
client
.copy_object()
.bucket(destination)
.key("copied")
.copy_source(&copy_source)
.send()
.await?;
opa.expect_bucket_tags("s3:GetObjectVersion", source, Some("finance")).await?;
opa.expect_bucket_tags("s3:PutObject", destination, Some("finance")).await?;
for directive in [TaggingDirective::Copy, TaggingDirective::Replace] {
client
.copy_object()
.bucket(destination)
.key("copied")
.copy_source(&copy_source)
.set_tagging_directive((directive == TaggingDirective::Replace).then_some(TaggingDirective::Replace))
.set_tagging((directive == TaggingDirective::Replace).then(|| "department=legal".to_owned()))
.send()
.await?;
opa.expect_bucket_tags("s3:GetObjectVersion", source, Some("finance")).await?;
// The access hook checks preliminary tags, then the copy use case checks
// the exact source reader's tags before publishing the destination.
for _ in 0..2 {
if directive == TaggingDirective::Copy {
opa.expect_versioned_tag_permission(source, "report", version).await?;
}
opa.expect_bucket_tags("s3:PutObject", destination, Some("finance")).await?;
}
let tags = admin.get_object_tagging().bucket(destination).key("copied").send().await?;
assert_eq!(tags.tag_set().len(), 1);
assert_eq!(tags.tag_set()[0].key(), "department");
assert_eq!(
tags.tag_set()[0].value(),
if directive == TaggingDirective::Copy {
"engineering"
} else {
"legal"
}
);
}
let upload = client
.create_multipart_upload()
@@ -847,6 +939,14 @@ async fn test_bucket_tags_opa_contract() -> TestResult {
assert_eq!(error.as_service_error().and_then(ProvideErrorMetadata::code), Some("AccessDenied"));
opa.expect_bucket_tags("s3:GetObjectVersion", source, Some("engineering"))
.await?;
let missing = admin
.head_object()
.bucket(destination)
.key("denied-source")
.send()
.await
.expect_err("a denied source must leave no destination object");
assert_eq!(missing.raw_response().map(|response| response.status().as_u16()), Some(404));
set_department(&client, source, "finance")
.await
.expect_err("requested tags must not authorize their own mutation");
@@ -885,7 +985,7 @@ async fn test_bucket_tags_opa_contract() -> TestResult {
assert_eq!(read.body.collect().await?.into_bytes().as_ref(), b"report");
opa.expect_bucket_tags("s3:GetObject", destination, Some("engineering"))
.await?;
admin
let invalid_write = admin
.put_bucket_tagging()
.bucket(destination)
.tagging(
@@ -895,7 +995,25 @@ async fn test_bucket_tags_opa_contract() -> TestResult {
.build()?,
)
.send()
.await
.expect_err("new duplicate tags must be rejected before publication");
assert_eq!(invalid_write.as_service_error().and_then(ProvideErrorMetadata::code), Some("InvalidTag"));
assert_eq!(invalid_write.raw_response().map(|response| response.status().as_u16()), Some(400));
set_department(&admin, source, "finance").await?;
seed_legacy_duplicate_bucket_tags(&mut env, destination).await?;
env.start_rustfs_server_without_cleanup_with_env(&server_env).await?;
// Prove this process can authorize the same caller and read a healthy bucket's persisted tags.
let read = client
.get_object()
.bucket(source)
.key("report")
.version_id(version)
.send()
.await?;
assert_eq!(read.body.collect().await?.into_bytes().as_ref(), b"report");
opa.expect_bucket_tags("s3:GetObjectVersion", source, Some("finance")).await?;
let error = client
.get_object()
.bucket(destination)
@@ -114,10 +114,10 @@ fn staged_commit(metadata: &Value, location: &str) -> Value {
})
}
async fn location_claim_preserves_other_users_objects_after_drop_and_restart(backing: &str) -> TestResult {
async fn location_claim_preserves_other_users_objects_after_drop_and_restart() -> TestResult {
init_logging();
let mut env = RustFSTestEnvironment::new().await?;
let server_env = [("RUSTFS_TABLE_CATALOG_BACKING", backing)];
let server_env = [("RUSTFS_TABLE_CATALOG_BACKING", "object")];
env.start_rustfs_server_without_cleanup_with_env(&[("RUSTFS_TABLE_CATALOG_BACKING", "object")])
.await?;
env.wait_for_server_ready().await?;
@@ -147,39 +147,6 @@ async fn location_claim_preserves_other_users_objects_after_drop_and_restart(bac
StatusCode::OK,
)
.await?;
if backing == "durable-strong" {
catalog_request(
&env,
&root,
Method::POST,
&format!("/iceberg/v1/{bucket}/catalog/warehouse-index/backfill"),
None,
StatusCode::NO_CONTENT,
)
.await?;
let preflight = catalog_request(
&env,
&root,
Method::GET,
&format!("/iceberg/v1/{bucket}/catalog/migration"),
None,
StatusCode::OK,
)
.await?;
assert_eq!(preflight["blockers"], json!([]), "{preflight}");
let migration = catalog_request(
&env,
&root,
Method::POST,
&format!("/iceberg/v1/{bucket}/catalog/migration"),
None,
StatusCode::OK,
)
.await?;
assert_eq!(migration["ready_to_enable_durable_strong"], json!(true), "{migration}");
env.restart_server_preserving_data(vec![], &server_env).await?;
env.wait_for_server_ready().await?;
}
admin_create_user(&env, &creator.0, &creator.1).await?;
admin_create_user(&env, &reader.0, &reader.1).await?;
let create_statement = json!({
@@ -341,7 +308,7 @@ async fn location_claim_preserves_other_users_objects_after_drop_and_restart(bac
StatusCode::FORBIDDEN,
)
.await?;
if backing == "object" {
{
catalog_request(
&env,
&creator,
@@ -408,10 +375,5 @@ async fn location_claim_preserves_other_users_objects_after_drop_and_restart(bac
#[tokio::test]
async fn object_catalog_location_claim_preserves_objects_after_drop_and_restart() -> TestResult {
location_claim_preserves_other_users_objects_after_drop_and_restart("object").await
}
#[tokio::test]
async fn durable_strong_catalog_location_claim_preserves_objects_after_drop_and_restart() -> TestResult {
location_claim_preserves_other_users_objects_after_drop_and_restart("durable-strong").await
location_claim_preserves_other_users_objects_after_drop_and_restart().await
}
@@ -48,6 +48,7 @@ const SOURCE_BINARY_ENV: &str = "RUSTFS_UPGRADE_SOURCE_BINARY";
const G09_EVIDENCE_DIR_ENV: &str = "RUSTFS_SCANNER_HEAL_G09_EVIDENCE_DIR";
const RC5_COMMIT: &str = "40a2470feb567201165a5b809b7598bb4b1f68f5";
const STABLE_1_0_1_COMMIT: &str = "6de965ae3c965a78ff819fbcd7acd4aa44177d92";
const STABLE_1_0_4_COMMIT: &str = "46337942d55f3a8e6a97b147ca98d9ea8115555d";
const SSE_MASTER_KEY_ENV: &str = "RUSTFS_SSE_S3_MASTER_KEY";
const SSE_MASTER_KEY: &str = "QkJCQkJCQkJCQkJCQkJCQkJCQkJCQkJCQkJCQkJCQkI=";
const PLAIN_BUCKET: &str = "upgrade-plain-data";
@@ -954,7 +955,8 @@ async fn exercise_mixed_cluster(
phase: &str,
current_node: usize,
previous_node: usize,
) -> TestResult {
previous_has_delete_recovery: bool,
) -> Result<Option<String>, BoxError> {
let clients = cluster.create_all_clients()?;
wait_for_upgrade_write_readiness(&clients, MIXED_BUCKET, phase, LISTING_CONVERGENCE_TIMEOUT).await?;
wait_for_drive_readmission(&clients, MIXED_BUCKET, phase, DRIVE_READMISSION_TIMEOUT).await?;
@@ -1009,6 +1011,30 @@ async fn exercise_mixed_cluster(
vec![u8::try_from(MULTIPART_WORKERS - 1)?; 64 * 1024]
);
// 1.0.4 already implements the delete decision protocol. Use a separate
// old-writer key so the original seed still spans the entire rollout.
if previous_has_delete_recovery {
let key = format!(".upgrade-delete/{phase}/previous-writer");
previous_client
.put_object()
.bucket(MIXED_BUCKET)
.key(&key)
.body(ByteStream::from_static(UPGRADE_READINESS_BODY))
.send()
.await?;
for client in &clients {
assert_eq!(read_object(client, MIXED_BUCKET, &key, None).await?.1, UPGRADE_READINESS_BODY);
}
upgrade_probe_client(current_client)
.delete_object()
.bucket(MIXED_BUCKET)
.key(&key)
.send()
.await?;
assert_mixed_deletes_visible(&clients, std::slice::from_ref(&key)).await?;
return Ok(Some(key));
}
// Every online participant must understand the delete decision protocol.
// An upgrade phase can serve reads/writes while intentionally refusing a
// modern DELETE; rejecting it must leave the old writer's seed intact.
@@ -1029,6 +1055,31 @@ async fn exercise_mixed_cluster(
);
}
Ok(None)
}
async fn assert_mixed_deletes_visible(clients: &[Client], keys: &[String]) -> TestResult {
for (node, client) in clients.iter().enumerate() {
for key in keys {
let error = upgrade_probe_client(client)
.get_object()
.bucket(MIXED_BUCKET)
.key(key)
.send()
.await
.expect_err("a successful mixed-version DELETE must remain visible through every reader");
assert_eq!(
error.raw_response().map(|response| response.status().as_u16()),
Some(404),
"node {node}: {key}"
);
assert_eq!(
error.as_service_error().and_then(ProvideErrorMetadata::code),
Some("NoSuchKey"),
"node {node}: {key}"
);
}
}
Ok(())
}
@@ -2240,6 +2291,9 @@ async fn direct_upgrade_from_previous_release_preserves_object_contracts() -> Te
async fn rolling_upgrade_from_previous_release_preserves_mixed_version_contracts() -> TestResult {
init_logging();
let previous_binary = source_binary()?;
// Bind the expected protocol to the verified release source, never to the
// DELETE result itself. The older CI source pins retain their 503 contract.
let previous_has_delete_recovery = binary_source_revision(&previous_binary).await? == STABLE_1_0_4_COMMIT;
let evidence_context = g09_evidence_context(&previous_binary).await?;
let current_binary = rustfs_binary_path();
let mut cluster = RustFSTestClusterEnvironment::new(MIXED_NODE_COUNT).await?;
@@ -2260,7 +2314,10 @@ async fn rolling_upgrade_from_previous_release_preserves_mixed_version_contracts
cluster.stop_node(0)?;
cluster.start_node_from_binary(0, &current_binary).await?;
exercise_mixed_cluster(&cluster, "one-current-node", 0, 1).await?;
let mut mixed_deleted_keys = Vec::new();
if let Some(key) = exercise_mixed_cluster(&cluster, "one-current-node", 0, 1, previous_has_delete_recovery).await? {
mixed_deleted_keys.push(key);
}
// One controlled out-of-policy attempt proves fail-closed behavior below
// native write quorum; positive MPU writers remain paused while mixed.
assert_mixed_pending_refusal(&cluster, &pending, &sources).await?;
@@ -2268,7 +2325,9 @@ async fn rolling_upgrade_from_previous_release_preserves_mixed_version_contracts
for (node_idx, phase) in [(1, "two-current-nodes"), (2, "one-previous-node")] {
cluster.stop_node(node_idx)?;
cluster.start_node_from_binary(node_idx, &current_binary).await?;
exercise_mixed_cluster(&cluster, phase, 0, 3).await?;
if let Some(key) = exercise_mixed_cluster(&cluster, phase, 0, 3, previous_has_delete_recovery).await? {
mixed_deleted_keys.push(key);
}
}
cluster.stop_node(3)?;
@@ -2278,6 +2337,7 @@ async fn rolling_upgrade_from_previous_release_preserves_mixed_version_contracts
let completed_pending = finish_pending_multipart(&clients, &pending, pending_parts.concat()).await?;
exercise_homogeneous_multipart(&cluster, "homogeneous-current").await?;
verify_load_bodies(&clients, "homogeneous-previous", "multipart").await?;
assert_mixed_deletes_visible(&clients, &mixed_deleted_keys).await?;
for (node_idx, client) in clients.iter().enumerate() {
assert_eq!(
@@ -2326,6 +2386,7 @@ async fn rolling_upgrade_from_previous_release_preserves_mixed_version_contracts
cluster.start_with_binary(&previous_binary).await?;
prepare_release_baseline(&mut cluster, &previous_binary, "rollback").await?;
let old_clients = cluster.create_all_clients()?;
assert_mixed_deletes_visible(&old_clients, &mixed_deleted_keys).await?;
assert_resumed_multipart(&old_clients, &completed_pending).await?;
for phase in ["homogeneous-previous", "homogeneous-current"] {
verify_load_bodies(&old_clients, phase, "multipart").await?;
@@ -2390,6 +2451,8 @@ async fn rolling_upgrade_from_previous_release_preserves_mixed_version_contracts
"phases": phases,
"objects_per_phase": objects_per_phase,
"mixed_load": "plain-put",
"mixed_delete_protocol_supported": previous_has_delete_recovery,
"mixed_deleted_keys": mixed_deleted_keys,
"homogeneous_multipart_phases": ["homogeneous-previous", "homogeneous-current", "homogeneous-rollback"],
"multipart_objects_per_homogeneous_phase": MULTIPART_WORKERS * MULTIPART_UPLOADS_PER_WORKER,
"assertions": [
@@ -2422,7 +2485,11 @@ async fn rolling_upgrade_from_previous_release_preserves_mixed_version_contracts
"16x16 multipart load runs in homogeneous previous, current and rollback phases",
"one-current-node Complete returns typed 503 and preserves actual source authority and parts",
"the same legacy upload completes and replays once fully upgraded without adding a version",
"modern DELETE fails closed and preserves every byte while old peers remain"
if previous_has_delete_recovery {
"mixed-version DELETE succeeds and stays absent through every reader after upgrade and rollback"
} else {
"modern DELETE fails closed and preserves every byte while old peers remain"
}
],
}),
)?;
+5
View File
@@ -31,6 +31,7 @@ workspace = true
[features]
default = []
fips = ["rustfs-s3-client/fips", "rustfs-utils/fips", "rustfs-tls-runtime/fips"]
diagnostic-read-version-caller = ["rustfs-io-metrics/diagnostic-read-version-caller"]
# Sampled wall/CPU diagnostics only; does not change the read backend.
diagnostic-local-batch = ["rustix/time"]
@@ -120,6 +121,7 @@ test-util = []
e2e-test-hooks = []
[dependencies]
hashbrown.workspace = true
hotpath.workspace = true
rustfs-filemeta.workspace = true
rustfs-utils = { workspace = true, features = ["full"] }
@@ -202,6 +204,9 @@ libc.workspace = true
# (backlog#1178); "thread" provides cleanup affinity; "fs" is a workspace default.
rustix = { workspace = true, features = ["process", "fs", "thread"] }
rustfs-madmin.workspace = true
rustfs-signer.workspace = true
rustfs-tls-runtime.workspace = true
quick-xml.workspace = true
reqwest = { workspace = true }
aws-sdk-s3 = { workspace = true, default-features = false, features = ["sigv4a", "default-https-client", "rt-tokio"] }
urlencoding = { workspace = true }
+43 -27
View File
@@ -191,7 +191,11 @@ pub mod bucket {
OBJECT_LOCK_CONFIG, UnreadableBucketConfig, is_unreadable_config_error, load_bucket_metadata,
table_catalog_path_hash, unreadable_config_refusal,
};
pub use crate::bucket::metadata::{BUCKET_DURABILITY_CONFIG, BUCKET_ON_DEMAND_MIGRATION_CONFIG};
pub use crate::bucket::metadata::{
BUCKET_DURABILITY_CONFIG, BUCKET_ON_DEMAND_MIGRATION_CONFIG, MAX_SWIFT_METADATA_COUNT, MAX_SWIFT_METADATA_VALUE_SIZE,
SWIFT_ACCOUNT_META_TAG_PREFIX, SWIFT_CONTAINER_META_TAG_PREFIX, SWIFT_TAGGING_MARKER,
};
pub use crate::bucket::metadata::{MAX_BUCKET_TAGS, is_swift_bucket_metadata_tag_key, validate_tag_set_entries};
}
pub mod config_parse_mode {
@@ -209,19 +213,20 @@ pub mod bucket {
pub mod metadata_sys {
pub use crate::bucket::metadata_sys::{
BUCKET_CONFIG_PUBLISH_HOOK, BucketConfigPublishHook, BucketMetadataMutationGuard, BucketMetadataSys,
ObjectLockConfigState, acquire_bucket_metadata_transaction_lock,
ObjectLockConfigState, ScannerBucketIncarnationReadGuard, acquire_bucket_metadata_transaction_lock,
acquire_bucket_metadata_transaction_lock_for_incarnation, acquire_scanner_bucket_incarnation_fence,
capture_bucket_metadata_incarnation, delete, delete_if_incarnation, delete_if_incarnation_at,
delete_under_transaction_lock, get, get_accelerate_config, get_bucket_policy, get_bucket_policy_raw,
get_bucket_targets_config, get_config_from_disk, get_cors_config, get_durability_config,
acquire_scanner_bucket_incarnation_read_fence, capture_bucket_metadata_incarnation, delete, delete_if_incarnation,
delete_if_incarnation_at, delete_under_transaction_lock, get, get_accelerate_config, get_bucket_policy,
get_bucket_policy_raw, get_bucket_targets_config, get_config_from_disk, get_cors_config, get_durability_config,
get_global_bucket_metadata_sys, get_lifecycle_config, get_logging_config, get_notification_config,
get_object_lock_config, get_object_lock_config_state, get_on_demand_migration_config,
get_on_demand_migration_config_in, get_public_access_block_config, get_quota_config, get_replication_config,
get_request_payment_config, get_sse_config, get_tagging_config, get_versioning_config, get_website_config,
init_bucket_metadata_sys, list_bucket_targets, reload_bucket_metadata, remove_bucket_metadata,
get_request_payment_config, get_s3_tagging_config, get_sse_config, get_tagging_config, get_versioning_config,
get_website_config, init_bucket_metadata_sys, list_bucket_targets, reload_bucket_metadata, remove_bucket_metadata,
remove_bucket_metadata_if_absent, set_bucket_metadata, update, update_bucket_targets_under_transaction_lock,
update_config_with, update_if_incarnation, update_if_incarnation_at, update_quota_if_incarnation,
update_quota_if_incarnation_at, update_under_transaction_lock, update_under_transaction_lock_at,
update_quota_if_incarnation_at, update_s3_tagging_config, update_under_transaction_lock,
update_under_transaction_lock_at,
};
#[cfg(feature = "test-util")]
pub use crate::bucket::metadata_sys::{ConfigWriteLockProbe, test_support};
@@ -331,8 +336,8 @@ pub mod bucket {
pub mod utils {
pub use crate::bucket::utils::{
check_bucket_and_object_names, check_list_objs_args, check_object_name_for_length_and_slash,
check_valid_bucket_name_strict, deserialize, has_bad_path_component, is_meta_bucketname, is_valid_object_prefix,
serialize,
check_valid_bucket_name_strict, deserialize, has_bad_path_component, has_xml_invalid_control_characters,
is_meta_bucketname, is_valid_object_prefix, serialize,
};
}
@@ -449,11 +454,18 @@ pub mod data_usage {
pub mod disk {
pub use crate::cluster::rpc::background_pipe::{BackgroundPipeReader, BackgroundPipeWriter, background_pipe};
pub use crate::cluster::rpc::background_reader_stream::BackgroundReaderStream;
pub use crate::disk::BOUNDED_DELETE_PREPARE_PROTOCOL_VERSION;
pub use crate::disk::background_memory::BackgroundEncodeBuffer;
pub use crate::disk::disk_store::get_object_disk_read_timeout;
#[cfg(feature = "test-util")]
pub use crate::disk::disk_store::wait_for_multipart_index_bootstrap;
pub use crate::disk::local::{ReplacementExecutionLease, ScanGuard};
#[cfg(feature = "test-util")]
pub use crate::disk::local::{local_durability_modes, set_local_rename_data_fail_after_metadata_commit};
pub use crate::disk::multipart_index::{
MULTIPART_INDEX_RPC_MAX_BYTES, MULTIPART_INDEX_RPC_MAX_REQUEST_BYTES, MultipartIndexPage, MultipartIndexRpcReply,
MultipartIndexRpcRequest,
};
pub use crate::disk::multipart_source::{
MULTIPART_SOURCE_PROTOCOL_VERSION, MultipartAbortCertificate, MultipartAbortProof, MultipartAbortProofKind,
MultipartAbortRound, MultipartSourceOperation, MultipartSourcePhase, MultipartSourceRequest, MultipartSourceResponse,
@@ -466,13 +478,15 @@ pub mod disk {
pub use crate::disk::os::{LocalPublicationPause, LocalPublicationStage};
pub use crate::disk::{
BATCH_READ_VERSION_MAX_ITEMS, BUCKET_META_PREFIX, BatchReadVersionItem, BatchReadVersionReq, BatchReadVersionResp,
CheckPartsResp, ConditionalFileUpdate, CurrentDeleteGeneration, CurrentDeleteTransition,
BoundedDeleteRecoveryDiscovery, BoundedDeleteRecoveryInspection, BoundedDeleteRecoverySettlement,
BoundedDeleteRecoverySettlementReply, CheckPartsResp, ConditionalFileUpdate, CurrentDeleteGeneration,
CurrentDeleteTransition, DELETE_RECOVERY_MAX_DISCOVERY_ENTRIES, DELETE_RECOVERY_MAX_INSPECTION_ENTRIES,
DELETE_RECOVERY_PROTOCOL_VERSION, DeleteOptions, DeleteRecoveryAction, DeleteRecoveryCursor, DeleteRecoveryDigest,
DeleteRecoveryInspection, DeleteRecoveryPage, DeleteRecoveryRpcOperation, DeleteRecoveryRpcReply, DeleteRecoveryState,
Disk, DiskAPI, DiskInfo, DiskInfoOptions, DiskLocation, DiskOption, DiskStore, FileInfoVersions, FileReader, FileWriter,
GuardedDeleteVersionsPayload, GuardedFileInfoVersions, HEALING_MARKER_PATH, NsScannerOpenRequest, OldCurrentSize,
POOL_META_NATIVE_CAS_METADATA_SUFFIX, PartTransactionAction, PoolMetaNativeBackup, PoolMetaNativeInspection,
PoolMetaNativeObject, PoolMetaNativeRecoveryRequest, PoolMetaNativeRecoveryResponse,
GuardedDeleteVersionsPayload, GuardedFileInfoVersions, HEAL_REPLACEMENT_RECOVERY_DIR, HEALING_MARKER_PATH,
NsScannerOpenRequest, OldCurrentSize, POOL_META_NATIVE_CAS_METADATA_SUFFIX, PartTransactionAction, PoolMetaNativeBackup,
PoolMetaNativeInspection, PoolMetaNativeObject, PoolMetaNativeRecoveryRequest, PoolMetaNativeRecoveryResponse,
QUOTA_MUTATION_FENCE_METADATA_SUFFIX, RUSTFS_META_BUCKET, RUSTFS_META_MULTIPART_BUCKET, RUSTFS_META_TMP_BUCKET,
ReadMultipleReq, ReadMultipleResp, ReadOptions, RenameDataResp, STORAGE_FORMAT_FILE, SnapshotLeaseToken,
UpdateMetadataOpts, VolumeInfo, WalkDirOptions, is_quota_mutation_fence_path, new_disk,
@@ -617,7 +631,7 @@ pub mod rebalance {
pub mod rio {
pub use crate::io_support::rio::{
DecryptReader, DynReader, EncryptReader, HardLimitReader, HashReader, ReadStream, Reader, WriteEncryption, WritePlan,
boxed_reader, compression_metadata_value, wrap_reader,
boxed_reader, compression_metadata_value, validate_compression_scheme, wrap_reader,
};
}
@@ -626,17 +640,17 @@ pub mod rpc {
pub use crate::cluster::rpc::{
AuthenticatedChannel, KMS_SIGNAL_SUBSYSTEM, LocalPeerS3Client, MAX_NETWORK_PROBE_BYTES, MAX_NETWORK_PROBE_DURATION,
NetworkPeerProbeClient, NetworkPeerProbeError, NetworkPeerProbeMeasurement, NetworkPeerTarget, PEER_RESTDRY_RUN,
PEER_RESTSIGNAL, PEER_RESTSUB_SYS, PeerRestClient, PeerS3Client, S3PeerSys, SERVICE_SIGNAL_REFRESH_CONFIG,
SERVICE_SIGNAL_RELOAD_DYNAMIC, ScannerBucketListing, ScannerDirtyUsageAcknowledgement, ScannerPeerActivity,
ScannerPeerDirtyUsageBucket, ScannerPeerDirtyUsageSnapshot, ScannerPublicationLease, ScannerScopedDirtyUsageAckEntry,
TONIC_RPC_PREFIX, TonicInterceptor, build_put_file_auth_trailer, check_and_record_signed_rpc_nonce,
decode_heal_bucket_rpc_options, encode_heal_bucket_rpc_options, gen_signature_headers, gen_tonic_replay_scope_headers,
gen_tonic_signature_headers, gen_tonic_signature_interceptor, node_service_time_out_client,
node_service_time_out_client_no_auth, normalize_tonic_rpc_audience, rpc_background_admission,
set_tonic_canonical_body_digest, sign_ns_scanner_capability, sign_ns_scanner_capability_with_checkpoint_coverage,
sign_ns_scanner_capability_with_tier_registry_generation, sign_put_file_capability, sign_tonic_rpc_response_proof,
tonic_boot_epoch_challenge, tonic_boot_epoch_response_headers, tonic_rpc_auth_failure_reason,
verify_ns_scanner_capability, verify_ns_scanner_capability_with_checkpoint_coverage,
PEER_RESTSIGNAL, PEER_RESTSUB_SYS, PeerRestClient, PeerS3Client, RpcReplayCacheAdmissionError, S3PeerSys,
SERVICE_SIGNAL_REFRESH_CONFIG, SERVICE_SIGNAL_RELOAD_DYNAMIC, ScannerBucketListing, ScannerDirtyUsageAcknowledgement,
ScannerPeerActivity, ScannerPeerDirtyUsageBucket, ScannerPeerDirtyUsageSnapshot, ScannerPublicationLease,
ScannerScopedDirtyUsageAckEntry, TONIC_RPC_PREFIX, TonicInterceptor, build_put_file_auth_trailer,
check_and_record_signed_rpc_nonce, decode_heal_bucket_rpc_options, encode_heal_bucket_rpc_options, gen_signature_headers,
gen_tonic_replay_scope_headers, gen_tonic_signature_headers, gen_tonic_signature_interceptor,
node_service_time_out_client, node_service_time_out_client_no_auth, normalize_tonic_rpc_audience,
rpc_auth_is_resource_exhausted, rpc_background_admission, set_tonic_canonical_body_digest, sign_ns_scanner_capability,
sign_ns_scanner_capability_with_checkpoint_coverage, sign_ns_scanner_capability_with_tier_registry_generation,
sign_put_file_capability, sign_tonic_rpc_response_proof, tonic_boot_epoch_challenge, tonic_boot_epoch_response_headers,
tonic_rpc_auth_failure_reason, verify_ns_scanner_capability, verify_ns_scanner_capability_with_checkpoint_coverage,
verify_ns_scanner_capability_with_tier_registry_generation, verify_put_file_auth_trailer, verify_put_file_capability,
verify_rpc_signature, verify_tonic_boot_epoch_response, verify_tonic_canonical_body_digest,
verify_tonic_mutation_body_digest, verify_tonic_mutation_body_digest_reject_unsigned, verify_tonic_rpc_response_proof,
@@ -654,7 +668,9 @@ pub mod set_disk {
#[cfg(feature = "test-util")]
pub mod test_util {
pub use crate::bucket::quota::reservation::fail_next_quota_ledger_save_for_test;
pub use crate::bucket::quota::reservation::{
fail_next_quota_ledger_save_for_test, initialize_quota_ledger_for_test, retry_quota_reconciliation_for_test,
};
pub use crate::set_disk::ConditionalReadBenchmarkMetadataGuard;
pub use crate::set_disk::{MultipartCommitBarrier, MultipartCommitPause, PutObjectCommitBarrier, PutObjectCommitPause};
+325 -5
View File
@@ -16,7 +16,8 @@ use crate::bucket::metadata::BucketMetadata;
use crate::bucket::metadata_sys::get_bucket_targets_config;
use crate::bucket::metadata_sys::get_replication_config;
use crate::bucket::remote_s3_client::{
PathStyle, REPLICATION_TARGET_RETRY_POLICY, RemoteCredentials, RemoteS3EndpointSpec, build_remote_s3_client,
PathStyle, REPLICATION_TARGET_RETRY_POLICY, RemoteCredentials, RemoteS3EndpointSpec, RemoteS3RetryPolicy,
UploadResponseDeadline, build_remote_s3_client,
};
use crate::bucket::replication::{ObjectLockIntegrity, object_lock_put_integrity, replication_etags_match};
use crate::bucket::replication::{ReplicationStatusType, ReplicationTargetConfigBridge};
@@ -119,6 +120,7 @@ impl From<&BucketTarget> for RemoteS3EndpointSpec {
ca_cert_pem: (!target.ca_cert_pem.trim().is_empty()).then(|| target.ca_cert_pem.clone()),
connect_timeout: None,
read_timeout: None,
operation_timeout: Some(TargetRequestTimeouts::from_env().request),
// Replication has no retry budget of its own on the request path,
// so it keeps the SDK's standard three attempts; stating it here
// pins the behaviour to this line instead of an SDK default.
@@ -1191,6 +1193,7 @@ impl BucketTargetSys {
health_check_duration: target.health_check_duration,
replicate_sync: target.replication_sync,
client: Arc::new(client),
timeouts: TargetRequestTimeouts::from_env(),
})
}
@@ -1922,6 +1925,83 @@ impl std::fmt::Display for S3ClientError {
}
}
/// Replication request budgets, snapshotted when a target client is built.
/// Streamed GET bodies are not subject to the upload-response budget.
#[derive(Debug, Clone, Copy)]
pub struct TargetRequestTimeouts {
request: Duration,
complete: Duration,
minimum_transfer_rate: u64,
}
impl Default for TargetRequestTimeouts {
fn default() -> Self {
Self {
request: Duration::from_secs(60),
complete: Duration::from_secs(15 * 60),
minimum_transfer_rate: 1024 * 1024,
}
}
}
impl TargetRequestTimeouts {
fn from_env() -> Self {
let defaults = Self::default();
let seconds = |name: &str, fallback: Duration| {
let value = rustfs_utils::get_env_u64(name, fallback.as_secs());
Duration::from_secs(if (1..=86400).contains(&value) {
value
} else {
fallback.as_secs()
})
};
let minimum_transfer_rate =
rustfs_utils::get_env_u64("RUSTFS_REPLICATION_MIN_TRANSFER_RATE_BYTES_PER_SEC", defaults.minimum_transfer_rate);
Self {
request: seconds("RUSTFS_REPLICATION_REQUEST_TIMEOUT_SECS", defaults.request),
complete: seconds("RUSTFS_REPLICATION_COMPLETE_TIMEOUT_SECS", defaults.complete),
minimum_transfer_rate: if minimum_transfer_rate == 0 {
defaults.minimum_transfer_rate
} else {
minimum_transfer_rate
},
}
}
fn upload_timeout(self, size: i64) -> Result<Duration, S3ClientError> {
let size = u64::try_from(size).map_err(|_| S3ClientError::new("replication upload size must be non-negative"))?;
// The SDK can make three attempts. Leave transmission headroom for
// all three; the separate response deadline still bounds a trickling
// target after each upload finishes.
let attempts = match REPLICATION_TARGET_RETRY_POLICY {
RemoteS3RetryPolicy::Disabled => 1,
RemoteS3RetryPolicy::Standard { max_attempts } => u64::from(max_attempts.max(1)),
};
let seconds = size
.div_ceil(self.minimum_transfer_rate)
.checked_mul(attempts)
.ok_or_else(|| S3ClientError::new("replication upload timeout exceeds supported duration"))?;
let timeout = self
.request
.checked_add(Duration::from_secs(seconds))
.ok_or_else(|| S3ClientError::new("replication upload timeout exceeds supported duration"))?;
if Instant::now().checked_add(timeout).is_none() {
return Err(S3ClientError::new("replication upload timeout exceeds supported duration"));
}
Ok(timeout)
}
fn upload_override(self, size: i64) -> Result<aws_sdk_s3::config::Builder, S3ClientError> {
Ok(aws_sdk_s3::config::Builder::new()
.timeout_config(
aws_smithy_types::timeout::TimeoutConfig::builder()
.operation_timeout(self.upload_timeout(size)?)
.build(),
)
.interceptor(UploadResponseDeadline(self.request)))
}
}
#[derive(Debug)]
pub struct TargetClient {
pub endpoint: String,
@@ -1935,6 +2015,7 @@ pub struct TargetClient {
pub health_check_duration: Duration,
pub replicate_sync: bool,
pub client: Arc<S3Client>,
pub timeouts: TargetRequestTimeouts,
}
impl TargetClient {
@@ -2360,12 +2441,11 @@ impl TargetClient {
Result::<_, aws_smithy_types::error::operation::BuildError>::Ok(req)
});
let mut config = self.timeouts.upload_override(size)?;
if forwards_source_checksum {
operation = operation.config_override(
aws_sdk_s3::config::Builder::new()
.request_checksum_calculation(aws_sdk_s3::config::RequestChecksumCalculation::WhenRequired),
);
config = config.request_checksum_calculation(aws_sdk_s3::config::RequestChecksumCalculation::WhenRequired);
}
operation = operation.config_override(config);
match operation.send().await {
Ok(output) => {
// Under SSE-KMS/DSSE or SSE-C the target's ETag is not the MD5
@@ -2501,6 +2581,7 @@ impl TargetClient {
}
Result::<_, aws_smithy_types::error::operation::BuildError>::Ok(req)
})
.config_override(self.timeouts.upload_override(size)?)
.send()
.await
{
@@ -2538,6 +2619,13 @@ impl TargetClient {
}
Result::<_, aws_smithy_types::error::operation::BuildError>::Ok(req)
})
.config_override(
aws_sdk_s3::config::Builder::new().timeout_config(
aws_smithy_types::timeout::TimeoutConfig::builder()
.operation_timeout(self.timeouts.complete)
.build(),
),
)
.send()
.await
{
@@ -2966,11 +3054,241 @@ mod tests {
health_check_duration: Duration::from_secs(5),
replicate_sync: false,
client: Arc::new(client),
timeouts: TargetRequestTimeouts::default(),
},
request_headers,
)
}
#[derive(Debug, Clone)]
struct TimedResponseConnector {
upload_delay: Duration,
response_delay: Duration,
prefix_bytes: usize,
response: bytes::Bytes,
frames: Arc<std::sync::atomic::AtomicUsize>,
dropped: Arc<std::sync::atomic::AtomicUsize>,
}
struct ResponseDrop(Arc<std::sync::atomic::AtomicUsize>);
impl Drop for ResponseDrop {
fn drop(&mut self) {
self.0.fetch_add(1, std::sync::atomic::Ordering::SeqCst);
}
}
impl SmithyHttpConnector for TimedResponseConnector {
fn call(&self, request: HttpRequest) -> HttpConnectorFuture {
let settings = self.clone();
HttpConnectorFuture::new(async move {
http_body_util::BodyExt::collect(request.into_body())
.await
.map_err(aws_smithy_runtime_api::client::result::ConnectorError::io)?;
tokio::time::sleep(settings.upload_delay).await;
let guard = ResponseDrop(Arc::clone(&settings.dropped));
let stream = futures::stream::unfold(
(settings.prefix_bytes, Some(settings.response), true, guard),
move |(remaining, mut body, first, guard)| {
let frames = Arc::clone(&settings.frames);
async move {
if remaining == 0 && body.is_none() {
return None;
}
if !first {
tokio::time::sleep(settings.response_delay).await;
}
let bytes = if remaining > 0 {
bytes::Bytes::from_static(b" ")
} else {
body.take()?
};
frames.fetch_add(1, std::sync::atomic::Ordering::SeqCst);
Some((
Ok::<_, std::io::Error>(http_body::Frame::data(bytes)),
(remaining.saturating_sub(1), body, false, guard),
))
}
},
)
.fuse();
let body = SdkBody::from_body_1_x(http_body_util::StreamBody::new(stream));
let mut response =
HttpResponse::new(aws_smithy_runtime_api::http::StatusCode::try_from(200u16).expect("status"), body);
response.headers_mut().insert("etag", "\"test-etag\"");
Ok(response)
})
}
}
fn timed_response_target(
upload_delay: Duration,
response_delay: Duration,
prefix_bytes: usize,
response: &'static [u8],
) -> (TargetClient, Arc<std::sync::atomic::AtomicUsize>, Arc<std::sync::atomic::AtomicUsize>) {
let frames = Arc::new(std::sync::atomic::AtomicUsize::new(0));
let dropped = Arc::new(std::sync::atomic::AtomicUsize::new(0));
let connector = SharedHttpConnector::new(TimedResponseConnector {
upload_delay,
response_delay,
prefix_bytes,
response: bytes::Bytes::from_static(response),
frames: Arc::clone(&frames),
dropped: Arc::clone(&dropped),
});
let http_client = http_client_fn(move |_settings, _components| connector.clone());
let (mut target, _) = header_recording_target_client(Vec::new());
target.timeouts = TargetRequestTimeouts {
request: Duration::from_secs(2),
complete: Duration::from_secs(10),
minimum_transfer_rate: 1024 * 1024,
};
target.client = Arc::new(S3Client::from_conf(
target
.client
.config()
.to_builder()
.http_client(http_client)
.retry_config(aws_sdk_s3::config::retry::RetryConfig::disabled())
.timeout_config(
aws_smithy_types::timeout::TimeoutConfig::builder()
.operation_timeout(target.timeouts.request)
.build(),
)
.build(),
));
(target, frames, dropped)
}
#[test]
fn replication_upload_budgets_preserve_large_transfers_and_reject_invalid_sizes() {
let policy = TargetRequestTimeouts::default();
assert_eq!(policy.upload_timeout(0).expect("empty upload"), Duration::from_secs(60));
assert_eq!(
policy.upload_timeout(5 * 1024 * 1024 * 1024).expect("maximum single PUT"),
Duration::from_secs(15420)
);
assert!(policy.upload_timeout(-1).is_err());
let slow = TargetRequestTimeouts {
minimum_transfer_rate: 1,
..policy
};
assert!(slow.upload_timeout(i64::MAX).is_err());
}
#[tokio::test(start_paused = true)]
async fn replication_upload_response_deadline_is_independent_of_size_and_checksum_override() {
for operation in ["put", "forwarded-checksum", "part"] {
let (target, frames, dropped) = timed_response_target(Duration::from_secs(4), Duration::from_millis(500), 100, b"");
let body = vec![b'x'; 2 * 1024 * 1024];
let size = i64::try_from(body.len()).expect("size");
let started = tokio::time::Instant::now();
let error = if operation == "part" {
target
.put_object_part(
"bucket",
"key",
"upload",
1,
size,
ByteStream::from(body),
&PutObjectPartOptions::default(),
)
.await
.expect_err("response deadline")
} else {
let mut opts = PutObjectOptions::default();
if operation == "forwarded-checksum" {
opts.user_metadata.insert(
"x-amz-checksum-sha256".to_owned(),
"OoJ3yNhRwv3wwtZoGqEIPrPX9xwTnfLl+ka0wStN1g0=".to_owned(),
);
}
target
.put_object("bucket", "key", size, ByteStream::from(body), &opts)
.await
.expect_err("response deadline")
};
assert!(
started.elapsed() >= Duration::from_secs(6),
"upload gets its full independent window: {error}"
);
assert!(
started.elapsed() < target.timeouts.upload_timeout(size).expect("budget"),
"response must end before the larger total deadline"
);
assert!(
frames.load(std::sync::atomic::Ordering::SeqCst) >= 2,
"trickle delivered progress before failure"
);
assert_eq!(dropped.load(std::sync::atomic::Ordering::SeqCst), 1, "transport body released");
}
}
#[tokio::test(start_paused = true)]
async fn replication_slow_upload_preserves_transfer_window_and_streaming_get_body() {
let (target, _, _) = timed_response_target(Duration::from_secs(4), Duration::from_millis(100), 5, b"");
let body = vec![b'x'; 2 * 1024 * 1024];
let size = i64::try_from(body.len()).expect("size");
target
.put_object("bucket", "key", size, ByteStream::from(body), &PutObjectOptions::default())
.await
.expect("slow upload, prompt response");
let (target, _, dropped) = timed_response_target(Duration::ZERO, Duration::from_secs(1), 5, b"payload");
let response = target
.get_object("bucket", "key", None, None, None, HeaderMap::new())
.await
.expect("GET headers");
let started = tokio::time::Instant::now();
let body = response
.body
.collect()
.await
.expect("streamed GET remains outside upload deadline")
.into_bytes();
assert!(started.elapsed() > target.timeouts.request);
assert_eq!(&body[..], b" payload");
assert_eq!(dropped.load(std::sync::atomic::Ordering::SeqCst), 1);
}
#[tokio::test(start_paused = true)]
async fn replication_complete_processing_whitespace_has_a_separate_bounded_window() {
const RESULT: &[u8] = b"<CompleteMultipartUploadResult xmlns=\"http://s3.amazonaws.com/doc/2006-03-01/\"><ETag>\"etag\"</ETag></CompleteMultipartUploadResult>";
for prefix in [5, 100] {
let (target, frames, dropped) = timed_response_target(Duration::ZERO, Duration::from_secs(1), prefix, RESULT);
let started = tokio::time::Instant::now();
let result = target
.complete_multipart_upload(
"bucket",
"key",
"upload",
vec![CompletedPart::builder().part_number(1).e_tag("etag").build()],
&PutObjectOptions::default(),
)
.await;
if prefix == 5 {
assert!(result.is_ok(), "legal processing whitespace: {result:?}");
} else {
assert!(result.is_err());
assert!(started.elapsed() <= Duration::from_secs(11));
}
assert!(started.elapsed() > target.timeouts.request);
assert!(frames.load(std::sync::atomic::Ordering::SeqCst) >= 2);
assert_eq!(dropped.load(std::sync::atomic::Ordering::SeqCst), 1);
}
}
#[tokio::test(start_paused = true)]
async fn replication_control_operation_deadline_covers_response_trickle() {
let (target, frames, dropped) =
timed_response_target(Duration::ZERO, Duration::from_millis(500), 100, b"<VersioningConfiguration/>");
let started = tokio::time::Instant::now();
assert!(target.get_bucket_versioning("bucket").await.is_err());
assert!(started.elapsed() <= Duration::from_secs(3));
assert!(frames.load(std::sync::atomic::Ordering::SeqCst) >= 2);
assert_eq!(dropped.load(std::sync::atomic::Ordering::SeqCst), 1);
}
fn streaming_test_body(payload: &'static [u8]) -> ByteStream {
let stream = tokio_util::io::ReaderStream::new(std::io::Cursor::new(payload));
let body = http_body_util::StreamBody::new(futures::StreamExt::map(stream, |r| r.map(http_body::Frame::data)));
@@ -3279,6 +3597,7 @@ mod tests {
health_check_duration: Duration::from_secs(5),
replicate_sync: false,
client: Arc::new(client),
timeouts: TargetRequestTimeouts::default(),
},
request_uris,
)
@@ -3591,6 +3910,7 @@ mod tests {
health_check_duration: Duration::from_secs(5),
replicate_sync: false,
client: Arc::new(client),
timeouts: TargetRequestTimeouts::default(),
})
}
@@ -5487,6 +5487,14 @@ pub async fn put_restore_opts(
if !oi.user_tags.is_empty() {
meta.insert(AMZ_OBJECT_TAGGING.to_string(), (*oi.user_tags).clone());
}
if let Some(expires) = oi.expires {
meta.insert(
"expires".to_string(),
expires
.format(&time::format_description::well_known::Rfc3339)
.map_err(std::io::Error::other)?,
);
}
let restore_expiry = lifecycle::expected_expiry_time(OffsetDateTime::now_utc(), rreq.days.unwrap_or(1));
meta.insert(
metadata_keys::RESTORE.to_string(),
@@ -5506,7 +5514,6 @@ pub async fn put_restore_opts(
// computed MD5 is not the object's public plaintext ETag.
preserve_etag: oi.etag.clone(),
shard_integrity_write_mode: Some(oi.shard_integrity_write_mode()),
//expires: oi.expires,
..Default::default()
})
}
@@ -7144,6 +7151,25 @@ mod tests {
0,
"tier generation lease should release after EOF"
);
let unread = get_transitioned_object_reader_with_tier_manager(
&object_info.bucket,
&object_info.name,
&None,
&HeaderMap::new(),
&object_info,
&ObjectOptions::default(),
&manager,
None,
)
.await
.expect("open an unread transitioned response");
assert_eq!(TierConfigMgr::active_operation_lease_count(&manager, &tier).await, 1);
drop(unread);
assert_eq!(
TierConfigMgr::active_operation_lease_count(&manager, &tier).await,
0,
"canceling an unread response must release the tier lease"
);
}
#[cfg(feature = "test-util")]
+188
View File
@@ -256,6 +256,128 @@ pub const BUCKET_POLICY_CONFIG: &str = "policy.json";
pub const BUCKET_NOTIFICATION_CONFIG: &str = "notification.xml";
pub const BUCKET_LIFECYCLE_CONFIG: &str = "lifecycle.xml";
pub const BUCKET_SSECONFIG: &str = "bucket-encryption.xml";
/// Validate tag entries supplied to the S3 API.
pub fn validate_tag_set_entries(tag_set: &[s3s::dto::Tag]) -> std::result::Result<(), &'static str> {
validate_tag_entries(tag_set, validate_s3_tag_entry)
}
fn validate_tag_entries(
tag_set: &[s3s::dto::Tag],
mut validate_entry: impl FnMut(&str, &str) -> std::result::Result<(), &'static str>,
) -> std::result::Result<(), &'static str> {
let mut keys = std::collections::HashSet::with_capacity(tag_set.len());
for tag in tag_set {
let key = tag
.key
.as_deref()
.filter(|key| !key.is_empty())
.ok_or("Tag key cannot be empty")?;
let value = tag.value.as_deref().ok_or("Tag value cannot be null")?;
validate_entry(key, value)?;
if !keys.insert(key) {
return Err("Cannot provide multiple Tags with the same key");
}
}
Ok(())
}
fn validate_s3_tag_entry(key: &str, value: &str) -> std::result::Result<(), &'static str> {
fn is_valid_tag_text(value: &str) -> bool {
static VALID_TAG_TEXT: std::sync::OnceLock<std::result::Result<regex::Regex, regex::Error>> = std::sync::OnceLock::new();
VALID_TAG_TEXT
.get_or_init(|| regex::Regex::new(r"^[\p{L}\p{N}\p{Z}+\-=._:/@]*$"))
.as_ref()
.is_ok_and(|pattern| pattern.is_match(value))
}
if key.encode_utf16().count() > 128 || !is_valid_tag_text(key) {
return Err("The TagKey you have provided is invalid");
}
if value.encode_utf16().count() > 256 || !is_valid_tag_text(value) {
return Err("The TagValue you have provided is invalid");
}
Ok(())
}
pub const MAX_BUCKET_TAGS: usize = 50;
/// Maximum number of metadata headers on a Swift resource.
pub const MAX_SWIFT_METADATA_COUNT: usize = 90;
/// Maximum byte length of a Swift metadata value.
pub const MAX_SWIFT_METADATA_VALUE_SIZE: usize = 256;
/// Prefix for Swift account metadata persisted in tagging.xml.
pub const SWIFT_ACCOUNT_META_TAG_PREFIX: &str = "swift-account-meta-";
/// Prefix for Swift container metadata persisted in tagging.xml.
pub const SWIFT_CONTAINER_META_TAG_PREFIX: &str = "swift-meta-";
/// Marks a bucket whose shared tag config contains Swift-owned metadata.
/// The key cannot be supplied through S3's tag alphabet.
pub const SWIFT_TAGGING_MARKER: &str = "!rustfs-swift-metadata";
/// Whether a persisted tag belongs to Swift's internal metadata namespace.
pub fn is_swift_bucket_metadata_tag_key(key: &str) -> bool {
matches!(
key,
"swift-acl-read" | "swift-acl-write" | "swift-versions-location" | SWIFT_TAGGING_MARKER
) || key.starts_with(SWIFT_ACCOUNT_META_TAG_PREFIX)
|| key.starts_with(SWIFT_CONTAINER_META_TAG_PREFIX)
}
fn validate_persisted_bucket_tags(tags: &[s3s::dto::Tag]) -> std::result::Result<(), &'static str> {
const SWIFT_ACL_MAX_BYTES: usize = 8192;
const SWIFT_META_NAME_MAX_BYTES: usize = 128;
const MAX_PERSISTED_TAGS: usize = MAX_BUCKET_TAGS + 2 * MAX_SWIFT_METADATA_COUNT + 4;
if tags.len() > MAX_PERSISTED_TAGS {
return Err("Too many persisted bucket tags");
}
// Swift stores ACLs, versioning and metadata in the same XML as S3 tags.
// Only those known keys may carry values outside the S3 tag alphabet.
let mut account_metadata = 0;
let mut container_metadata = 0;
let mut s3_tags = 0;
validate_tag_entries(tags, |key, value| {
let swift_metadata = key
.strip_prefix(SWIFT_ACCOUNT_META_TAG_PREFIX)
.map(|name| (name, &mut account_metadata))
.or_else(|| {
key.strip_prefix(SWIFT_CONTAINER_META_TAG_PREFIX)
.map(|name| (name, &mut container_metadata))
});
if let Some((name, count)) = swift_metadata
&& name.len() <= SWIFT_META_NAME_MAX_BYTES
&& http::HeaderName::from_bytes(name.as_bytes()).is_ok()
&& value.len() <= MAX_SWIFT_METADATA_VALUE_SIZE
&& http::HeaderValue::from_str(value).is_ok()
{
*count += 1;
if *count > MAX_SWIFT_METADATA_COUNT {
return Err("Too many Swift metadata tags");
}
return Ok(());
}
if key == SWIFT_TAGGING_MARKER && value == "1" {
return Ok(());
}
let valid_swift_header = match key {
"swift-acl-read" | "swift-acl-write" => value.len() <= SWIFT_ACL_MAX_BYTES,
"swift-versions-location" => value.len() <= 256 && !value.is_empty() && !value.contains('/'),
_ => false,
};
if valid_swift_header && http::HeaderValue::from_str(value).is_ok() {
return Ok(());
}
s3_tags += 1;
if s3_tags > MAX_BUCKET_TAGS {
return Err("Cannot have more than 50 tags per bucket");
}
validate_s3_tag_entry(key, value)
})
}
pub const BUCKET_TAGGING_CONFIG: &str = "tagging.xml";
pub const BUCKET_QUOTA_CONFIG_FILE: &str = "quota.json";
pub const OBJECT_LOCK_CONFIG: &str = "object-lock.xml";
@@ -990,6 +1112,13 @@ impl BucketMetadata {
self.encryption_config_updated_at = updated;
}
BUCKET_TAGGING_CONFIG => {
if !data.is_empty() {
let tagging = deserialize::<Tagging>(&data).map_err(|_| {
Error::InvalidArgument(self.name.clone(), String::new(), "Invalid bucket tagging XML".to_owned())
})?;
validate_persisted_bucket_tags(&tagging.tag_set)
.map_err(|message| Error::InvalidArgument(self.name.clone(), String::new(), message.to_owned()))?;
}
self.tagging_config_xml = data;
// Drop the parsed form (like lifecycle above) so clearing the
// payload can't leave stale parsed tags to be cached.
@@ -1781,9 +1910,68 @@ mod test {
assert_eq!(metadata.bucket_incarnation_id, incarnation);
}
#[test]
fn persisted_bucket_tags_keep_separate_s3_and_swift_limits() {
use s3s::dto::Tag;
let tag = |key: String, value: &str| Tag {
key: Some(key),
value: Some(value.to_owned()),
};
let acl = tag("swift-acl-read".to_owned(), ".r:*,AUTH_other");
assert_eq!(
validate_tag_set_entries(std::slice::from_ref(&acl)),
Err("The TagValue you have provided is invalid")
);
let mut tags = vec![acl.clone()];
tags.extend((0..MAX_BUCKET_TAGS).map(|i| tag(format!("team-{i}"), "blue")));
tags.extend((0..MAX_SWIFT_METADATA_COUNT).map(|i| tag(format!("{SWIFT_CONTAINER_META_TAG_PREFIX}item!{i}"), "a,b")));
assert_eq!(validate_persisted_bucket_tags(&tags), Ok(()));
tags.push(tag(format!("{SWIFT_CONTAINER_META_TAG_PREFIX}overflow"), "a,b"));
assert_eq!(validate_persisted_bucket_tags(&tags), Err("Too many Swift metadata tags"));
tags.pop();
tags.push(acl);
assert_eq!(
validate_persisted_bucket_tags(&tags),
Err("Cannot provide multiple Tags with the same key")
);
tags.pop();
tags[0].value = Some(".r:*,".repeat(1700));
assert_eq!(validate_persisted_bucket_tags(&tags), Err("The TagValue you have provided is invalid"));
}
/// backlog#2292: a replicated config is stamped with the source
/// `updated_at` it was given, not the local clock, while the plain
/// `update_config` entry keeps stamping the local clock.
#[test]
fn bucket_tag_updates_reject_unreadable_shapes_without_replacing_metadata() {
let mut metadata = BucketMetadata::new("validated-tags");
let valid = br#"<Tagging><TagSet><Tag><Key>team</Key><Value></Value></Tag></TagSet></Tagging>"#.to_vec();
metadata
.update_config(BUCKET_TAGGING_CONFIG, valid.clone())
.expect("empty value is valid");
let updated = metadata.tagging_config_updated_at;
for invalid in [
"<Tagging><TagSet><Tag><Key>team</Key></Tag></TagSet></Tagging>".to_owned(),
"<Tagging><TagSet><Tag><Key></Key><Value>x</Value></Tag></TagSet></Tagging>".to_owned(),
"<Tagging><TagSet><Tag><Key>team</Key><Value>x</Value></Tag><Tag><Key>team</Key><Value>y</Value></Tag></TagSet></Tagging>".to_owned(),
format!("<Tagging><TagSet>{}</TagSet></Tagging>", (0..51).map(|i| format!("<Tag><Key>k{i}</Key><Value>v</Value></Tag>")).collect::<String>()),
] {
assert!(metadata.update_config_at(BUCKET_TAGGING_CONFIG, invalid.into_bytes(), updated + time::Duration::hours(1)).is_err());
assert_eq!(metadata.tagging_config_xml, valid);
assert_eq!(metadata.tagging_config_updated_at, updated);
}
metadata.tagging_config_xml = b"legacy corrupt tags".to_vec();
metadata
.update_config(BUCKET_TAGGING_CONFIG, Vec::new())
.expect("deletion repairs legacy corruption");
assert!(metadata.tagging_config_xml.is_empty());
}
#[test]
fn update_config_at_stamps_the_given_time_and_update_config_stamps_now() {
let source_time = OffsetDateTime::now_utc() - time::Duration::hours(3);
+428 -49
View File
@@ -22,6 +22,7 @@ use super::target::BucketTargets;
use crate::bucket::bucket_target_sys::BucketTargetSys;
use crate::bucket::metadata::{load_bucket_metadata_parse, load_bucket_metadata_parse_with_presence};
use crate::bucket::utils::is_meta_bucketname;
use crate::diagnostics::object_lock::ObjectLockAttempt;
use crate::disk::RUSTFS_META_BUCKET;
use crate::error::{Error, Result, is_err_bucket_not_found, is_err_strict_volume_not_found};
use crate::object_api::ObjectOptions;
@@ -599,7 +600,9 @@ pub(crate) async fn get_bucket_incarnation_id_for_options_in(
}
let sys = bucket_metadata_sys_of(ctx)?;
let sys = sys.read().await.clone();
let guard = acquire_bucket_metadata_transaction_read_lock_for_options_in(ctx, bucket, opts).await?;
let guard =
acquire_bucket_metadata_transaction_read_lock_for_options_in(ctx, bucket, opts, "bucket_incarnation_options_read")
.await?;
sys.get_bucket_incarnation_id_under_transaction_lock(bucket, &guard).await
}
@@ -866,6 +869,30 @@ impl BucketMetadataMutationGuard {
}
}
/// Read-only scanner authority for one bucket incarnation.
///
/// Lock order is bucket lifecycle (read), then metadata transaction (read).
/// This guard cannot authorize a configuration mutation or scanner ACK write.
pub struct ScannerBucketIncarnationReadGuard {
bucket: String,
incarnation_id: Uuid,
lifecycle_guard: rustfs_lock::NamespaceLockGuard,
transaction_guard: rustfs_lock::NamespaceLockGuard,
}
impl ScannerBucketIncarnationReadGuard {
/// Returns the verified identity while both read fences remain valid.
///
/// # Errors
/// Returns an error if either lock has lost its lease.
pub fn checked_bucket_incarnation(&self) -> Result<(&str, Uuid)> {
if self.lifecycle_guard.is_lock_lost() || self.transaction_guard.is_lock_lost() {
return Err(Error::other("scanner bucket incarnation read lock was lost"));
}
Ok((&self.bucket, self.incarnation_id))
}
}
async fn acquire_config_write_guard(sys: Arc<RwLock<BucketMetadataSys>>, bucket: &str) -> Result<BucketMetadataMutationGuard> {
acquire_config_write_guard_for_incarnation(sys, bucket, None).await
}
@@ -892,6 +919,68 @@ pub async fn acquire_scanner_bucket_incarnation_fence(
acquire_config_write_guard_with_migration(sys, bucket, Some(expected_incarnation_id), false).await
}
/// Validate scanner read authority without serializing compatible metadata readers.
///
/// The returned guard retains both read locks. It never migrates metadata or
/// creates an incarnation; mutation callers must use the exclusive scanner fence.
///
/// # Errors
/// Returns an error for invalid bucket/owner identities, unavailable locks or
/// storage, absent incarnation metadata, or a deleted/recreated bucket.
pub async fn acquire_scanner_bucket_incarnation_read_fence(
bucket: &str,
expected_incarnation_id: Uuid,
expected_owner_id: Uuid,
) -> Result<ScannerBucketIncarnationReadGuard> {
acquire_scanner_bucket_incarnation_read_fence_with_sys(
get_bucket_metadata_sys()?,
bucket,
expected_incarnation_id,
expected_owner_id,
)
.await
}
async fn acquire_scanner_bucket_incarnation_read_fence_with_sys(
sys: Arc<RwLock<BucketMetadataSys>>,
bucket: &str,
expected_incarnation_id: Uuid,
expected_owner_id: Uuid,
) -> Result<ScannerBucketIncarnationReadGuard> {
super::utils::check_valid_bucket_name(bucket)?;
let metadata_sys = sys.read().await.clone();
if expected_owner_id.is_nil() || metadata_sys.object_store().id != expected_owner_id || expected_incarnation_id.is_nil() {
return Err(Error::other("scanner bucket incarnation owner does not match"));
}
let lifecycle_guard = metadata_sys.object_store().acquire_bucket_lifecycle_read_lock(bucket).await?;
let transaction_guard = await_bucket_namespace_operation(
Some(&lifecycle_guard),
bucket,
"scanner incarnation read transaction lock acquisition",
acquire_metadata_transaction_guard(
metadata_sys.object_store(),
bucket,
"scanner_incarnation_probe",
rustfs_lock::LockType::Shared,
),
)
.await?;
let incarnation_id = validate_bucket_metadata_guard_incarnation(
&metadata_sys,
bucket,
Some(expected_incarnation_id),
&lifecycle_guard,
&transaction_guard,
)
.await?;
Ok(ScannerBucketIncarnationReadGuard {
bucket: bucket.to_owned(),
incarnation_id,
lifecycle_guard,
transaction_guard,
})
}
async fn acquire_config_write_guard_with_migration(
sys: Arc<RwLock<BucketMetadataSys>>,
bucket: &str,
@@ -917,15 +1006,47 @@ async fn acquire_config_write_guard_with_migration(
Some(&lifecycle_guard),
bucket,
"bucket config transaction lock acquisition",
acquire_transaction_lock_with_sys(&sys, bucket),
acquire_transaction_lock_with_sys_for_purpose(
&sys,
bucket,
if migrate {
"bucket_config_update"
} else {
"scanner_incarnation_validation"
},
),
)
.await?;
let current_incarnation_id = validate_bucket_metadata_guard_incarnation(
&metadata_sys,
bucket,
expected_incarnation_id,
&lifecycle_guard,
&transaction_guard,
)
.await?;
Ok(BucketMetadataMutationGuard {
bucket: bucket.to_string(),
incarnation_id: current_incarnation_id,
lifecycle_guard,
transaction_guard,
})
}
async fn validate_bucket_metadata_guard_incarnation(
metadata_sys: &BucketMetadataSys,
bucket: &str,
expected_incarnation_id: Option<Uuid>,
lifecycle_guard: &rustfs_lock::NamespaceLockGuard,
transaction_guard: &rustfs_lock::NamespaceLockGuard,
) -> Result<Uuid> {
await_bucket_namespace_operation(
Some(&lifecycle_guard),
Some(lifecycle_guard),
bucket,
"bucket config existence validation",
await_bucket_namespace_operation(
Some(&transaction_guard),
Some(transaction_guard),
bucket,
"bucket config existence transaction validation",
async {
@@ -943,11 +1064,11 @@ async fn acquire_config_write_guard_with_migration(
)
.await?;
let current_incarnation_id = await_bucket_namespace_operation(
Some(&lifecycle_guard),
Some(lifecycle_guard),
bucket,
"bucket config incarnation validation",
await_bucket_namespace_operation(
Some(&transaction_guard),
Some(transaction_guard),
bucket,
"bucket config incarnation transaction validation",
load_bucket_incarnation(metadata_sys.object_store(), bucket),
@@ -960,12 +1081,7 @@ async fn acquire_config_write_guard_with_migration(
return Err(Error::BucketNotFound(bucket.to_string()));
}
Ok(BucketMetadataMutationGuard {
bucket: bucket.to_string(),
incarnation_id: current_incarnation_id,
lifecycle_guard,
transaction_guard,
})
Ok(current_incarnation_id)
}
/// Rewrite one config file while the caller already holds this bucket's
@@ -1174,11 +1290,23 @@ async fn delete_under_config_write_guard(
/// targets one, so a writer on another node cannot land a whole-file save in
/// the middle of this read-modify-write.
pub async fn update_config_with<F>(bucket: &str, config_file: &str, mutate: F) -> Result<OffsetDateTime>
where
F: FnOnce(&BucketMetadata) -> Result<Vec<u8>> + Send,
{
update_config_with_expected(bucket, config_file, None, mutate).await
}
async fn update_config_with_expected<F>(
bucket: &str,
config_file: &str,
expected_incarnation_id: Option<Uuid>,
mutate: F,
) -> Result<OffsetDateTime>
where
F: FnOnce(&BucketMetadata) -> Result<Vec<u8>> + Send,
{
let sys = get_bucket_metadata_sys()?;
let guard = Box::pin(acquire_config_write_guard(sys.clone(), bucket)).await?;
let guard = Box::pin(acquire_config_write_guard_for_incarnation(sys.clone(), bucket, expected_incarnation_id)).await?;
guard.ensure_valid(bucket)?;
let metadata_sys = sys.read().await.clone();
let updated = await_bucket_namespace_operation(
@@ -1197,6 +1325,70 @@ where
Ok(updated)
}
fn swift_manages_bucket_tags(tagging: &Tagging) -> bool {
tagging
.tag_set
.iter()
.any(|tag| tag.key.as_deref() == Some(super::metadata::SWIFT_TAGGING_MARKER) && tag.value.as_deref() == Some("1"))
}
/// Read the S3 tag set while keeping Swift's internal tags private.
pub async fn get_s3_tagging_config(bucket: &str) -> Result<(Tagging, OffsetDateTime)> {
let (mut tagging, updated) = get_tagging_config(bucket).await?;
if swift_manages_bucket_tags(&tagging) {
tagging.tag_set.retain(|tag| {
!tag.key
.as_deref()
.is_some_and(super::metadata::is_swift_bucket_metadata_tag_key)
});
}
Ok((tagging, updated))
}
/// Replace or clear S3 tags atomically while retaining Swift metadata.
///
/// The expected incarnation, when present, is checked under the bucket
/// lifecycle and metadata guards. A failed write leaves the previous config
/// intact. Repeated writes replace the same S3 set.
pub async fn update_s3_tagging_config(
bucket: &str,
tagging: Option<Tagging>,
expected_incarnation_id: Option<Uuid>,
) -> Result<OffsetDateTime> {
let tags = tagging.map(|tagging| tagging.tag_set).unwrap_or_default();
let invalid = |message: &str| Error::InvalidArgument(bucket.to_owned(), String::new(), message.to_owned());
if tags.len() > super::metadata::MAX_BUCKET_TAGS {
return Err(invalid("Cannot have more than 50 tags per bucket"));
}
super::metadata::validate_tag_set_entries(&tags).map_err(invalid)?;
update_config_with_expected(bucket, BUCKET_TAGGING_CONFIG, expected_incarnation_id, |metadata| {
let mut merged = metadata.tagging_config.clone().unwrap_or_default();
if swift_manages_bucket_tags(&merged) {
if tags.iter().any(|tag| {
tag.key
.as_deref()
.is_some_and(super::metadata::is_swift_bucket_metadata_tag_key)
}) {
return Err(invalid("Tag key is reserved for Swift metadata on this bucket"));
}
merged.tag_set.retain(|tag| {
tag.key
.as_deref()
.is_some_and(super::metadata::is_swift_bucket_metadata_tag_key)
});
} else {
merged.tag_set.clear();
}
merged.tag_set.extend(tags);
if merged.tag_set.is_empty() {
Ok(Vec::new())
} else {
super::utils::serialize(&merged).map_err(Error::other)
}
})
.await
}
/// Acquire a bucket's metadata transaction lock, held across a whole
/// read-modify-write of its metadata file.
///
@@ -1235,13 +1427,18 @@ pub(crate) async fn acquire_bucket_metadata_transaction_lock_in(
pub(crate) async fn acquire_bucket_metadata_transaction_read_lock_in(
ctx: &crate::runtime::instance::InstanceContext,
bucket: &str,
) -> Result<rustfs_lock::NamespaceLockGuard> {
acquire_bucket_metadata_transaction_read_lock_in_for_purpose(ctx, bucket, "bucket_metadata_read").await
}
pub(crate) async fn acquire_bucket_metadata_transaction_read_lock_in_for_purpose(
ctx: &crate::runtime::instance::InstanceContext,
bucket: &str,
op: &'static str,
) -> Result<rustfs_lock::NamespaceLockGuard> {
let sys = bucket_metadata_sys_of(ctx)?;
let api = sys.read().await.object_store();
let lock = api
.new_ns_lock(RUSTFS_META_BUCKET, &bucket_metadata_transaction_lock_key(bucket))
.await?;
Ok(lock.get_read_lock(crate::set_disk::get_lock_acquire_timeout()).await?)
acquire_metadata_transaction_guard(api, bucket, op, rustfs_lock::LockType::Shared).await
}
/// Readers already holding an Object Lock snapshot must share its guard:
@@ -1250,6 +1447,7 @@ pub(crate) async fn acquire_bucket_metadata_transaction_read_lock_for_options_in
ctx: &crate::runtime::instance::InstanceContext,
bucket: &str,
opts: &ObjectOptions,
op: &'static str,
) -> Result<Arc<rustfs_lock::NamespaceLockGuard>> {
if let Some(snapshot) = opts.object_lock_config_snapshot.as_ref() {
let store = object_store_in(ctx).await?;
@@ -1259,12 +1457,22 @@ pub(crate) async fn acquire_bucket_metadata_transaction_read_lock_for_options_in
Error::other("Object Lock snapshot does not hold a valid metadata transaction fence for this bucket")
});
}
Ok(Arc::new(acquire_bucket_metadata_transaction_read_lock_in(ctx, bucket).await?))
Ok(Arc::new(
acquire_bucket_metadata_transaction_read_lock_in_for_purpose(ctx, bucket, op).await?,
))
}
async fn acquire_transaction_lock_with_sys(
sys: &Arc<RwLock<BucketMetadataSys>>,
bucket: &str,
) -> Result<rustfs_lock::NamespaceLockGuard> {
acquire_transaction_lock_with_sys_for_purpose(sys, bucket, "bucket_metadata_transaction_write").await
}
async fn acquire_transaction_lock_with_sys_for_purpose(
sys: &Arc<RwLock<BucketMetadataSys>>,
bucket: &str,
op: &'static str,
) -> Result<rustfs_lock::NamespaceLockGuard> {
// Resolve the store under a short-lived read guard: this runs before the
// write guard in `acquire_config_write_guards`, and must not still hold a
@@ -1273,12 +1481,22 @@ async fn acquire_transaction_lock_with_sys(
let lock = api
.new_ns_lock(RUSTFS_META_BUCKET, &bucket_metadata_transaction_lock_key(bucket))
.await?;
let acquire = lock.get_write_lock(crate::set_disk::get_lock_acquire_timeout());
let timeout = crate::set_disk::get_lock_acquire_timeout();
let mut attempt = ObjectLockAttempt::start(
op,
RUSTFS_META_BUCKET,
&bucket_metadata_transaction_lock_key(bucket),
None,
&lock,
"write",
timeout,
);
let acquire = lock.get_write_lock(timeout);
#[cfg(any(test, feature = "test-util"))]
{
tokio::pin!(acquire);
let mut notified = false;
let guard = futures::future::poll_fn(|cx| match std::future::Future::poll(acquire.as_mut(), cx) {
let result = futures::future::poll_fn(|cx| match std::future::Future::poll(acquire.as_mut(), cx) {
std::task::Poll::Pending => {
if !notified {
notify_config_write_lock_attempt(bucket);
@@ -1288,11 +1506,38 @@ async fn acquire_transaction_lock_with_sys(
}
std::task::Poll::Ready(result) => std::task::Poll::Ready(result),
})
.await?;
Ok(guard)
.await;
attempt.observe_acquisition(&result);
Ok(result?)
}
#[cfg(not(any(test, feature = "test-util")))]
Ok(acquire.await?)
{
let result = acquire.await;
attempt.observe_acquisition(&result);
Ok(result?)
}
}
pub(crate) async fn acquire_metadata_transaction_guard(
api: Arc<ECStore>,
bucket: &str,
op: &'static str,
lock_type: rustfs_lock::LockType,
) -> Result<rustfs_lock::NamespaceLockGuard> {
let object = bucket_metadata_transaction_lock_key(bucket);
let lock = api.new_ns_lock(RUSTFS_META_BUCKET, &object).await?;
let timeout = crate::set_disk::get_lock_acquire_timeout();
let mode = match lock_type {
rustfs_lock::LockType::Shared => "read",
rustfs_lock::LockType::Exclusive => "write",
};
let mut attempt = ObjectLockAttempt::start(op, RUSTFS_META_BUCKET, &object, None, &lock, mode, timeout);
let result = match lock_type {
rustfs_lock::LockType::Shared => lock.get_read_lock(timeout).await,
rustfs_lock::LockType::Exclusive => lock.get_write_lock(timeout).await,
};
attempt.observe_acquisition(&result);
Ok(result?)
}
/// The lock resource name is deliberately still the `bucket-targets` one it
@@ -2794,13 +3039,13 @@ impl BucketMetadataSys {
}
async fn find_bucket_incarnation_id_from_disk(&self, bucket: &str) -> Result<Option<Uuid>> {
let transaction_lock = self
.object_store()
.new_ns_lock(RUSTFS_META_BUCKET, &bucket_metadata_transaction_lock_key(bucket))
.await?;
let _transaction_guard = transaction_lock
.get_read_lock(crate::set_disk::get_lock_acquire_timeout())
.await?;
let _transaction_guard = acquire_metadata_transaction_guard(
self.object_store(),
bucket,
"bucket_incarnation_disk_read",
rustfs_lock::LockType::Shared,
)
.await?;
self.find_bucket_incarnation_id_under_transaction_lock(bucket, &_transaction_guard)
.await
}
@@ -2876,13 +3121,13 @@ impl BucketMetadataSys {
/// revalidated at write quorum, so a stale snapshot can neither revert an
/// acknowledged configuration update nor outlive a delete/recreate.
async fn migrate_bucket_creation_commit(&self, bucket: &str) -> Result<BucketMetadataAuthority> {
let transaction_lock = self
.object_store()
.new_ns_lock(RUSTFS_META_BUCKET, &bucket_metadata_transaction_lock_key(bucket))
.await?;
let transaction_guard = transaction_lock
.get_write_lock(crate::set_disk::get_lock_acquire_timeout())
.await?;
let transaction_guard = acquire_metadata_transaction_guard(
self.object_store(),
bucket,
"bucket_creation_commit_migration",
rustfs_lock::LockType::Exclusive,
)
.await?;
let namespace_lock = self.object_store().new_ns_lock(bucket, bucket).await?;
let namespace_guard = namespace_lock
.get_read_lock(crate::set_disk::get_lock_acquire_timeout())
@@ -2932,13 +3177,13 @@ impl BucketMetadataSys {
}
async fn migrate_legacy_metadata(&self, bucket: &str) -> Result<BucketMetadataAuthority> {
let transaction_lock = self
.object_store()
.new_ns_lock(RUSTFS_META_BUCKET, &bucket_metadata_transaction_lock_key(bucket))
.await?;
let _transaction_guard = transaction_lock
.get_write_lock(crate::set_disk::get_lock_acquire_timeout())
.await?;
let _transaction_guard = acquire_metadata_transaction_guard(
self.object_store(),
bucket,
"bucket_metadata_legacy_migration",
rustfs_lock::LockType::Exclusive,
)
.await?;
let authority = self
.load_authoritative_metadata_from_disk_under_transaction_lock(bucket)
@@ -3428,7 +3673,7 @@ mod tests {
),
(
BUCKET_TAGGING_CONFIG,
r#"<Tagging><TagSet><Tag><Key>environment</Key><Value>测试-🦀</Value></Tag></TagSet></Tagging>"#.as_bytes(),
r#"<Tagging><TagSet><Tag><Key>environment</Key><Value>测试-rust</Value></Tag></TagSet></Tagging>"#.as_bytes(),
),
(BUCKET_REPLICATION_CONFIG, NEW_WRITER_REPLICATION_XML),
(
@@ -3977,6 +4222,12 @@ mod tests {
.await
.is_err()
);
assert!(
acquire_scanner_bucket_incarnation_read_fence_with_sys(sys.clone(), bucket, Uuid::new_v4(), store.id)
.await
.is_err(),
"read-only scanner validation must not migrate an absent incarnation"
);
assert!(load_bucket_incarnation(store, bucket).await.expect("read sidecar").is_none());
assert!(
sys.read()
@@ -4033,6 +4284,128 @@ mod tests {
);
}
#[tokio::test(flavor = "multi_thread", worker_threads = 4)]
#[serial]
async fn scanner_incarnation_read_fence_shares_reads_and_blocks_mutations() {
let (_dirs, store) = isolated_store_over_temp_disks().await;
init_bucket_metadata_sys(store.clone(), Vec::new()).await;
let sys = bucket_metadata_sys_of(&store.ctx).expect("metadata owner");
let bucket = "scanner-read-fence";
store
.make_bucket(bucket, &MakeBucketOptions::default())
.await
.expect("create bucket");
let incarnation = store
.bucket_incarnation_id_from_disk(bucket)
.await
.expect("bucket incarnation");
let guard = acquire_scanner_bucket_incarnation_read_fence_with_sys(sys.clone(), bucket, incarnation, store.id)
.await
.expect("read-only scanner fence");
assert_eq!(guard.checked_bucket_incarnation().expect("valid read fences"), (bucket, incarnation));
let transaction_lock = store
.new_ns_lock(RUSTFS_META_BUCKET, &bucket_metadata_transaction_lock_key(bucket))
.await
.expect("metadata transaction namespace");
assert!(
transaction_lock.get_write_lock(Duration::from_millis(20)).await.is_err(),
"scanner read authority must exclude metadata writers"
);
let lifecycle_lock = store
.new_ns_lock(bucket, crate::storage_api_contracts::bucket::BUCKET_LIFECYCLE_LOCK_OBJECT)
.await
.expect("bucket lifecycle namespace");
assert!(
lifecycle_lock.get_write_lock(Duration::from_millis(20)).await.is_err(),
"scanner read authority must exclude delete and recreation"
);
let another = timeout(
Duration::from_secs(1),
acquire_scanner_bucket_incarnation_read_fence_with_sys(sys.clone(), bucket, incarnation, store.id),
)
.await
.expect("scanner readers must not serialize")
.expect("second read-only scanner fence");
let compatible = timeout(
Duration::from_secs(1),
acquire_bucket_metadata_transaction_read_lock_in(&store.ctx, bucket),
)
.await
.expect("metadata readers must remain compatible")
.expect("metadata transaction reader");
drop(compatible);
drop(another);
let mut mutation = Box::pin(acquire_config_write_guard_with_migration(sys.clone(), bucket, Some(incarnation), false));
assert!(matches!(futures::poll!(&mut mutation), std::task::Poll::Pending));
let mut deletion = Box::pin(store.acquire_bucket_lifecycle_write_lock(bucket));
assert!(matches!(futures::poll!(&mut deletion), std::task::Poll::Pending));
drop(deletion);
drop(guard);
let mutation = timeout(Duration::from_secs(1), mutation)
.await
.expect("metadata mutation must progress after scanner read release")
.expect("exclusive mutation fence");
drop(mutation);
let lifecycle = timeout(Duration::from_secs(1), store.acquire_bucket_lifecycle_write_lock(bucket))
.await
.expect("bucket delete fence must progress after readers release")
.expect("exclusive lifecycle fence");
drop(lifecycle);
for (owner, expected) in [
(Uuid::new_v4(), incarnation),
(Uuid::nil(), incarnation),
(store.id, Uuid::nil()),
] {
assert!(
acquire_scanner_bucket_incarnation_read_fence_with_sys(sys.clone(), bucket, expected, owner)
.await
.is_err(),
"invalid owner or nil incarnation must not receive authority"
);
}
assert!(
acquire_scanner_bucket_incarnation_read_fence_with_sys(sys.clone(), bucket, Uuid::new_v4(), store.id)
.await
.is_err(),
"a stale incarnation must not receive authority"
);
store
.delete_bucket(bucket, &DeleteBucketOptions::default())
.await
.expect("delete bucket");
assert!(
acquire_scanner_bucket_incarnation_read_fence_with_sys(sys.clone(), bucket, incarnation, store.id)
.await
.is_err(),
"deleted bucket must not receive authority"
);
store
.make_bucket(bucket, &MakeBucketOptions::default())
.await
.expect("recreate bucket");
let replacement = store
.bucket_incarnation_id_from_disk(bucket)
.await
.expect("replacement incarnation");
assert_ne!(replacement, incarnation);
assert!(
acquire_scanner_bucket_incarnation_read_fence_with_sys(sys.clone(), bucket, incarnation, store.id)
.await
.is_err(),
"recreation must reject the previous scanner identity"
);
let guard = acquire_scanner_bucket_incarnation_read_fence_with_sys(sys, bucket, replacement, store.id)
.await
.expect("replacement read-only scanner fence");
assert_eq!(
guard.checked_bucket_incarnation().expect("replacement remains fenced"),
(bucket, replacement)
);
}
#[tokio::test]
async fn old_node_metadata_rewrite_cannot_replace_bucket_incarnation_sidecar() {
let (dirs, ecstore) = isolated_store_over_temp_disks().await;
@@ -4556,8 +4929,13 @@ mod tests {
std::fs::create_dir_all(dir.path().join(bucket)).expect("bucket volume should be created");
}
let mut bm = BucketMetadata::new(bucket);
bm.update_config(config_file, raw.to_vec())
.expect("raw config should be stored");
if config_file == crate::bucket::metadata::BUCKET_TAGGING_CONFIG {
// Model legacy persisted corruption without passing the write-time validator.
bm.tagging_config_xml = raw.to_vec();
} else {
bm.update_config(config_file, raw.to_vec())
.expect("raw config should be stored");
}
sys.persist_new_and_set(bm).await.expect("initial metadata should persist");
}
@@ -5738,7 +6116,8 @@ mod tests {
// update only needs one interleaving to show up.
const ROUNDS: usize = 8;
for round in 0..ROUNDS {
let tagging = format!("<Tagging><Round>{round}</Round></Tagging>").into_bytes();
let tagging =
format!("<Tagging><TagSet><Tag><Key>round</Key><Value>{round}</Value></Tag></TagSet></Tagging>").into_bytes();
let policy = format!(r#"{{"Version":"2012-10-17","Round":{round}}}"#).into_bytes();
let start = Arc::new(tokio::sync::Barrier::new(2));
@@ -5811,7 +6190,7 @@ mod tests {
let blocked = tokio::spawn({
let node_b = node_b.clone();
async move { update_with_sys(node_b, bucket, BUCKET_TAGGING_CONFIG, b"<Tagging/>".to_vec()).await }
async move { update_with_sys(node_b, bucket, BUCKET_TAGGING_CONFIG, b"<Tagging><TagSet/></Tagging>".to_vec()).await }
});
// Long enough for the write to have finished had it not waited: the
@@ -119,6 +119,21 @@ async fn durable_quota_reporting_retires_degraded_floors_without_changing_ledger
// Exercise the real storage entry point before testing the new
// reporting semantics: the same ledger must admit and refuse
// these writes even when a scanner floor claims a million bytes.
if protocol == 1 {
let mut bootstrap = PutObjReader::from_vec(vec![0x5a; 512]);
assert!(
matches!(
store
.put_object(&bucket, "bootstrap", &mut bootstrap, &ObjectOptions::default())
.await,
Err(StorageError::SlowDown)
),
"a missing authoritative ledger must not admit a write from scanner reporting"
);
crate::bucket::quota::reservation::retry_quota_reconciliation_for_test(store.clone(), &bucket)
.await
.expect("bootstrap authoritative usage through the production worker");
}
let mut too_large = PutObjReader::from_vec(vec![0x5a; 8192]);
assert!(
matches!(
File diff suppressed because it is too large Load Diff
+87 -2
View File
@@ -40,7 +40,9 @@ use aws_smithy_runtime_api::client::http::{
HttpConnector as SmithyHttpConnector, HttpConnectorFuture, SharedHttpConnector, http_client_fn,
};
use aws_smithy_runtime_api::client::interceptors::Intercept;
use aws_smithy_runtime_api::client::interceptors::context::BeforeTransmitInterceptorContextMut;
use aws_smithy_runtime_api::client::interceptors::context::{
BeforeDeserializationInterceptorContextMut, BeforeTransmitInterceptorContextMut,
};
use aws_smithy_runtime_api::client::orchestrator::{HttpRequest, HttpResponse};
use aws_smithy_runtime_api::client::result::ConnectorError;
use aws_smithy_runtime_api::client::runtime_components::RuntimeComponents;
@@ -155,6 +157,8 @@ pub struct RemoteS3EndpointSpec {
pub ca_cert_pem: Option<String>,
pub connect_timeout: Option<Duration>,
pub read_timeout: Option<Duration>,
/// Entire SDK call, including non-streaming response bodies and retries.
pub operation_timeout: Option<Duration>,
/// How many wire requests one logical call may cost. Every consumer
/// declares it; see [`RemoteS3RetryPolicy`].
pub retry: RemoteS3RetryPolicy,
@@ -252,6 +256,78 @@ pub(crate) fn remote_sdk_credentials(credentials: &RemoteCredentials, now: Syste
Ok(builder.build())
}
/// Only installed on non-streaming upload operations. The budget begins at
/// response headers, independently of the potentially long request upload.
#[derive(Debug)]
pub(crate) struct UploadResponseDeadline(pub Duration);
impl Intercept for UploadResponseDeadline {
fn name(&self) -> &'static str {
"RustfsUploadResponseDeadline"
}
fn modify_before_deserialization(
&self,
context: &mut BeforeDeserializationInterceptorContextMut<'_>,
_runtime_components: &RuntimeComponents,
_cfg: &mut ConfigBag,
) -> Result<(), BoxError> {
let response = context.response_mut();
let inner = std::mem::replace(response.body_mut(), SdkBody::taken());
*response.body_mut() = SdkBody::from_body_1_x(UploadResponseBody {
inner: Some(inner),
deadline: Box::pin(tokio::time::sleep(self.0)),
});
Ok(())
}
}
struct UploadResponseBody {
inner: Option<SdkBody>,
deadline: std::pin::Pin<Box<tokio::time::Sleep>>,
}
impl http_body::Body for UploadResponseBody {
type Data = bytes::Bytes;
type Error = BoxError;
fn poll_frame(
mut self: std::pin::Pin<&mut Self>,
cx: &mut std::task::Context<'_>,
) -> std::task::Poll<Option<Result<http_body::Frame<Self::Data>, Self::Error>>> {
use std::future::Future as _;
use std::task::Poll;
if self.inner.is_none() {
return Poll::Ready(None);
}
// The absolute response budget never renews on data, empty frames,
// or trailers. A retry gets its own bounded response window.
if self.deadline.as_mut().poll(cx).is_ready() {
self.inner.take();
return Poll::Ready(Some(Err(std::io::Error::new(
std::io::ErrorKind::TimedOut,
"replication response body deadline exceeded",
)
.into())));
}
let Some(inner) = self.inner.as_mut() else {
return Poll::Ready(None);
};
let result = std::pin::Pin::new(inner).poll_frame(cx);
if matches!(result, Poll::Ready(None) | Poll::Ready(Some(Err(_)))) {
self.inner.take();
}
result
}
fn is_end_stream(&self) -> bool {
self.inner.as_ref().is_none_or(http_body::Body::is_end_stream)
}
fn size_hint(&self) -> http_body::SizeHint {
self.inner.as_ref().map(http_body::Body::size_hint).unwrap_or_default()
}
}
/// Appends a caller-identifying token to the SDK `User-Agent`. Runs after
/// signing: SigV4 excludes `user-agent` from the canonical request, so the
/// signature stays valid.
@@ -309,7 +385,7 @@ pub async fn build_remote_s3_config(spec: &RemoteS3EndpointSpec) -> Result<aws_s
config_builder = config_builder.http_client(http_client);
}
if spec.connect_timeout.is_some() || spec.read_timeout.is_some() {
if spec.connect_timeout.is_some() || spec.read_timeout.is_some() || spec.operation_timeout.is_some() {
let mut timeouts = TimeoutConfig::builder();
if let Some(connect_timeout) = spec.connect_timeout {
timeouts = timeouts.connect_timeout(connect_timeout);
@@ -317,6 +393,9 @@ pub async fn build_remote_s3_config(spec: &RemoteS3EndpointSpec) -> Result<aws_s
if let Some(read_timeout) = spec.read_timeout {
timeouts = timeouts.read_timeout(read_timeout);
}
if let Some(operation_timeout) = spec.operation_timeout {
timeouts = timeouts.operation_timeout(operation_timeout);
}
config_builder = config_builder.timeout_config(timeouts.build());
}
@@ -674,6 +753,7 @@ mod tests {
ca_cert_pem: None,
connect_timeout: None,
read_timeout: None,
operation_timeout: None,
retry: RemoteS3RetryPolicy::Disabled,
user_agent_suffix: "",
}
@@ -1027,11 +1107,16 @@ mod tests {
spec.user_agent_suffix = "RustFS-Test/0.0";
spec.connect_timeout = Some(Duration::from_secs(5));
spec.read_timeout = Some(Duration::from_secs(5));
spec.operation_timeout = Some(Duration::from_secs(12));
let config = build_remote_s3_config(&spec)
.await
.expect("spec should build")
.http_client(http_client)
.build();
assert_eq!(
config.timeout_config().and_then(TimeoutConfig::operation_timeout),
Some(Duration::from_secs(12))
);
S3Client::from_conf(config)
.head_bucket()
.bucket("bucket")
@@ -33,6 +33,7 @@ pub(crate) struct ReplicationMetadataStore;
impl ReplicationMetadataStore {
pub(crate) const MRF_REPLICATION_FILE: &'static str = "config/replication/mrf.bin";
pub(crate) const MRF_REPLICATION_RECOVERY_LOCK: &'static str = "config/replication/mrf.bin.recovery";
pub(crate) const MRF_REPLICATION_REPLAY_STAMP: &'static str = "config/replication/mrf.bin.replay";
pub(crate) const FORCE_DELETE_REPLICATION_FILE: &'static str = "config/replication/force-delete.bin";
pub(crate) const FORCE_DELETE_REPLICATION_TRANSACTION_LOCK: &'static str = "config/replication/force-delete.bin.transaction";
@@ -116,6 +117,10 @@ mod tests {
ReplicationMetadataStore::MRF_REPLICATION_RECOVERY_LOCK,
"config/replication/mrf.bin.recovery"
);
assert_eq!(
ReplicationMetadataStore::MRF_REPLICATION_REPLAY_STAMP,
"config/replication/mrf.bin.replay"
);
assert_eq!(
ReplicationMetadataStore::FORCE_DELETE_REPLICATION_FILE,
"config/replication/force-delete.bin"
@@ -16,8 +16,8 @@ use super::replication_config_store::ReplicationConfigStore;
use super::replication_error_boundary::{Error as EcstoreError, is_err_object_not_found, is_err_version_not_found};
use super::replication_filemeta_boundary::{
MrfOpKind, MrfReplicateEntry, REPLICATE_HEAL_DELETE, ReplicateDecision, ReplicateObjectInfo, ReplicatedTargetInfo,
ReplicationStatusType, ReplicationType, ReplicationWorkerOperation, ResyncDecision, replicate_decision_for_admitted_targets,
replication_statuses_map, version_purge_statuses_map,
ReplicationGenerationSnapshot, ReplicationStatusType, ReplicationType, ReplicationWorkerOperation, ResyncDecision,
replicate_decision_for_admitted_targets, replication_statuses_map, version_purge_statuses_map,
};
use super::replication_lock_boundary::ReplicationLockTiming;
use super::replication_logging::{EVENT_REPLICATION_CONFIG_LOOKUP_SKIPPED, LOG_COMPONENT_ECSTORE, LOG_SUBSYSTEM_REPLICATION};
@@ -86,6 +86,10 @@ const MRF_CORRUPT_FILE_PREFIX: &str = "config/replication/mrf.corrupt";
const MRF_PENDING_CAP: usize = 200_000;
const MRF_RETRY_INITIAL_DELAY: Duration = Duration::from_millis(100);
const MRF_RETRY_MAX_DELAY: Duration = Duration::from_secs(5);
/// Interval between runtime replays of the MRF journal, cluster-wide. A pass
/// re-attempts every retained entry, so a target that stays down costs one
/// attempt per entry per interval.
const MRF_REPLAY_INTERVAL: Duration = Duration::from_secs(300);
#[derive(Debug, Default)]
pub struct DurableMrfBacklog {
@@ -615,7 +619,7 @@ async fn read_mrf_entries<S: ReplicationObjectIO>(storage: Arc<S>) -> Result<Vec
}
/// Acknowledge only the generation read by the recovery leader. Entries appended
/// after that generation are retained as a suffix and replayed on the next startup.
/// after that generation are retained as a suffix and replayed on the next pass.
/// Lock order: recovery leader lock -> MRF journal object lock.
async fn acknowledge_mrf_recovery<S: ReplicationStorage>(
storage: Arc<S>,
@@ -667,7 +671,7 @@ async fn acknowledge_mrf_recovery<S: ReplicationStorage>(
Err(EcstoreError::PreconditionFailed)
}
/// Acquires the MRF recovery leader lock for the startup replay.
/// Acquires the MRF recovery leader lock for one journal replay pass.
/// Returns `None` (after logging) when the lock cannot be created or another
/// node is already processing the backlog.
async fn acquire_mrf_recovery_guard<S: ReplicationStorage>(storage: &Arc<S>) -> Option<rustfs_lock::NamespaceLockGuard> {
@@ -1018,7 +1022,7 @@ fn admitted_mrf_replicate_object(oi: ObjectInfo, entry: &MrfReplicateEntry, op_t
}
/// Acknowledges the replayed MRF prefix and returns the retained backlog.
/// On acknowledgement failure the backlog is preserved for the next startup and
/// On acknowledgement failure the backlog is preserved for the next replay pass and
/// re-read (falling back to the replayed snapshot) so the published summary stays accurate.
async fn resolve_retained_mrf_entries<S: ReplicationStorage>(
storage: &Arc<S>,
@@ -1033,7 +1037,7 @@ async fn resolve_retained_mrf_entries<S: ReplicationStorage>(
component = LOG_COMPONENT_ECSTORE,
subsystem = LOG_SUBSYSTEM_REPLICATION,
error = %error,
"Failed to acknowledge the MRF recovery prefix; preserving it for the next startup"
"Failed to acknowledge the MRF recovery prefix; preserving it for the next replay pass"
);
match read_mrf_entries(storage.clone()).await {
Ok(current) => current,
@@ -1051,6 +1055,120 @@ async fn resolve_retained_mrf_entries<S: ReplicationStorage>(
}
}
fn unix_millis_now() -> i64 {
i64::try_from(OffsetDateTime::now_utc().unix_timestamp_nanos() / 1_000_000).unwrap_or(i64::MAX)
}
/// Whether a periodic pass may run, given the start time of the last pass on
/// any node. A stamp more than one interval ahead of this node's clock is skew
/// or a bad value, so it does not suppress replay.
fn mrf_replay_due(last_started_ms: Option<i64>, now_ms: i64, interval: Duration) -> bool {
let Some(last_started_ms) = last_started_ms else {
return true;
};
let interval_ms = i64::try_from(interval.as_millis()).unwrap_or(i64::MAX);
now_ms.saturating_sub(last_started_ms) >= interval_ms || last_started_ms.saturating_sub(now_ms) >= interval_ms
}
async fn read_mrf_replay_stamp<S: ReplicationStorage>(storage: &Arc<S>) -> Option<i64> {
let data = ReplicationConfigStore::read(storage.clone(), ReplicationMetadataStore::MRF_REPLICATION_REPLAY_STAMP)
.await
.ok()?;
std::str::from_utf8(&data).ok()?.trim().parse().ok()
}
async fn write_mrf_replay_stamp<S: ReplicationStorage>(storage: &Arc<S>, started_ms: i64) {
if let Err(error) = ReplicationConfigStore::save(
storage.clone(),
ReplicationMetadataStore::MRF_REPLICATION_REPLAY_STAMP,
started_ms.to_string().into_bytes(),
)
.await
{
warn!(
component = LOG_COMPONENT_ECSTORE,
subsystem = LOG_SUBSYSTEM_REPLICATION,
error = %error,
"Failed to record the MRF replay start time"
);
}
}
/// One MRF journal replay pass. The recovery leader reads the journal,
/// re-drives every entry, and acknowledges the replayed prefix; entries that
/// could not be admitted, and entries appended meanwhile, stay for the next pass.
///
/// Every node runs the periodic loop, and the leader lock only serializes
/// passes. With `min_interval`, a pass is skipped when a pass on any node
/// started within that interval, read from the shared replay stamp, so a
/// cluster replays once per interval rather than once per node. The startup
/// pass passes `None` and always runs. A pass that finds entries records its
/// start time. Returns whether the journal was read.
/// Lock order: recovery leader lock -> replay stamp object lock.
async fn replay_mrf_journal<S: ReplicationStorage>(storage: &Arc<S>, min_interval: Option<Duration>) -> bool {
let Some(recovery_guard) = acquire_mrf_recovery_guard(storage).await else {
return false;
};
let started_ms = unix_millis_now();
if let Some(interval) = min_interval
&& !mrf_replay_due(read_mrf_replay_stamp(storage).await, started_ms, interval)
{
return false;
}
let Some(entries) = load_mrf_recovery_entries(storage).await else {
return true;
};
if !entries.is_empty() {
write_mrf_replay_stamp(storage, started_ms).await;
}
set_durable_mrf_backlog_snapshot(durable_mrf_backlog_summary_from_entries(&entries));
let total = entries.len();
let mut queued_count = 0usize;
let mut retry_entries = Vec::new();
for entry in entries.iter() {
let Some(admission) = replay_mrf_entry(entry, storage, &mut retry_entries).await else {
continue;
};
if admission == ReplicationQueueAdmission::Missed {
retry_entries.push(entry.clone());
} else if admission == ReplicationQueueAdmission::Queued {
queued_count += 1;
}
}
let retained = resolve_retained_mrf_entries(storage, &recovery_guard, &entries, &retry_entries).await;
let retained_count = retained.len();
set_durable_mrf_backlog_snapshot(durable_mrf_backlog_summary_from_entries(&retained));
if queued_count > 0 {
info!(
component = LOG_COMPONENT_ECSTORE,
subsystem = LOG_SUBSYSTEM_REPLICATION,
recovered = queued_count,
total,
retained = retained_count,
"Replayed MRF entries admitted for retry"
);
}
true
}
/// Re-runs [`replay_mrf_journal`] every `period`. The recovery leader lock keeps
/// one pass at a time across the cluster, including the startup pass, and the
/// shared replay stamp spaces passes from different nodes by `period`.
fn spawn_mrf_replay_loop<S: ReplicationStorage>(storage: Arc<S>, period: Duration) -> JoinHandle<()> {
tokio::spawn(async move {
loop {
tokio::time::sleep(period).await;
replay_mrf_journal(&storage, Some(period)).await;
}
})
}
#[derive(Debug, thiserror::Error)]
#[error("replication resync {active_resync_id} is already active for {bucket}/{arn}")]
struct ResyncActiveConflictError {
@@ -1171,6 +1289,7 @@ impl<S: ReplicationStorage> ReplicationPool<S> {
// Start background tasks
pool.start_mrf_persister().await;
pool.start_mrf_processor().await;
pool.start_mrf_replayer().await;
pool.start_force_delete_processor().await;
pool
@@ -1462,7 +1581,7 @@ impl<S: ReplicationStorage> ReplicationPool<S> {
/// Queues a replica task
pub async fn queue_replica_task(&self, ri: ReplicateObjectInfo) -> ReplicationQueueAdmission {
// A version that is already queued or being uploaded is not driven a
// A source generation already queued or being uploaded is not driven a
// second time: the scanner heal pass sees it as PENDING until the
// first upload lands and would otherwise re-queue it every cycle
// (backlog#2362). The key is released when the worker finishes, or
@@ -1639,55 +1758,26 @@ impl<S: ReplicationStorage> ReplicationPool<S> {
/// Starts the MRF processor — one-shot at startup.
///
/// Reads the on-disk MRF file, re-injects admitted entries as Heal operations, and
/// rewrites any entries that could not be admitted for a later startup retry.
/// Replays the on-disk MRF journal once; [`Self::start_mrf_replayer`] repeats
/// the pass while the server runs.
async fn start_mrf_processor(&self) {
let storage = self.storage.clone();
let handle = tokio::spawn(async move {
let Some(recovery_guard) = acquire_mrf_recovery_guard(&storage).await else {
return;
};
let Some(entries) = load_mrf_recovery_entries(&storage).await else {
return;
};
set_durable_mrf_backlog_snapshot(durable_mrf_backlog_summary_from_entries(&entries));
let total = entries.len();
let mut queued_count = 0usize;
let mut retry_entries = Vec::new();
for entry in entries.iter() {
let Some(admission) = replay_mrf_entry(entry, &storage, &mut retry_entries).await else {
continue;
};
if admission == ReplicationQueueAdmission::Missed {
retry_entries.push(entry.clone());
} else if admission == ReplicationQueueAdmission::Queued {
queued_count += 1;
}
}
let retained = resolve_retained_mrf_entries(&storage, &recovery_guard, &entries, &retry_entries).await;
let retained_count = retained.len();
set_durable_mrf_backlog_snapshot(durable_mrf_backlog_summary_from_entries(&retained));
if queued_count > 0 {
info!(
component = LOG_COMPONENT_ECSTORE,
subsystem = LOG_SUBSYSTEM_REPLICATION,
recovered = queued_count,
total,
retained = retained_count,
"Replayed MRF entries admitted for retry"
);
}
replay_mrf_journal(&storage, None).await;
});
self.task_handles.lock().await.push(handle);
}
/// Starts the runtime MRF replayer — ongoing background task.
///
/// Entries reach the journal while the server runs (worker-queue overflow,
/// delete-marker purge intents whose watch window drained with targets still
/// dirty). Replaying only at startup left them waiting for a restart.
async fn start_mrf_replayer(&self) {
let handle = spawn_mrf_replay_loop(self.storage.clone(), MRF_REPLAY_INTERVAL);
self.task_handles.lock().await.push(handle);
}
async fn start_force_delete_processor(&self) {
let storage = self.storage.clone();
let handle =
@@ -1755,7 +1845,7 @@ impl<S: ReplicationStorage> ReplicationPool<S> {
/// writes them to the on-disk MRF file every flush interval (default 10s,
/// overridable via `RUSTFS_REPL_MRF_FLUSH_INTERVAL_MS`) or when 1 000 new
/// entries accumulate. Runtime writers append under the journal lock; only the
/// startup recovery leader may remove an acknowledged prefix.
/// recovery leader of a replay pass may remove an acknowledged prefix.
async fn start_mrf_persister(&self) {
let Some(mut rx) = self.mrf_save_rx.lock().await.take() else {
return;
@@ -2314,28 +2404,47 @@ impl<S: ReplicationStorage> ReplicationPool<S> {
}
/// Object versions currently queued or being uploaded, keyed by bucket,
/// object name and version. `queue_replica_task` admits a version only once
/// object name, version and source generation. `queue_replica_task` admits a generation only once
/// while it is in flight; the scanner heal pass and MRF replays that arrive
/// in the meantime are `Skipped` instead of driving a second complete upload
/// (backlog#2362). Entries are removed when the worker finishes the task or
/// when no worker accepted it.
#[derive(Debug, Default)]
pub(crate) struct ReplicationInFlight {
keys: std::sync::Mutex<std::collections::HashSet<(String, String, Option<uuid::Uuid>)>>,
keys: std::sync::Mutex<std::collections::HashSet<ReplicationInFlightKey>>,
}
#[derive(Debug, Clone, PartialEq, Eq, Hash)]
struct ReplicationInFlightKey {
bucket: String,
object: String,
version_id: Option<uuid::Uuid>,
generation: ReplicationGenerationSnapshot,
}
impl From<&ReplicateObjectInfo> for ReplicationInFlightKey {
fn from(info: &ReplicateObjectInfo) -> Self {
Self {
bucket: info.bucket.clone(),
object: info.name.clone(),
version_id: info.version_id,
generation: info.replication_generation.clone(),
}
}
}
impl ReplicationInFlight {
fn lock(&self) -> std::sync::MutexGuard<'_, std::collections::HashSet<(String, String, Option<uuid::Uuid>)>> {
fn lock(&self) -> std::sync::MutexGuard<'_, std::collections::HashSet<ReplicationInFlightKey>> {
self.keys.lock().unwrap_or_else(|poisoned| poisoned.into_inner())
}
/// Claim `ri`; `false` when the same version is already in flight.
/// Claim `ri`; a metadata mutation is independent of its earlier generation.
fn try_begin(&self, ri: &ReplicateObjectInfo) -> bool {
self.lock().insert((ri.bucket.clone(), ri.name.clone(), ri.version_id))
self.lock().insert(ReplicationInFlightKey::from(ri))
}
fn finish(&self, ri: &ReplicateObjectInfo) {
self.lock().remove(&(ri.bucket.clone(), ri.name.clone(), ri.version_id));
self.lock().remove(&ReplicationInFlightKey::from(ri));
}
#[cfg(test)]
@@ -2348,26 +2457,21 @@ impl ReplicationInFlight {
/// including when replication panics.
struct ReplicationInFlightGuard {
in_flight: Arc<ReplicationInFlight>,
key: ReplicateObjectInfo,
key: ReplicationInFlightKey,
}
impl ReplicationInFlightGuard {
fn new(in_flight: Arc<ReplicationInFlight>, ri: &ReplicateObjectInfo) -> Self {
Self {
in_flight,
key: ReplicateObjectInfo {
bucket: ri.bucket.clone(),
name: ri.name.clone(),
version_id: ri.version_id,
..Default::default()
},
key: ReplicationInFlightKey::from(ri),
}
}
}
impl Drop for ReplicationInFlightGuard {
fn drop(&mut self) {
self.in_flight.finish(&self.key);
self.in_flight.lock().remove(&self.key);
}
}
@@ -3421,6 +3525,8 @@ mod tests {
allow_first_read: Notify,
read_count: AtomicUsize,
delete_storage_calls: AtomicUsize,
delete_error: StdMutex<EcstoreError>,
replay_stamp: StdMutex<Option<Vec<u8>>>,
reported_size: StdMutex<Option<i64>>,
stream_read_bytes: Arc<AtomicUsize>,
write_count: AtomicUsize,
@@ -3488,6 +3594,24 @@ mod tests {
_h: Self::HeaderMap,
_opts: &Self::ObjectOptions,
) -> Result<Self::GetObjectReader, Self::Error> {
if object.ends_with(ReplicationMetadataStore::MRF_REPLICATION_REPLAY_STAMP) {
let stamp = self
.shared
.replay_stamp
.lock()
.expect("test replay stamp lock should not be poisoned")
.clone()
.ok_or(EcstoreError::FileNotFound)?;
return Ok(Self::GetObjectReader {
object_info: ObjectInfo {
size: i64::try_from(stamp.len()).expect("test stamp length should fit i64"),
..Default::default()
},
stream: Box::new(Cursor::new(stamp)),
buffered_body: None,
body_source: Default::default(),
});
}
if !object.ends_with("/.replication/resync.bin")
&& !object.ends_with("config/replication/mrf.bin")
&& !object.ends_with("config/replication/force-delete.bin")
@@ -3551,6 +3675,16 @@ mod tests {
let lock = self.new_ns_lock(bucket, object).await?;
Some(lock.get_write_lock(Duration::from_secs(10)).await?)
};
if object.ends_with(ReplicationMetadataStore::MRF_REPLICATION_REPLAY_STAMP) {
let mut stamp = Vec::new();
data.stream.read_to_end(&mut stamp).await.map_err(EcstoreError::from)?;
*self
.shared
.replay_stamp
.lock()
.expect("test replay stamp lock should not be poisoned") = Some(stamp);
return Ok(ObjectInfo::default());
}
if opts.http_preconditions.is_some()
&& let Some(replacement) = self
.shared
@@ -3672,7 +3806,12 @@ mod tests {
_opts: Self::ObjectOptions,
) -> Result<Self::ObjectInfo, Self::Error> {
self.shared.delete_storage_calls.fetch_add(1, Ordering::SeqCst);
Err(EcstoreError::NotImplemented)
Err(self
.shared
.delete_error
.lock()
.expect("delete error lock should not be poisoned")
.clone())
}
async fn delete_objects(
@@ -3929,6 +4068,48 @@ mod tests {
assert_eq!(current_queue(&pool, "in-flight-bucket").await, (3, 3 * 4096));
}
#[tokio::test]
async fn queue_replica_task_admits_metadata_changes_during_an_initial_upload() {
let pool = new_test_replication_pool(Arc::new(LoadResyncNodeStore::new("node-a", empty_resync_shared_state()))).await;
let (tx, mut rx) = mpsc::channel(4);
pool.workers.write().await.push(tx);
let initial = ReplicateObjectInfo {
bucket: "pending-tags-bucket".to_string(),
name: "object".to_string(),
version_id: Some(uuid::Uuid::new_v4()),
size: 4096,
replication_generation: ReplicationGenerationSnapshot {
mutation_id: Some(uuid::Uuid::new_v4().to_string()),
..Default::default()
},
..Default::default()
};
let metadata = ReplicateObjectInfo {
op_type: ReplicationType::Metadata,
user_tags: String::new(),
replication_generation: ReplicationGenerationSnapshot {
mutation_id: Some(uuid::Uuid::new_v4().to_string()),
..Default::default()
},
..initial.clone()
};
assert_eq!(pool.queue_replica_task(initial.clone()).await, ReplicationQueueAdmission::Queued);
assert_eq!(pool.queue_replica_task(metadata.clone()).await, ReplicationQueueAdmission::Queued);
assert_eq!(pool.queue_replica_task(metadata.clone()).await, ReplicationQueueAdmission::Skipped);
assert_eq!(pool.in_flight.len(), 2);
drop(ReplicationInFlightGuard::new(pool.in_flight.clone(), &initial));
assert_eq!(pool.in_flight.len(), 1);
assert_eq!(pool.queue_replica_task(metadata.clone()).await, ReplicationQueueAdmission::Skipped);
for expected in [&initial, &metadata] {
let ReplicationOperation::Object(queued) = rx.recv().await.expect("queued operation") else {
panic!("expected object replication");
};
assert_eq!(queued.replication_generation, expected.replication_generation);
}
drop(ReplicationInFlightGuard::new(pool.in_flight.clone(), &metadata));
assert_eq!(pool.in_flight.len(), 0);
}
#[tokio::test]
async fn queue_replica_task_releases_the_version_when_no_worker_accepts_it() {
let pool = new_test_replication_pool(Arc::new(LoadResyncNodeStore::new("node-a", empty_resync_shared_state()))).await;
@@ -4171,6 +4352,8 @@ mod tests {
allow_first_read: Notify::new(),
read_count: AtomicUsize::new(0),
delete_storage_calls: AtomicUsize::new(0),
delete_error: StdMutex::new(EcstoreError::NotImplemented),
replay_stamp: StdMutex::new(None),
reported_size: StdMutex::new(None),
stream_read_bytes: Arc::new(AtomicUsize::new(0)),
write_count: AtomicUsize::new(0),
@@ -4221,6 +4404,103 @@ mod tests {
}
}
#[tokio::test]
async fn delete_worker_already_purged_source_completes_without_retry() {
let arn = "arn:test";
for delete_marker_version in [false, true] {
for version_id in [Uuid::new_v4(), Uuid::nil()] {
for error in [
EcstoreError::VersionNotFound("source".to_string(), "purged".to_string(), version_id.to_string()),
EcstoreError::ObjectNotFound("source".to_string(), "purged".to_string()),
EcstoreError::FileVersionNotFound,
EcstoreError::FileNotFound,
] {
let shared = empty_resync_shared_state();
*shared.delete_error.lock().expect("delete error lock should not be poisoned") = error;
let storage = Arc::new(LoadResyncNodeStore::new("purged-source", shared.clone()));
let delete = DeletedObjectReplicationInfo {
bucket: "source".to_string(),
delete_object: ReplicationDeletedObject {
object_name: "purged".to_string(),
version_id: (!delete_marker_version).then_some(version_id),
delete_marker_version_id: delete_marker_version.then_some(version_id),
replication_state: Some(ReplicationState {
replicate_decision_str: test_replicate_decision(&[arn]).to_string(),
purge_targets: HashMap::from([(arn.to_string(), VersionPurgeStatusType::Complete)]),
..Default::default()
}),
..Default::default()
},
op_type: ReplicationType::Delete,
..Default::default()
};
assert!(
replicate_delete_with_outcome(delete, storage).await,
"a confirmed target purge must complete when the source is already absent (marker={delete_marker_version}, version={version_id})"
);
assert_eq!(
shared.delete_storage_calls.load(Ordering::SeqCst),
2 + usize::from(!delete_marker_version && !version_id.is_nil()),
"the worker must lock and attempt writeback, reading a non-null data version first"
);
}
}
}
}
#[tokio::test]
async fn delete_worker_missing_source_does_not_hide_unfinished_replication() {
let arn = "arn:test";
for (version_purge, target_status, error) in [
(
true,
VersionPurgeStatusType::Pending,
EcstoreError::ObjectNotFound("source".to_string(), "purged".to_string()),
),
(true, VersionPurgeStatusType::Complete, EcstoreError::ErasureWriteQuorum),
(
false,
VersionPurgeStatusType::Empty,
EcstoreError::ObjectNotFound("source".to_string(), "purged".to_string()),
),
] {
let shared = empty_resync_shared_state();
*shared.delete_error.lock().expect("delete error lock should not be poisoned") = error;
let storage = Arc::new(LoadResyncNodeStore::new("unfinished-delete", shared));
let delete = DeletedObjectReplicationInfo {
bucket: "source".to_string(),
delete_object: ReplicationDeletedObject {
object_name: "purged".to_string(),
version_id: version_purge.then(Uuid::new_v4),
delete_marker: !version_purge,
replication_state: Some(ReplicationState {
replicate_decision_str: test_replicate_decision(&[arn]).to_string(),
targets: if version_purge {
HashMap::new()
} else {
HashMap::from([(arn.to_string(), ReplicationStatusType::Completed)])
},
purge_targets: if version_purge {
HashMap::from([(arn.to_string(), target_status)])
} else {
HashMap::new()
},
..Default::default()
}),
..Default::default()
},
op_type: ReplicationType::Delete,
..Default::default()
};
assert!(
!replicate_delete_with_outcome(delete, storage).await,
"unfinished target work, source I/O failures, and marker creation must remain retryable"
);
}
}
#[tokio::test]
async fn delete_worker_preserves_persisted_admission_without_live_config() {
let arn = "arn:test";
@@ -5304,6 +5584,8 @@ mod tests {
allow_first_read: Notify::new(),
read_count: AtomicUsize::new(0),
delete_storage_calls: AtomicUsize::new(0),
delete_error: StdMutex::new(EcstoreError::NotImplemented),
replay_stamp: StdMutex::new(None),
reported_size: StdMutex::new(None),
stream_read_bytes: Arc::new(AtomicUsize::new(0)),
write_count: AtomicUsize::new(0),
@@ -6268,6 +6550,108 @@ mod tests {
assert_eq!(retained.target_arns, entry.target_arns);
}
#[tokio::test]
async fn mrf_replay_loop_drains_entries_persisted_after_startup() {
let shared = empty_resync_shared_state();
let storage = Arc::new(LoadResyncNodeStore::new("mrf-runtime-replay", shared.clone()));
// A force-delete intent from a superseded generation replays as
// Skipped, so a pass that reaches it acknowledges it off the journal.
let stale_intent = |object: &str| MrfReplicateEntry {
bucket: "mrf-runtime-replay".to_string(),
object: object.to_string(),
op: MrfOpKind::Delete,
force_delete_id: Some(uuid::Uuid::new_v4()),
force_delete_local_commit: false,
target_arns: vec!["arn:rustfs:replication:target-a".to_string()],
..Default::default()
};
let journal_drained = || async {
tokio::time::timeout(Duration::from_secs(5), async {
while !shared.data.lock().expect("test data lock should not be poisoned").is_empty() {
tokio::time::sleep(Duration::from_millis(10)).await;
}
})
.await
.is_ok()
};
let handle = spawn_mrf_replay_loop(storage, Duration::from_millis(20));
for object in ["first", "second"] {
*shared.data.lock().expect("test data lock should not be poisoned") =
encode_mrf_file(&[stale_intent(object)]).expect("MRF entry should encode");
assert!(
journal_drained().await,
"an MRF entry persisted while the server runs must be replayed without a restart ({object})"
);
}
handle.abort();
}
#[tokio::test]
async fn mrf_periodic_replay_runs_once_per_interval_across_nodes() {
let shared = empty_resync_shared_state();
let node_a = Arc::new(LoadResyncNodeStore::new("mrf-node-a", shared.clone()));
let node_b = Arc::new(LoadResyncNodeStore::new("mrf-node-b", shared.clone()));
let interval = Duration::from_secs(1);
let persist_backlog = |object: &str| {
let entry = MrfReplicateEntry {
bucket: "mrf-node-replay".to_string(),
object: object.to_string(),
op: MrfOpKind::Delete,
force_delete_id: Some(uuid::Uuid::new_v4()),
force_delete_local_commit: false,
target_arns: vec!["arn:rustfs:replication:target-a".to_string()],
..Default::default()
};
*shared.data.lock().expect("test data lock should not be poisoned") =
encode_mrf_file(&[entry]).expect("MRF entry should encode");
};
let journal_is_empty = || shared.data.lock().expect("test data lock should not be poisoned").is_empty();
persist_backlog("first");
assert!(replay_mrf_journal(&node_a, Some(interval)).await, "node A's periodic pass should replay");
assert!(journal_is_empty(), "node A's pass should acknowledge the replayed entry");
// Node B's timer fires within the same interval with a backlog present.
persist_backlog("second");
assert!(
!replay_mrf_journal(&node_b, Some(interval)).await,
"a node must not replay within one interval of another node's pass"
);
assert!(!journal_is_empty(), "the backlog should wait for the next interval");
assert!(
replay_mrf_journal(&node_b, None).await,
"the startup pass should replay regardless of the shared stamp"
);
assert!(journal_is_empty(), "the startup pass should acknowledge the replayed entry");
persist_backlog("third");
tokio::time::sleep(interval).await;
assert!(
replay_mrf_journal(&node_a, Some(interval)).await,
"a periodic pass should replay once the interval has elapsed"
);
assert!(journal_is_empty(), "the due pass should acknowledge the replayed entry");
}
#[test]
fn mrf_replay_due_tolerates_missing_and_skewed_stamps() {
let interval = Duration::from_secs(300);
let now = 1_000_000_000;
assert!(mrf_replay_due(None, now, interval));
assert!(!mrf_replay_due(Some(now - 299_999), now, interval));
assert!(mrf_replay_due(Some(now - 300_000), now, interval));
assert!(
!mrf_replay_due(Some(now + 1_000), now, interval),
"small clock skew should still space passes"
);
assert!(
mrf_replay_due(Some(now + 300_000), now, interval),
"a stamp an interval ahead must not suppress replay"
);
}
#[test]
fn mrf_entry_delete_marker_roundtrip() {
let dm_vid = Uuid::new_v4();
@@ -2809,17 +2809,22 @@ pub(crate) async fn replicate_delete_with_outcome<S: ReplicationStorage>(
true
}
Err(e) => {
error!(
event = EVENT_RESYNC_TARGET_OPERATION_FAILED,
component = LOG_COMPONENT_ECSTORE,
subsystem = LOG_SUBSYSTEM_REPLICATION_RESYNC,
bucket = %bucket,
arn = %dobj.target_arn,
object = %dobj.delete_object.object_name,
operation = "apply_replication_delete_state",
error = %e,
"Replication target operation failed"
);
// A batch purge or another worker may have removed this version
// already. Remote failures still control the final outcome below.
let source_already_purged = is_version_purge && (is_err_object_not_found(&e) || is_err_version_not_found(&e));
if !source_already_purged {
error!(
event = EVENT_RESYNC_TARGET_OPERATION_FAILED,
component = LOG_COMPONENT_ECSTORE,
subsystem = LOG_SUBSYSTEM_REPLICATION_RESYNC,
bucket = %bucket,
arn = %dobj.target_arn,
object = %dobj.delete_object.object_name,
operation = "apply_replication_delete_state",
error = %e,
"Replication target operation failed"
);
}
send_event(EventArgs {
event_name,
bucket_name: bucket.clone(),
@@ -2832,7 +2837,7 @@ pub(crate) async fn replicate_delete_with_outcome<S: ReplicationStorage>(
},
..Default::default()
});
false
source_already_purged
}
};
@@ -3010,7 +3015,7 @@ const DELETE_MARKER_PURGE_WATCH_INTERVAL: TokioDuration = TokioDuration::from_se
/// If the marker disappears (deleted before or while the replica landed),
/// purge the replicated marker from the targets, retrying failed targets on
/// later rounds. When the window drains with targets still dirty, persist the
/// purge intent as a durable MRF entry so the next startup replays it through
/// purge intent as a durable MRF entry so the next MRF replay pass re-drives it through
/// `purge_stale_delete_marker_targets`.
async fn watch_and_purge_source_delete_marker<S: ReplicationStorage>(
bucket: String,
@@ -3101,7 +3106,7 @@ async fn enqueue_delete_marker_purge_mrf(dobj: &DeletedObjectReplicationInfo, fa
/// Heal callers retry for the full watch window because the startup MRF
/// processor runs before bucket metadata (and thus target clients) finishes
/// initializing — the first pass can see `target_client_missing` and a later
/// round resolves the client; the replay loop is serial and startup-only, so
/// round resolves the client; replay passes are serial background work, so
/// blocking it for up to the window per dirty entry is acceptable. Live
/// callers run on replication workers where a down target would pin a worker
/// for the whole window, so they attempt once and lean on the durable intent
@@ -6604,6 +6609,7 @@ mod tests {
health_check_duration: std::time::Duration::from_secs(5),
replicate_sync: false,
client: Arc::new(aws_sdk_s3::Client::from_conf(config)),
timeouts: Default::default(),
})
}
+12
View File
@@ -259,6 +259,10 @@ pub fn check_object_name_for_length_and_slash(bucket: &str, object: &str) -> Res
pub fn check_new_object_args(bucket: &str, object: &str, data_movement: bool) -> Result<()> {
check_put_object_args(bucket, object)?;
if !data_movement && !is_meta_bucketname(bucket) && has_xml_invalid_control_characters(object) {
return Err(StorageError::ObjectNameInvalid(bucket.to_owned(), object.to_owned()));
}
if cfg!(target_os = "windows") && !data_movement && !is_meta_bucketname(bucket) && object.contains('\\') {
return Err(StorageError::InvalidArgument(
bucket.to_owned(),
@@ -270,6 +274,14 @@ pub fn check_new_object_args(bucket: &str, object: &str, data_movement: bool) ->
Ok(())
}
/// New public keys must remain representable in an unencoded XML listing.
/// Existing keys still use the read/delete validators so they can be removed.
pub fn has_xml_invalid_control_characters(object: &str) -> bool {
object
.chars()
.any(|character| character < ' ' && !matches!(character, '\t' | '\n' | '\r'))
}
pub fn check_copy_obj_args(bucket: &str, object: &str) -> Result<()> {
check_bucket_and_object_names(bucket, object)
}
+297 -33
View File
@@ -34,7 +34,7 @@ use tokio::sync::Mutex as TokioMutex;
use tokio::task::JoinSet;
use tokio::time::timeout;
use tokio_util::sync::CancellationToken;
use tracing::{debug, error, warn};
use tracing::{Instrument, debug, error, warn};
const LOG_COMPONENT_ECSTORE: &str = "ecstore";
const LOG_SUBSYSTEM_METACACHE: &str = "metacache";
@@ -217,6 +217,8 @@ impl Drop for NotifyOnDrop {
#[derive(Default)]
pub struct ListPathRawOptions {
/// Retain native listing admission through producer cancellation/draining.
pub read_budget: Option<Arc<tokio::sync::OwnedSemaphorePermit>>,
pub disks: Vec<Option<DiskStore>>,
pub fallback_disks: Vec<Option<DiskStore>>,
pub bucket: String,
@@ -224,7 +226,10 @@ pub struct ListPathRawOptions {
pub recursive: bool,
pub incl_deleted: bool,
pub skip_hidden_prefix_check: bool,
pub hidden_entry_budget: usize,
pub pending_prefix: Option<String>,
pub filter_prefix: Option<String>,
pub identifier_tree_leaf: Option<String>,
pub forward_to: Option<String>,
pub min_disks: usize,
/// Deliver every replica to the partial callback, including matching headers.
@@ -253,6 +258,7 @@ pub struct ListPathRawOptions {
impl Clone for ListPathRawOptions {
fn clone(&self) -> Self {
Self {
read_budget: self.read_budget.clone(),
disks: self.disks.clone(),
fallback_disks: self.fallback_disks.clone(),
bucket: self.bucket.clone(),
@@ -260,7 +266,10 @@ impl Clone for ListPathRawOptions {
recursive: self.recursive,
incl_deleted: self.incl_deleted,
skip_hidden_prefix_check: self.skip_hidden_prefix_check,
hidden_entry_budget: self.hidden_entry_budget,
pending_prefix: self.pending_prefix.clone(),
filter_prefix: self.filter_prefix.clone(),
identifier_tree_leaf: self.identifier_tree_leaf.clone(),
forward_to: self.forward_to.clone(),
min_disks: self.min_disks,
preserve_replica_metadata: self.preserve_replica_metadata,
@@ -289,8 +298,11 @@ fn walk_dir_options(opts: &ListPathRawOptions) -> WalkDirOptions {
recursive: opts.recursive,
incl_deleted: opts.incl_deleted,
skip_hidden_prefix_check: opts.skip_hidden_prefix_check,
hidden_entry_budget: opts.hidden_entry_budget,
pending_prefix: opts.pending_prefix.clone(),
report_notfound: opts.report_not_found,
filter_prefix: opts.filter_prefix.clone(),
identifier_tree_leaf: opts.identifier_tree_leaf.clone(),
forward_to: opts.forward_to.clone(),
limit: opts.per_disk_limit,
skip_total_timeout: opts.skip_walkdir_total_timeout,
@@ -301,13 +313,15 @@ fn walk_dir_options(opts: &ListPathRawOptions) -> WalkDirOptions {
}
/// How a merged raw listing stream ended.
#[derive(Clone, Copy, Debug, Default, PartialEq, Eq)]
#[derive(Clone, Debug, Default, PartialEq, Eq)]
pub struct ListPathRawOutcome {
/// At least one producer stopped at its per-disk limit instead of at the
/// end of its namespace, so the closed stream is not proof of exhaustion.
/// Carried in-band by the metacache stream trailer, so remote producers
/// report it as well.
pub limit_reached: bool,
/// Inclusive sorted boundary proved by every reader before a bounded stop.
pub scan_cut: Option<String>,
}
pub async fn list_path_raw(rx: CancellationToken, opts: ListPathRawOptions) -> disk::error::Result<()> {
@@ -363,10 +377,15 @@ async fn list_path_raw_inner(
let producer_errs: Arc<[OnceLock<DiskError>]> = (0..opts.disks.len()).map(|_| OnceLock::new()).collect::<Vec<_>>().into();
let cancel_rx = CancellationToken::new();
let walk_options = Arc::new(walk_dir_options(&opts));
for (disk_idx, disk) in opts.disks.iter().enumerate() {
let opdisk = disk.clone();
let opts_clone = opts.clone();
let walk_options = walk_options.clone();
#[cfg(test)]
let test_reader_behavior = opts.test_reader_behaviors.get(disk_idx).cloned();
#[cfg(test)]
let test_disk_io_context_observed = opts.test_disk_io_context_observed.clone();
let fallback_claim_tracker = fallback_claim_tracker.clone();
let fds_clone = fds.clone();
#[cfg(test)]
@@ -376,17 +395,19 @@ async fn list_path_raw_inner(
let disk_io_context_clone = disk_io_context.clone();
let (rd, wr) = tokio::io::duplex(64 * 1024);
readers.push(MetacacheReader::new(rd));
jobs.spawn(with_optional_disk_io_context(disk_io_context_clone, async move {
let read_budget = opts.read_budget.clone();
let producer = with_optional_disk_io_context(disk_io_context_clone, async move {
let _read_budget = read_budget;
let mut wr = PublishedBytesWriter::new(wr);
#[cfg(test)]
if let Some(observer) = &opts_clone.test_disk_io_context_observed {
if let Some(observer) = &test_disk_io_context_observed {
observer.store(
crate::disk::current_disk_io_context().is_some_and(|context| context.class == DiskIoClass::Background),
std::sync::atomic::Ordering::Release,
);
}
#[cfg(test)]
let test_primary_error = if let Some(behavior) = opts_clone.test_reader_behaviors.get(disk_idx).cloned() {
let test_primary_error = if let Some(behavior) = test_reader_behavior {
match behavior {
TestReaderBehavior::Eof => return Ok(()),
TestReaderBehavior::Entries(entries) => {
@@ -443,8 +464,6 @@ async fn list_path_raw_inner(
None
};
let wakl_opts = walk_dir_options(&opts_clone);
let mut need_fallback = false;
let mut last_err = None;
#[cfg(test)]
@@ -459,7 +478,7 @@ async fn list_path_raw_inner(
if !need_fallback && let Some(disk) = opdisk {
let primary_walk_started = std::time::Instant::now();
match disk.walk_dir(wakl_opts, &mut wr).await {
match disk.walk_dir((*walk_options).clone(), &mut wr).await {
Ok(_res) => {
rustfs_io_metrics::record_stage_duration(
"metacache_walk_dir_primary",
@@ -483,8 +502,8 @@ async fn list_path_raw_inner(
event = EVENT_METACACHE_LISTING,
component = LOG_COMPONENT_ECSTORE,
subsystem = LOG_SUBSYSTEM_METACACHE,
bucket = %opts_clone.bucket,
path = %opts_clone.path,
bucket = %walk_options.bucket,
path = %walk_options.base_dir,
disk_index = disk_idx,
state = "walk_dir_missing_path",
error = ?err,
@@ -495,8 +514,8 @@ async fn list_path_raw_inner(
event = EVENT_METACACHE_LISTING,
component = LOG_COMPONENT_ECSTORE,
subsystem = LOG_SUBSYSTEM_METACACHE,
bucket = %opts_clone.bucket,
path = %opts_clone.path,
bucket = %walk_options.bucket,
path = %walk_options.base_dir,
disk_index = disk_idx,
state = "walk_dir_failed",
error = ?err,
@@ -598,8 +617,8 @@ async fn list_path_raw_inner(
event = EVENT_METACACHE_LISTING,
component = LOG_COMPONENT_ECSTORE,
subsystem = LOG_SUBSYSTEM_METACACHE,
bucket = %opts_clone.bucket,
path = %opts_clone.path,
bucket = %walk_options.bucket,
path = %walk_options.base_dir,
disk_index = disk_idx,
state = "fallback_disk_missing_for_path",
error = ?err,
@@ -610,8 +629,8 @@ async fn list_path_raw_inner(
event = EVENT_METACACHE_LISTING,
component = LOG_COMPONENT_ECSTORE,
subsystem = LOG_SUBSYSTEM_METACACHE,
bucket = %opts_clone.bucket,
path = %opts_clone.path,
bucket = %walk_options.bucket,
path = %walk_options.base_dir,
disk_index = disk_idx,
state = "fallback_disk_missing",
error = ?err,
@@ -626,7 +645,7 @@ async fn list_path_raw_inner(
}
let fallback_walk_started = std::time::Instant::now();
match disk.as_ref().walk_dir(walk_dir_options(&opts_clone), &mut wr).await {
match disk.as_ref().walk_dir((*walk_options).clone(), &mut wr).await {
Ok(_r) => {
rustfs_io_metrics::record_stage_duration(
"metacache_walk_dir_fallback",
@@ -652,8 +671,8 @@ async fn list_path_raw_inner(
event = EVENT_METACACHE_LISTING,
component = LOG_COMPONENT_ECSTORE,
subsystem = LOG_SUBSYSTEM_METACACHE,
bucket = %opts_clone.bucket,
path = %opts_clone.path,
bucket = %walk_options.bucket,
path = %walk_options.base_dir,
disk_index = disk_idx,
state = "fallback_walk_dir_missing_path",
error = ?err,
@@ -664,8 +683,8 @@ async fn list_path_raw_inner(
event = EVENT_METACACHE_LISTING,
component = LOG_COMPONENT_ECSTORE,
subsystem = LOG_SUBSYSTEM_METACACHE,
bucket = %opts_clone.bucket,
path = %opts_clone.path,
bucket = %walk_options.bucket,
path = %walk_options.base_dir,
disk_index = disk_idx,
state = "fallback_walk_dir_failed",
error = ?err,
@@ -687,12 +706,13 @@ async fn list_path_raw_inner(
// warn!("list_path_raw: while need_fallback done");
Ok(())
}));
});
jobs.spawn(producer.in_current_span());
}
let revjob_rx = rx.clone();
let mut revjobs = JoinSet::new();
revjobs.spawn(async move {
let merge = async move {
// Consumer-side peek timeout, and a caveat worth understanding
// (rustfs/backlog#1217).
//
@@ -722,6 +742,8 @@ async fn list_path_raw_inner(
// Sticky across rounds: once any reader's trailer says its producer
// stopped at the per-disk limit, the merged stream may hide more keys.
let mut limit_reached = false;
let mut scan_cut = None;
let mut last_scanned_key: Option<String> = None;
loop {
let mut current = MetaCacheEntry::default();
@@ -733,7 +755,7 @@ async fn list_path_raw_inner(
// );
if revjob_rx.is_cancelled() {
return Ok(ListPathRawOutcome { limit_reached });
return Ok(ListPathRawOutcome { limit_reached, scan_cut });
}
let mut top_entries: Vec<Option<MetaCacheEntry>> = vec![None; readers.len()];
@@ -793,7 +815,12 @@ async fn list_path_raw_inner(
continue;
}
// eof
limit_reached |= r.limit_reached() == Some(true);
if r.limit_reached() == Some(true) {
limit_reached = true;
scan_cut = Some(last_scanned_key.clone().ok_or_else(|| {
DiskError::other("bounded listing reader ended without a scan boundary")
})?);
}
at_eof += 1;
// warn!("list_path_raw: peek eof, disk: {}", i);
continue;
@@ -867,7 +894,7 @@ async fn list_path_raw_inner(
*r = MetacacheReader::new(detached_rd);
continue;
}
PeekOutcome::Cancelled => return Ok(ListPathRawOutcome { limit_reached }),
PeekOutcome::Cancelled => return Ok(ListPathRawOutcome { limit_reached, scan_cut }),
}
};
@@ -926,7 +953,7 @@ async fn list_path_raw_inner(
if let Some(finished_fn) = opts.finished.as_ref() {
tokio::select! {
biased;
_ = revjob_rx.cancelled() => return Ok(ListPathRawOutcome { limit_reached }),
_ = revjob_rx.cancelled() => return Ok(ListPathRawOutcome { limit_reached, scan_cut }),
_ = finished_fn(&errs) => {}
}
}
@@ -976,6 +1003,14 @@ async fn list_path_raw_inner(
return Err(classify_listing_quorum_failure(&failures));
}
// The next head from a bounded reader is unknown. The previous
// globally consumed group is a safe cut; consuming another reader's
// later head would turn this absence of evidence into a skipped key.
// Check quorum failures above before accepting the bounded result.
if limit_reached {
break;
}
// Break if all at EOF or error.
if at_eof + has_err == readers.len() {
if has_err > 0
@@ -983,7 +1018,7 @@ async fn list_path_raw_inner(
{
tokio::select! {
biased;
_ = revjob_rx.cancelled() => return Ok(ListPathRawOutcome { limit_reached }),
_ = revjob_rx.cancelled() => return Ok(ListPathRawOutcome { limit_reached, scan_cut }),
_ = finished_fn(&errs) => {}
}
}
@@ -994,6 +1029,7 @@ async fn list_path_raw_inner(
break;
}
last_scanned_key = Some(current.name.clone());
if agree == readers.len() && !opts.preserve_replica_metadata {
for r in readers.iter_mut() {
if !r.skip_current() {
@@ -1005,7 +1041,7 @@ async fn list_path_raw_inner(
// warn!("list_path_raw: agreed_fn start, current: {:?}", &current.name);
tokio::select! {
biased;
_ = revjob_rx.cancelled() => return Ok(ListPathRawOutcome { limit_reached }),
_ = revjob_rx.cancelled() => return Ok(ListPathRawOutcome { limit_reached, scan_cut }),
result = agreed_fn(current) => result?
}
// warn!("list_path_raw: agreed_fn done");
@@ -1029,13 +1065,14 @@ async fn list_path_raw_inner(
if let Some(partial_fn) = opts.partial.as_ref() {
tokio::select! {
biased;
_ = revjob_rx.cancelled() => return Ok(ListPathRawOutcome { limit_reached }),
_ = revjob_rx.cancelled() => return Ok(ListPathRawOutcome { limit_reached, scan_cut }),
_ = partial_fn(MetaCacheEntries(top_entries), &errs) => {}
}
}
}
Ok(ListPathRawOutcome { limit_reached })
});
Ok(ListPathRawOutcome { limit_reached, scan_cut })
};
revjobs.spawn(merge.in_current_span());
let merge_started = std::time::Instant::now();
let Some(revjob) = revjobs.join_next().await else {
@@ -1199,6 +1236,39 @@ mod tests {
assert!(!options.skip_total_timeout);
}
#[test]
fn identifier_tree_selection_survives_job_clone_and_disk_request() {
let request = ListPathRawOptions {
path: "catalog/namespaces/".to_string(),
identifier_tree_leaf: Some("table-entry.json".to_string()),
recursive: true,
..Default::default()
};
let cloned = request.clone();
let walk = walk_dir_options(&cloned);
assert_eq!(walk.identifier_tree_leaf, request.identifier_tree_leaf);
assert_eq!(walk.base_dir, request.path);
assert!(walk.recursive);
}
#[test]
fn cloned_reader_options_preserve_hidden_scan_budget_and_pending_prefix() {
let request = ListPathRawOptions {
bucket: "versioned".to_owned(),
hidden_entry_budget: 100,
pending_prefix: Some("pending/".to_owned()),
forward_to: Some("pending/0099".to_owned()),
..Default::default()
};
// Both walks are built once from the shared options; the per-job clone
// must carry every field that the walk derives from the request.
let job = request.clone();
let walk = walk_dir_options(&job);
assert_eq!(walk.hidden_entry_budget, 100);
assert_eq!(walk.pending_prefix.as_deref(), request.pending_prefix.as_deref());
assert_eq!(walk.forward_to.as_deref(), Some("pending/0099"));
}
#[tokio::test]
async fn list_path_raw_empty_disks_returns_read_quorum() {
let err = list_path_raw(CancellationToken::new(), ListPathRawOptions::default())
@@ -1249,6 +1319,163 @@ mod tests {
);
}
#[tokio::test]
async fn list_path_raw_fanout_retains_only_linear_disk_references() {
let root = tempfile::tempdir().expect("create disk root");
let endpoint = crate::disk::endpoint::Endpoint::try_from(root.path().to_str().expect("UTF-8 disk path"))
.expect("parse disk endpoint");
let local = Arc::new(
crate::disk::local::LocalDisk::new(&endpoint, false)
.await
.expect("open local disk"),
);
let wrapper = crate::disk::disk_store::LocalDiskWrapper::new(local, false);
let disk = Arc::new(crate::disk::Disk::Local(Box::new(wrapper)));
let baseline = Arc::strong_count(&disk);
for primary_count in [1, 4, 16] {
for fallback_count in [0, 2] {
let entered: Vec<_> = (0..primary_count).map(|_| Arc::new(tokio::sync::Notify::new())).collect();
let stopped: Vec<_> = (0..primary_count).map(|_| Arc::new(tokio::sync::Notify::new())).collect();
let behaviors = entered
.iter()
.zip(&stopped)
.map(|(entered, stopped)| TestReaderBehavior::StallWithSignals {
entered: entered.clone(),
stopped: stopped.clone(),
})
.collect();
let cancel = CancellationToken::new();
let listing = tokio::spawn(list_path_raw(
cancel.clone(),
ListPathRawOptions {
disks: vec![Some(disk.clone()); primary_count],
fallback_disks: vec![Some(disk.clone()); fallback_count],
min_disks: primary_count,
test_reader_behaviors: behaviors,
peek_timeout: Some(Duration::from_secs(60)),
..Default::default()
},
));
for signal in &entered {
timeout(Duration::from_secs(2), signal.notified())
.await
.expect("every producer must enter the real fanout before counting references");
}
// The merge owns the configured primary/fallback slots, each producer
// owns its primary, and the shared fallback queue owns each fallback.
let expected = baseline + 2 * (primary_count + fallback_count);
let retained = Arc::strong_count(&disk);
cancel.cancel();
timeout(Duration::from_secs(2), listing)
.await
.expect("cancellation must stop the listing")
.expect("listing task must join")
.expect("external cancellation must finish cleanly");
for signal in &stopped {
timeout(Duration::from_secs(2), signal.notified())
.await
.expect("cancellation must drop every producer");
}
assert_eq!(Arc::strong_count(&disk), baseline, "cancellation must release every fanout reference");
assert_eq!(
retained, expected,
"fanout must retain linear references for {primary_count} primary and {fallback_count} fallback slots"
);
}
}
}
#[tokio::test(flavor = "current_thread")]
async fn list_path_raw_producer_and_merge_preserve_request_span() {
use tracing_subscriber::{Layer, layer::SubscriberExt};
struct ProducerSpan(Arc<AtomicBool>);
#[derive(Default)]
struct StateField(bool);
impl tracing::field::Visit for StateField {
fn record_debug(&mut self, _: &tracing::field::Field, _: &dyn std::fmt::Debug) {}
fn record_str(&mut self, field: &tracing::field::Field, value: &str) {
self.0 |= field.name() == "state" && value == "walk_dir_missing_path";
}
}
impl<S> Layer<S> for ProducerSpan
where
S: tracing::Subscriber + for<'a> tracing_subscriber::registry::LookupSpan<'a>,
{
fn on_event(&self, event: &tracing::Event<'_>, context: tracing_subscriber::layer::Context<'_, S>) {
let mut state = StateField::default();
event.record(&mut state);
if state.0
&& context
.lookup_current()
.is_some_and(|span| span.name() == "flink_list_request")
{
self.0.store(true, std::sync::atomic::Ordering::Release);
}
}
}
// The subscriber is thread-local; keep the spawned tasks on this runtime thread.
let producer_observed = Arc::new(AtomicBool::new(false));
let subscriber = tracing_subscriber::registry().with(ProducerSpan(producer_observed.clone()));
let _subscriber = tracing::subscriber::set_default(subscriber);
let request = tracing::info_span!("flink_list_request", request_id = "list-span-regression");
let root = tempfile::tempdir().expect("create disk root");
let endpoint = crate::disk::endpoint::Endpoint::try_from(root.path().to_str().expect("UTF-8 disk path"))
.expect("parse disk endpoint");
let local = Arc::new(
crate::disk::local::LocalDisk::new(&endpoint, false)
.await
.expect("open local disk"),
);
let wrapper = crate::disk::disk_store::LocalDiskWrapper::new(local, false);
let disk = Arc::new(crate::disk::Disk::Local(Box::new(wrapper)));
let error = list_path_raw(
CancellationToken::new(),
ListPathRawOptions {
disks: vec![Some(disk)],
bucket: "missing-bucket".to_owned(),
min_disks: 1,
..Default::default()
},
)
.instrument(request.clone())
.await
.expect_err("missing bucket must fail the real disk producer");
assert_eq!(error, DiskError::VolumeNotFound);
assert!(
producer_observed.load(std::sync::atomic::Ordering::Acquire),
"disk error must retain the request span"
);
let merge_observed = Arc::new(AtomicBool::new(false));
let merge_sink = merge_observed.clone();
let request_id = request.id();
list_path_raw(
CancellationToken::new(),
ListPathRawOptions {
disks: vec![None],
min_disks: 1,
test_reader_behaviors: vec![TestReaderBehavior::Entries(vec![fallback_test_entry()])],
agreed: Some(Box::new(move |_| {
merge_sink.store(tracing::Span::current().id() == request_id, std::sync::atomic::Ordering::Release);
Box::pin(async { Ok(()) })
})),
..Default::default()
},
)
.instrument(request)
.await
.expect("merge should deliver the test entry");
assert!(
merge_observed.load(std::sync::atomic::Ordering::Acquire),
"merge callback must retain the request span"
);
}
#[tokio::test]
async fn scanner_joinset_walk_uses_the_local_disk_background_gate() {
let root = tempfile::tempdir().expect("local disk root should be created");
@@ -1316,6 +1543,7 @@ mod tests {
outcome.limit_reached,
"a reader whose trailer reports a limit stop must mark the merged stream as possibly incomplete"
);
assert_eq!(outcome.scan_cut.as_deref(), Some("bucket/object"));
}
#[tokio::test]
@@ -1692,6 +1920,42 @@ mod tests {
assert_eq!(seen.lock().expect("seen mutex poisoned").as_slice(), &["bucket/object".to_string()]);
}
#[tokio::test]
async fn list_read_budget_is_retained_by_raw_producer_until_cancellation() {
let semaphore = Arc::new(tokio::sync::Semaphore::new(4));
let grant = Arc::new(semaphore.clone().acquire_many_owned(4).await.expect("listing grant"));
let weak_grant = Arc::downgrade(&grant);
let entered = Arc::new(tokio::sync::Notify::new());
let stopped = Arc::new(tokio::sync::Notify::new());
let worker = tokio::spawn(list_path_raw(
CancellationToken::new(),
ListPathRawOptions {
read_budget: Some(grant),
disks: vec![None],
min_disks: 1,
test_reader_behaviors: vec![TestReaderBehavior::StallWithSignals {
entered: entered.clone(),
stopped: stopped.clone(),
}],
..Default::default()
},
));
timeout(Duration::from_secs(5), entered.notified())
.await
.expect("producer entered");
assert!(weak_grant.strong_count() >= 2, "producer must retain a grant independently of the parent");
assert_eq!(semaphore.available_permits(), 0);
worker.abort();
assert!(worker.await.expect_err("parent cancelled").is_cancelled());
timeout(Duration::from_secs(5), stopped.notified())
.await
.expect("producer stopped");
let _next = timeout(Duration::from_secs(5), semaphore.acquire_many(4))
.await
.expect("cancelled listing releases all permits")
.expect("budget stays open");
}
#[tokio::test]
async fn list_path_raw_aborts_unresponsive_producer_after_quorum_eof() {
let result = timeout(
File diff suppressed because it is too large Load Diff
+11 -11
View File
@@ -35,17 +35,17 @@ pub use client::{
// Re-exported through `api::rpc`; not every item is consumed inside this crate.
#[allow(unused_imports)]
pub use http_auth::{
RPC_BACKGROUND_ADMISSION_HEADER, TONIC_RPC_PREFIX, build_auth_headers, build_put_file_auth_trailer,
check_and_record_signed_rpc_nonce, gen_signature_headers, gen_tonic_replay_scope_headers, gen_tonic_signature_headers,
normalize_tonic_rpc_audience, rpc_background_admission, set_tonic_canonical_body_digest, set_tonic_mutation_body_digest,
set_tonic_rolling_canonical_body_digest, set_tonic_rolling_mutation_body_digest, sign_ns_scanner_capability,
sign_ns_scanner_capability_with_checkpoint_coverage, sign_ns_scanner_capability_with_tier_registry_generation,
sign_put_file_capability, sign_tonic_rpc_response_proof, tonic_boot_epoch_challenge, tonic_boot_epoch_response_headers,
tonic_rpc_auth_failure_reason, verify_ns_scanner_capability, verify_ns_scanner_capability_with_checkpoint_coverage,
verify_ns_scanner_capability_with_tier_registry_generation, verify_put_file_auth_trailer, verify_put_file_capability,
verify_rpc_signature, verify_tonic_boot_epoch_response, verify_tonic_canonical_body_digest,
verify_tonic_mutation_body_digest, verify_tonic_mutation_body_digest_reject_unsigned, verify_tonic_rpc_response_proof,
verify_tonic_rpc_signature, verify_tonic_rpc_signature_with_bootstrap,
RPC_BACKGROUND_ADMISSION_HEADER, RpcReplayCacheAdmissionError, TONIC_RPC_PREFIX, build_auth_headers,
build_put_file_auth_trailer, check_and_record_signed_rpc_nonce, gen_signature_headers, gen_tonic_replay_scope_headers,
gen_tonic_signature_headers, normalize_tonic_rpc_audience, rpc_auth_is_resource_exhausted, rpc_background_admission,
set_tonic_canonical_body_digest, set_tonic_mutation_body_digest, set_tonic_rolling_canonical_body_digest,
set_tonic_rolling_mutation_body_digest, sign_ns_scanner_capability, sign_ns_scanner_capability_with_checkpoint_coverage,
sign_ns_scanner_capability_with_tier_registry_generation, sign_put_file_capability, sign_tonic_rpc_response_proof,
tonic_boot_epoch_challenge, tonic_boot_epoch_response_headers, tonic_rpc_auth_failure_reason, verify_ns_scanner_capability,
verify_ns_scanner_capability_with_checkpoint_coverage, verify_ns_scanner_capability_with_tier_registry_generation,
verify_put_file_auth_trailer, verify_put_file_capability, verify_rpc_signature, verify_tonic_boot_epoch_response,
verify_tonic_canonical_body_digest, verify_tonic_mutation_body_digest, verify_tonic_mutation_body_digest_reject_unsigned,
verify_tonic_rpc_response_proof, verify_tonic_rpc_signature, verify_tonic_rpc_signature_with_bootstrap,
};
#[cfg(test)]
pub(crate) use internode_data_transport::TcpHttpInternodeDataTransport;

Some files were not shown because too many files have changed in this diff Show More