Compare commits

..

38 Commits

Author SHA1 Message Date
overtrue 1fc1982976 chore: restore main's Cargo and baseline files after merging main 2026-11-11 19:22:46 +08:00
overtrue e2a927a301 Merge remote-tracking branch 'origin/main' into gw/T3-reference-binary 2026-11-11 19:22:22 +08:00
Chris An ff84f2b5bf fix(ci): fix mint runner and Dockerfile.source rc arch (#9310)
* fix(ci): run the mint lane on ubuntu-latest, which has a Docker daemon

* fix(docker): let BuildKit populate TARGETARCH for the rc download

* fix(docker): derive the rc architecture when TARGETARCH is not set
2026-11-11 06:18:31 -05:00
Chris An def5418168 test(e2e): expect ServiceUnavailable for degraded ListParts after #9149 (#9313)
* test(e2e): expect ServiceUnavailable for degraded ListParts after #9149

PR #9149 made ServiceUnavailable the default API code for storage
read-quorum failures and limited SlowDownRead to GET/HEAD object reads.
ListParts maps its storage error through ApiError::from, so a degraded
ListParts now answers 503 ServiceUnavailable. The two distributed
acceptance cases still expected the previous SlowDownRead code and have
failed every nightly e2e-distributed run since 2026-11-05.

The 503 status and the fail-closed (non-empty-list) assertions are
unchanged; only the error code tracks the intended contract.

* style(e2e): apply rustfmt to the ListParts error-code assertion
2026-11-11 06:18:24 -05:00
Chris An 85f5cc9ead feat(gateway-adapter): host, bucket-owner and request-id adapters (#9289)
* feat(server): add the RUSTFS_S3_STACK selector and dual-stack e2e lane (#8411)

* refactor(server): group the s3s-compat tower layers under legacy_compat (#8410)

Move the ten tower layers that patch s3s behaviour out of
rustfs/src/server/layer.rs into rustfs/src/server/legacy_compat/, one file
per layer with its tests. Each file header names the s3s behaviour it
patches, the gateway mechanism that replaces it (Replaced by:), and the task
that removes it (Removed by: T2.12, rustfs/backlog#2771). The layers keep
their code, their order in external_service_stack!, and their behaviour; no
layer is deleted.

Add server::http::tests::stack_census, which renders the ordered service
chain of the real external_service_stack! expansion and compares it with a
committed constant, so a reorder, insertion or removal is a visible diff.

Route the two s3s error types the moved layers use through a
storage_api::server::legacy_compat shim so the split adds no s3s-importing
file, and lower the s3s footprint baselines to the measured values (files 209
to 208, s3_error! lines 1586 to 1585). Keep the server runtime-source bypass
guard covering the moved code.

Refs rustfs/backlog#2787, rustfs/backlog#2734

* feat(s3-types): add the RustFS-owned S3 error type and s3_error! macro (#8406)

`rustfs-s3-types` now provides `S3ErrorCode` (generated by build.rs from
src/codes.txt, the grep-derived call surface, plus `Custom`), `S3Error`,
`S3Result`, `StdError` and the `s3_error` macro with the same call syntax
as s3s, so the T1.2 codemod is a `use` swap. Statuses are hand-written and
pinned to the s3s behaviour RustFS ships today by an oracle test that uses
s3s as a dev-dependency only. The `From` conversions for the legacy edge
live behind the off-by-default `compat-s3s` feature (the orphan rule
forbids them in rustfs/src/error.rs); `rustfs` enables it.

Refs rustfs/backlog#2742, rustfs/backlog#2734

* ci: add the per-crate test-count ratchet (#8404)

Quick Checks now fails any PR that lowers the number of test attributes
in a workspace crate or adds #[ignore], measured per crate against the
exact rows of .config/test-count-baseline.txt (rustfs/backlog#2734
section 1 rule 3). The counter widens the spec's literal
#[(tokio::)?test] to the tokio attributes that carry arguments and both
spellings of #[test_case(...)], and anchors at line start so
commented-out attributes do not count. The initial baseline covers 52
crates, 21077 tests and 164 ignored.

Refs rustfs/backlog#2738
Refs rustfs/backlog#2734

* test(ecstore): pin the persisted bucket-config XML bytes for 13 families (#8408)

Before any DTO or codec is replaced, pin the exact bytes that
deserialize followed by serialize produces for every persisted
bucket-configuration XML family in its three source shapes (AWS
documentation, MinIO-written, RustFS-written), and the bytes RustFS
writes today for a typed value. Byte equality only: a structural
comparison would let a codec replacement change the persisted encoding
unnoticed.

The generator script refuses to overwrite existing goldens without
--force, refuses on dirty codec paths, cannot report success from its
write mode, and records the generating commit in the fixtures README.

Refs rustfs/backlog#2744, rustfs/backlog#2734

* ci: raise the test-count baseline to the integration tree (#8426)

* ci: make the s3s footprint baseline exact and lower-only (#8403)

* ci: make the s3s footprint baseline exact and lower-only

The s3s footprint guard compared three counters against ceilings held in
the script itself, so a ceiling could sit above the tree and a drift was
fixable by raising it (rustfs/rustfs#5964 raised the s3_error! ceiling
to go green). The three values now live in
.config/s3s-footprint-baseline.txt as files, s3_error_lines and
ecstore_files, the script reads only that file, and a measurement that
differs from its baseline in either direction fails: growth prints
"s3s footprint grew: <measured> > <baseline>", shrinkage prints
"lower the baseline to <measured> in this PR". The counting method is
unchanged; the baseline is the current measurement of
gateway/integration (209 / 1585 / 36). The header records the end state:
once all three reach 0, T4.2 switches the script to failing on any
occurrence.

Refs rustfs/backlog#2739, rustfs/backlog#2734

* ci: add the s3s edge allowlist mode to the footprint guard (#8405)

`scripts/check_s3s_footprint.sh --mode allowlist` fails on every Rust
file that references s3s (`s3s::` or `s3s_sigv4`) and is not matched by
an entry of the new `.config/s3s-edge-allowlist.txt`; `--dry-run` lists
those files and exits 0. The allowlist carries the initial edge from the
spec, and the script rejects an entry that names or matches anything
under crates/ other than crates/e2e_test. The baseline mode and the
argument-less CI invocation are unchanged.

Refs rustfs/backlog#2741, rustfs/backlog#2734

* test(ecstore): reach the bucket-config codec through the test facade (#8428)

* refactor(s3-types): own the HTTP Body and drop s3s from three crates (#8431)

Add rustfs_s3_types::Body, a BoxBody<Bytes, StdError> wrapper with
empty() and From<Bytes | Vec<u8> | String | &'static str>, and the
legacy From conversions to and from s3s::Body behind compat-s3s.

The signer is now generic over the request body: it only reads and
writes the request head, so it needs no body type and its callers keep
theirs. s3-client sends its requests with the owned Body, madmin signs a
bodiless head, and targets drops its two S3Request/S3Response header
wrappers in favour of extract_params_header.

rustfs-signer, rustfs-madmin and rustfs-targets no longer depend on
s3s; the s3s footprint baseline drops from 209 to 202 files.

Refs rustfs/backlog#2747, rustfs/backlog#2734

* refactor(ecstore): move S3ErrorCode to rustfs-s3-types by codemod (#8432)

* test(ecstore): pin the InvalidRange responses of new_getobjectreader

Both error branches of new_getobjectreader build an ErrorResponse with S3ErrorCode::InvalidRange, and no test observed either code. Pin the code, its wire name and the location fields of each branch before the code type moves to rustfs-s3-types (rustfs/backlog#2743), so a changed code on either branch fails a test. The ecstore test-count row rises by the two tests.

* chore(scripts): add the s3s-to-rustfs-s3-types error codemod

Task T1.2 of rustfs/backlog#2734 (rustfs/backlog#2743) moves S3Error, S3ErrorCode, S3Result and s3_error! from s3s to rustfs-s3-types by script, one crate closure per PR. The codemod rewrites use trees (splitting mixed s3s imports) and qualified code paths, never comments or literals, adds the rustfs-s3-types dependency, drops s3s only from a crate that no longer names it, refuses every file of the s3s edge allowlist, takes --hold for a file whose type is fixed outside the closure, and prints a per-crate census. A second run changes nothing. Its self-test runs under make script-tests.

* refactor(ecstore): take S3ErrorCode from rustfs-s3-types

Output of scripts/codemods/s3_error_to_s3_types.py over crates/ecstore, crates/lifecycle, crates/scanner and crates/replication (rustfs/backlog#2743 batches 1 and 2). The ErrorResponse code field, the two InvalidRange constructions in new_getobjectreader and an unused tier import now use rustfs_s3_types::S3ErrorCode; the variant names and wire names are unchanged. lifecycle, scanner and replication reference none of the four error names. warm_backend.rs and warm_backend_s3.rs are held: they compare against rustfs_s3_client::ErrorResponse.code, which stays the s3s type until s3-client moves.

* ci(footprint): count only s3_error! lines still on s3s's macro

s3_error_lines counted every line holding the text `s3_error!`, which
also matches the RustFS-owned `rustfs_s3_types::s3_error!`. A codemod
that moves call sites onto rustfs-s3-types therefore could not lower it
(rustfs/backlog#2743, decision 3; rustfs/rustfs#8432, gotcha 2).

The counter now applies a textual, per-file and per-line rule that errs
on the side of counting (full rule in the script header):
- occurrences spelled `rustfs_s3_types::s3_error!` never count;
- a line holding `s3s::s3_error!` always counts;
- any other `s3_error!` line counts unless its file is cleared, that is,
  it imports the macro from rustfs_s3_types by name on a line starting
  with the use item (or defines it with `macro_rules! s3_error`) and does
  not also bring s3s's macro in (`use s3s::s3_error`, a `use s3s::{..}`
  tree naming it or a crate-root `*`, `use s3s::*`, or
  `#[macro_use] extern crate s3s`);
- fail closed: a file matching neither form (`use super::*`, a
  crate-local re-export, a renamed or glob rustfs_s3_types import, a
  commented-out import, a comment-only mention) counts.

Census of the 1585 lines on 96fc06776: 1478 in 64 files that import the
macro from s3s; 107 in 15 files matching neither form (105 through
`use super::*`, 1 qualified `s3s::s3_error!`, 1 comment line); none in
files that import it from rustfs_s3_types or define it. Every line stays
counted, so the baseline remains s3_error_lines=1585 (files=202 and
ecstore_files=36 unchanged). Rewriting rustfs/src/auth.rs's import to
rustfs_s3_types lowers the new counter to 1567; the old one stayed 1585.

`--self-test` (wired into `make script-tests`) copies the script into
22 fixture trees: 6 positive cases and 16 negative ones (lines that must
not count, forms that must not clear a file, and a missing baseline,
missing key or failing classification search that must fail). Each tree
has a control line that must count, and each run gets s3s text on stdin.
Seven cases were red against the old counter. 19 mutants each turn their
case red: old definition, flipped import test, dropped fail-closed
branch, dropped e2e_test exclusion, dropped s3s veto, dropped qualified
s3s rule, dropped rustfs-qualified strip, unanchored rustfs import,
renamed import accepted, nested glob taken as root glob, single-line
s3s trees only, dropped s3s glob forms, dropped macro_use form, dropped
definition clearing, ignored classification rg errors, error-line search
without its path, classification searches without their path, missing
baseline skipped, missing key skipped.

* test(e2e): compare two RUSTFS_S3_STACK servers over every S3 operation (#8435)

Add dual_stack_differential to e2e_test. It starts two rustfs processes
from the same binary, stack A on RUSTFS_S3_STACK=legacy and stack B on
RUSTFS_E2E_STACK_B (default legacy until rustfs/backlog#2734 T2.5),
sends both the same aws-sdk-s3 script covering the 73 methods of
impl S3 for FS (a normal request and a 4xx negative each), and compares
status, error code, an allowlist of response headers and a few typed
body fields. Error message text is never compared. Each run writes
<target>/dual-stack/<run>.json with compared_ops and the (op, field,
a, b) differences; any difference fails the test.

differential_detects_an_injected_difference starts B with
RUSTFS_REGION=eu-west-1 and requires GetBucketLocation to be reported.
The comparator is pure and unit-tested without a server.

Wire the module into the e2e-smoke filter and update the e2e-smoke and
e2e-full selection digests and the e2e_test test-count row.

Refs rustfs/backlog#2740

* feat(app): add the RustFS-owned RequestEnvelope and its s3s constructor (#8437)

* feat(app): add the RustFS-owned RequestEnvelope and its s3s constructor

* ci: classify the s3s edge adapters in storage/s3_compat as interface

* feat(s3-types): add Body::from_stream and move Swift off s3s::Body (#8438)

rustfs/backlog#2747 batch 3. `Body::from_stream` wraps a chunk stream of
unknown length over `StreamBody`/`BoxBody`: one data frame per chunk, the
first error ends the body, the size hint stays unknown, and dropping the
body drops the stream. Parity tests pin it to the s3s `StreamingBlob::wrap`
it replaces, at the `http_body` level and byte for byte through hyper.

The Swift handlers in rustfs-protocols build their responses as the owned
`Body`. `SwiftService` converts them into the wrapped S3 service's response
body type through `From<Body>`, so rustfs keeps serving `s3s::Body` at the
legacy edge unchanged. Request bodies stay Send-only (`UnsyncBoxBody`); the
two storage writers and the DLO/SLO segment streams get `Sync` from
`sync_wrapper::SyncStream`, which is what s3s's `http_body_unsync` mutex
provided before. The DTO and error uses of s3s in protocols are untouched
(T1.5 and T1.2).

* refactor(protocols): move error types to rustfs-s3-types by codemod (#8439)

T1.2 batch 3 (rustfs/backlog#2743): s3select-api, s3select-query and
the error items of protocols move off s3s by scripts/codemods, run with
--hold crates/protocols/src/sftp/errors.rs. That file stays on s3s: it
downcasts the backend error, whose runtime type is still s3s::S3Error.

- rustfs/src/protocols/client.rs: the two webdav session methods now
  return rustfs_s3_types::S3Result and convert the s3s error with From.
- codemod: a child module that calls s3_error! through `use super::*`
  gets its own `use rustfs_s3_types::s3_error;` once its parent imports
  that macro by name; otherwise it is listed as GLOB.
- s3-types: a table test pins code, status and the s3s error the legacy
  edge renders for every variant a moved call site names.
- s3_error_lines 1585 -> 1584 (a lifecycle comment reworded).

* test(connect): restore one-second drive request budgets

Apply the exact test-only repair already landed on main in 5972a09cc3. The filesystem probe and production CLI success test request one second and keep their existing one-second duration and parameter checks. All outcome, I/O, error, cleanup and resource-limit assertions remain.

Scope: rustfs/tests/connect_perf_drive.rs only. Patch-id: cb0a1dfc8f68a67727525adea7458e2cee7d09f8.

Validation environment: LANG=en_US.UTF-8, CARGO_BUILD_JOBS=1, CARGO_INCREMENTAL=0, exclusive APFS-cloned CARGO_TARGET_DIR.

Quick Checks: every command from .github/actions/quick-checks/action.yml, with captured output and exit status below.

$ shellcheck --version && actionlint
ShellCheck - shell script analysis tool
version: 0.11.0
license: GNU General Public License, version 3
website: https://www.shellcheck.net
exit_code=0

$ cargo fmt --all --check
exit_code=0

$ ./scripts/check_unsafe_code_allowances.sh
exit_code=0

$ ./scripts/check_layer_dependencies.sh
Layer dependency guard passed (no new reverse dependencies/cycles).
exit_code=0

$ ./scripts/check_architecture_migration_rules.sh
Architecture migration rules passed.
exit_code=0

$ ./scripts/check_logging_guardrails.sh
✅ Logging guardrails check passed
exit_code=0

$ ./scripts/check_error_other_format_ratchet.sh
error(format!) ratchet passed (total call sites: 582).
exit_code=0

$ ./scripts/check_no_tokio_io_uring.sh
exit_code=0

$ ./scripts/check_extension_schema_boundaries.sh
Extension schema boundaries passed.
exit_code=0

$ ./scripts/check_body_cache_whitelist.sh
Body-cache whitelist guard passed.
exit_code=0

$ ./scripts/check_s3s_footprint.sh
s3s footprint OK: files importing s3s is 195 (baseline files=195)
s3s footprint OK: s3s s3_error! lines is 1584 (baseline s3_error_lines=1584)
s3s footprint OK: ecstore files referencing s3s is 36 (baseline ecstore_files=36)
✅ s3s footprint baseline check passed
exit_code=0

$ ./scripts/check_test_count_ratchet.sh
test-count ratchet OK: 52 crates, 21222 tests, 164 ignored match .config/test-count-baseline.txt
exit_code=0

$ ./scripts/check_fips_wording.sh
FIPS wording guard passed (README.md and CHANGELOG.md contain no forbidden claims).
KMS wording guard passed (crates/kms claims no Vault KV2 Transit wrapping).
exit_code=0

$ ./scripts/check_embedded_secrets.sh
Embedded secret guard passed (no private key material or provider credential literal in the tree).
exit_code=0

$ make script-tests
Running script tests...
entrypoint credential policy tests passed
internode grpc A/B bench env tests passed
object batch benchmark enhanced tests passed
PASS package service script contract
hotpath warp A/B gate tests passed
hotpath warp ABBA tests passed
Scanner validation artifacts written to /var/folders/0w/s4wxsn2n4w9b6f3cd4xrz53w0000gn/T/tmp.1pFe6Kl1Yu/out
Scanner validation artifacts written to /var/folders/0w/s4wxsn2n4w9b6f3cd4xrz53w0000gn/T/tmp.1pFe6Kl1Yu/out-missing-pid
.............................
----------------------------------------------------------------------
Ran 29 tests in 181.286s

OK
{"decision":"verified","pending_gates":["G01","G03","G04","G05","G06","G07","G08","G09","G10","G11","G12","G13","G14","P1","P2","P3","P4","R-D","R-E","R-L"],"pending_lanes":["authority-coverage","checkpoint-and-crash","ec8-4-multiset","maintenance-producers","mixed-version-rollback","mrf-responsibility","scheduler-pressure","status-and-outcome"],"rejected_errors":[],"rejected_gate":null,"release_approved":false,"release_schema_capable":true,"schema":1,"verified_gate":"G02"}
{"decision":"verified","pending_gates":["G01","G02","G03","G04","G05","G06","G07","G08","G09","G10","G11","G12","G13","G14","P1","P2","P3","P4","R-D","R-L"],"pending_lanes":["authority-coverage","checkpoint-and-crash","ec8-4-multiset","maintenance-producers","mixed-version-rollback","mrf-responsibility","scheduler-pressure","status-and-outcome"],"rejected_errors":[],"rejected_gate":null,"release_approved":false,"release_schema_capable":true,"schema":1,"verified_gate":"R-E"}
{"decision":"verified","pending_gates":["G01","G03","G04","G05","G06","G07","G08","G09","G10","G11","G12","G13","G14","P1","P2","P3","P4","R-D","R-E","R-L"],"pending_lanes":["authority-coverage","checkpoint-and-crash","ec8-4-multiset","maintenance-producers","mixed-version-rollback","mrf-responsibility","scheduler-pressure","status-and-outcome"],"rejected_errors":[],"rejected_gate":null,"release_approved":false,"release_schema_capable":true,"schema":1,"verified_gate":"G02"}
{"decision":"verified","pending_gates":["G01","G02","G03","G04","G05","G06","G07","G08","G09","G10","G11","G12","G13","G14","P1","P2","P3","P4","R-D","R-L"],"pending_lanes":["authority-coverage","checkpoint-and-crash","ec8-4-multiset","maintenance-producers","mixed-version-rollback","mrf-responsibility","scheduler-pressure","status-and-outcome"],"rejected_errors":[],"rejected_gate":null,"release_approved":false,"release_schema_capable":true,"schema":1,"verified_gate":"R-E"}
{"decision":"verified","pending_gates":["G02","G03","G04","G05","G06","G07","G08","G09","G10","G11","G12","G13","G14","P1","P2","P3","P4","R-D","R-E","R-L"],"pending_lanes":["authority-coverage","checkpoint-and-crash","ec8-4-multiset","maintenance-producers","mixed-version-rollback","mrf-responsibility","scheduler-pressure","status-and-outcome"],"rejected_errors":[],"rejected_gate":null,"release_approved":false,"release_schema_capable":true,"schema":1,"verified_gate":"G01"}
{"decision":"verified","pending_gates":["G01","G02","G04","G05","G06","G07","G08","G09","G10","G11","G12","G13","G14","P1","P2","P3","P4","R-D","R-E","R-L"],"pending_lanes":["authority-coverage","checkpoint-and-crash","ec8-4-multiset","maintenance-producers","mixed-version-rollback","mrf-responsibility","scheduler-pressure","status-and-outcome"],"rejected_errors":[],"rejected_gate":null,"release_approved":false,"release_schema_capable":true,"schema":1,"verified_gate":"G03"}
{"decision":"verified","pending_gates":["G01","G02","G03","G04","G05","G06","G07","G08","G09","G10","G11","G12","G13","G14","P1","P2","P3","P4","R-D","R-E"],"pending_lanes":["authority-coverage","checkpoint-and-crash","ec8-4-multiset","maintenance-producers","mixed-version-rollback","mrf-responsibility","scheduler-pressure","status-and-outcome"],"rejected_errors":[],"rejected_gate":null,"release_approved":false,"release_schema_capable":true,"schema":1,"verified_gate":"R-L"}
{"decision":"verified","pending_gates":["G01","G02","G03","G04","G05","G06","G07","G08","G09","G10","G11","G12","G13","P1","P2","P3","P4","R-D","R-E","R-L"],"pending_lanes":["authority-coverage","checkpoint-and-crash","ec8-4-multiset","maintenance-producers","mixed-version-rollback","mrf-responsibility","scheduler-pressure","status-and-outcome"],"rejected_errors":[],"rejected_gate":null,"release_approved":false,"release_schema_capable":true,"schema":1,"verified_gate":"G14"}
{"decision":"verified","pending_gates":["G01","G02","G03","G04","G05","G06","G07","G08","G09","G10","G11","G12","G13","P1","P2","P3","P4","R-D","R-E","R-L"],"pending_lanes":["authority-coverage","checkpoint-and-crash","ec8-4-multiset","maintenance-producers","mixed-version-rollback","mrf-responsibility","scheduler-pressure","status-and-outcome"],"rejected_errors":[],"rejected_gate":null,"release_approved":false,"release_schema_capable":true,"schema":1,"verified_gate":"G14"}
{"decision":"verified","pending_gates":["G01","G02","G03","G04","G05","G06","G07","G08","G09","G11","G12","G13","G14","P1","P2","P3","P4","R-D","R-E","R-L"],"pending_lanes":["authority-coverage","checkpoint-and-crash","ec8-4-multiset","maintenance-producers","mixed-version-rollback","mrf-responsibility","scheduler-pressure","status-and-outcome"],"rejected_errors":[],"rejected_gate":null,"release_approved":false,"release_schema_capable":true,"schema":1,"verified_gate":"G10"}
{"decision":"verified","pending_gates":["G01","G02","G03","G04","G05","G06","G07","G08","G09","G10","G11","G12","G13","G14","P2","P3","P4","R-D","R-E","R-L"],"pending_lanes":["authority-coverage","checkpoint-and-crash","ec8-4-multiset","maintenance-producers","mixed-version-rollback","mrf-responsibility","scheduler-pressure","status-and-outcome"],"rejected_errors":[],"rejected_gate":null,"release_approved":false,"release_schema_capable":true,"schema":1,"verified_gate":"P1"}
{"decision":"verified","pending_gates":["G01","G02","G03","G04","G05","G06","G07","G08","G09","G10","G11","G12","G13","G14","P1","P3","P4","R-D","R-E","R-L"],"pending_lanes":["authority-coverage","checkpoint-and-crash","ec8-4-multiset","maintenance-producers","mixed-version-rollback","mrf-responsibility","scheduler-pressure","status-and-outcome"],"rejected_errors":[],"rejected_gate":null,"release_approved":false,"release_schema_capable":true,"schema":1,"verified_gate":"P2"}
{"decision":"verified","pending_gates":["G01","G02","G03","G04","G05","G06","G07","G08","G09","G10","G11","G12","G13","G14","P1","P2","P4","R-D","R-E","R-L"],"pending_lanes":["authority-coverage","checkpoint-and-crash","ec8-4-multiset","maintenance-producers","mixed-version-rollback","mrf-responsibility","scheduler-pressure","status-and-outcome"],"rejected_errors":[],"rejected_gate":null,"release_approved":false,"release_schema_capable":true,"schema":1,"verified_gate":"P3"}
{"decision":"verified","pending_gates":["G01","G02","G03","G04","G06","G07","G08","G09","G10","G11","G12","G13","G14","P1","P2","P3","P4","R-D","R-E","R-L"],"pending_lanes":["authority-coverage","checkpoint-and-crash","ec8-4-multiset","maintenance-producers","mixed-version-rollback","mrf-responsibility","scheduler-pressure","status-and-outcome"],"rejected_errors":[],"rejected_gate":null,"release_approved":false,"release_schema_capable":true,"schema":1,"verified_gate":"G05"}
{"decision":"verified","pending_gates":["G01","G02","G03","G04","G05","G07","G08","G09","G10","G11","G12","G13","G14","P1","P2","P3","P4","R-D","R-E","R-L"],"pending_lanes":["authority-coverage","checkpoint-and-crash","ec8-4-multiset","maintenance-producers","mixed-version-rollback","mrf-responsibility","scheduler-pressure","status-and-outcome"],"rejected_errors":[],"rejected_gate":null,"release_approved":false,"release_schema_capable":true,"schema":1,"verified_gate":"G06"}
{"decision":"verified","pending_gates":["G01","G02","G03","G04","G05","G06","G07","G08","G09","G10","G11","G12","G13","G14","P1","P2","P3","P4","R-E","R-L"],"pending_lanes":["authority-coverage","checkpoint-and-crash","ec8-4-multiset","maintenance-producers","mixed-version-rollback","mrf-responsibility","scheduler-pressure","status-and-outcome"],"rejected_errors":[],"rejected_gate":null,"release_approved":false,"release_schema_capable":true,"schema":1,"verified_gate":"R-D"}
Status-and-outcome release descriptor verified: /private/var/folders/0w/s4wxsn2n4w9b6f3cd4xrz53w0000gn/T/tmp8rwn1pe0/descriptor/release-bundle-status-outcome.json
{"decision":"verified","pending_gates":["G01","G02","G03","G04","G06","G07","G08","G09","G10","G11","G12","G13","G14","P1","P2","P3","P4","R-D","R-E","R-L"],"pending_lanes":["authority-coverage","checkpoint-and-crash","ec8-4-multiset","maintenance-producers","mixed-version-rollback","mrf-responsibility","scheduler-pressure","status-and-outcome"],"rejected_errors":[],"rejected_gate":null,"release_approved":false,"release_schema_capable":true,"schema":1,"verified_gate":"G05"}
{"decision":"verified","pending_gates":["G01","G02","G03","G04","G05","G07","G08","G09","G10","G11","G12","G13","G14","P1","P2","P3","P4","R-D","R-E","R-L"],"pending_lanes":["authority-coverage","checkpoint-and-crash","ec8-4-multiset","maintenance-producers","mixed-version-rollback","mrf-responsibility","scheduler-pressure","status-and-outcome"],"rejected_errors":[],"rejected_gate":null,"release_approved":false,"release_schema_capable":true,"schema":1,"verified_gate":"G06"}
{"decision":"verified","pending_gates":["G01","G02","G03","G04","G05","G06","G07","G08","G09","G10","G11","G12","G13","G14","P1","P2","P3","P4","R-E","R-L"],"pending_lanes":["authority-coverage","checkpoint-and-crash","ec8-4-multiset","maintenance-producers","mixed-version-rollback","mrf-responsibility","scheduler-pressure","status-and-outcome"],"rejected_errors":[],"rejected_gate":null,"release_approved":false,"release_schema_capable":true,"schema":1,"verified_gate":"R-D"}
{"decision":"verified","pending_gates":["G01","G02","G03","G04","G05","G06","G08","G09","G10","G11","G12","G13","G14","P1","P2","P3","P4","R-D","R-E","R-L"],"pending_lanes":["authority-coverage","checkpoint-and-crash","ec8-4-multiset","maintenance-producers","mixed-version-rollback","mrf-responsibility","scheduler-pressure","status-and-outcome"],"rejected_errors":[],"rejected_gate":null,"release_approved":false,"release_schema_capable":true,"schema":1,"verified_gate":"G07"}
{"decision":"verified","pending_gates":["G01","G02","G03","G04","G05","G06","G07","G09","G10","G11","G12","G13","G14","P1","P2","P3","P4","R-D","R-E","R-L"],"pending_lanes":["authority-coverage","checkpoint-and-crash","ec8-4-multiset","maintenance-producers","mixed-version-rollback","mrf-responsibility","scheduler-pressure","status-and-outcome"],"rejected_errors":[],"rejected_gate":null,"release_approved":false,"release_schema_capable":true,"schema":1,"verified_gate":"G08"}
{"decision":"verified","pending_gates":["G01","G02","G03","G04","G05","G06","G07","G08","G09","G10","G11","G12","G13","G14","P1","P2","P3","R-D","R-E","R-L"],"pending_lanes":["authority-coverage","checkpoint-and-crash","ec8-4-multiset","maintenance-producers","mixed-version-rollback","mrf-responsibility","scheduler-pressure","status-and-outcome"],"rejected_errors":[],"rejected_gate":null,"release_approved":false,"release_schema_capable":true,"schema":1,"verified_gate":"P4"}
exact 1MiB handoff ABBA tests passed
pinned paired ABBA bench tests passed
==> /var/folders/0w/s4wxsn2n4w9b6f3cd4xrz53w0000gn/T/tmp.mYmuI01hNq/bin/path_containment (-max_total_time=7, -seed=123456789, -artifact_prefix=/var/folders/0w/s4wxsn2n4w9b6f3cd4xrz53w0000gn/T/tmp.mYmuI01hNq/explicit-artifacts/path_containment/, corpus=/private/tmp/gw-orchestrator-2734/ci-drive/fuzz/corpus/path_containment)
-seed=123456789
target=path_containment
seed=123456789
max_total_time=7
git_revision=4d15b816236630b2c42f321ffc9ed3654edc94e3
git_dirty=true
runner_mode=prebuilt
==> /var/folders/0w/s4wxsn2n4w9b6f3cd4xrz53w0000gn/T/tmp.mYmuI01hNq/bin/path_containment (-max_total_time=1, -seed=2322617706, -artifact_prefix=/var/folders/0w/s4wxsn2n4w9b6f3cd4xrz53w0000gn/T/tmp.mYmuI01hNq/auto-artifacts/path_containment/, corpus=/private/tmp/gw-orchestrator-2734/ci-drive/fuzz/corpus/path_containment)
-seed=2322617706
==> cargo +nightly fuzz run path_containment (-max_total_time=9, -seed=987654321, -artifact_prefix=/var/folders/0w/s4wxsn2n4w9b6f3cd4xrz53w0000gn/T/tmp.mYmuI01hNq/cargo-artifacts/path_containment/)
+nightly
fuzz
run
path_containment
-seed=987654321
seed=987654321
runner_mode=cargo-fuzz
fuzz runner seed manifest tests passed
✅ scripts/python_bin.sh resolver checks passed
self-test ok: clean fixture with every exemption present passes
self-test ok: all 11 pattern families fire
self-test ok: moving a pinned source is reported
Embedded secret guard self-test passed.
self-test ok: p1-s3s-import: 'use s3s::s3_error;' counts every line, the rustfs_s3_types module's too (s3_error_lines=3)
self-test ok: p2-s3s-mixed-tree: a multi-line s3s use tree naming s3_error counts (s3_error_lines=3)
self-test ok: p3-s3s-glob: a crate-root glob of s3s, bare or inside a tree, counts (s3_error_lines=5)
self-test ok: p4-macro-use: '#[macro_use] extern crate s3s' counts (s3_error_lines=3)
self-test ok: p5-qualified-s3s: a qualified s3s::s3_error! line counts in a cleared file (s3_error_lines=2)
self-test ok: p6-exact-pass: a baseline equal to the tree passes (exit 0)
self-test ok: n1-rustfs-import: 'use rustfs_s3_types::s3_error;' alone counts nothing (s3_error_lines=1)
self-test ok: n2-rustfs-mixed-tree: a rustfs_s3_types tree plus an s3s tree without the macro counts nothing (s3_error_lines=1)
self-test ok: n3-defines: the file defining the macro counts nothing (s3_error_lines=1)
self-test ok: n4-qualified-rustfs: a qualified rustfs_s3_types::s3_error! line counts nothing (s3_error_lines=1)
self-test ok: n5-renamed-call: a call through a renamed macro is not text-matched (out of scope) (s3_error_lines=1)
self-test ok: n6-comment-string: a comment or string mentioning s3_error! in a cleared file counts nothing (s3_error_lines=1)
self-test ok: n7-e2e-excluded: crates/e2e_test is excluded (s3_error_lines=1)
self-test ok: n8-super-glob: an unrecognised import form ('use super::*;') counts (s3_error_lines=3)
self-test ok: n9-facade: a crate-local re-export counts (s3_error_lines=2)
self-test ok: n10-renamed-import: a renamed rustfs_s3_types import does not clear s3_error! lines (s3_error_lines=2)
self-test ok: n11-commented-import: a commented-out rustfs_s3_types import does not clear (s3_error_lines=2)
self-test ok: n12-rustfs-glob: a glob import of rustfs_s3_types does not clear (s3_error_lines=2)
self-test ok: n13-comment-only: a comment mentioning s3_error! in a file with no import counts (s3_error_lines=2)
self-test ok: f1-baseline-missing: a missing baseline file fails (exit 1)
self-test ok: f2-key-missing: a baseline without s3_error_lines fails (exit 1)
self-test ok: f3-rg-fails: a failing classification search fails the guard (exit 1)
✅ s3s footprint self-test passed
test_core_gate_rejects_missing_ignored_filtered_and_corrupt_inputs (__main__.SelfTests.test_core_gate_rejects_missing_ignored_filtered_and_corrupt_inputs) ... ok
test_e2e_requires_registration (__main__.SelfTests.test_e2e_requires_registration) ... ok
test_fuzz_runtime_uses_matrix_target (__main__.SelfTests.test_fuzz_runtime_uses_matrix_target) ... ok
test_ilm_lane_keeps_the_measured_cargo_build_budget (__main__.SelfTests.test_ilm_lane_keeps_the_measured_cargo_build_budget) ... ok
test_profile_listing_binds_platform_digest (__main__.SelfTests.test_profile_listing_binds_platform_digest) ... ok
test_profile_listing_enforces_selection (__main__.SelfTests.test_profile_listing_enforces_selection) ... ok
test_quick_checks_commands_propagate_failure (__main__.SelfTests.test_quick_checks_commands_propagate_failure) ... ok
test_quick_checks_rejects_caller_and_execution_bypasses (__main__.SelfTests.test_quick_checks_rejects_caller_and_execution_bypasses) ... ok
test_runner_readiness_fails_closed (__main__.SelfTests.test_runner_readiness_fails_closed) ... ok
test_s3_runner_rejects_unbounded_failure_locals (__main__.SelfTests.test_s3_runner_rejects_unbounded_failure_locals) ... ok
test_scanner_heal_begin_requires_embedded_source_provenance (__main__.SelfTests.test_scanner_heal_begin_requires_embedded_source_provenance) ... ok
test_scanner_heal_case_does_not_approve_pending_release (__main__.SelfTests.test_scanner_heal_case_does_not_approve_pending_release) ... ok
test_scanner_heal_case_lane_is_required_for_runner_profile (__main__.SelfTests.test_scanner_heal_case_lane_is_required_for_runner_profile) ... ok
test_scanner_heal_case_only_schema_cannot_approve_release (__main__.SelfTests.test_scanner_heal_case_only_schema_cannot_approve_release) ... ok
test_scanner_heal_crash_case_rejects_restart_oracle (__main__.SelfTests.test_scanner_heal_crash_case_rejects_restart_oracle) ... ok
test_scanner_heal_ec84_case_rejects_wrong_erasure_geometry (__main__.SelfTests.test_scanner_heal_ec84_case_rejects_wrong_erasure_geometry) ... ok
test_scanner_heal_finish_collects_oracles_from_registry (__main__.SelfTests.test_scanner_heal_finish_collects_oracles_from_registry) ... ok
test_scanner_heal_multipool_case_accepts_deferred_optional_outage_write (__main__.SelfTests.test_scanner_heal_multipool_case_accepts_deferred_optional_outage_write) ... ok
test_scanner_heal_pending_gate_cannot_map_to_implemented_lane (__main__.SelfTests.test_scanner_heal_pending_gate_cannot_map_to_implemented_lane) ... ok
test_scanner_heal_receipts_reject_reuse_and_missing_builds (__main__.SelfTests.test_scanner_heal_receipts_reject_reuse_and_missing_builds) ... ok
test_scanner_heal_rejects_boolean_fractional_and_out_of_geometry_integers (__main__.SelfTests.test_scanner_heal_rejects_boolean_fractional_and_out_of_geometry_integers) ... ok
test_scanner_heal_rejects_broken_execution_and_artifacts (__main__.SelfTests.test_scanner_heal_rejects_broken_execution_and_artifacts) ... ok
test_scanner_heal_rejects_copied_junit_and_wrong_suite_build (__main__.SelfTests.test_scanner_heal_rejects_copied_junit_and_wrong_suite_build) ... ok
test_scanner_heal_release_bundle_accepts_complete_measured_evidence (__main__.SelfTests.test_scanner_heal_release_bundle_accepts_complete_measured_evidence) ... ok
test_scanner_heal_release_bundle_assembler_copies_measured_artifacts (__main__.SelfTests.test_scanner_heal_release_bundle_assembler_copies_measured_artifacts) ... ok
test_scanner_heal_release_bundle_assembler_merges_measured_descriptors (__main__.SelfTests.test_scanner_heal_release_bundle_assembler_merges_measured_descriptors) ... ok
test_scanner_heal_release_bundle_assembler_rejects_duplicate_descriptor_gate (__main__.SelfTests.test_scanner_heal_release_bundle_assembler_rejects_duplicate_descriptor_gate) ... ok
test_scanner_heal_release_bundle_assembler_rejects_fixture_descriptor (__main__.SelfTests.test_scanner_heal_release_bundle_assembler_rejects_fixture_descriptor) ... ok
test_scanner_heal_release_bundle_assembler_rejects_hash_mismatch (__main__.SelfTests.test_scanner_heal_release_bundle_assembler_rejects_hash_mismatch) ... ok
test_scanner_heal_release_bundle_assembler_rejects_mixed_descriptor_revisions (__main__.SelfTests.test_scanner_heal_release_bundle_assembler_rejects_mixed_descriptor_revisions) ... ok
test_scanner_heal_release_bundle_enforces_topology_duration_profile_and_versions (__main__.SelfTests.test_scanner_heal_release_bundle_enforces_topology_duration_profile_and_versions) ... ok
test_scanner_heal_release_bundle_fixture_writer_is_not_release_evidence (__main__.SelfTests.test_scanner_heal_release_bundle_fixture_writer_is_not_release_evidence) ... ok
test_scanner_heal_release_bundle_gate_accepts_single_measured_gate (__main__.SelfTests.test_scanner_heal_release_bundle_gate_accepts_single_measured_gate) ... ok
test_scanner_heal_release_bundle_gate_rejects_missing_field (__main__.SelfTests.test_scanner_heal_release_bundle_gate_rejects_missing_field) ... ok
test_scanner_heal_release_bundle_gate_rejects_unknown_gate (__main__.SelfTests.test_scanner_heal_release_bundle_gate_rejects_unknown_gate) ... ok
test_scanner_heal_release_bundle_rejects_synthetic_or_missing_fields (__main__.SelfTests.test_scanner_heal_release_bundle_rejects_synthetic_or_missing_fields) ... ok
test_scanner_heal_release_bundle_requires_domain_evidence (__main__.SelfTests.test_scanner_heal_release_bundle_requires_domain_evidence) ... ok
test_scanner_heal_release_bundle_requires_field_provenance (__main__.SelfTests.test_scanner_heal_release_bundle_requires_field_provenance) ... ok
test_scanner_heal_release_bundle_requires_json_artifact_provenance (__main__.SelfTests.test_scanner_heal_release_bundle_requires_json_artifact_provenance) ... ok
test_scanner_heal_release_bundle_requires_mixed_version_field_roles (__main__.SelfTests.test_scanner_heal_release_bundle_requires_mixed_version_field_roles) ... ok
test_scanner_heal_release_bundle_requires_same_gate_measurement_window (__main__.SelfTests.test_scanner_heal_release_bundle_requires_same_gate_measurement_window) ... ok
test_scanner_heal_release_consumes_multiple_registry_oracles (__main__.SelfTests.test_scanner_heal_release_consumes_multiple_registry_oracles) ... ok
test_scanner_heal_release_matrix_lanes_remain_pending_without_real_evidence (__main__.SelfTests.test_scanner_heal_release_matrix_lanes_remain_pending_without_real_evidence) ... ok
test_scanner_heal_release_requirements_cannot_be_cleared (__main__.SelfTests.test_scanner_heal_release_requirements_cannot_be_cleared) ... ok
test_scanner_heal_release_status_rejects_focused_case_run (__main__.SelfTests.test_scanner_heal_release_status_rejects_focused_case_run) ... ok
test_scanner_heal_release_status_rejects_synthetic_case (__main__.SelfTests.test_scanner_heal_release_status_rejects_synthetic_case) ... ok
test_scanner_heal_required_evidence_fields_cannot_be_removed (__main__.SelfTests.test_scanner_heal_required_evidence_fields_cannot_be_removed) ... ok
test_scheduled_alerts_require_completion_watchdog (__main__.SelfTests.test_scheduled_alerts_require_completion_watchdog) ... ok
test_update_profile_selection_changes_only_requested_platform (__main__.SelfTests.test_update_profile_selection_changes_only_requested_platform) ... ok
test_validate_includes_quick_checks (__main__.SelfTests.test_validate_includes_quick_checks) ... ok
test_vault_tests_require_cross_process_serialization (__main__.SelfTests.test_vault_tests_require_cross_process_serialization) ... ok
test_workflow_readiness_requires_dependency_probe (__main__.SelfTests.test_workflow_readiness_requires_dependency_probe) ... ok

----------------------------------------------------------------------
Ran 52 tests in 14.301s

OK
e2e-full selection OK: 1 tests, sha256=29e30e6ed2ed38d9e5b0a7cdd9536abf5c719f7021a02b578ae1381563a3c273
...................................
----------------------------------------------------------------------
Ran 35 tests in 1.161s

OK
...................
----------------------------------------------------------------------
Ran 19 tests in 17.397s

OK
..........
----------------------------------------------------------------------
Ran 10 tests in 0.183s

OK
test_actual_selector_bootstrap_uses_base_policy_and_fails_closed (__main__.SelfTests.test_actual_selector_bootstrap_uses_base_policy_and_fails_closed) ... ok
test_documentation_paths_do_not_hide_build_or_fixture_changes (__main__.SelfTests.test_documentation_paths_do_not_hide_build_or_fixture_changes) ... ok
test_e2e_scope_keeps_production_and_shared_configuration_full (__main__.SelfTests.test_e2e_scope_keeps_production_and_shared_configuration_full) ... ok
test_e2e_shortcut_rejects_new_direct_renamed_and_target_dependencies (__main__.SelfTests.test_e2e_shortcut_rejects_new_direct_renamed_and_target_dependencies) ... ok
test_e2e_shortcut_traverses_implicit_path_dependencies (__main__.SelfTests.test_e2e_shortcut_traverses_implicit_path_dependencies) ... ok
test_event_contract_requires_complete_candidate_and_optional_lanes (__main__.SelfTests.test_event_contract_requires_complete_candidate_and_optional_lanes) ... ok
test_every_wrong_result_missing_job_or_selection_fails_closed (__main__.SelfTests.test_every_wrong_result_missing_job_or_selection_fails_closed) ... ok
test_full_e2e_gate_preserves_workflow_branch_and_event_scope (__main__.SelfTests.test_full_e2e_gate_preserves_workflow_branch_and_event_scope) ... ok
test_git_range_includes_deleted_source_and_rename_origins (__main__.SelfTests.test_git_range_includes_deleted_source_and_rename_origins) ... ok
test_job_ids_and_display_names_cannot_hide_validation (__main__.SelfTests.test_job_ids_and_display_names_cannot_hide_validation) ... ok
test_report_only_stack_lane_mirrors_e2e_smoke_outside_the_aggregate (__main__.SelfTests.test_report_only_stack_lane_mirrors_e2e_smoke_outside_the_aggregate) ... ok
test_repository_wiring_and_missing_dependency_regression (__main__.SelfTests.test_repository_wiring_and_missing_dependency_regression) ... ok
test_verify_command_preserves_failures (__main__.SelfTests.test_verify_command_preserves_failures) ... ok

----------------------------------------------------------------------
Ran 13 tests in 3.280s

OK
security coverage self-test passed
test_api_failures_preserve_other_evidence_and_never_enter_grace (__main__.SelfTests.test_api_failures_preserve_other_evidence_and_never_enter_grace) ... ok
test_attempt_outcome_does_not_replace_recent_success (__main__.SelfTests.test_attempt_outcome_does_not_replace_recent_success) ... ok
test_cancelled_attempt_cannot_refresh_expired_success (__main__.SelfTests.test_cancelled_attempt_cannot_refresh_expired_success) ... ok
test_cli_requires_the_repository_default_branch (__main__.SelfTests.test_cli_requires_the_repository_default_branch) ... ok
test_config_rejects_duplicate_and_invalid_entries (__main__.SelfTests.test_config_rejects_duplicate_and_invalid_entries) ... ok
test_freshness_boundaries (__main__.SelfTests.test_freshness_boundaries) ... ok
test_grace_requires_two_successful_queries_with_no_history (__main__.SelfTests.test_grace_requires_two_successful_queries_with_no_history) ... ok
test_invalid_api_evidence_fails_closed (__main__.SelfTests.test_invalid_api_evidence_fails_closed) ... ok
test_report_retains_every_workflow (__main__.SelfTests.test_report_retains_every_workflow) ... ok
test_requests_filter_schedule_default_branch_and_success_on_server (__main__.SelfTests.test_requests_filter_schedule_default_branch_and_success_on_server) ... ok

----------------------------------------------------------------------
Ran 10 tests in 0.051s

OK
..............................
----------------------------------------------------------------------
Ran 30 tests in 22.354s

OK
..........
----------------------------------------------------------------------
Ran 10 tests in 2.969s

OK
.................................
----------------------------------------------------------------------
Ran 33 tests in 0.160s

OK
Online functional runner labels: pf-testing
.......
----------------------------------------------------------------------
Ran 7 tests in 0.002s

OK
.......
----------------------------------------------------------------------
Ran 7 tests in 0.001s

OK
....
----------------------------------------------------------------------
Ran 4 tests in 0.128s

OK
...
----------------------------------------------------------------------
Ran 3 tests in 0.547s

OK
PASS: cold stampede script self-test
exit_code=0

$ python3 ./scripts/test_object_generation_protocol_model.py
test_a_recovery_barrier_cannot_choose_an_empty_value (__main__.ObjectGenerationProtocolModel.test_a_recovery_barrier_cannot_choose_an_empty_value) ... ok
test_all_bounded_two_proposer_interleavings_choose_at_most_one_value (__main__.ObjectGenerationProtocolModel.test_all_bounded_two_proposer_interleavings_choose_at_most_one_value) ... ok
test_an_unavailable_decision_quorum_cannot_choose_a_candidate (__main__.ObjectGenerationProtocolModel.test_an_unavailable_decision_quorum_cannot_choose_a_candidate) ... ok
test_counter_allocation_requires_persistence_and_is_consumed_once (__main__.ObjectGenerationProtocolModel.test_counter_allocation_requires_persistence_and_is_consumed_once) ... ok
test_counter_restart_burns_persisted_allocations_before_or_after_issue (__main__.ObjectGenerationProtocolModel.test_counter_restart_burns_persisted_allocations_before_or_after_issue) ... ok
test_crash_after_promise_before_reply_does_not_restore_an_older_ballot (__main__.ObjectGenerationProtocolModel.test_crash_after_promise_before_reply_does_not_restore_an_older_ballot) ... ok
test_duplicate_or_unconfigured_voters_cannot_form_a_quorum (__main__.ObjectGenerationProtocolModel.test_duplicate_or_unconfigured_voters_cannot_form_a_quorum) ... ok
test_empty_or_unobserved_minority_slot_can_retry_without_allocating_another_revision (__main__.ObjectGenerationProtocolModel.test_empty_or_unobserved_minority_slot_can_retry_without_allocating_another_revision) ... ok
test_equal_ballot_cannot_accept_different_bytes_and_lower_ballot_is_rejected (__main__.ObjectGenerationProtocolModel.test_equal_ballot_cannot_accept_different_bytes_and_lower_ballot_is_rejected) ... ok
test_forgetting_tombstone_authority_allows_an_aba_create (__main__.ObjectGenerationProtocolModel.test_forgetting_tombstone_authority_allows_an_aba_create) ... ok
test_issuing_before_persistence_reuses_a_ballot_after_restart (__main__.ObjectGenerationProtocolModel.test_issuing_before_persistence_reuses_a_ballot_after_restart) ... ok
test_lost_ack_and_restart_preserve_the_chosen_operation (__main__.ObjectGenerationProtocolModel.test_lost_ack_and_restart_preserve_the_chosen_operation) ... ok
test_minority_accepted_work_is_adopted_and_finished (__main__.ObjectGenerationProtocolModel.test_minority_accepted_work_is_adopted_and_finished) ... ok
test_omitting_highest_accepted_adoption_has_a_double_choice_counterexample (__main__.ObjectGenerationProtocolModel.test_omitting_highest_accepted_adoption_has_a_double_choice_counterexample) ... ok
test_quorum_choice_does_not_require_an_uncontacted_disk_to_learn_it (__main__.ObjectGenerationProtocolModel.test_quorum_choice_does_not_require_an_uncontacted_disk_to_learn_it) ... ok
test_recovery_failure_or_stale_expected_head_does_not_allocate_a_successor (__main__.ObjectGenerationProtocolModel.test_recovery_failure_or_stale_expected_head_does_not_allocate_a_successor) ... ok
test_replacing_voters_with_empty_state_can_choose_a_conflicting_value (__main__.ObjectGenerationProtocolModel.test_replacing_voters_with_empty_state_can_choose_a_conflicting_value) ... ok
test_skipping_predecessor_recovery_chooses_a_broken_revision_chain (__main__.ObjectGenerationProtocolModel.test_skipping_predecessor_recovery_chooses_a_broken_revision_chain) ... ok
test_stale_publication_rollback_and_gc_cannot_change_the_winner (__main__.ObjectGenerationProtocolModel.test_stale_publication_rollback_and_gc_cannot_change_the_winner) ... ok
test_superseded_operation_identity_cannot_be_reused_for_different_input (__main__.ObjectGenerationProtocolModel.test_superseded_operation_identity_cannot_be_reused_for_different_input) ... ok
test_superseded_operation_retry_returns_its_original_successor (__main__.ObjectGenerationProtocolModel.test_superseded_operation_retry_returns_its_original_successor) ... ok
test_tombstone_and_null_marker_payloads_do_not_reset_authority_revision (__main__.ObjectGenerationProtocolModel.test_tombstone_and_null_marker_payloads_do_not_reset_authority_revision) ... ok
test_tombstone_rejects_a_delayed_never_present_create (__main__.ObjectGenerationProtocolModel.test_tombstone_rejects_a_delayed_never_present_create) ... ok
test_unlearned_predecessor_is_recovered_before_the_second_slot (__main__.ObjectGenerationProtocolModel.test_unlearned_predecessor_is_recovered_before_the_second_slot) ... ok

----------------------------------------------------------------------
Ran 24 tests in 0.958s

OK
model: states=51023, transitions=167520, terminal=732, chosen_candidates=2
negative control: double-choice trace=((0, 'prepare', 0), (0, 'prepare', 1), (0, 'prepare', 2), (0, 'accept', 0), (0, 'accept', 1), (0, 'accept', 2), (1, 'prepare', 0), (1, 'prepare', 1), (1, 'prepare', 2), (1, 'accept', 0), (1, 'accept', 1), (1, 'accept', 2))
multi-slot round schedules: 640
exit_code=0

$ python3 ./scripts/check_test_wiring.py
OK: e2e modules, runner selection, fuzz matrices, profiles, scheduled alerts, and Quick Checks are wired
exit_code=0

$ ./scripts/check_no_planning_docs.sh
exit_code=0

$ ./scripts/check_uring_lane_lib_only.sh
OK: no 'uring_'-matching test functions under crates/ecstore/tests; --lib is safe
exit_code=0

Initial complete gate
$ cargo nextest run --profile ci -p rustfs --test connect_perf_drive --no-fail-fast
 Nextest run ID e07fc5b9-c6c8-4cdd-ae61-54839bf24437 with nextest profile: ci
    Starting 10 tests across 1 binary
        PASS [   0.016s] rustfs::connect_perf_drive cancellation_and_single_collector_leave_no_scratch
        PASS [   0.011s] rustfs::connect_perf_drive cleanup_failure_overrides_cancellation_and_remains_attributed
        PASS [   0.009s] rustfs::connect_perf_drive consent_and_resource_budgets_fail_before_scratch_io
        PASS [   0.009s] rustfs::connect_perf_drive invalid_roots_and_existing_scratch_fail_without_clobbering
        PASS [  16.950s] rustfs::connect_perf_drive production_cli_measures_and_signs_exact_binary_provenance
        PASS [   4.025s] rustfs::connect_perf_drive production_cli_rejects_missing_l1_consent_and_limits_before_artifacts
        PASS [   0.008s] rustfs::connect_perf_drive read_only_root_and_full_drive_errors_are_classified
        PASS [   0.015s] rustfs::connect_perf_drive real_local_write_read_is_measured_and_scratch_is_removed
        PASS [   0.007s] rustfs::connect_perf_drive runtime_timeout_and_full_drive_use_schema_valid_failed_reason
        PASS [   0.035s] rustfs::connect_perf_drive successful_result_has_signed_bounded_private_offline_export
────────────
     Summary [  21.087s] 10 tests run: 10 passed, 0 skipped
exit_code=0

Restored complete gate
$ cargo nextest run --profile ci -p rustfs --test connect_perf_drive --no-fail-fast
 Nextest run ID 233b7307-5361-4eb7-94d5-b384f17d2d6a with nextest profile: ci
    Starting 10 tests across 1 binary
        PASS [   0.009s] rustfs::connect_perf_drive cancellation_and_single_collector_leave_no_scratch
        PASS [   0.008s] rustfs::connect_perf_drive cleanup_failure_overrides_cancellation_and_remains_attributed
        PASS [   0.008s] rustfs::connect_perf_drive consent_and_resource_budgets_fail_before_scratch_io
        PASS [   0.008s] rustfs::connect_perf_drive invalid_roots_and_existing_scratch_fail_without_clobbering
        PASS [  19.518s] rustfs::connect_perf_drive production_cli_measures_and_signs_exact_binary_provenance
        PASS [   4.146s] rustfs::connect_perf_drive production_cli_rejects_missing_l1_consent_and_limits_before_artifacts
        PASS [   0.009s] rustfs::connect_perf_drive read_only_root_and_full_drive_errors_are_classified
        PASS [   0.018s] rustfs::connect_perf_drive real_local_write_read_is_measured_and_scratch_is_removed
        PASS [   0.009s] rustfs::connect_perf_drive runtime_timeout_and_full_drive_use_schema_valid_failed_reason
        PASS [   0.054s] rustfs::connect_perf_drive successful_result_has_signed_bounded_private_offline_export
────────────
     Summary [  23.787s] 10 tests run: 10 passed, 0 skipped
exit_code=0

Mutation M1: success_measurement reports max(actual elapsed, requested budget + 1), forcing 1001 ms for the one-second request. The retained duration assertion at rustfs/tests/connect_perf_drive.rs:105 fails.
$ cargo nextest run --profile ci -p rustfs --test connect_perf_drive --no-fail-fast -E 'test(real_local_write_read_is_measured_and_scratch_is_removed)'
info: experimental features enabled: setup-scripts
   Compiling rustfs v1.0.1 (/private/tmp/gw-orchestrator-2734/ci-drive/rustfs)
warning: linker stderr: ld: __eh_frame section too large (max 16MB) to encode dwarf unwind offsets in compact unwind table, performance of exception handling might be affected
  |
  = note: `#[warn(linker_messages)]` on by default

warning: `rustfs` (bin "rustfs") generated 1 warning
warning: `rustfs` (bin "rustfs-cli") generated 1 warning (1 duplicate)
    Finished `test` profile [unoptimized + debuginfo] target(s) in 7m 06s
────────────
 Nextest run ID 9f618777-7bf6-4cbc-8803-dd7f50f449ed with nextest profile: ci
    Starting 1 test across 1 binary (9 tests skipped)
        FAIL [   0.016s] rustfs::connect_perf_drive real_local_write_read_is_measured_and_scratch_is_removed
  stdout ───

    running 1 test
    test real_local_write_read_is_measured_and_scratch_is_removed ... FAILED

    failures:

    failures:
        real_local_write_read_is_measured_and_scratch_is_removed

    test result: FAILED. 0 passed; 1 failed; 0 ignored; 0 measured; 9 filtered out; finished in 0.01s

  stderr ───

    thread 'real_local_write_read_is_measured_and_scratch_is_removed' (106986846) panicked at rustfs/tests/connect_perf_drive.rs:105:5:
    assertion failed: (1..=1_000).contains(&data.duration_millis)
    note: run with `RUST_BACKTRACE=1` environment variable to display a backtrace

────────────
     Summary [   0.017s] 1 test run: 0 passed, 1 failed, 9 skipped
        FAIL [   0.016s] rustfs::connect_perf_drive real_local_write_read_is_measured_and_scratch_is_removed
error: test run failed
exit_code=100

Mutation M2: target_parameters reports requested duration + 1. The in-process assertion at rustfs/tests/connect_perf_drive.rs:111 and CLI JSON assertion at rustfs/tests/connect_perf_drive.rs:527 both reject 1001 against 1000.
$ cargo nextest run --profile ci -p rustfs --test connect_perf_drive --no-fail-fast -E 'test(real_local_write_read_is_measured_and_scratch_is_removed) | test(production_cli_measures_and_signs_exact_binary_provenance)'
info: experimental features enabled: setup-scripts
   Compiling rustfs v1.0.1 (/private/tmp/gw-orchestrator-2734/ci-drive/rustfs)
warning: linker stderr: ld: __eh_frame section too large (max 16MB) to encode dwarf unwind offsets in compact unwind table, performance of exception handling might be affected
  |
  = note: `#[warn(linker_messages)]` on by default

warning: `rustfs` (bin "rustfs") generated 1 warning
warning: `rustfs` (bin "rustfs-cli") generated 1 warning (1 duplicate)
    Finished `test` profile [unoptimized + debuginfo] target(s) in 7m 20s
────────────
 Nextest run ID b73d46ab-c910-4e1e-8063-7fc7e332ec15 with nextest profile: ci
    Starting 2 tests across 1 binary (8 tests skipped)
        FAIL [  13.968s] rustfs::connect_perf_drive production_cli_measures_and_signs_exact_binary_provenance
  stdout ───

    running 1 test
    test production_cli_measures_and_signs_exact_binary_provenance ... FAILED

    failures:

    failures:
        production_cli_measures_and_signs_exact_binary_provenance

    test result: FAILED. 0 passed; 1 failed; 0 ignored; 0 measured; 9 filtered out; finished in 13.96s

  stderr ───

    thread 'production_cli_measures_and_signs_exact_binary_provenance' (107015462) panicked at rustfs/tests/connect_perf_drive.rs:527:5:
    assertion `left == right` failed
      left: Number(1001)
     right: 1000
    note: run with `RUST_BACKTRACE=1` environment variable to display a backtrace

        FAIL [   0.015s] rustfs::connect_perf_drive real_local_write_read_is_measured_and_scratch_is_removed
  stdout ───

    running 1 test
    test real_local_write_read_is_measured_and_scratch_is_removed ... FAILED

    failures:

    failures:
        real_local_write_read_is_measured_and_scratch_is_removed

    test result: FAILED. 0 passed; 1 failed; 0 ignored; 0 measured; 9 filtered out; finished in 0.01s

  stderr ───

    thread 'real_local_write_read_is_measured_and_scratch_is_removed' (107016667) panicked at rustfs/tests/connect_perf_drive.rs:111:5:
    assertion `left == right` failed
      left: 1001
     right: 1000
    note: run with `RUST_BACKTRACE=1` environment variable to display a backtrace

────────────
     Summary [  13.985s] 2 tests run: 0 passed, 2 failed, 8 skipped
        FAIL [  13.968s] rustfs::connect_perf_drive production_cli_measures_and_signs_exact_binary_provenance
        FAIL [   0.015s] rustfs::connect_perf_drive real_local_write_read_is_measured_and_scratch_is_removed
error: test run failed
exit_code=100

Transient production mutations were restored byte-for-byte before the complete green rerun. Original/restored SHA256: 8d389e6d8836c041aab8c6a93e56addd65ba3f70b33264e1c74f639d0572a2da.

$ git diff --check

exit_code=0

$ git diff --cached --check

exit_code=0

Independent orchestrator acceptance
Tested code tree: e67c72fb01

Tested tree: 16fbbd2ea4
shellcheck --version && actionlint -> exit 0
cargo fmt --all --check -> exit 0
./scripts/check_unsafe_code_allowances.sh -> exit 0
./scripts/check_layer_dependencies.sh -> exit 0
./scripts/check_architecture_migration_rules.sh -> exit 0
./scripts/check_logging_guardrails.sh -> exit 0
./scripts/check_error_other_format_ratchet.sh -> exit 0
./scripts/check_no_tokio_io_uring.sh -> exit 0
./scripts/check_extension_schema_boundaries.sh -> exit 0
./scripts/check_body_cache_whitelist.sh -> exit 0
./scripts/check_s3s_footprint.sh -> exit 0
./scripts/check_test_count_ratchet.sh -> exit 0
./scripts/check_fips_wording.sh -> exit 0
./scripts/check_embedded_secrets.sh -> exit 0
make script-tests -> exit 0
python3 ./scripts/test_object_generation_protocol_model.py -> exit 0
python3 ./scripts/check_test_wiring.py -> exit 0
./scripts/check_no_planning_docs.sh -> exit 0
./scripts/check_uring_lane_lib_only.sh -> exit 0

$ LANG=en_US.UTF-8 CARGO_BUILD_JOBS=1 CARGO_INCREMENTAL=0 cargo nextest run --profile ci -p rustfs --test connect_perf_drive --no-fail-fast
info: experimental features enabled: setup-scripts
warning: linker stderr: ld: __eh_frame section too large (max 16MB) to encode dwarf unwind offsets in compact unwind table, performance of exception handling might be affected
  |
  = note: `#[warn(linker_messages)]` on by default

warning: `rustfs` (bin "rustfs-cli") generated 1 warning
warning: `rustfs` (bin "rustfs") generated 1 warning (1 duplicate)
    Finished `test` profile [unoptimized + debuginfo] target(s) in 0.69s
────────────
 Nextest run ID cd1edf4c-f1c0-456d-ad1a-76cc00ce05c9 with nextest profile: ci
    Starting 10 tests across 1 binary
        PASS [   0.012s] rustfs::connect_perf_drive cancellation_and_single_collector_leave_no_scratch
        PASS [   0.010s] rustfs::connect_perf_drive cleanup_failure_overrides_cancellation_and_remains_attributed
        PASS [   0.010s] rustfs::connect_perf_drive consent_and_resource_budgets_fail_before_scratch_io
        PASS [   0.009s] rustfs::connect_perf_drive invalid_roots_and_existing_scratch_fail_without_clobbering
        PASS [  20.328s] rustfs::connect_perf_drive production_cli_measures_and_signs_exact_binary_provenance
        PASS [   5.527s] rustfs::connect_perf_drive production_cli_rejects_missing_l1_consent_and_limits_before_artifacts
        PASS [   0.013s] rustfs::connect_perf_drive read_only_root_and_full_drive_errors_are_classified
        PASS [   0.019s] rustfs::connect_perf_drive real_local_write_read_is_measured_and_scratch_is_removed
        PASS [   0.011s] rustfs::connect_perf_drive runtime_timeout_and_full_drive_use_schema_valid_failed_reason
        PASS [   0.049s] rustfs::connect_perf_drive successful_result_has_signed_bounded_private_offline_export
────────────
     Summary [  25.991s] 10 tests run: 10 passed, 0 skipped

exit_code=0

All 54 tracked Cargo.toml files parse without duplicate keys. git diff --check passed. The production source remains unchanged; the two deliberate implementation mutations above were restored before this complete green run.

* fix(test): compare legacy body wire parity as raw bytes

Lossy UTF-8 comparison concealed a changed binary response byte.
Compare the already captured Vec<u8> values directly.

Validation:
- Binary mutant: migrated payload 0x84 vs legacy 0x83; old comparator
  passed 1/1, raw comparator failed 0/1 (exit 100); restored green.
- Non-Swift body-drain mutant: untouched-body delegation test failed
  with empty vs payload (exit 100); restored handler without a diff.
- cargo nextest run --profile ci -p rustfs-s3-types -p rustfs-protocols
  --features rustfs-protocols/swift --no-fail-fast:
  543 tests run, 543 passed, 0 skipped after both mutants restored.
- cargo fmt --all --check and git diff --check passed.

Orchestrator acceptance (LANG=en_US.UTF-8):
cargo nextest run --profile ci -p rustfs-s3-types -p rustfs-protocols --features rustfs-protocols/swift --no-fail-fast
Summary [2.445s] 543 tests run: 543 passed, 0 skipped.

All shared quick checks on the final code tree:
shellcheck --version && actionlint -> exit 0
cargo fmt --all --check -> exit 0
./scripts/check_unsafe_code_allowances.sh -> exit 0
./scripts/check_layer_dependencies.sh -> exit 0
./scripts/check_architecture_migration_rules.sh -> exit 0
./scripts/check_logging_guardrails.sh -> exit 0
./scripts/check_error_other_format_ratchet.sh -> exit 0
./scripts/check_no_tokio_io_uring.sh -> exit 0
./scripts/check_extension_schema_boundaries.sh -> exit 0
./scripts/check_body_cache_whitelist.sh -> exit 0
./scripts/check_s3s_footprint.sh -> exit 0
./scripts/check_test_count_ratchet.sh -> exit 0
./scripts/check_fips_wording.sh -> exit 0
./scripts/check_embedded_secrets.sh -> exit 0
make script-tests -> exit 0
python3 ./scripts/test_object_generation_protocol_model.py -> exit 0
python3 ./scripts/check_test_wiring.py -> exit 0
./scripts/check_no_planning_docs.sh -> exit 0
./scripts/check_uring_lane_lib_only.sh -> exit 0

Final independent acceptance after the drive-test CI repair
Parent: a99d8f4f89
Tested code tree: 93cbd96bf0

Tested tree: 075ae5161c2b62a96812bc1e3c904a7663b4f2f8
shellcheck --version && actionlint -> exit 0
cargo fmt --all --check -> exit 0
./scripts/check_unsafe_code_allowances.sh -> exit 0
./scripts/check_layer_dependencies.sh -> exit 0
./scripts/check_architecture_migration_rules.sh -> exit 0
./scripts/check_logging_guardrails.sh -> exit 0
./scripts/check_error_other_format_ratchet.sh -> exit 0
./scripts/check_no_tokio_io_uring.sh -> exit 0
./scripts/check_extension_schema_boundaries.sh -> exit 0
./scripts/check_body_cache_whitelist.sh -> exit 0
./scripts/check_s3s_footprint.sh -> exit 0
./scripts/check_test_count_ratchet.sh -> exit 0
./scripts/check_fips_wording.sh -> exit 0
./scripts/check_embedded_secrets.sh -> exit 0
make script-tests -> exit 0
python3 ./scripts/test_object_generation_protocol_model.py -> exit 0
python3 ./scripts/check_test_wiring.py -> exit 0
./scripts/check_no_planning_docs.sh -> exit 0
./scripts/check_uring_lane_lib_only.sh -> exit 0

$ LANG=en_US.UTF-8 CARGO_BUILD_JOBS=1 CARGO_INCREMENTAL=0 cargo nextest run --profile ci -p rustfs-s3-types -p rustfs-protocols --features rustfs-protocols/swift --no-fail-fast
    Finished `test` profile [unoptimized + debuginfo] target(s) in 10m 08s
 Nextest run ID 5c7ff102-ec6d-479a-86e0-810859b28bbf with nextest profile: ci
    Starting 543 tests across 8 binaries
     Summary [   6.729s] 543 tests run: 543 passed, 0 skipped
exit_code=0

The only source change is the raw-byte wire comparator. Cargo manifests and the ratchet baselines remain unchanged from the independently validated parent. The invalid-byte and consumed-request mutations remain recorded above; their implementation changes were restored.

* fix: repair integration checks and IPC test readiness

Wait for the RPC/locks test sockets to pass the existing owner and mode
checks inside the original 5s readiness window and 10ms poll interval.

Use the existing admin S3 carrier for the new recovery and quota error
calls. Regenerate the test-count baseline for main's 118 existing tests
across seven crates. Pair the private pagination helper's raw-scan inputs
and remove the redundant Copy clone from the retirement event.

Keep the original assertions, responses, authorization, locks and storage
operations. The pagination body and raw-scan expressions at all five calls
are byte-identical; reinserting only the Copy clone restores the entire
base heal file. All ignored baseline values and s3s counts stay unchanged.

Validation base: c47432e353
Validation tree: f081084bc5
Patch SHA-256: 2dcf0028691bfb2772edfb38e262829b04f4151832dcce823c9dd9f7434b52de
Trace production-prefix SHA-256: fd994f218a21429d62866f4efb8eee1ee57b58f4a51865aa35d05a54ef1ccb45

All commands used LANG=en_US.UTF-8, CARGO_BUILD_JOBS=1,
CARGO_INCREMENTAL=0 and CARGO_NET_OFFLINE=true in one exclusive target.
Every gate retained the same tree and six source hashes before and after.
54 Cargo.toml files parse without duplicate keys.

cargo clippy -p rustfs --all-targets -- -D warnings
    Finished `dev` profile [unoptimized + debuginfo] target(s) in 12m 50s
Exit: 0. Wall time: 770.855s.

The original 19 quick-check steps:
shellcheck --version && actionlint -> exit 0
cargo fmt --all --check -> exit 0
./scripts/check_unsafe_code_allowances.sh -> exit 0
./scripts/check_layer_dependencies.sh -> exit 0
./scripts/check_architecture_migration_rules.sh -> exit 0
./scripts/check_logging_guardrails.sh -> exit 0
./scripts/check_error_other_format_ratchet.sh -> exit 0
./scripts/check_no_tokio_io_uring.sh -> exit 0
./scripts/check_extension_schema_boundaries.sh -> exit 0
./scripts/check_body_cache_whitelist.sh -> exit 0
./scripts/check_s3s_footprint.sh -> exit 0
./scripts/check_test_count_ratchet.sh -> exit 0
./scripts/check_fips_wording.sh -> exit 0
./scripts/check_embedded_secrets.sh -> exit 0
make script-tests -> exit 0
python3 ./scripts/test_object_generation_protocol_model.py -> exit 0
python3 ./scripts/check_test_wiring.py -> exit 0
./scripts/check_no_planning_docs.sh -> exit 0
./scripts/check_uring_lane_lib_only.sh -> exit 0
Wall time: 286.337s.

cargo nextest run --profile ci -p rustfs --no-fail-fast
     Summary [ 217.017s] 5382 tests run: 5382 passed (4 leaky), 40 skipped
Exit: 0. Wall time: 1324.414s.

cargo nextest run --profile ci -p rustfs-s3-types -p rustfs-protocols --features rustfs-protocols/swift --no-fail-fast
     Summary [   2.189s] 543 tests run: 543 passed, 0 skipped
Exit: 0. Wall time: 123.192s.

cargo nextest run --profile ci -p rustfs-ecstore list_objects_paginate --no-fail-fast
     Summary [   0.170s] 8 tests run: 8 passed, 5921 skipped
Exit: 0. Wall time: 261.324s.

cargo nextest run --profile ci -p rustfs-heal durable_mrf_anchor_retires_only_after_exact_bucket_generation_is_deleted --no-fail-fast
     Summary [   0.805s] 1 test run: 1 passed, 835 skipped
Exit: 0. Wall time: 199.801s.

Baseline regeneration:
./scripts/check_test_count_ratchet.sh --update
wrote .config/test-count-baseline.txt: 52 crates, 21340 tests, 164 ignored
Test-only increases: ecstore48, heal3, iam14, io-metrics2, obs3, policy14,
rustfs34. Existing ignored values and all other rows are byte-identical.
s3s footprint OK: files importing s3s is 195 (baseline files=195)
s3s footprint OK: s3s s3_error! lines is 1584 (baseline s3_error_lines=1584)
s3s footprint OK: ecstore files referencing s3s is 36 (baseline ecstore_files=36)

Historical runtime red controls on f622 (same retained two assertions):
cargo nextest run --profile ci -p rustfs --lib -E 'test(=connect::diagnostics::trace_runtime::tests::local_top_rpc_reads_service_http_completions_and_signs_offline) | test(=connect::diagnostics::trace_runtime::tests::local_top_locks_reads_service_manager_and_signs_offline)' --no-fail-fast
Original path readiness with a temporary 500ms bind/seal gap:
     Summary [   0.863s] 2 tests run: 0 passed, 2 failed, 5113 skipped
Exit: 100.
Sealed readiness in that same gap:
     Summary [   5.621s] 2 tests run: 2 passed, 5113 skipped
Exit: 0.
Compiled M1: replace the helper predicate with path.exists():
     Summary [   0.870s] 2 tests run: 0 passed, 2 failed, 5113 skipped
Exit: 100.
Restore the sealed predicate byte-for-byte:
     Summary [   5.640s] 2 tests run: 2 passed, 5113 skipped
Exit: 0.
Both reds observe mode755 with owner_matches=true and fail at the original
RPC/locks unwraps with StateSecurity. M1 failure locations: trace_runtime.rs
2547:29 and 2713:29. No new assertions were introduced.

cargo nextest run --profile ci -p rustfs --lib -E 'test(connect::diagnostics::trace_runtime::tests::)' --no-fail-fast
     Summary [  20.748s] 28 tests run: 28 passed (1 leaky), 5087 skipped
Exit: 0. This security sweep is historical; the full final unit gate above
covers the actual c474 candidate.
The first disk-stopped compile ran no tests and is excluded from red credit.
All temporary seal instrumentation is removed from the final source.

Final unit cleanup markers: 4 leaky, 40 original skips (exit0). The ld
__eh_frame linker warning also appears verbatim in the original f622 run
and the earlier c474 run; there are no new source warnings.

* fix: remove redundant pending closure in disk test

Pass the capture-free pending future factory as a function item in the existing slow-operation test. This fixes the redundant_closure error at disk_store.rs:2914 that stopped strict Clippy before the feature tests could run.

The single-line test edit preserves the factory's result, warning and timeout behavior. Reversing the replacement reproduces the parent file byte for byte. All 220 assertion lines and the production prefix are unchanged; no tests, assertions, guards, profiles or baselines were added or relaxed.

Original Linux CI evidence (run 37888552268 at 53245be9f):
- RIO job113685483992: strict Clippy exit101, redundant closure at disk_store.rs:2914; feature tests skipped.
- Workspace Test and Lint job113685484058: same strict Clippy exit101; nextest and doc tests skipped.
These are lint failures, not runtime mutation kills or known flakes.

Native macOS arm64 validation on tree7a5bf47e7533b29479bc85207416102f1337c566, LANG=en_US.UTF-8, one build job, CARGO_INCREMENTAL=0 and offline dependencies:

shellcheck --version && actionlint -> exit 0
cargo fmt --all --check -> exit 0
./scripts/check_unsafe_code_allowances.sh -> exit 0
./scripts/check_layer_dependencies.sh -> exit 0
./scripts/check_architecture_migration_rules.sh -> exit 0
./scripts/check_logging_guardrails.sh -> exit 0
./scripts/check_error_other_format_ratchet.sh -> exit 0
./scripts/check_no_tokio_io_uring.sh -> exit 0
./scripts/check_extension_schema_boundaries.sh -> exit 0
./scripts/check_body_cache_whitelist.sh -> exit 0
./scripts/check_s3s_footprint.sh -> exit 0
./scripts/check_test_count_ratchet.sh -> exit 0
./scripts/check_fips_wording.sh -> exit 0
./scripts/check_embedded_secrets.sh -> exit 0
make script-tests -> exit 0
python3 ./scripts/test_object_generation_protocol_model.py -> exit 0
python3 ./scripts/check_test_wiring.py -> exit 0
./scripts/check_no_planning_docs.sh -> exit 0
./scripts/check_uring_lane_lib_only.sh -> exit 0
All 19 checks: exit0, 312.161s.

$ cargo clippy -p rustfs -p rustfs-ecstore --all-targets --features rio-v2 -- -D warnings
Finished `dev` profile [unoptimized + debuginfo] target(s) in 20m 11s
exit0, 1213.210s total.

$ cargo nextest run --profile ci -p rustfs-ecstore --features rio-v2 slow_disk_operation_warns_once_while_pending_and_preserves_result --no-fail-fast
Finished `test` profile [unoptimized + debuginfo] target(s) in 4m 34s
PASS [0.026s] rustfs-ecstore disk::disk_store::tests::slow_disk_operation_warns_once_while_pending_and_preserves_result
Summary [   0.170s] 1 test run: 1 passed, 5943 skipped
exit0, 281.920s total; 5943 existing filter skips.

warning: linker stderr: ld: __eh_frame section too large (max 16MB) to encode dwarf unwind offsets in compact unwind table, performance of exception handling might be affected
This linker warning is retained; flags, profiles and allowances are unchanged.

54 tracked Cargo.toml files parsed without duplicate keys. git diff --check passed. No new assertions were introduced, so no new runtime mutation is claimed. Native compilation and the existing helper test do not substitute for actual Linux RIO execution; the new integration CI run still needs to recover before the RPC merge.

Preservation and output hashes:
parent file SHA256: 8e59ee745e3aaaa22848b943d3b10f6353b515956282cde1f3af2c394200c9ac
final file SHA256: b69ba3237061b161448ae8b62b615885d5770b98a5b4a2479bcfbd70f1bd1f17
unchanged production prefix SHA256: 7903a6e288252c420231242d7b58a25fa2a7140ea3dd430cbe7f0dde11369dcb
RIO Linux failure log SHA256:421addb9868fd88e4109a922311e54f91f0dfd70862f8ba7fcaba193482ed90b
Workspace Linux failure log SHA256:e0aea8d0f014c022252a9534968e083c98cfc1fd760da445ecd04ec3f522e26a
19 output SHA256:35a54f28d707956ed1edb29d492c4dfe8a9fe753d103b111b5a2f1b5c41c2b98
RIO Clippy output SHA256:d6046e3868f2b3992fe11cdd04e9e98eb50a87d1234d9a70dbfeb154ff61638d
slow-operation output SHA256:605b7b258d2fd09c6ef2ab2ffc90d433d0509480f1e45125d20c823759e83608

Refs rustfs/backlog#2734.

* test(ecstore): isolate tier-free recovery fixture startup

The fixture constructor can take the production recovery lock and wait for 60 seconds before this test installs its queue and seeds metadata. Disable its background recovery only while creating this fixture, then restore the environment before calling the existing production recovery entrypoint.

Only the existing test setup changes. Its production entrypoint, default lock parameters, original assertions, 30-second timeout, cancellation and cleanup remain intact. No tests or assertions were added. Production and store/init.rs are byte-identical to the parent.

Independent acceptance ran on merge tree 8ec358cb42ba9ebb3bcfeafcef048a3f0366355b; the runner header reports parent HEAD, while before/after index and source hashes confirm this candidate. Worker task branch d8055766b8 has the same tree.

Every command used LANG=en_US.UTF-8. Task compilation also used CARGO_BUILD_JOBS=1, CARGO_INCREMENTAL=0, CARGO_NET_OFFLINE=true, an independent target, and empty RUSTC_WRAPPER/RUSTC_WORKSPACE_WRAPPER to keep compilers in its owned process group without changing repository configuration.

Command:
shellcheck --version && actionlint
Exit 0; observed output excerpt:
version: 0.11.0
Log SHA-256: fc9eec16b4191d058a295813ca9afbe9d2b37b59975da860170726be162a3924

Command:
cargo fmt --all --check
Exit 0; observed output excerpt:
(no stdout or stderr)
Log SHA-256: e3b0c44298fc1c149afbf4c8996fb92427ae41e4649b934ca495991b7852b855

Command:
./scripts/check_unsafe_code_allowances.sh
Exit 0; observed output excerpt:
(no stdout or stderr)
Log SHA-256: e3b0c44298fc1c149afbf4c8996fb92427ae41e4649b934ca495991b7852b855

Command:
./scripts/check_layer_dependencies.sh
Exit 0; observed output excerpt:
Layer dependency guard passed (no new reverse dependencies/cycles).
Log SHA-256: 69c6cb1896c0586b28d1167380d3260d99d8017a4265c8df5a2348f289e93a12

Command:
./scripts/check_architecture_migration_rules.sh
Exit 0; observed output excerpt:
Architecture migration rules passed.
Log SHA-256: 28790e70d070607ed4de0704f292a65030be92ec224ef106d4b72fd8296f896e

Command:
./scripts/check_logging_guardrails.sh
Exit 0; observed output excerpt:
✅ Logging guardrails check passed
Log SHA-256: 7adf41abff8ed8f7617decf86498888ddde78937099dd9ce47f6d359dfc2d317

Command:
./scripts/check_error_other_format_ratchet.sh
Exit 0; observed output excerpt:
error(format!) ratchet passed (total call sites: 579).
Log SHA-256: 66172aef18f150ba0605af122e26ffcf91f6dd86f43e38b05e2e982ce8d783ee

Command:
./scripts/check_no_tokio_io_uring.sh
Exit 0; observed output excerpt:
(no stdout or stderr)
Log SHA-256: e3b0c44298fc1c149afbf4c8996fb92427ae41e4649b934ca495991b7852b855

Command:
./scripts/check_extension_schema_boundaries.sh
Exit 0; observed output excerpt:
Extension schema boundaries passed.
Log SHA-256: f13d0acd0e2ce5d9815241b12b4dab7d01cc11b9b65bcf6dfb7812c878532a51

Command:
./scripts/check_body_cache_whitelist.sh
Exit 0; observed output excerpt:
Body-cache whitelist guard passed.
Log SHA-256: 6b828847b0383950ee6eef9cd846d0c322c945e47efd4c7d1ad881178810d6bb

Command:
./scripts/check_s3s_footprint.sh
Exit 0; observed output excerpt:
s3s footprint OK: files importing s3s is 195 (baseline files=195)
s3s footprint OK: s3s s3_error! lines is 1584 (baseline s3_error_lines=1584)
s3s footprint OK: ecstore files referencing s3s is 36 (baseline ecstore_files=36)
✅ s3s footprint baseline check passed
Log SHA-256: 47f53d04ef645618563fe1f22abc64667f19078f1974feedcca46e4f2c016844

Command:
./scripts/check_test_count_ratchet.sh
Exit 0; observed output excerpt:
test-count ratchet OK: 52 crates, 21340 tests, 164 ignored match .config/test-count-baseline.txt
Log SHA-256: 618a5e330e933de0c6ff484546df401457024a88bb5986d11d588794e33dab60

Command:
./scripts/check_fips_wording.sh
Exit 0; observed output excerpt:
FIPS wording guard passed (README.md and CHANGELOG.md contain no forbidden claims).
KMS wording guard passed (crates/kms claims no Vault KV2 Transit wrapping).
Log SHA-256: 99221cc9979c480ec4f46f50d76c52b18c72fe499f12e15e4f4a22a1b8bfb140

Command:
./scripts/check_embedded_secrets.sh
Exit 0; observed output excerpt:
Embedded secret guard passed (no private key material or provider credential literal in the tree).
Log SHA-256: 437a3ceba32094d2901889965ad191e411aad32cdb18bf0b8659e818011c406b

Command:
make script-tests
Exit 0; observed output excerpt:
test_cli_requires_the_repository_default_branch (__main__.SelfTests.test_cli_requires_the_repository_default_branch) ... ok
test_config_rejects_duplicate_and_invalid_entries (__main__.SelfTests.test_config_rejects_duplicate_and_invalid_entries) ... ok
test_freshness_boundaries (__main__.SelfTests.test_freshness_boundaries) ... ok
test_grace_requires_two_successful_queries_with_no_history (__main__.SelfTests.test_grace_requires_two_successful_queries_with_no_history) ... ok
test_invalid_api_evidence_fails_closed (__main__.SelfTests.test_invalid_api_evidence_fails_closed) ... ok
test_report_retains_every_workflow (__main__.SelfTests.test_report_retains_every_workflow) ... ok
test_requests_filter_schedule_default_branch_and_success_on_server (__main__.SelfTests.test_requests_filter_schedule_default_branch_and_success_on_server) ... ok
Ran 10 tests in 0.042s
OK
Ran 30 tests in 20.627s
OK
Ran 10 tests in 2.268s
OK
Ran 33 tests in 0.127s
OK
Ran 7 tests in 0.002s
OK
Ran 7 tests in 0.001s
OK
Ran 4 tests in 0.108s
OK
Ran 3 tests in 0.564s
OK
PASS: cold stampede script self-test
Log SHA-256: 75fda2d069a373c63667b908baaf99e6591aab9099a739b0c4c22ceb6da8d68c

Command:
python3 ./scripts/test_object_generation_protocol_model.py
Exit 0; observed output excerpt:
test_an_unavailable_decision_quorum_cannot_choose_a_candidate (__main__.ObjectGenerationProtocolModel.test_an_unavailable_decision_quorum_cannot_choose_a_candidate) ... ok
test_counter_allocation_requires_persistence_and_is_consumed_once (__main__.ObjectGenerationProtocolModel.test_counter_allocation_requires_persistence_and_is_consumed_once) ... ok
test_counter_restart_burns_persisted_allocations_before_or_after_issue (__main__.ObjectGenerationProtocolModel.test_counter_restart_burns_persisted_allocations_before_or_after_issue) ... ok
test_crash_after_promise_before_reply_does_not_restore_an_older_ballot (__main__.ObjectGenerationProtocolModel.test_crash_after_promise_before_reply_does_not_restore_an_older_ballot) ... ok
test_duplicate_or_unconfigured_voters_cannot_form_a_quorum (__main__.ObjectGenerationProtocolModel.test_duplicate_or_unconfigured_voters_cannot_form_a_quorum) ... ok
test_empty_or_unobserved_minority_slot_can_retry_without_allocating_another_revision (__main__.ObjectGenerationProtocolModel.test_empty_or_unobserved_minority_slot_can_retry_without_allocating_another_revision) ... ok
test_equal_ballot_cannot_accept_different_bytes_and_lower_ballot_is_rejected (__main__.ObjectGenerationProtocolModel.test_equal_ballot_cannot_accept_different_bytes_and_lower_ballot_is_rejected) ... ok
test_forgetting_tombstone_authority_allows_an_aba_create (__main__.ObjectGenerationProtocolModel.test_forgetting_tombstone_authority_allows_an_aba_create) ... ok
test_issuing_before_persistence_reuses_a_ballot_after_restart (__main__.ObjectGenerationProtocolModel.test_issuing_before_persistence_reuses_a_ballot_after_restart) ... ok
test_lost_ack_and_restart_preserve_the_chosen_operation (__main__.ObjectGenerationProtocolModel.test_lost_ack_and_restart_preserve_the_chosen_operation) ... ok
test_minority_accepted_work_is_adopted_and_finished (__main__.ObjectGenerationProtocolModel.test_minority_accepted_work_is_adopted_and_finished) ... ok
test_omitting_highest_accepted_adoption_has_a_double_choice_counterexample (__main__.ObjectGenerationProtocolModel.test_omitting_highest_accepted_adoption_has_a_double_choice_counterexample) ... ok
test_quorum_choice_does_not_require_an_uncontacted_disk_to_learn_it (__main__.ObjectGenerationProtocolModel.test_quorum_choice_does_not_require_an_uncontacted_disk_to_learn_it) ... ok
test_recovery_failure_or_stale_expected_head_does_not_allocate_a_successor (__main__.ObjectGenerationProtocolModel.test_recovery_failure_or_stale_expected_head_does_not_allocate_a_successor) ... ok
test_replacing_voters_with_empty_state_can_choose_a_conflicting_value (__main__.ObjectGenerationProtocolModel.test_replacing_voters_with_empty_state_can_choose_a_conflicting_value) ... ok
test_skipping_predecessor_recovery_chooses_a_broken_revision_chain (__main__.ObjectGenerationProtocolModel.test_skipping_predecessor_recovery_chooses_a_broken_revision_chain) ... ok
test_stale_publication_rollback_and_gc_cannot_change_the_winner (__main__.ObjectGenerationProtocolModel.test_stale_publication_rollback_and_gc_cannot_change_the_winner) ... ok
test_superseded_operation_identity_cannot_be_reused_for_different_input (__main__.ObjectGenerationProtocolModel.test_superseded_operation_identity_cannot_be_reused_for_different_input) ... ok
test_superseded_operation_retry_returns_its_original_successor (__main__.ObjectGenerationProtocolModel.test_superseded_operation_retry_returns_its_original_successor) ... ok
test_tombstone_and_null_marker_payloads_do_not_reset_authority_revision (__main__.ObjectGenerationProtocolModel.test_tombstone_and_null_marker_payloads_do_not_reset_authority_revision) ... ok
test_tombstone_rejects_a_delayed_never_present_create (__main__.ObjectGenerationProtocolModel.test_tombstone_rejects_a_delayed_never_present_create) ... ok
test_unlearned_predecessor_is_recovered_before_the_second_slot (__main__.ObjectGenerationProtocolModel.test_unlearned_predecessor_is_recovered_before_the_second_slot) ... ok
Ran 24 tests in 0.929s
OK
Log SHA-256: 0b4be926efb0609d497ae58930340153f01957b606206cbbdb3e0d47eb29d3a2

Command:
python3 ./scripts/check_test_wiring.py
Exit 0; observed output excerpt:
OK: e2e modules, runner selection, fuzz matrices, profiles, scheduled alerts, and Quick Checks are wired
Log SHA-256: 21f676f458451525535ab0ef73d6a9c0d9774e16d76a4d19e396ad34936dd39c

Command:
./scripts/check_no_planning_docs.sh
Exit 0; observed output excerpt:
(no stdout or stderr)
Log SHA-256: e3b0c44298fc1c149afbf4c8996fb92427ae41e4649b934ca495991b7852b855

Command:
./scripts/check_uring_lane_lib_only.sh
Exit 0; observed output excerpt:
OK: no 'uring_'-matching test functions under crates/ecstore/tests; --lib is safe
Log SHA-256: 43f3f851e27ec2ab9e34d6b05fefdede38af992699c473904bfe6f53a46d52e3

Command:
cargo nextest run --profile ci -p rustfs-ecstore --lib --features rio-v2 -E 'test(=bucket::lifecycle::bucket_lifecycle_ops::tests::tier_free_version_recovery_production_entrypoint_enqueues_seeded_item)' --no-fail-fast
Exit 0; wall 234.399s.
Observed output:
    Finished `test` profile [unoptimized + debuginfo] target(s) in 3m 49s
    Starting 1 test across 1 binary (5871 tests skipped)
  SETUP PASS [   0.006s] ecstore-base-stack: sh -c 'echo RUST_MIN_STACK=4194304 >> "$NEXTEST_ENV"'
        PASS [   0.440s] rustfs-ecstore bucket::lifecycle::bucket_lifecycle_ops::tests::tier_free_version_recovery_production_entrypoint_enqueues_seeded_item
     Summary [   0.586s] 1 test run: 1 passed, 5871 skipped
Log SHA-256: 34e973f86e8dc2102c1a0e483e0ac802ce7df3d1b40a603932f62c0805c4de27

Command:
cargo nextest run --profile ci -p rustfs-ecstore --lib --features rio-v2 -E 'test(=bucket::lifecycle::bucket_lifecycle_ops::tests::tier_free_version_recovery_eight_coordinators_hold_one_leader_and_take_over) | test(=store::bucket::tests::creation_commit_backfill_runs_under_the_config_write_fence) | test(=store::bucket::tests::orphaned_bucket_recovery_requires_every_volume_absent_and_is_retryable)' --no-fail-fast
Exit 0; wall 4.010s.
Observed output:
    Finished `test` profile [unoptimized + debuginfo] target(s) in 0.43s
    Starting 3 tests across 1 binary (5869 tests skipped)
  SETUP PASS [   0.005s] ecstore-base-stack: sh -c 'echo RUST_MIN_STACK=4194304 >> "$NEXTEST_ENV"'
        PASS [   0.068s] rustfs-ecstore bucket::lifecycle::bucket_lifecycle_ops::tests::tier_free_version_recovery_eight_coordinators_hold_one_leader_and_take_over
        PASS [   0.637s] rustfs-ecstore store::bucket::tests::creation_commit_backfill_runs_under_the_config_write_fence
        PASS [   0.846s] rustfs-ecstore store::bucket::tests::orphaned_bucket_recovery_requires_every_volume_absent_and_is_retryable
     Summary [   0.987s] 3 tests run: 3 passed, 5869 skipped
Log SHA-256: b5da2a927ff18ca9aa217762f7186a6d4371bd9e8feedd7b77940e10b64dffac

Command:
cargo clippy -p rustfs -p rustfs-ecstore --all-targets --features rio-v2 -- -D warnings
Exit 0; wall 1084.508s.
Observed output:
    Finished `dev` profile [unoptimized + debuginfo] target(s) in 18m 03s
Log SHA-256: 28f98be7970faaa71cd118da1da4e6e2e3d23b9dda7e6449eaa7a37e741260b5

Compiled mutation command (unchanged for each red and restored run):
cargo nextest run --profile ci -p rustfs-ecstore --lib --features rio-v2 -E 'test(=bucket::lifecycle::bucket_lifecycle_ops::tests::tier_free_version_recovery_production_entrypoint_enqueues_seeded_item)' --no-fail-fast

M1: reverse only this test-local constructor flag from false to true. The mutant compiled, then failed the original recovered-item queue assertion after 30.481s. Restoring the entire observed candidate passed in 0.460s. Red log 55bf3b1535529ac0bcfb8071f9b56f9f40631007a85c315367e7e93786d65293; restored log 204d5fbe4574a7917f5e740a795d864fe28a7e7fb5e33d27b0a9cf68e2f17f52.

M2: replace only the actual awaited enqueue with false, consuming its original inputs. The mutant compiled, then failed the same original queue assertion after 30.516s. Restoring the entire observed candidate passed in 0.457s. Red log 89186330413215e231e2b06bf013cc31097ea8f4a106d91421a36d6c8986a7fd; restored log e848b42d2302401afa5dbcd9e37bf76d2e56a96ab6de5a488513b997f53c5e9b.

All temporary observation and mutation code was removed before final acceptance. Historical baseline and diagnostic failures remain recorded; they are not mutation credit. The separate Object Lock intent-retry diagnostic still fails, and these native macOS checks do not establish Linux CI recovery.

* refactor(rpc): serve internode rpc bodies with the owned Body (#8441)

* test(rpc): pin the wire bytes of every internode rpc response body

rustfs/backlog#2747 batch 4, step 1 of 2, on the s3s bodies the rpc paths
serve today. Each response body builder in storage/rpc/http_service.rs is
served through hyper's HTTP/1.1 server and the raw bytes the peer reads are
compared with goldens captured from this commit: status, content-length or
chunked framing, chunk boundaries, the terminating chunk, HEAD, and the cut
connection a failure after data produces. The capability responses carry a
per-process proof, so their length header and payload are checked against
the body they encode instead.

The read_file_stream response builder moves into read_file_stream_response
so the test reaches the same code the handler runs; behaviour is unchanged.

* refactor(rpc): serve internode rpc bodies with the owned Body

rustfs/backlog#2747 batch 4, step 2 of 2. The internode rpc HTTP paths
build their responses with rustfs_s3_types::Body: Body::from_stream
replaces Body::from(StreamingBlob::wrap(..)) at the six stream sites, and
the in-memory and empty bodies keep their constructors. InternodeRpcService
converts its responses into the wrapped service's response body through
From<Body>, so the server keeps composing it with the s3s service as
before. The wire-byte goldens from the previous commit pass unchanged;
storage/rpc no longer names s3s.

* fix(rpc): simplify wire test response builder type

Name the boxed response builder used by the capability wire test so
all-target Clippy can check it without a type-complexity warning.

Validation (LANG=en_US.UTF-8, CARGO_INCREMENTAL=0):
- cargo clippy -p rustfs --all-targets -- -D warnings: exit 0.
- cargo nextest run --profile ci -p rustfs --lib -E 'test(/storage::rpc::/)': 279 tests run, 279 passed, 4842 skipped.
- All 19 quick-check commands, including script-tests, both Python scripts and cargo fmt --all --check: exit 0.
- Footprint baseline regenerated: files=194, s3_error_lines=1584, ecstore_files=36; test-count baseline regenerated: 52 crates, 21228 tests, 164 ignored.
- All 54 Cargo.toml files parse without duplicate keys.

No assertion was added or changed by this lint fix. R1-R8 are retained
in the existing PR verification record.

* fix(ecstore): resume fenced Object Lock creation intents

* fix(ecstore): bound creation retry futures on the stack

* fix(ecstore): require native proof for creation retries

* fix(ecstore): bound bucket creation stack usage (#8466)

* fix(ecstore): retain creation fences through rollback

Keep lifecycle, metadata-transaction, and namespace guards around rollback and metadata cleanup. Preserve the original publication error when a guard is lost. Build the timestamp retry fixture through the real interrupted producer so it checks a valid pending creation.

Validated source tree: 4e8e207999.
Two independent source reviews found no blocking regression.

Validation (LANG=en_US.UTF-8):
- bash -c 'shellcheck --version && actionlint' — exit 0
- cargo fmt --all --check — exit 0
- ./scripts/check_unsafe_code_allowances.sh — exit 0
- ./scripts/check_layer_dependencies.sh — exit 0
- ./scripts/check_architecture_migration_rules.sh — exit 0
- ./scripts/check_logging_guardrails.sh — exit 0
- ./scripts/check_error_other_format_ratchet.sh — exit 0
- ./scripts/check_no_tokio_io_uring.sh — exit 0
- ./scripts/check_extension_schema_boundaries.sh — exit 0
- ./scripts/check_body_cache_whitelist.sh — exit 0
- ./scripts/check_s3s_footprint.sh — exit 0
- ./scripts/check_test_count_ratchet.sh — exit 0
- ./scripts/check_fips_wording.sh — exit 0
- ./scripts/check_embedded_secrets.sh — exit 0
- make script-tests — exit 0
- python3 ./scripts/test_object_generation_protocol_model.py — exit 0
- python3 ./scripts/check_test_wiring.py — exit 0
- ./scripts/check_no_planning_docs.sh — exit 0
- ./scripts/check_uring_lane_lib_only.sh — exit 0
- cargo nextest run --profile ci -p rustfs-ecstore --lib --locked --no-fail-fast — 5894 passed, 0 skipped
- cargo check -p rustfs-ecstore --all-targets --locked — exit 0
- cargo nextest run --profile ci -p rustfs --lib --locked --no-fail-fast -E 'test(=admin::handlers::table_catalog::tests::cold_strong_registration_completes_while_other_bucket_backup_waits) | test(=admin::handlers::table_catalog::tests::commit_backend_borrowed_put_preserves_owner_authorization_and_precondition) | test(=admin::handlers::table_catalog::tests::disable_table_bucket_preserves_objects_and_can_be_reenabled) | test(=admin::handlers::table_catalog::tests::load_table_handler_negotiates_vended_credentials_without_breaking_metadata_only_callers) | test(=admin::handlers::table_catalog::tests::namespace_property_handler_updates_object_backed_catalog_and_maps_errors) | test(=app::object::get::tests::execute_get_object_resumes_from_relocated_pool_without_splicing_body) | test(=table_catalog::store::tests::catalog_writer_loss_prevents_recovery_storage_commit)' — 7 passed
- cargo nextest run --profile ci -p rustfs --lib --locked --no-fail-fast -E 'test(unsupported_remote_disk_)' — 2 passed

Additional fault-injection evidence remains incomplete. Timestamp M1 reached its original assertion; its restore run is being collected. M2–M5 and the three planned rollback faults have not run. An automated risk check interrupted the timestamp proof task; no assertions or validation settings were weakened.

* fix(ecstore): box nested bucket rollback state

Keep the guarded rollback future behind a heap allocation so bucket creation does not store its nested state inline. The rollback body, all three fence references and pending-proof revocation order are unchanged.

The same three original Linux tests overflowed on both e611 CI attempts. Local format, ecstore test compilation and all 19 quick-check steps pass; the original Linux feature lanes still need to validate this candidate.

* test(ecstore): isolate rollback guard loss fixtures

Run the three rollback metadata-cleanup cases in separate test processes so reused /proc/self/fd aliases from earlier cases cannot invalidate their creation witnesses. Keep the original fixture, guards, assertions and io_uring test selection, and verify that each child runs the intended case successfully.

Update the generated test-count baseline for the added macro arm.

Validation: cargo check -p rustfs-ecstore --tests; cargo fmt --all --check; all 19 quick-check commands, including make script-tests and the two Python checks. Linux io_uring acceptance and fault controls remain pending.

* refactor(replication): move the replication config onto gateway types (#8443)

* wip: T1.4 replication family

* refactor(replication): move the replication config onto gateway types

The replication configuration family leaves s3s: rustfs-replication no longer depends on s3s, and ECStore bucket metadata, the scanner and the rustfs app/admin layers hold rustfs_gateway_types::persistence::PersistedReplicationConfiguration. The persisted XML goes through the gateway persistence codec behind a new BucketConfigXml trait (the 39 byte goldens pass with unchanged fixtures); the legacy s3s edge converts once in storage/ecfs.rs through storage/s3_compat/replication.rs; the scanner data usage cache keeps its MessagePack bytes through a serde mirror pinned by bytes recorded from s3s 5761ddfe. rustfs-gateway-types 0.32.0 is pinned once (rev 772d3385). Refs rustfs/backlog#2745.

* test: use S3 serializer for list response tests

* fix(ci): restore integration test compilation

Gate the diagnostic test import with its feature, initialize the conditional
MetadataWriteStore fields explicitly, and decode the scanner replication fixture
through its existing cache boundary. Preserve the original fixture and assertions.

Validation was run on tree 544acb80c0 at 6003d4b8.
LANG=en_US.UTF-8; no source changes follow validation.

Quick checks: 19/19 passed.
- shellcheck --version && actionlint: exit 0
- cargo fmt --all --check: exit 0
- ./scripts/check_unsafe_code_allowances.sh: exit 0
- ./scripts/check_layer_dependencies.sh: exit 0
- ./scripts/check_architecture_migration_rules.sh: exit 0
- ./scripts/check_logging_guardrails.sh: exit 0
- ./scripts/check_error_other_format_ratchet.sh: exit 0
- ./scripts/check_no_tokio_io_uring.sh: exit 0
- ./scripts/check_extension_schema_boundaries.sh: exit 0
- ./scripts/check_body_cache_whitelist.sh: exit 0
- ./scripts/check_s3s_footprint.sh: exit 0
- ./scripts/check_test_count_ratchet.sh: exit 0
- ./scripts/check_fips_wording.sh: exit 0
- ./scripts/check_embedded_secrets.sh: exit 0
- make script-tests: exit 0
- python3 ./scripts/test_object_generation_protocol_model.py: exit 0
- python3 ./scripts/check_test_wiring.py: exit 0
- ./scripts/check_no_planning_docs.sh: exit 0
- ./scripts/check_uring_lane_lib_only.sh: exit 0

Darwin validation:
- cargo clippy --locked -p rustfs-ecstore -p rustfs-scanner --all-targets -- -D warnings: exit 0
- cargo nextest run --locked -p rustfs-ecstore -p rustfs-scanner --lib -E 'test(config::com::tests::save_config_preserves_retryable_system_volume_errors) | test(scanner_io::tests::replication_maintenance_refreshes_compacted_stats_across_cycles_and_retries)' --test-threads 1 --no-fail-fast: exit 0, 2/2 passed
- cargo clippy --locked -p rustfs-ecstore -p rustfs-scanner --all-targets --features rustfs-ecstore/diagnostic-read-version-caller,rustfs-scanner/diagnostic-read-version-caller -- -D warnings: exit 101
- cargo nextest run --locked -p rustfs-ecstore -p rustfs-scanner --lib --features rustfs-ecstore/diagnostic-read-version-caller,rustfs-scanner/diagnostic-read-version-caller -E 'test(config::com::tests::save_config_preserves_retryable_system_volume_errors) | test(scanner_io::tests::replication_maintenance_refreshes_compacted_stats_across_cycles_and_retries) | test(cluster::rpc::runtime_sources::tests::) | test(config::com::tests::diagnostic_config_write_retains_origin_and_api_for_results_and_cancellation)' --test-threads 1 --no-fail-fast: exit 0, 14/14 passed

Diagnostic strict Clippy remains red on four unchanged production lints:
runtime_sources.rs if_same_then_else and three redundant_locals reports in
set_disk/core/metadata_read.rs and set_disk/read.rs. Their source ranges and
feature wiring match the base; the base was not compiled separately.

Linux aarch64 validation (Rust/Clippy 1.99.0, UID 501, jobs=1):
- cargo clippy --all-targets -- -D warnings: exit 0; Finished dev profile in 68m 12s
- cargo clippy -p rustfs -p rustfs-ecstore --all-targets --features rio-v2 -- -D warnings: exit 0; Finished dev profile in 49m 45s
Source HEAD/tree and clean status matched before and after both commands.
No compiler or stack overrides were used. protoc 35.1 was installed from its
verified ARM64 release; flatc was absent and not invoked by the workspace build scripts.
The local environment differs from hosted CI.

* fix(ecstore): recover delete batches with missing versions

Treat an explicit non-null version as a no-op when it is absent from both recovery sources. Keep rejecting missing null versions, delete-marker mismatches and conflicting local generations.

Add returning-disk and partial-receipt regressions, extend journal boundary checks, and regenerate the ecstore test-count baseline. Production faults exercise the 24 added assertion sites and 7 semantic checks: 33 faults failed at the expected sites and passed after full restoration.

Validation (LANG=en_US.UTF-8, one build job):

cargo nextest run --locked -p rustfs-ecstore --lib -E 'test(delete_recovery) | test(quorum_delete_cleanup_coordinator_binds_single_and_batch) | test(crash_delete_retry_batch_reclaims_inline_and_reserved_external)' --test-threads 1 --no-fail-fast
Summary [  34.580s] 69 tests run: 69 passed, 6058 skipped; exit 0

cargo clippy --locked -p rustfs-ecstore --all-targets -- -D warnings
Finished `dev` profile [unoptimized + debuginfo] target(s) in 5m 37s; exit 0

All 19 final Quick Checks completed with exit 0:

- shellcheck --version && actionlint
- cargo fmt --all --check
- ./scripts/check_unsafe_code_allowances.sh
- ./scripts/check_layer_dependencies.sh
- ./scripts/check_architecture_migration_rules.sh
- ./scripts/check_logging_guardrails.sh
- ./scripts/check_error_other_format_ratchet.sh
- ./scripts/check_no_tokio_io_uring.sh
- ./scripts/check_extension_schema_boundaries.sh
- ./scripts/check_body_cache_whitelist.sh
- ./scripts/check_s3s_footprint.sh
- ./scripts/check_test_count_ratchet.sh
- ./scripts/check_fips_wording.sh
- ./scripts/check_embedded_secrets.sh
- make script-tests
- python3 ./scripts/test_object_generation_protocol_model.py
- python3 ./scripts/check_test_wiring.py
- ./scripts/check_no_planning_docs.sh
- ./scripts/check_uring_lane_lib_only.sh

test-count ratchet OK: 52 crates, 22009 tests, 165 ignored match .config/test-count-baseline.txt
PASS: cold stampede script self-test
model: states=51023, transitions=167520, terminal=732, chosen_candidates=2
multi-slot round schedules: 640
OK: e2e modules, runner selection, fuzz matrices, profiles, scheduled alerts, and Quick Checks are wired

* fix(ecstore): retry tier commit intent lock contention

Retry transient namespace lock timeouts within the existing three-attempt
CAS budget. Reload and verify the original mutation identity after
contention, and preserve the final typed error when attempts are exhausted.
Cover save/read contention, exhausted retries, replacement identities and
CAS reconciliation without changing the existing timeout or lock policy.

Validation:
- cargo nextest run --locked -p rustfs-ecstore --features test-util --lib -E 'test(tier_mutation)' --test-threads 1 --no-fail-fast: 52 passed.
- cargo nextest run --locked -p rustfs-ecstore --features test-util --lib -E 'test(intent_advance_) | test(=services::tier::tier::tests::caller_cancellation_after_durable_prepare_does_not_hide_the_mutation)' --test-threads 1 --no-fail-fast: 5 passed.
- cargo clippy --locked -p rustfs-ecstore --all-targets -- -D warnings: passed.
- cargo clippy --locked -p rustfs-ecstore --all-targets --features test-util -- -D warnings: passed.
- 27 mutation pairs failed at the intended assertions and passed after restoring the production code.
- bash scripts/check_test_count_ratchet.sh --update, then bash scripts/check_test_count_ratchet.sh: 52 crates, 22013 tests, 165 ignored; ecstore increases from 6581 to 6585.

* refactor(lifecycle): replace s3s domain types with Gateway DTOs

Keep lifecycle rules, restore requests, Object Lock, and scanner cached tags in domain types. Convert at the existing HTTP boundary while preserving legacy XML, scanner cache bytes, date ordering, and tombstones.

Linux test validation before the final import-path-only adjustment:

cargo nextest run -p rustfs-ecstore bucket_config_goldens
39 passed; 6211 skipped by selector.

cargo nextest run --lib -p rustfs -E 'test(/test_zero_rule_lifecycle_tombstone_recognition/) | test(/test_local_lifecycle_staleness_axis_selection/) | test(/test_lifecycle_staleness_preserves_legacy_submillisecond_axis/) | test(/test_merge_incoming_lifecycle/) | test(/test_lifecycle_expiry_statement_matrix/) | test(/test_lifecycle_expiry_subset_xml/) | test(/put_bucket_lifecycle_validation_errors_keep_their_s3_code/) | test(/execute_put_bucket_lifecycle_configuration_rejects_missing_configuration/) | test(/execute_restore_object_rejects_missing_restore_request/) | test(/execute_restore_object_rejects_select_type/) | test(/execute_restore_object_maps_failures_to_typed_s3_errors/) | test(/execute_restore_object_returns_internal_error_when_store_uninitialized/) | test(/restore_request_conversion_preserves_legacy_output_values/) | test(/restore_request_conversion_preserves_presence_and_raw_enums/) | test(/restore_request_conversion_distinguishes_absent_and_empty_containers/)'
25 passed; 5318 skipped by selector.

cargo nextest run --profile ci -p rustfs-ecstore -p rustfs-lifecycle -p rustfs-replication -p rustfs-scanner
7872 passed; 21 skipped under the existing CI profile.

Previously verified mutation controls: 39 production faults, 41 RED/GREEN pairs.

cargo clippy --locked -p rustfs -p rustfs-ecstore -p rustfs-lifecycle -p rustfs-scanner --all-targets -- -D warnings
Passed in 40m 45s before the import-path adjustment, then in 1m 59s on
the final source with warnings denied. The final adjustment imports the
existing helper through the object domain facade; fmt and the architecture
check also passed.

Part of rustfs/backlog#2745. The remaining ecstore configuration families follow separately.

* refactor(admin): move error types to rustfs-s3-types (#8440)

* refactor(protocols): move error types to rustfs-s3-types by codemod

T1.2 batch 3 (rustfs/backlog#2743): s3select-api, s3select-query and
the error items of protocols move off s3s by scripts/codemods, run with
--hold crates/protocols/src/sftp/errors.rs. That file stays on s3s: it
downcasts the backend error, whose runtime type is still s3s::S3Error.

- rustfs/src/protocols/client.rs: the two webdav session methods now
  return rustfs_s3_types::S3Result and convert the s3s error with From.
- codemod: a child module that calls s3_error! through `use super::*`
  gets its own `use rustfs_s3_types::s3_error;` once its parent imports
  that macro by name; otherwise it is listed as GLOB.
- s3-types: a table test pins code, status and the s3s error the legacy
  edge renders for every variant a moved call site names.
- s3_error_lines 1585 -> 1584 (a lifecycle comment reworded).

* refactor(admin): move error types to rustfs-s3-types (WIP, no build)

T1.2 batch 4 (rustfs/backlog#2743), work in progress: rustfs check
still fails with 16 errors (listed in the PR body).

- codemod: --facade PATH treats a crate-local re-export of s3s's error
  items like s3s for the four names (9 new self-tests, 9 mutants red).
- codemod output for rustfs/src/admin and rustfs/src/site_replication
  (--facade crate::storage_api::site_replication::s3); 6 glob children
  get an explicit rustfs_s3_types::s3_error import.
- admin/router.rs: the Operation trait returns rustfs_s3_types::S3Result.
- error.rs: From<ApiError> builds the rustfs-s3-types error; the s3s
  conversion is derived from it.

* fix(admin): convert error types at the s3s boundary

Batch 4 of T1.2 left the admin and site_replication modules returning
rustfs_s3_types::S3Error while their callers and callees on the storage
and auth side still speak s3s::S3Error. Convert at each edge with the
existing From impls in rustfs-s3-types (compat_s3s.rs) instead of
changing any code, message or status:

- admin/auth.rs, admin/handlers/object_zip_download.rs,
  admin/handlers/table_catalog/mod.rs: map the s3s result returned by
  check_key_valid / authorize_request / authorize_internal_object_request
  into the admin error type with .map_err(Into::into).
- storage/ecfs.rs: the s3s-facing side maps the admin site_replication
  gate results the other way.
- admin/handlers/pools.rs: ApiError.code stays s3s::S3ErrorCode, so the
  rustfs code is converted with .into() when the struct is built, in
  production and in the two tests that build or compare it.
- admin/handlers/heal.rs, admin/runtime_sources.rs: the router returns
  s3s errors (S3Route impl); the expected rustfs code is converted with
  .into() before the comparison.
- admin/handlers/table_catalog/mod.rs: iceberg_rest_error takes
  &'static str so the Cow<'static, str> custom code borrows it; every
  caller already passes an ICEBERG_ERROR_* constant or a &'static str.
- admin/handlers/quota.rs: drop a now same-type .into() after s3_error!.
- storage_api.rs, site_replication/mod.rs: drop re-exports and an import
  that no longer have users (clippy -D warnings).

cargo check -p rustfs --all-targets and cargo clippy -p rustfs
--all-targets -- -D warnings are clean; the s3s footprint baseline is
unchanged at files=172, s3_error_lines=303, ecstore_files=28.

Refs rustfs/backlog#2743, rustfs/backlog#2734.

* refactor(ecstore): move bucket configs onto gateway persisted types (#8709)

* refactor(ecstore): move bucket configs onto gateway persisted types

Finish the ECStore batch of T1.4 (rustfs/backlog#2745). The remaining
eleven bucket configuration families (Website, CORS, Tagging, Versioning,
ObjectLock, Encryption, Notification, Logging, Accelerate, RequestPayment,
PublicAccessBlock) now read and write their stored XML through the
rustfs-gateway-types persistence and cors_tagging codecs instead of the
s3s codec, so the legacy s3s bucket-config codec and its
s3gate-metadata-xml compat marker are removed.

- ecstore: BucketConfigXml is implemented by the gateway codec for every
  family; metadata, metadata_sys, versioning, object lock, replication and
  lifecycle consumers take the persisted types. Region moves to
  rustfs_s3_types::Region, and the ETag conversion to s3s moves to the
  rustfs edge with its tests. Only warm_backend.rs and warm_backend_s3.rs
  still reference s3s, held for T1.5 (rustfs/backlog#2746).
- scanner: object lock cache, item actions and data usage use the
  persisted types.
- rustfs: the legacy S3 edge converts each family between s3s DTOs and the
  persisted types in storage/storage_api.rs by writing one codec's XML and
  reading it with the other; Tagging converts field by field.
- Bump the rustfs-gateway-types pin to 21283e87.

The 39 persisted-byte goldens are unchanged and pass, so a document the
old codec wrote reads back and re-encodes to the same bytes. s3s footprint:
files 181 -> 146, ecstore_files 28 -> 2, s3_error_lines unchanged.

* test(app): prepare the PutBucketEncryption config from the request

The uninitialized-store test passed an empty persisted configuration, which the use case now validates before it reaches the store check, so it answered MalformedXML instead of InternalError. Convert the request's configuration the way the S3 edge does.

* style(app): format the PutBucketEncryption test

* refactor(s3-client): move remote error codes and DTO enums off s3s (#8783)

Part of T1.5 (rustfs/backlog#2746), batch 1.

* refactor(s3select): run queries on gateway DTO request shapes (#8794)

* refactor(s3select): run queries on an engine request built from gateway DTO shapes

Part of T1.5 (rustfs/backlog#2746), batch 2.

* fix(s3select): keep the select request conversion in the app layer

The layer guard forbids app -> storage; the conversion moves from storage/s3_compat/select.rs to app/select_request.rs unchanged.

* fix(heal): retry transient metadata conflicts (#8812)

* fix(heal): retry transient metadata conflicts

* chore: update heal test count baseline

* test: reserve nextest capacity for progress probes (#8818)

* test(e2e): use a non-default region for the differential negative control

Since main's region-aware CreateBucket and GetBucketLocation (#8739, #8743), us-east-1 answers an empty LocationConstraint and 200 for an owned re-create, so the control no longer reported its own region. Also refresh the e2e-full linux selection from the CI listing (682 tests).

* refactor(protocols): move StorageBackend to gateway types (#8825)

* refactor(protocols): move storage backend to gateway types

* test: update e2e test count baseline

* fix: preserve copy source keys and unknown lengths

* test: update protocols test count baseline

* fix(protocols): preserve typed SFTP backend errors

* test(protocols): use gateway list input in policy tests

* fix(protocols): clear clippy failures after T1.5 merge (#8831)

fix(protocols): clear clippy failures after merge

* fix(ecstore): finalize restore metadata after local copy-back (#8836)

* fix(ecstore): finalize restore metadata after local copy-back

* test(ecstore): ratchet added restore finalization cases

* fix(ecstore): keep restore copyback compatible with older peers

* fix(ecstore): gate restore copyback on fleet compatibility

* fix(ecstore): retain transition identity during restore copyback

* fix(ecstore): route committed restore copyback locally

* fix restore copyback cleanup

* fix(ecstore): honor restore probe test fallback

* refactor(storage): migrate option errors to domain types (#8839)

* refactor(s3): migrate private multipart validation errors (#8844)

* refactor(auth): derive policy conditions from AuthzFacts (#8442)

* refactor(auth): extract authorization facts

* test(e2e): update smoke selection baseline

* test(e2e): update full selection baseline

* refactor(admin): migrate health and prefix usage to owned requests (#8887)

* refactor(admin): add owned health router boundary

* refactor(admin): migrate prefix usage to owned request

* refactor(admin): complete the owned admin request boundary (#8949)

* refactor(admin): complete the owned admin request boundary

Every type implementing the owned admin `Operation` now serves the legacy
router through one blanket impl that converts the s3s request and response
at that single edge, so handlers register with `AdminOperation(&Handler {})`
and need no adapter statics; `LegacyAdminAdapter` is removed.

`AdminRequest` gains the accessors handlers read today (`remote_addr`,
`client_info`, `request_context`, `req_info`, `req_info_mut`,
`install_req_info`, `envelope_mut`, `take_body`), the request helpers gain
envelope forms (`authenticate_envelope_credentials`,
`object_store_from_envelope`, the table-catalog `*_from_envelope` helpers,
`KmsAdminAudit::from_envelope`, `AccountAuditContext::from_envelope`,
`CallerIdentity::resolve_envelope`), `read_compatible_admin_body` accepts
either body type, and the owned `Body` gains `store_all_limited` and
`Default`. `admin::operation::test_support::TestRequest` builds owned
requests for handler tests. Unused accessors carry a per-method dead-code
expectation that each migrating handler batch removes.

No handler behavior changes; the remaining migrations follow in batches.

* chore(admin): add the T1.12 admin handler codemod

The script rewrites an admin handler file from the s3s request and response
types to the owned boundary (imports, signatures, request field and helper
calls, body access, response construction, and test request literals) and
leaves a TODO comment at every site it cannot rewrite mechanically. It is
idempotent and is removed once T1.12 completes.

* refactor(admin): migrate replication and KMS handlers to owned requests (#8951)

Move the T1.12 batch 2 admin handlers (site_replication, kms_keys, event,
replication, diagnostics, on_demand_migration, audit, account, kms_rekey,
kms_key_lifecycle, object_data_cache, tls_debug, integrity) from the s3s
request/response types to the owned AdminRequest/AdminResponse boundary.
Handler logic, authorization order, error codes and response bytes are
unchanged; the request/response plumbing now goes through the envelope
accessors and helpers introduced by the owned boundary.

The shared response helpers (utils::json_response, utils::empty_response,
handlers::admin_json_response, target_descriptor::build_json_response,
profile::authorize_profile_request) and the site replication repair
executor still answer with the s3s types, so the migrated files carry
owned-boundary forms of those helpers and convert the repair response at
its call edge. The streaming console-log body and the two devnull drains
move from the s3s byte stream to the owned Body stream/frame API.

Drop the dead_code expectations for the owned accessors and envelope
helpers this batch starts using, and lower the s3s footprint baseline to
the measured 99 files.

Part of rustfs/backlog#2788 (rustfs/backlog#2734 T1.12).

* refactor(admin): add owned forms of the shared admin response helpers

Handler batches were copying `json_response`, `empty_response`,
`admin_json_response`, and `build_json_response` into each migrated file
because the shared helpers still return the s3s response. Provide owned
forms next to each legacy helper (`owned_json_response`,
`owned_empty_response`, `owned_admin_json_response`,
`owned_build_json_response`, `authorize_profile_envelope`) with parity
tests against the legacy shape, derive `Debug` for `AdminResponse` so
tests can use `expect_err`, and fix three codemod rules: strip owned
string conversions in test credentials, annotate non-literal URI parses,
and leave a rebound request parameter immutable.

* refactor(admin): use the shared owned response helpers in batch 2 (#8954)

Replace the per-file owned-boundary copies of json_response,
empty_response, admin_json_response, build_json_response and
authorize_profile_request in the batch 2 admin handlers with the shared
owned_* helpers and authorize_profile_envelope that the boundary branch
now provides, and restore expect_err in the handler tests now that
AdminResponse implements Debug; the `.err().expect(..)` stand-ins were
failing the base branch's clippy step (clippy::err_expect under
-D warnings). The site replication repair call edge
keeps its conversion because the repair executor still answers with the
s3s response type.

Drop the dead_code expectations on the shared owned helpers this batch
now uses in non-test code.

Part of rustfs/backlog#2788 (rustfs/backlog#2734 T1.12).

* fix(admin): preserve legacy body limit error responses

* refactor(admin): move policy, ILM and batch 3 handlers to owned requests (#8956)

Batch 3 of T1.12 (rustfs/backlog#2788, parent rustfs/backlog#2734) moves
14 admin handler files to AdminRequest/AdminResponse and the owned
Operation trait: ilm_transition, policies, idp_compat, config_admin,
plugins_instances, profile_admin, group, heal, object_zip_download,
kms_key_metadata, durability, inspect_archive, sts and
gateway_key_inventory. Handler logic, authorization order, error codes
and response bytes are unchanged; test request literals use
test_support::TestRequest.

The handlers use the shared owned helpers (owned_json_response,
owned_admin_json_response, owned_build_json_response,
authorize_profile_envelope). The dead_code expectations on the
AdminRequest accessors and RequestEnvelope::uri_mut they now exercise
are removed. The legacy target_descriptor::build_json_response keeps
only its equivalence test caller and carries the matching expectation
until it retires.

object_zip_download builds the s3s request the S3 gate still reads from
the owned request facts; heal authenticates through
authenticate_envelope_credentials; the trace, inspect-archive and zip
bodies use Body::from_stream. extract_plugin_instance_filters takes the
URI so extensions.rs keeps calling it, and the two idp_compat
delegations stay on the legacy trait until their delegates migrate.

The s3s footprint baseline drops from 99 to 90 files.

* test(admin): anchor the replication diff body contract on the call

The source-contract test looked for `let body = read_compatible_admin_body(`
on one line; rustfmt wrapped that binding after the handler moved to the
owned request boundary, so the marker was no longer found. Anchor on the
call itself so formatting cannot change what the contract checks.

* refactor(admin): migrate table catalog and batch 1 to owned requests (#8955)

* fix(admin): preserve catalog body limit response bytes

* refactor(admin): migrate identity and cluster handlers to owned requests (#8966)

* refactor(admin): migrate identity and cluster handlers to owned requests

Move the T1.12 batch 4 admin handlers (user, service_account, oidc, tier,
system, pools, quota, kms_dynamic, kms_backup, rebalance, module_switch,
bucket_meta, cluster_snapshot, realtime, bucket_recovery, profile) from the
s3s request/response types to the owned AdminRequest/AdminResponse
boundary. All 90 impl Operation blocks now take AdminRequest and return
AdminResponse; handler logic, authorization order, error codes and response
bytes are unchanged. The handlers use the shared owned helpers
(owned_json_response, owned_admin_json_response, authorize_profile_envelope,
the envelope-form runtime sources and audit constructors) instead of local
copies.

profile.rs drops the s3s forms of authorize_profile_request and
profile_not_implemented_response now that profile_admin.rs is on the
envelope gate too; profile_not_implemented_response answers with an
AdminResponse. The realtime metrics stream moves from the s3s ByteStream
adapter to Body::from_stream, and the rebalance idle-404 document still
renders through s3s::S3Error::to_http_response with its body converted at
the edge.

The idp_compat delegators ImportIamV2 and AddServiceAccountLdap move to the
owned trait as well: with their targets on both traits, the method-call
delegation became ambiguous between the owned impl and the router blanket
impl, so they now take the owned request directly.

KmsAdminAudit::from_request loses its last production caller with this batch
and is deleted; its comparison test now asserts the envelope facts directly.

Lower the s3s footprint baseline to the measured 68 files.

Part of rustfs/backlog#2788 (rustfs/backlog#2734 T1.12).

* refactor(admin): remove the legacy admin helpers batch 4 superseded

Batch 4 moved the last production callers of the s3s-typed admin helpers
to their owned forms, which left `admin_json_response`, `json_response`,
`authenticate_request`, `authorize_admin_request`, `object_store_from_req`
and `object_store_from_extensions` unused, along with the `Body`,
`S3Request` and `S3Response` re-exports of the admin `s3` facade. Remove
them. The owned forms keep their documentation; the behavior tests of the
removed `json_response` now exercise `owned_json_response`, and the
compat-prefix test asserts the owned response directly.

Two tests existed only to compare a removed legacy helper with its owned
twin (`owned_json_response_matches_the_legacy_shape`) or to cover the
removed legacy gate (`authorize_admin_request_without_credentials_is_rejected`);
they go with the helpers, and the owned equivalents stay. The rustfs
test-count row drops from 5957 to 5955 and the s3s footprint from 68 to 66
files.

* refactor(admin): drop the legacy target descriptor JSON response helper (#8968)

`target_descriptor::build_json_response` returned the s3s response and
had no production caller left once the event, audit and plugin instance
handlers moved to `owned_build_json_response`; it survived only for the
legacy/owned equivalence test. Remove it, give the owned helper its own
documentation, and make the test assert the owned response directly:
status, `application/json`, the echoed `x-request-id` when present and
only then, and the body bytes. The file no longer imports s3s, so the
footprint baseline drops from 66 to 65 files.

* test(ecstore): pin native history on cancellation under the cancellable queue

main moved the local write path onto the disk blocking queue, which skips an
operation once its waiter is dropped, so a borrowed write cancelled while it is
queued no longer creates its file. The integration-only native publication
test still expected the queued open to run. Assert the new execution model
instead (the open never runs) while keeping its safety claims: the Started
record is taken before queueing and the cancelled publication never regrants
native history.

* chore(gateway): raise test-count rows for tests merged from main

* test(s3select): use the owned JSON input types in merged planner tests

* fix(ecstore): scope the merged direct-writer test hook to test builds

* fix(ecstore): create parents before admitted truncate opens

Align the heal tests with removal-disabled preservation and local-only outcome
fields, and refresh the e2e-full selection for the synced tests.

* chore(gateway): update test-count baseline after main sync

Main #9241 removed the node pool-metadata readiness cache with 8 tests
and added 6, so the rustfs row drops from 6440 to 6438. The ecstore and
s3select rows rise for tests added by #9237, #9239 and #9241.

* feat(gateway-adapter): add IAM-backed CredentialProvider

Add the ring-2 crate rustfs-gateway-adapter with IamCredentials, the
gateway CredentialProvider answered by RustFS IAM (rustfs/backlog#2760,
credentials half; PolicySource is deferred).

- Pin rustfs-gateway at the existing gateway rev with
  default-features = false so the server crate stays out of the build.
- IamKeyLookup is a narrow IAM trait implemented for IamSys so the
  adapter is testable without an in-memory IAM backend.
- Root credentials are injected at construction and resolve before IAM
  is ready.
- Mapping: keystone: prefix -> NotFound (deliberate change from the
  legacy empty secret); IAM not ready or not initialized -> Unavailable;
  unknown key or NoSuch{User,ServiceAccount,TempAccount,Account} ->
  NotFound; other IAM errors -> Backend; invalid or "off" principals
  resolve disabled; STS credentials carry a SessionBinding, and an STS
  credential without an expiry is a Backend fault rather than a
  never-ending session.
- No authorization in lookup; no Debug on secret-holding types; IAM
  error text is dropped after classification.

* feat(scripts): add junit_diff.py for dual-stack JUnit comparison (#9275)

* feat(gateway-adapter): add IAM-backed PolicySource

Add IamPolicySource, the gateway PolicySource answered by RustFS IAM
(rustfs/backlog#2760, policy half). One snapshot per request carries a
RustfsPrincipal: principal kind, account, effective username, parent
user, credential groups, live group memberships, token claims and
is_owner.

- Anonymous callers get an empty snapshot without asking IAM.
- IAM not ready, not initialized or failing is PolicyError::unavailable,
  never an empty snapshot.
- Unknown, disabled and session-kind-mismatched principals are refused
  with constant labels; unreadable tokens are malformed.
- Claims are read from the stored token (root secret for STS, the
  account's own secret for service accounts), and is_owner mirrors the
  legacy has_root_access rule.
- RustfsPrincipal holds no secret or session token and has no Debug.

* ci(gateway): make the test-count ratchet a floor, not exact (#9278)

ci(gateway): make the test-count ratchet a floor instead of an exact baseline

Tests below a row still fail; tests above it, a member without a row and ignores below the row now pass with a note. Ignores above a row still fail. Adds contract tests wired into make script-tests.

* feat(gateway-adapter): add host, bucket-owner and request-id adapters

* fix(gateway-adapter): avoid ecstore path in doc comment

* fix(gateway-adapter): keep the s3s path pattern out of docs

* chore(gateway): record ignored microbenchmarks merged from main (#9290)

chore(gateway): record the ignored microbenchmarks merged from main

main added three throughput microbenchmarks that are ignored by design
(#9217): one each in rustfs-checksums, rustfs-rio and rustfs-utils. The
test-count ratchet treats a higher ignored count as a failure until the
row is raised, so raise those three rows to 1 and tighten the test rows
to the merged tree with --update.

* test(gateway-adapter): make the test carrier Clone for Extensions::insert

* fix(gateway-adapter): share live root and refuse short secrets

Address the security review of the T2.1 adapters (rustfs/backlog#2760).

- Add RootCredentials (validated: root secret and STS signing key at
  least MIN_SECRET_KEY_LEN bytes), the RootSource trait and the settable
  RootCell. IamAdapters::new is now the only constructor of
  IamCredentials and IamPolicySource, so both read one live root source
  per request; no root source answers unavailable.
- A looked-up identity whose secret is shorter than 8 bytes is NotFound
  in the credential provider and DISABLED_PRINCIPAL in the policy
  source, whatever the lookup reports as valid.
- STS claims are read with the STS signing key from the root source
  instead of assuming it equals the root secret.
- Cover the off-status-alone branch, the OIDC owner exclusion, service
  accounts of root, the name live groups are read under, and another
  session's token.

* ci(s3-tests): add reference parity lane against pinned 1.0.5 (#9288)

* fix(gateway-adapter): effective secret length and unsplittable pair

Address the confirmation review of the T2.1 adapters
(rustfs/backlog#2760).

- One rule for every secret: its length after trailing NUL bytes are
  removed must be at least MIN_SECRET_KEY_LEN. HMAC zero-pads its key,
  so trailing NULs add nothing; interior NULs count. Applied to the root
  secret, the STS signing key and every looked-up credential.
- IamAdapters keeps both halves private and hands them to a builder
  together through the consuming install(builder, make_authenticator);
  compile-fail doctests show a mixed pair cannot be assembled.
- Any Fn() -> Option<Arc<RootCredentials>> is a RootSource, so the
  binary reads its live root accessor directly; RootCell is now a
  test-only helper.

* test(gateway-adapter): use bucket names the rules accept in host tests

* refactor(gateway-adapter): never refuse on request id; owner is the reported constant

* docs(gateway-adapter): state what install guarantees about the provider handle

---------

Co-authored-by: rustfs-cla-assistant[bot] <273121464+rustfs-cla-assistant[bot]@users.noreply.github.com>
2026-11-11 06:16:20 -05:00
overtrue 613cfac753 Merge remote-tracking branch 'origin/gateway/integration' into gw/T3-reference-binary 2026-11-11 14:24:38 +08:00
rustfs-cla-assistant[bot] 4c7cc8a943 Merge remote-tracking branch 'origin/main' into gateway/integration 2026-11-11 05:25:16 +00:00
hector 75292c6224 ci: add manual packaging of the origin/main tip (#9291)
Dispatch-only workflow that compiles the current main commit with a
caller-supplied version and uploads zip, deb, and rpm artifacts.

Co-authored-by: hector <majinghe@users.noreply.github.com>
2026-11-11 04:55:02 +00:00
Chris An 16e3071f84 feat(gateway-adapter): IAM CredentialProvider and PolicySource (#9284)
* feat(server): add the RUSTFS_S3_STACK selector and dual-stack e2e lane (#8411)

* refactor(server): group the s3s-compat tower layers under legacy_compat (#8410)

Move the ten tower layers that patch s3s behaviour out of
rustfs/src/server/layer.rs into rustfs/src/server/legacy_compat/, one file
per layer with its tests. Each file header names the s3s behaviour it
patches, the gateway mechanism that replaces it (Replaced by:), and the task
that removes it (Removed by: T2.12, rustfs/backlog#2771). The layers keep
their code, their order in external_service_stack!, and their behaviour; no
layer is deleted.

Add server::http::tests::stack_census, which renders the ordered service
chain of the real external_service_stack! expansion and compares it with a
committed constant, so a reorder, insertion or removal is a visible diff.

Route the two s3s error types the moved layers use through a
storage_api::server::legacy_compat shim so the split adds no s3s-importing
file, and lower the s3s footprint baselines to the measured values (files 209
to 208, s3_error! lines 1586 to 1585). Keep the server runtime-source bypass
guard covering the moved code.

Refs rustfs/backlog#2787, rustfs/backlog#2734

* feat(s3-types): add the RustFS-owned S3 error type and s3_error! macro (#8406)

`rustfs-s3-types` now provides `S3ErrorCode` (generated by build.rs from
src/codes.txt, the grep-derived call surface, plus `Custom`), `S3Error`,
`S3Result`, `StdError` and the `s3_error` macro with the same call syntax
as s3s, so the T1.2 codemod is a `use` swap. Statuses are hand-written and
pinned to the s3s behaviour RustFS ships today by an oracle test that uses
s3s as a dev-dependency only. The `From` conversions for the legacy edge
live behind the off-by-default `compat-s3s` feature (the orphan rule
forbids them in rustfs/src/error.rs); `rustfs` enables it.

Refs rustfs/backlog#2742, rustfs/backlog#2734

* ci: add the per-crate test-count ratchet (#8404)

Quick Checks now fails any PR that lowers the number of test attributes
in a workspace crate or adds #[ignore], measured per crate against the
exact rows of .config/test-count-baseline.txt (rustfs/backlog#2734
section 1 rule 3). The counter widens the spec's literal
#[(tokio::)?test] to the tokio attributes that carry arguments and both
spellings of #[test_case(...)], and anchors at line start so
commented-out attributes do not count. The initial baseline covers 52
crates, 21077 tests and 164 ignored.

Refs rustfs/backlog#2738
Refs rustfs/backlog#2734

* test(ecstore): pin the persisted bucket-config XML bytes for 13 families (#8408)

Before any DTO or codec is replaced, pin the exact bytes that
deserialize followed by serialize produces for every persisted
bucket-configuration XML family in its three source shapes (AWS
documentation, MinIO-written, RustFS-written), and the bytes RustFS
writes today for a typed value. Byte equality only: a structural
comparison would let a codec replacement change the persisted encoding
unnoticed.

The generator script refuses to overwrite existing goldens without
--force, refuses on dirty codec paths, cannot report success from its
write mode, and records the generating commit in the fixtures README.

Refs rustfs/backlog#2744, rustfs/backlog#2734

* ci: raise the test-count baseline to the integration tree (#8426)

* ci: make the s3s footprint baseline exact and lower-only (#8403)

* ci: make the s3s footprint baseline exact and lower-only

The s3s footprint guard compared three counters against ceilings held in
the script itself, so a ceiling could sit above the tree and a drift was
fixable by raising it (rustfs/rustfs#5964 raised the s3_error! ceiling
to go green). The three values now live in
.config/s3s-footprint-baseline.txt as files, s3_error_lines and
ecstore_files, the script reads only that file, and a measurement that
differs from its baseline in either direction fails: growth prints
"s3s footprint grew: <measured> > <baseline>", shrinkage prints
"lower the baseline to <measured> in this PR". The counting method is
unchanged; the baseline is the current measurement of
gateway/integration (209 / 1585 / 36). The header records the end state:
once all three reach 0, T4.2 switches the script to failing on any
occurrence.

Refs rustfs/backlog#2739, rustfs/backlog#2734

* ci: add the s3s edge allowlist mode to the footprint guard (#8405)

`scripts/check_s3s_footprint.sh --mode allowlist` fails on every Rust
file that references s3s (`s3s::` or `s3s_sigv4`) and is not matched by
an entry of the new `.config/s3s-edge-allowlist.txt`; `--dry-run` lists
those files and exits 0. The allowlist carries the initial edge from the
spec, and the script rejects an entry that names or matches anything
under crates/ other than crates/e2e_test. The baseline mode and the
argument-less CI invocation are unchanged.

Refs rustfs/backlog#2741, rustfs/backlog#2734

* test(ecstore): reach the bucket-config codec through the test facade (#8428)

* refactor(s3-types): own the HTTP Body and drop s3s from three crates (#8431)

Add rustfs_s3_types::Body, a BoxBody<Bytes, StdError> wrapper with
empty() and From<Bytes | Vec<u8> | String | &'static str>, and the
legacy From conversions to and from s3s::Body behind compat-s3s.

The signer is now generic over the request body: it only reads and
writes the request head, so it needs no body type and its callers keep
theirs. s3-client sends its requests with the owned Body, madmin signs a
bodiless head, and targets drops its two S3Request/S3Response header
wrappers in favour of extract_params_header.

rustfs-signer, rustfs-madmin and rustfs-targets no longer depend on
s3s; the s3s footprint baseline drops from 209 to 202 files.

Refs rustfs/backlog#2747, rustfs/backlog#2734

* refactor(ecstore): move S3ErrorCode to rustfs-s3-types by codemod (#8432)

* test(ecstore): pin the InvalidRange responses of new_getobjectreader

Both error branches of new_getobjectreader build an ErrorResponse with S3ErrorCode::InvalidRange, and no test observed either code. Pin the code, its wire name and the location fields of each branch before the code type moves to rustfs-s3-types (rustfs/backlog#2743), so a changed code on either branch fails a test. The ecstore test-count row rises by the two tests.

* chore(scripts): add the s3s-to-rustfs-s3-types error codemod

Task T1.2 of rustfs/backlog#2734 (rustfs/backlog#2743) moves S3Error, S3ErrorCode, S3Result and s3_error! from s3s to rustfs-s3-types by script, one crate closure per PR. The codemod rewrites use trees (splitting mixed s3s imports) and qualified code paths, never comments or literals, adds the rustfs-s3-types dependency, drops s3s only from a crate that no longer names it, refuses every file of the s3s edge allowlist, takes --hold for a file whose type is fixed outside the closure, and prints a per-crate census. A second run changes nothing. Its self-test runs under make script-tests.

* refactor(ecstore): take S3ErrorCode from rustfs-s3-types

Output of scripts/codemods/s3_error_to_s3_types.py over crates/ecstore, crates/lifecycle, crates/scanner and crates/replication (rustfs/backlog#2743 batches 1 and 2). The ErrorResponse code field, the two InvalidRange constructions in new_getobjectreader and an unused tier import now use rustfs_s3_types::S3ErrorCode; the variant names and wire names are unchanged. lifecycle, scanner and replication reference none of the four error names. warm_backend.rs and warm_backend_s3.rs are held: they compare against rustfs_s3_client::ErrorResponse.code, which stays the s3s type until s3-client moves.

* ci(footprint): count only s3_error! lines still on s3s's macro

s3_error_lines counted every line holding the text `s3_error!`, which
also matches the RustFS-owned `rustfs_s3_types::s3_error!`. A codemod
that moves call sites onto rustfs-s3-types therefore could not lower it
(rustfs/backlog#2743, decision 3; rustfs/rustfs#8432, gotcha 2).

The counter now applies a textual, per-file and per-line rule that errs
on the side of counting (full rule in the script header):
- occurrences spelled `rustfs_s3_types::s3_error!` never count;
- a line holding `s3s::s3_error!` always counts;
- any other `s3_error!` line counts unless its file is cleared, that is,
  it imports the macro from rustfs_s3_types by name on a line starting
  with the use item (or defines it with `macro_rules! s3_error`) and does
  not also bring s3s's macro in (`use s3s::s3_error`, a `use s3s::{..}`
  tree naming it or a crate-root `*`, `use s3s::*`, or
  `#[macro_use] extern crate s3s`);
- fail closed: a file matching neither form (`use super::*`, a
  crate-local re-export, a renamed or glob rustfs_s3_types import, a
  commented-out import, a comment-only mention) counts.

Census of the 1585 lines on 96fc06776: 1478 in 64 files that import the
macro from s3s; 107 in 15 files matching neither form (105 through
`use super::*`, 1 qualified `s3s::s3_error!`, 1 comment line); none in
files that import it from rustfs_s3_types or define it. Every line stays
counted, so the baseline remains s3_error_lines=1585 (files=202 and
ecstore_files=36 unchanged). Rewriting rustfs/src/auth.rs's import to
rustfs_s3_types lowers the new counter to 1567; the old one stayed 1585.

`--self-test` (wired into `make script-tests`) copies the script into
22 fixture trees: 6 positive cases and 16 negative ones (lines that must
not count, forms that must not clear a file, and a missing baseline,
missing key or failing classification search that must fail). Each tree
has a control line that must count, and each run gets s3s text on stdin.
Seven cases were red against the old counter. 19 mutants each turn their
case red: old definition, flipped import test, dropped fail-closed
branch, dropped e2e_test exclusion, dropped s3s veto, dropped qualified
s3s rule, dropped rustfs-qualified strip, unanchored rustfs import,
renamed import accepted, nested glob taken as root glob, single-line
s3s trees only, dropped s3s glob forms, dropped macro_use form, dropped
definition clearing, ignored classification rg errors, error-line search
without its path, classification searches without their path, missing
baseline skipped, missing key skipped.

* test(e2e): compare two RUSTFS_S3_STACK servers over every S3 operation (#8435)

Add dual_stack_differential to e2e_test. It starts two rustfs processes
from the same binary, stack A on RUSTFS_S3_STACK=legacy and stack B on
RUSTFS_E2E_STACK_B (default legacy until rustfs/backlog#2734 T2.5),
sends both the same aws-sdk-s3 script covering the 73 methods of
impl S3 for FS (a normal request and a 4xx negative each), and compares
status, error code, an allowlist of response headers and a few typed
body fields. Error message text is never compared. Each run writes
<target>/dual-stack/<run>.json with compared_ops and the (op, field,
a, b) differences; any difference fails the test.

differential_detects_an_injected_difference starts B with
RUSTFS_REGION=eu-west-1 and requires GetBucketLocation to be reported.
The comparator is pure and unit-tested without a server.

Wire the module into the e2e-smoke filter and update the e2e-smoke and
e2e-full selection digests and the e2e_test test-count row.

Refs rustfs/backlog#2740

* feat(app): add the RustFS-owned RequestEnvelope and its s3s constructor (#8437)

* feat(app): add the RustFS-owned RequestEnvelope and its s3s constructor

* ci: classify the s3s edge adapters in storage/s3_compat as interface

* feat(s3-types): add Body::from_stream and move Swift off s3s::Body (#8438)

rustfs/backlog#2747 batch 3. `Body::from_stream` wraps a chunk stream of
unknown length over `StreamBody`/`BoxBody`: one data frame per chunk, the
first error ends the body, the size hint stays unknown, and dropping the
body drops the stream. Parity tests pin it to the s3s `StreamingBlob::wrap`
it replaces, at the `http_body` level and byte for byte through hyper.

The Swift handlers in rustfs-protocols build their responses as the owned
`Body`. `SwiftService` converts them into the wrapped S3 service's response
body type through `From<Body>`, so rustfs keeps serving `s3s::Body` at the
legacy edge unchanged. Request bodies stay Send-only (`UnsyncBoxBody`); the
two storage writers and the DLO/SLO segment streams get `Sync` from
`sync_wrapper::SyncStream`, which is what s3s's `http_body_unsync` mutex
provided before. The DTO and error uses of s3s in protocols are untouched
(T1.5 and T1.2).

* refactor(protocols): move error types to rustfs-s3-types by codemod (#8439)

T1.2 batch 3 (rustfs/backlog#2743): s3select-api, s3select-query and
the error items of protocols move off s3s by scripts/codemods, run with
--hold crates/protocols/src/sftp/errors.rs. That file stays on s3s: it
downcasts the backend error, whose runtime type is still s3s::S3Error.

- rustfs/src/protocols/client.rs: the two webdav session methods now
  return rustfs_s3_types::S3Result and convert the s3s error with From.
- codemod: a child module that calls s3_error! through `use super::*`
  gets its own `use rustfs_s3_types::s3_error;` once its parent imports
  that macro by name; otherwise it is listed as GLOB.
- s3-types: a table test pins code, status and the s3s error the legacy
  edge renders for every variant a moved call site names.
- s3_error_lines 1585 -> 1584 (a lifecycle comment reworded).

* test(connect): restore one-second drive request budgets

Apply the exact test-only repair already landed on main in 5972a09cc3. The filesystem probe and production CLI success test request one second and keep their existing one-second duration and parameter checks. All outcome, I/O, error, cleanup and resource-limit assertions remain.

Scope: rustfs/tests/connect_perf_drive.rs only. Patch-id: cb0a1dfc8f68a67727525adea7458e2cee7d09f8.

Validation environment: LANG=en_US.UTF-8, CARGO_BUILD_JOBS=1, CARGO_INCREMENTAL=0, exclusive APFS-cloned CARGO_TARGET_DIR.

Quick Checks: every command from .github/actions/quick-checks/action.yml, with captured output and exit status below.

$ shellcheck --version && actionlint
ShellCheck - shell script analysis tool
version: 0.11.0
license: GNU General Public License, version 3
website: https://www.shellcheck.net
exit_code=0

$ cargo fmt --all --check
exit_code=0

$ ./scripts/check_unsafe_code_allowances.sh
exit_code=0

$ ./scripts/check_layer_dependencies.sh
Layer dependency guard passed (no new reverse dependencies/cycles).
exit_code=0

$ ./scripts/check_architecture_migration_rules.sh
Architecture migration rules passed.
exit_code=0

$ ./scripts/check_logging_guardrails.sh
✅ Logging guardrails check passed
exit_code=0

$ ./scripts/check_error_other_format_ratchet.sh
error(format!) ratchet passed (total call sites: 582).
exit_code=0

$ ./scripts/check_no_tokio_io_uring.sh
exit_code=0

$ ./scripts/check_extension_schema_boundaries.sh
Extension schema boundaries passed.
exit_code=0

$ ./scripts/check_body_cache_whitelist.sh
Body-cache whitelist guard passed.
exit_code=0

$ ./scripts/check_s3s_footprint.sh
s3s footprint OK: files importing s3s is 195 (baseline files=195)
s3s footprint OK: s3s s3_error! lines is 1584 (baseline s3_error_lines=1584)
s3s footprint OK: ecstore files referencing s3s is 36 (baseline ecstore_files=36)
✅ s3s footprint baseline check passed
exit_code=0

$ ./scripts/check_test_count_ratchet.sh
test-count ratchet OK: 52 crates, 21222 tests, 164 ignored match .config/test-count-baseline.txt
exit_code=0

$ ./scripts/check_fips_wording.sh
FIPS wording guard passed (README.md and CHANGELOG.md contain no forbidden claims).
KMS wording guard passed (crates/kms claims no Vault KV2 Transit wrapping).
exit_code=0

$ ./scripts/check_embedded_secrets.sh
Embedded secret guard passed (no private key material or provider credential literal in the tree).
exit_code=0

$ make script-tests
Running script tests...
entrypoint credential policy tests passed
internode grpc A/B bench env tests passed
object batch benchmark enhanced tests passed
PASS package service script contract
hotpath warp A/B gate tests passed
hotpath warp ABBA tests passed
Scanner validation artifacts written to /var/folders/0w/s4wxsn2n4w9b6f3cd4xrz53w0000gn/T/tmp.1pFe6Kl1Yu/out
Scanner validation artifacts written to /var/folders/0w/s4wxsn2n4w9b6f3cd4xrz53w0000gn/T/tmp.1pFe6Kl1Yu/out-missing-pid
.............................
----------------------------------------------------------------------
Ran 29 tests in 181.286s

OK
{"decision":"verified","pending_gates":["G01","G03","G04","G05","G06","G07","G08","G09","G10","G11","G12","G13","G14","P1","P2","P3","P4","R-D","R-E","R-L"],"pending_lanes":["authority-coverage","checkpoint-and-crash","ec8-4-multiset","maintenance-producers","mixed-version-rollback","mrf-responsibility","scheduler-pressure","status-and-outcome"],"rejected_errors":[],"rejected_gate":null,"release_approved":false,"release_schema_capable":true,"schema":1,"verified_gate":"G02"}
{"decision":"verified","pending_gates":["G01","G02","G03","G04","G05","G06","G07","G08","G09","G10","G11","G12","G13","G14","P1","P2","P3","P4","R-D","R-L"],"pending_lanes":["authority-coverage","checkpoint-and-crash","ec8-4-multiset","maintenance-producers","mixed-version-rollback","mrf-responsibility","scheduler-pressure","status-and-outcome"],"rejected_errors":[],"rejected_gate":null,"release_approved":false,"release_schema_capable":true,"schema":1,"verified_gate":"R-E"}
{"decision":"verified","pending_gates":["G01","G03","G04","G05","G06","G07","G08","G09","G10","G11","G12","G13","G14","P1","P2","P3","P4","R-D","R-E","R-L"],"pending_lanes":["authority-coverage","checkpoint-and-crash","ec8-4-multiset","maintenance-producers","mixed-version-rollback","mrf-responsibility","scheduler-pressure","status-and-outcome"],"rejected_errors":[],"rejected_gate":null,"release_approved":false,"release_schema_capable":true,"schema":1,"verified_gate":"G02"}
{"decision":"verified","pending_gates":["G01","G02","G03","G04","G05","G06","G07","G08","G09","G10","G11","G12","G13","G14","P1","P2","P3","P4","R-D","R-L"],"pending_lanes":["authority-coverage","checkpoint-and-crash","ec8-4-multiset","maintenance-producers","mixed-version-rollback","mrf-responsibility","scheduler-pressure","status-and-outcome"],"rejected_errors":[],"rejected_gate":null,"release_approved":false,"release_schema_capable":true,"schema":1,"verified_gate":"R-E"}
{"decision":"verified","pending_gates":["G02","G03","G04","G05","G06","G07","G08","G09","G10","G11","G12","G13","G14","P1","P2","P3","P4","R-D","R-E","R-L"],"pending_lanes":["authority-coverage","checkpoint-and-crash","ec8-4-multiset","maintenance-producers","mixed-version-rollback","mrf-responsibility","scheduler-pressure","status-and-outcome"],"rejected_errors":[],"rejected_gate":null,"release_approved":false,"release_schema_capable":true,"schema":1,"verified_gate":"G01"}
{"decision":"verified","pending_gates":["G01","G02","G04","G05","G06","G07","G08","G09","G10","G11","G12","G13","G14","P1","P2","P3","P4","R-D","R-E","R-L"],"pending_lanes":["authority-coverage","checkpoint-and-crash","ec8-4-multiset","maintenance-producers","mixed-version-rollback","mrf-responsibility","scheduler-pressure","status-and-outcome"],"rejected_errors":[],"rejected_gate":null,"release_approved":false,"release_schema_capable":true,"schema":1,"verified_gate":"G03"}
{"decision":"verified","pending_gates":["G01","G02","G03","G04","G05","G06","G07","G08","G09","G10","G11","G12","G13","G14","P1","P2","P3","P4","R-D","R-E"],"pending_lanes":["authority-coverage","checkpoint-and-crash","ec8-4-multiset","maintenance-producers","mixed-version-rollback","mrf-responsibility","scheduler-pressure","status-and-outcome"],"rejected_errors":[],"rejected_gate":null,"release_approved":false,"release_schema_capable":true,"schema":1,"verified_gate":"R-L"}
{"decision":"verified","pending_gates":["G01","G02","G03","G04","G05","G06","G07","G08","G09","G10","G11","G12","G13","P1","P2","P3","P4","R-D","R-E","R-L"],"pending_lanes":["authority-coverage","checkpoint-and-crash","ec8-4-multiset","maintenance-producers","mixed-version-rollback","mrf-responsibility","scheduler-pressure","status-and-outcome"],"rejected_errors":[],"rejected_gate":null,"release_approved":false,"release_schema_capable":true,"schema":1,"verified_gate":"G14"}
{"decision":"verified","pending_gates":["G01","G02","G03","G04","G05","G06","G07","G08","G09","G10","G11","G12","G13","P1","P2","P3","P4","R-D","R-E","R-L"],"pending_lanes":["authority-coverage","checkpoint-and-crash","ec8-4-multiset","maintenance-producers","mixed-version-rollback","mrf-responsibility","scheduler-pressure","status-and-outcome"],"rejected_errors":[],"rejected_gate":null,"release_approved":false,"release_schema_capable":true,"schema":1,"verified_gate":"G14"}
{"decision":"verified","pending_gates":["G01","G02","G03","G04","G05","G06","G07","G08","G09","G11","G12","G13","G14","P1","P2","P3","P4","R-D","R-E","R-L"],"pending_lanes":["authority-coverage","checkpoint-and-crash","ec8-4-multiset","maintenance-producers","mixed-version-rollback","mrf-responsibility","scheduler-pressure","status-and-outcome"],"rejected_errors":[],"rejected_gate":null,"release_approved":false,"release_schema_capable":true,"schema":1,"verified_gate":"G10"}
{"decision":"verified","pending_gates":["G01","G02","G03","G04","G05","G06","G07","G08","G09","G10","G11","G12","G13","G14","P2","P3","P4","R-D","R-E","R-L"],"pending_lanes":["authority-coverage","checkpoint-and-crash","ec8-4-multiset","maintenance-producers","mixed-version-rollback","mrf-responsibility","scheduler-pressure","status-and-outcome"],"rejected_errors":[],"rejected_gate":null,"release_approved":false,"release_schema_capable":true,"schema":1,"verified_gate":"P1"}
{"decision":"verified","pending_gates":["G01","G02","G03","G04","G05","G06","G07","G08","G09","G10","G11","G12","G13","G14","P1","P3","P4","R-D","R-E","R-L"],"pending_lanes":["authority-coverage","checkpoint-and-crash","ec8-4-multiset","maintenance-producers","mixed-version-rollback","mrf-responsibility","scheduler-pressure","status-and-outcome"],"rejected_errors":[],"rejected_gate":null,"release_approved":false,"release_schema_capable":true,"schema":1,"verified_gate":"P2"}
{"decision":"verified","pending_gates":["G01","G02","G03","G04","G05","G06","G07","G08","G09","G10","G11","G12","G13","G14","P1","P2","P4","R-D","R-E","R-L"],"pending_lanes":["authority-coverage","checkpoint-and-crash","ec8-4-multiset","maintenance-producers","mixed-version-rollback","mrf-responsibility","scheduler-pressure","status-and-outcome"],"rejected_errors":[],"rejected_gate":null,"release_approved":false,"release_schema_capable":true,"schema":1,"verified_gate":"P3"}
{"decision":"verified","pending_gates":["G01","G02","G03","G04","G06","G07","G08","G09","G10","G11","G12","G13","G14","P1","P2","P3","P4","R-D","R-E","R-L"],"pending_lanes":["authority-coverage","checkpoint-and-crash","ec8-4-multiset","maintenance-producers","mixed-version-rollback","mrf-responsibility","scheduler-pressure","status-and-outcome"],"rejected_errors":[],"rejected_gate":null,"release_approved":false,"release_schema_capable":true,"schema":1,"verified_gate":"G05"}
{"decision":"verified","pending_gates":["G01","G02","G03","G04","G05","G07","G08","G09","G10","G11","G12","G13","G14","P1","P2","P3","P4","R-D","R-E","R-L"],"pending_lanes":["authority-coverage","checkpoint-and-crash","ec8-4-multiset","maintenance-producers","mixed-version-rollback","mrf-responsibility","scheduler-pressure","status-and-outcome"],"rejected_errors":[],"rejected_gate":null,"release_approved":false,"release_schema_capable":true,"schema":1,"verified_gate":"G06"}
{"decision":"verified","pending_gates":["G01","G02","G03","G04","G05","G06","G07","G08","G09","G10","G11","G12","G13","G14","P1","P2","P3","P4","R-E","R-L"],"pending_lanes":["authority-coverage","checkpoint-and-crash","ec8-4-multiset","maintenance-producers","mixed-version-rollback","mrf-responsibility","scheduler-pressure","status-and-outcome"],"rejected_errors":[],"rejected_gate":null,"release_approved":false,"release_schema_capable":true,"schema":1,"verified_gate":"R-D"}
Status-and-outcome release descriptor verified: /private/var/folders/0w/s4wxsn2n4w9b6f3cd4xrz53w0000gn/T/tmp8rwn1pe0/descriptor/release-bundle-status-outcome.json
{"decision":"verified","pending_gates":["G01","G02","G03","G04","G06","G07","G08","G09","G10","G11","G12","G13","G14","P1","P2","P3","P4","R-D","R-E","R-L"],"pending_lanes":["authority-coverage","checkpoint-and-crash","ec8-4-multiset","maintenance-producers","mixed-version-rollback","mrf-responsibility","scheduler-pressure","status-and-outcome"],"rejected_errors":[],"rejected_gate":null,"release_approved":false,"release_schema_capable":true,"schema":1,"verified_gate":"G05"}
{"decision":"verified","pending_gates":["G01","G02","G03","G04","G05","G07","G08","G09","G10","G11","G12","G13","G14","P1","P2","P3","P4","R-D","R-E","R-L"],"pending_lanes":["authority-coverage","checkpoint-and-crash","ec8-4-multiset","maintenance-producers","mixed-version-rollback","mrf-responsibility","scheduler-pressure","status-and-outcome"],"rejected_errors":[],"rejected_gate":null,"release_approved":false,"release_schema_capable":true,"schema":1,"verified_gate":"G06"}
{"decision":"verified","pending_gates":["G01","G02","G03","G04","G05","G06","G07","G08","G09","G10","G11","G12","G13","G14","P1","P2","P3","P4","R-E","R-L"],"pending_lanes":["authority-coverage","checkpoint-and-crash","ec8-4-multiset","maintenance-producers","mixed-version-rollback","mrf-responsibility","scheduler-pressure","status-and-outcome"],"rejected_errors":[],"rejected_gate":null,"release_approved":false,"release_schema_capable":true,"schema":1,"verified_gate":"R-D"}
{"decision":"verified","pending_gates":["G01","G02","G03","G04","G05","G06","G08","G09","G10","G11","G12","G13","G14","P1","P2","P3","P4","R-D","R-E","R-L"],"pending_lanes":["authority-coverage","checkpoint-and-crash","ec8-4-multiset","maintenance-producers","mixed-version-rollback","mrf-responsibility","scheduler-pressure","status-and-outcome"],"rejected_errors":[],"rejected_gate":null,"release_approved":false,"release_schema_capable":true,"schema":1,"verified_gate":"G07"}
{"decision":"verified","pending_gates":["G01","G02","G03","G04","G05","G06","G07","G09","G10","G11","G12","G13","G14","P1","P2","P3","P4","R-D","R-E","R-L"],"pending_lanes":["authority-coverage","checkpoint-and-crash","ec8-4-multiset","maintenance-producers","mixed-version-rollback","mrf-responsibility","scheduler-pressure","status-and-outcome"],"rejected_errors":[],"rejected_gate":null,"release_approved":false,"release_schema_capable":true,"schema":1,"verified_gate":"G08"}
{"decision":"verified","pending_gates":["G01","G02","G03","G04","G05","G06","G07","G08","G09","G10","G11","G12","G13","G14","P1","P2","P3","R-D","R-E","R-L"],"pending_lanes":["authority-coverage","checkpoint-and-crash","ec8-4-multiset","maintenance-producers","mixed-version-rollback","mrf-responsibility","scheduler-pressure","status-and-outcome"],"rejected_errors":[],"rejected_gate":null,"release_approved":false,"release_schema_capable":true,"schema":1,"verified_gate":"P4"}
exact 1MiB handoff ABBA tests passed
pinned paired ABBA bench tests passed
==> /var/folders/0w/s4wxsn2n4w9b6f3cd4xrz53w0000gn/T/tmp.mYmuI01hNq/bin/path_containment (-max_total_time=7, -seed=123456789, -artifact_prefix=/var/folders/0w/s4wxsn2n4w9b6f3cd4xrz53w0000gn/T/tmp.mYmuI01hNq/explicit-artifacts/path_containment/, corpus=/private/tmp/gw-orchestrator-2734/ci-drive/fuzz/corpus/path_containment)
-seed=123456789
target=path_containment
seed=123456789
max_total_time=7
git_revision=4d15b816236630b2c42f321ffc9ed3654edc94e3
git_dirty=true
runner_mode=prebuilt
==> /var/folders/0w/s4wxsn2n4w9b6f3cd4xrz53w0000gn/T/tmp.mYmuI01hNq/bin/path_containment (-max_total_time=1, -seed=2322617706, -artifact_prefix=/var/folders/0w/s4wxsn2n4w9b6f3cd4xrz53w0000gn/T/tmp.mYmuI01hNq/auto-artifacts/path_containment/, corpus=/private/tmp/gw-orchestrator-2734/ci-drive/fuzz/corpus/path_containment)
-seed=2322617706
==> cargo +nightly fuzz run path_containment (-max_total_time=9, -seed=987654321, -artifact_prefix=/var/folders/0w/s4wxsn2n4w9b6f3cd4xrz53w0000gn/T/tmp.mYmuI01hNq/cargo-artifacts/path_containment/)
+nightly
fuzz
run
path_containment
-seed=987654321
seed=987654321
runner_mode=cargo-fuzz
fuzz runner seed manifest tests passed
✅ scripts/python_bin.sh resolver checks passed
self-test ok: clean fixture with every exemption present passes
self-test ok: all 11 pattern families fire
self-test ok: moving a pinned source is reported
Embedded secret guard self-test passed.
self-test ok: p1-s3s-import: 'use s3s::s3_error;' counts every line, the rustfs_s3_types module's too (s3_error_lines=3)
self-test ok: p2-s3s-mixed-tree: a multi-line s3s use tree naming s3_error counts (s3_error_lines=3)
self-test ok: p3-s3s-glob: a crate-root glob of s3s, bare or inside a tree, counts (s3_error_lines=5)
self-test ok: p4-macro-use: '#[macro_use] extern crate s3s' counts (s3_error_lines=3)
self-test ok: p5-qualified-s3s: a qualified s3s::s3_error! line counts in a cleared file (s3_error_lines=2)
self-test ok: p6-exact-pass: a baseline equal to the tree passes (exit 0)
self-test ok: n1-rustfs-import: 'use rustfs_s3_types::s3_error;' alone counts nothing (s3_error_lines=1)
self-test ok: n2-rustfs-mixed-tree: a rustfs_s3_types tree plus an s3s tree without the macro counts nothing (s3_error_lines=1)
self-test ok: n3-defines: the file defining the macro counts nothing (s3_error_lines=1)
self-test ok: n4-qualified-rustfs: a qualified rustfs_s3_types::s3_error! line counts nothing (s3_error_lines=1)
self-test ok: n5-renamed-call: a call through a renamed macro is not text-matched (out of scope) (s3_error_lines=1)
self-test ok: n6-comment-string: a comment or string mentioning s3_error! in a cleared file counts nothing (s3_error_lines=1)
self-test ok: n7-e2e-excluded: crates/e2e_test is excluded (s3_error_lines=1)
self-test ok: n8-super-glob: an unrecognised import form ('use super::*;') counts (s3_error_lines=3)
self-test ok: n9-facade: a crate-local re-export counts (s3_error_lines=2)
self-test ok: n10-renamed-import: a renamed rustfs_s3_types import does not clear s3_error! lines (s3_error_lines=2)
self-test ok: n11-commented-import: a commented-out rustfs_s3_types import does not clear (s3_error_lines=2)
self-test ok: n12-rustfs-glob: a glob import of rustfs_s3_types does not clear (s3_error_lines=2)
self-test ok: n13-comment-only: a comment mentioning s3_error! in a file with no import counts (s3_error_lines=2)
self-test ok: f1-baseline-missing: a missing baseline file fails (exit 1)
self-test ok: f2-key-missing: a baseline without s3_error_lines fails (exit 1)
self-test ok: f3-rg-fails: a failing classification search fails the guard (exit 1)
✅ s3s footprint self-test passed
test_core_gate_rejects_missing_ignored_filtered_and_corrupt_inputs (__main__.SelfTests.test_core_gate_rejects_missing_ignored_filtered_and_corrupt_inputs) ... ok
test_e2e_requires_registration (__main__.SelfTests.test_e2e_requires_registration) ... ok
test_fuzz_runtime_uses_matrix_target (__main__.SelfTests.test_fuzz_runtime_uses_matrix_target) ... ok
test_ilm_lane_keeps_the_measured_cargo_build_budget (__main__.SelfTests.test_ilm_lane_keeps_the_measured_cargo_build_budget) ... ok
test_profile_listing_binds_platform_digest (__main__.SelfTests.test_profile_listing_binds_platform_digest) ... ok
test_profile_listing_enforces_selection (__main__.SelfTests.test_profile_listing_enforces_selection) ... ok
test_quick_checks_commands_propagate_failure (__main__.SelfTests.test_quick_checks_commands_propagate_failure) ... ok
test_quick_checks_rejects_caller_and_execution_bypasses (__main__.SelfTests.test_quick_checks_rejects_caller_and_execution_bypasses) ... ok
test_runner_readiness_fails_closed (__main__.SelfTests.test_runner_readiness_fails_closed) ... ok
test_s3_runner_rejects_unbounded_failure_locals (__main__.SelfTests.test_s3_runner_rejects_unbounded_failure_locals) ... ok
test_scanner_heal_begin_requires_embedded_source_provenance (__main__.SelfTests.test_scanner_heal_begin_requires_embedded_source_provenance) ... ok
test_scanner_heal_case_does_not_approve_pending_release (__main__.SelfTests.test_scanner_heal_case_does_not_approve_pending_release) ... ok
test_scanner_heal_case_lane_is_required_for_runner_profile (__main__.SelfTests.test_scanner_heal_case_lane_is_required_for_runner_profile) ... ok
test_scanner_heal_case_only_schema_cannot_approve_release (__main__.SelfTests.test_scanner_heal_case_only_schema_cannot_approve_release) ... ok
test_scanner_heal_crash_case_rejects_restart_oracle (__main__.SelfTests.test_scanner_heal_crash_case_rejects_restart_oracle) ... ok
test_scanner_heal_ec84_case_rejects_wrong_erasure_geometry (__main__.SelfTests.test_scanner_heal_ec84_case_rejects_wrong_erasure_geometry) ... ok
test_scanner_heal_finish_collects_oracles_from_registry (__main__.SelfTests.test_scanner_heal_finish_collects_oracles_from_registry) ... ok
test_scanner_heal_multipool_case_accepts_deferred_optional_outage_write (__main__.SelfTests.test_scanner_heal_multipool_case_accepts_deferred_optional_outage_write) ... ok
test_scanner_heal_pending_gate_cannot_map_to_implemented_lane (__main__.SelfTests.test_scanner_heal_pending_gate_cannot_map_to_implemented_lane) ... ok
test_scanner_heal_receipts_reject_reuse_and_missing_builds (__main__.SelfTests.test_scanner_heal_receipts_reject_reuse_and_missing_builds) ... ok
test_scanner_heal_rejects_boolean_fractional_and_out_of_geometry_integers (__main__.SelfTests.test_scanner_heal_rejects_boolean_fractional_and_out_of_geometry_integers) ... ok
test_scanner_heal_rejects_broken_execution_and_artifacts (__main__.SelfTests.test_scanner_heal_rejects_broken_execution_and_artifacts) ... ok
test_scanner_heal_rejects_copied_junit_and_wrong_suite_build (__main__.SelfTests.test_scanner_heal_rejects_copied_junit_and_wrong_suite_build) ... ok
test_scanner_heal_release_bundle_accepts_complete_measured_evidence (__main__.SelfTests.test_scanner_heal_release_bundle_accepts_complete_measured_evidence) ... ok
test_scanner_heal_release_bundle_assembler_copies_measured_artifacts (__main__.SelfTests.test_scanner_heal_release_bundle_assembler_copies_measured_artifacts) ... ok
test_scanner_heal_release_bundle_assembler_merges_measured_descriptors (__main__.SelfTests.test_scanner_heal_release_bundle_assembler_merges_measured_descriptors) ... ok
test_scanner_heal_release_bundle_assembler_rejects_duplicate_descriptor_gate (__main__.SelfTests.test_scanner_heal_release_bundle_assembler_rejects_duplicate_descriptor_gate) ... ok
test_scanner_heal_release_bundle_assembler_rejects_fixture_descriptor (__main__.SelfTests.test_scanner_heal_release_bundle_assembler_rejects_fixture_descriptor) ... ok
test_scanner_heal_release_bundle_assembler_rejects_hash_mismatch (__main__.SelfTests.test_scanner_heal_release_bundle_assembler_rejects_hash_mismatch) ... ok
test_scanner_heal_release_bundle_assembler_rejects_mixed_descriptor_revisions (__main__.SelfTests.test_scanner_heal_release_bundle_assembler_rejects_mixed_descriptor_revisions) ... ok
test_scanner_heal_release_bundle_enforces_topology_duration_profile_and_versions (__main__.SelfTests.test_scanner_heal_release_bundle_enforces_topology_duration_profile_and_versions) ... ok
test_scanner_heal_release_bundle_fixture_writer_is_not_release_evidence (__main__.SelfTests.test_scanner_heal_release_bundle_fixture_writer_is_not_release_evidence) ... ok
test_scanner_heal_release_bundle_gate_accepts_single_measured_gate (__main__.SelfTests.test_scanner_heal_release_bundle_gate_accepts_single_measured_gate) ... ok
test_scanner_heal_release_bundle_gate_rejects_missing_field (__main__.SelfTests.test_scanner_heal_release_bundle_gate_rejects_missing_field) ... ok
test_scanner_heal_release_bundle_gate_rejects_unknown_gate (__main__.SelfTests.test_scanner_heal_release_bundle_gate_rejects_unknown_gate) ... ok
test_scanner_heal_release_bundle_rejects_synthetic_or_missing_fields (__main__.SelfTests.test_scanner_heal_release_bundle_rejects_synthetic_or_missing_fields) ... ok
test_scanner_heal_release_bundle_requires_domain_evidence (__main__.SelfTests.test_scanner_heal_release_bundle_requires_domain_evidence) ... ok
test_scanner_heal_release_bundle_requires_field_provenance (__main__.SelfTests.test_scanner_heal_release_bundle_requires_field_provenance) ... ok
test_scanner_heal_release_bundle_requires_json_artifact_provenance (__main__.SelfTests.test_scanner_heal_release_bundle_requires_json_artifact_provenance) ... ok
test_scanner_heal_release_bundle_requires_mixed_version_field_roles (__main__.SelfTests.test_scanner_heal_release_bundle_requires_mixed_version_field_roles) ... ok
test_scanner_heal_release_bundle_requires_same_gate_measurement_window (__main__.SelfTests.test_scanner_heal_release_bundle_requires_same_gate_measurement_window) ... ok
test_scanner_heal_release_consumes_multiple_registry_oracles (__main__.SelfTests.test_scanner_heal_release_consumes_multiple_registry_oracles) ... ok
test_scanner_heal_release_matrix_lanes_remain_pending_without_real_evidence (__main__.SelfTests.test_scanner_heal_release_matrix_lanes_remain_pending_without_real_evidence) ... ok
test_scanner_heal_release_requirements_cannot_be_cleared (__main__.SelfTests.test_scanner_heal_release_requirements_cannot_be_cleared) ... ok
test_scanner_heal_release_status_rejects_focused_case_run (__main__.SelfTests.test_scanner_heal_release_status_rejects_focused_case_run) ... ok
test_scanner_heal_release_status_rejects_synthetic_case (__main__.SelfTests.test_scanner_heal_release_status_rejects_synthetic_case) ... ok
test_scanner_heal_required_evidence_fields_cannot_be_removed (__main__.SelfTests.test_scanner_heal_required_evidence_fields_cannot_be_removed) ... ok
test_scheduled_alerts_require_completion_watchdog (__main__.SelfTests.test_scheduled_alerts_require_completion_watchdog) ... ok
test_update_profile_selection_changes_only_requested_platform (__main__.SelfTests.test_update_profile_selection_changes_only_requested_platform) ... ok
test_validate_includes_quick_checks (__main__.SelfTests.test_validate_includes_quick_checks) ... ok
test_vault_tests_require_cross_process_serialization (__main__.SelfTests.test_vault_tests_require_cross_process_serialization) ... ok
test_workflow_readiness_requires_dependency_probe (__main__.SelfTests.test_workflow_readiness_requires_dependency_probe) ... ok

----------------------------------------------------------------------
Ran 52 tests in 14.301s

OK
e2e-full selection OK: 1 tests, sha256=29e30e6ed2ed38d9e5b0a7cdd9536abf5c719f7021a02b578ae1381563a3c273
...................................
----------------------------------------------------------------------
Ran 35 tests in 1.161s

OK
...................
----------------------------------------------------------------------
Ran 19 tests in 17.397s

OK
..........
----------------------------------------------------------------------
Ran 10 tests in 0.183s

OK
test_actual_selector_bootstrap_uses_base_policy_and_fails_closed (__main__.SelfTests.test_actual_selector_bootstrap_uses_base_policy_and_fails_closed) ... ok
test_documentation_paths_do_not_hide_build_or_fixture_changes (__main__.SelfTests.test_documentation_paths_do_not_hide_build_or_fixture_changes) ... ok
test_e2e_scope_keeps_production_and_shared_configuration_full (__main__.SelfTests.test_e2e_scope_keeps_production_and_shared_configuration_full) ... ok
test_e2e_shortcut_rejects_new_direct_renamed_and_target_dependencies (__main__.SelfTests.test_e2e_shortcut_rejects_new_direct_renamed_and_target_dependencies) ... ok
test_e2e_shortcut_traverses_implicit_path_dependencies (__main__.SelfTests.test_e2e_shortcut_traverses_implicit_path_dependencies) ... ok
test_event_contract_requires_complete_candidate_and_optional_lanes (__main__.SelfTests.test_event_contract_requires_complete_candidate_and_optional_lanes) ... ok
test_every_wrong_result_missing_job_or_selection_fails_closed (__main__.SelfTests.test_every_wrong_result_missing_job_or_selection_fails_closed) ... ok
test_full_e2e_gate_preserves_workflow_branch_and_event_scope (__main__.SelfTests.test_full_e2e_gate_preserves_workflow_branch_and_event_scope) ... ok
test_git_range_includes_deleted_source_and_rename_origins (__main__.SelfTests.test_git_range_includes_deleted_source_and_rename_origins) ... ok
test_job_ids_and_display_names_cannot_hide_validation (__main__.SelfTests.test_job_ids_and_display_names_cannot_hide_validation) ... ok
test_report_only_stack_lane_mirrors_e2e_smoke_outside_the_aggregate (__main__.SelfTests.test_report_only_stack_lane_mirrors_e2e_smoke_outside_the_aggregate) ... ok
test_repository_wiring_and_missing_dependency_regression (__main__.SelfTests.test_repository_wiring_and_missing_dependency_regression) ... ok
test_verify_command_preserves_failures (__main__.SelfTests.test_verify_command_preserves_failures) ... ok

----------------------------------------------------------------------
Ran 13 tests in 3.280s

OK
security coverage self-test passed
test_api_failures_preserve_other_evidence_and_never_enter_grace (__main__.SelfTests.test_api_failures_preserve_other_evidence_and_never_enter_grace) ... ok
test_attempt_outcome_does_not_replace_recent_success (__main__.SelfTests.test_attempt_outcome_does_not_replace_recent_success) ... ok
test_cancelled_attempt_cannot_refresh_expired_success (__main__.SelfTests.test_cancelled_attempt_cannot_refresh_expired_success) ... ok
test_cli_requires_the_repository_default_branch (__main__.SelfTests.test_cli_requires_the_repository_default_branch) ... ok
test_config_rejects_duplicate_and_invalid_entries (__main__.SelfTests.test_config_rejects_duplicate_and_invalid_entries) ... ok
test_freshness_boundaries (__main__.SelfTests.test_freshness_boundaries) ... ok
test_grace_requires_two_successful_queries_with_no_history (__main__.SelfTests.test_grace_requires_two_successful_queries_with_no_history) ... ok
test_invalid_api_evidence_fails_closed (__main__.SelfTests.test_invalid_api_evidence_fails_closed) ... ok
test_report_retains_every_workflow (__main__.SelfTests.test_report_retains_every_workflow) ... ok
test_requests_filter_schedule_default_branch_and_success_on_server (__main__.SelfTests.test_requests_filter_schedule_default_branch_and_success_on_server) ... ok

----------------------------------------------------------------------
Ran 10 tests in 0.051s

OK
..............................
----------------------------------------------------------------------
Ran 30 tests in 22.354s

OK
..........
----------------------------------------------------------------------
Ran 10 tests in 2.969s

OK
.................................
----------------------------------------------------------------------
Ran 33 tests in 0.160s

OK
Online functional runner labels: pf-testing
.......
----------------------------------------------------------------------
Ran 7 tests in 0.002s

OK
.......
----------------------------------------------------------------------
Ran 7 tests in 0.001s

OK
....
----------------------------------------------------------------------
Ran 4 tests in 0.128s

OK
...
----------------------------------------------------------------------
Ran 3 tests in 0.547s

OK
PASS: cold stampede script self-test
exit_code=0

$ python3 ./scripts/test_object_generation_protocol_model.py
test_a_recovery_barrier_cannot_choose_an_empty_value (__main__.ObjectGenerationProtocolModel.test_a_recovery_barrier_cannot_choose_an_empty_value) ... ok
test_all_bounded_two_proposer_interleavings_choose_at_most_one_value (__main__.ObjectGenerationProtocolModel.test_all_bounded_two_proposer_interleavings_choose_at_most_one_value) ... ok
test_an_unavailable_decision_quorum_cannot_choose_a_candidate (__main__.ObjectGenerationProtocolModel.test_an_unavailable_decision_quorum_cannot_choose_a_candidate) ... ok
test_counter_allocation_requires_persistence_and_is_consumed_once (__main__.ObjectGenerationProtocolModel.test_counter_allocation_requires_persistence_and_is_consumed_once) ... ok
test_counter_restart_burns_persisted_allocations_before_or_after_issue (__main__.ObjectGenerationProtocolModel.test_counter_restart_burns_persisted_allocations_before_or_after_issue) ... ok
test_crash_after_promise_before_reply_does_not_restore_an_older_ballot (__main__.ObjectGenerationProtocolModel.test_crash_after_promise_before_reply_does_not_restore_an_older_ballot) ... ok
test_duplicate_or_unconfigured_voters_cannot_form_a_quorum (__main__.ObjectGenerationProtocolModel.test_duplicate_or_unconfigured_voters_cannot_form_a_quorum) ... ok
test_empty_or_unobserved_minority_slot_can_retry_without_allocating_another_revision (__main__.ObjectGenerationProtocolModel.test_empty_or_unobserved_minority_slot_can_retry_without_allocating_another_revision) ... ok
test_equal_ballot_cannot_accept_different_bytes_and_lower_ballot_is_rejected (__main__.ObjectGenerationProtocolModel.test_equal_ballot_cannot_accept_different_bytes_and_lower_ballot_is_rejected) ... ok
test_forgetting_tombstone_authority_allows_an_aba_create (__main__.ObjectGenerationProtocolModel.test_forgetting_tombstone_authority_allows_an_aba_create) ... ok
test_issuing_before_persistence_reuses_a_ballot_after_restart (__main__.ObjectGenerationProtocolModel.test_issuing_before_persistence_reuses_a_ballot_after_restart) ... ok
test_lost_ack_and_restart_preserve_the_chosen_operation (__main__.ObjectGenerationProtocolModel.test_lost_ack_and_restart_preserve_the_chosen_operation) ... ok
test_minority_accepted_work_is_adopted_and_finished (__main__.ObjectGenerationProtocolModel.test_minority_accepted_work_is_adopted_and_finished) ... ok
test_omitting_highest_accepted_adoption_has_a_double_choice_counterexample (__main__.ObjectGenerationProtocolModel.test_omitting_highest_accepted_adoption_has_a_double_choice_counterexample) ... ok
test_quorum_choice_does_not_require_an_uncontacted_disk_to_learn_it (__main__.ObjectGenerationProtocolModel.test_quorum_choice_does_not_require_an_uncontacted_disk_to_learn_it) ... ok
test_recovery_failure_or_stale_expected_head_does_not_allocate_a_successor (__main__.ObjectGenerationProtocolModel.test_recovery_failure_or_stale_expected_head_does_not_allocate_a_successor) ... ok
test_replacing_voters_with_empty_state_can_choose_a_conflicting_value (__main__.ObjectGenerationProtocolModel.test_replacing_voters_with_empty_state_can_choose_a_conflicting_value) ... ok
test_skipping_predecessor_recovery_chooses_a_broken_revision_chain (__main__.ObjectGenerationProtocolModel.test_skipping_predecessor_recovery_chooses_a_broken_revision_chain) ... ok
test_stale_publication_rollback_and_gc_cannot_change_the_winner (__main__.ObjectGenerationProtocolModel.test_stale_publication_rollback_and_gc_cannot_change_the_winner) ... ok
test_superseded_operation_identity_cannot_be_reused_for_different_input (__main__.ObjectGenerationProtocolModel.test_superseded_operation_identity_cannot_be_reused_for_different_input) ... ok
test_superseded_operation_retry_returns_its_original_successor (__main__.ObjectGenerationProtocolModel.test_superseded_operation_retry_returns_its_original_successor) ... ok
test_tombstone_and_null_marker_payloads_do_not_reset_authority_revision (__main__.ObjectGenerationProtocolModel.test_tombstone_and_null_marker_payloads_do_not_reset_authority_revision) ... ok
test_tombstone_rejects_a_delayed_never_present_create (__main__.ObjectGenerationProtocolModel.test_tombstone_rejects_a_delayed_never_present_create) ... ok
test_unlearned_predecessor_is_recovered_before_the_second_slot (__main__.ObjectGenerationProtocolModel.test_unlearned_predecessor_is_recovered_before_the_second_slot) ... ok

----------------------------------------------------------------------
Ran 24 tests in 0.958s

OK
model: states=51023, transitions=167520, terminal=732, chosen_candidates=2
negative control: double-choice trace=((0, 'prepare', 0), (0, 'prepare', 1), (0, 'prepare', 2), (0, 'accept', 0), (0, 'accept', 1), (0, 'accept', 2), (1, 'prepare', 0), (1, 'prepare', 1), (1, 'prepare', 2), (1, 'accept', 0), (1, 'accept', 1), (1, 'accept', 2))
multi-slot round schedules: 640
exit_code=0

$ python3 ./scripts/check_test_wiring.py
OK: e2e modules, runner selection, fuzz matrices, profiles, scheduled alerts, and Quick Checks are wired
exit_code=0

$ ./scripts/check_no_planning_docs.sh
exit_code=0

$ ./scripts/check_uring_lane_lib_only.sh
OK: no 'uring_'-matching test functions under crates/ecstore/tests; --lib is safe
exit_code=0

Initial complete gate
$ cargo nextest run --profile ci -p rustfs --test connect_perf_drive --no-fail-fast
 Nextest run ID e07fc5b9-c6c8-4cdd-ae61-54839bf24437 with nextest profile: ci
    Starting 10 tests across 1 binary
        PASS [   0.016s] rustfs::connect_perf_drive cancellation_and_single_collector_leave_no_scratch
        PASS [   0.011s] rustfs::connect_perf_drive cleanup_failure_overrides_cancellation_and_remains_attributed
        PASS [   0.009s] rustfs::connect_perf_drive consent_and_resource_budgets_fail_before_scratch_io
        PASS [   0.009s] rustfs::connect_perf_drive invalid_roots_and_existing_scratch_fail_without_clobbering
        PASS [  16.950s] rustfs::connect_perf_drive production_cli_measures_and_signs_exact_binary_provenance
        PASS [   4.025s] rustfs::connect_perf_drive production_cli_rejects_missing_l1_consent_and_limits_before_artifacts
        PASS [   0.008s] rustfs::connect_perf_drive read_only_root_and_full_drive_errors_are_classified
        PASS [   0.015s] rustfs::connect_perf_drive real_local_write_read_is_measured_and_scratch_is_removed
        PASS [   0.007s] rustfs::connect_perf_drive runtime_timeout_and_full_drive_use_schema_valid_failed_reason
        PASS [   0.035s] rustfs::connect_perf_drive successful_result_has_signed_bounded_private_offline_export
────────────
     Summary [  21.087s] 10 tests run: 10 passed, 0 skipped
exit_code=0

Restored complete gate
$ cargo nextest run --profile ci -p rustfs --test connect_perf_drive --no-fail-fast
 Nextest run ID 233b7307-5361-4eb7-94d5-b384f17d2d6a with nextest profile: ci
    Starting 10 tests across 1 binary
        PASS [   0.009s] rustfs::connect_perf_drive cancellation_and_single_collector_leave_no_scratch
        PASS [   0.008s] rustfs::connect_perf_drive cleanup_failure_overrides_cancellation_and_remains_attributed
        PASS [   0.008s] rustfs::connect_perf_drive consent_and_resource_budgets_fail_before_scratch_io
        PASS [   0.008s] rustfs::connect_perf_drive invalid_roots_and_existing_scratch_fail_without_clobbering
        PASS [  19.518s] rustfs::connect_perf_drive production_cli_measures_and_signs_exact_binary_provenance
        PASS [   4.146s] rustfs::connect_perf_drive production_cli_rejects_missing_l1_consent_and_limits_before_artifacts
        PASS [   0.009s] rustfs::connect_perf_drive read_only_root_and_full_drive_errors_are_classified
        PASS [   0.018s] rustfs::connect_perf_drive real_local_write_read_is_measured_and_scratch_is_removed
        PASS [   0.009s] rustfs::connect_perf_drive runtime_timeout_and_full_drive_use_schema_valid_failed_reason
        PASS [   0.054s] rustfs::connect_perf_drive successful_result_has_signed_bounded_private_offline_export
────────────
     Summary [  23.787s] 10 tests run: 10 passed, 0 skipped
exit_code=0

Mutation M1: success_measurement reports max(actual elapsed, requested budget + 1), forcing 1001 ms for the one-second request. The retained duration assertion at rustfs/tests/connect_perf_drive.rs:105 fails.
$ cargo nextest run --profile ci -p rustfs --test connect_perf_drive --no-fail-fast -E 'test(real_local_write_read_is_measured_and_scratch_is_removed)'
info: experimental features enabled: setup-scripts
   Compiling rustfs v1.0.1 (/private/tmp/gw-orchestrator-2734/ci-drive/rustfs)
warning: linker stderr: ld: __eh_frame section too large (max 16MB) to encode dwarf unwind offsets in compact unwind table, performance of exception handling might be affected
  |
  = note: `#[warn(linker_messages)]` on by default

warning: `rustfs` (bin "rustfs") generated 1 warning
warning: `rustfs` (bin "rustfs-cli") generated 1 warning (1 duplicate)
    Finished `test` profile [unoptimized + debuginfo] target(s) in 7m 06s
────────────
 Nextest run ID 9f618777-7bf6-4cbc-8803-dd7f50f449ed with nextest profile: ci
    Starting 1 test across 1 binary (9 tests skipped)
        FAIL [   0.016s] rustfs::connect_perf_drive real_local_write_read_is_measured_and_scratch_is_removed
  stdout ───

    running 1 test
    test real_local_write_read_is_measured_and_scratch_is_removed ... FAILED

    failures:

    failures:
        real_local_write_read_is_measured_and_scratch_is_removed

    test result: FAILED. 0 passed; 1 failed; 0 ignored; 0 measured; 9 filtered out; finished in 0.01s

  stderr ───

    thread 'real_local_write_read_is_measured_and_scratch_is_removed' (106986846) panicked at rustfs/tests/connect_perf_drive.rs:105:5:
    assertion failed: (1..=1_000).contains(&data.duration_millis)
    note: run with `RUST_BACKTRACE=1` environment variable to display a backtrace

────────────
     Summary [   0.017s] 1 test run: 0 passed, 1 failed, 9 skipped
        FAIL [   0.016s] rustfs::connect_perf_drive real_local_write_read_is_measured_and_scratch_is_removed
error: test run failed
exit_code=100

Mutation M2: target_parameters reports requested duration + 1. The in-process assertion at rustfs/tests/connect_perf_drive.rs:111 and CLI JSON assertion at rustfs/tests/connect_perf_drive.rs:527 both reject 1001 against 1000.
$ cargo nextest run --profile ci -p rustfs --test connect_perf_drive --no-fail-fast -E 'test(real_local_write_read_is_measured_and_scratch_is_removed) | test(production_cli_measures_and_signs_exact_binary_provenance)'
info: experimental features enabled: setup-scripts
   Compiling rustfs v1.0.1 (/private/tmp/gw-orchestrator-2734/ci-drive/rustfs)
warning: linker stderr: ld: __eh_frame section too large (max 16MB) to encode dwarf unwind offsets in compact unwind table, performance of exception handling might be affected
  |
  = note: `#[warn(linker_messages)]` on by default

warning: `rustfs` (bin "rustfs") generated 1 warning
warning: `rustfs` (bin "rustfs-cli") generated 1 warning (1 duplicate)
    Finished `test` profile [unoptimized + debuginfo] target(s) in 7m 20s
────────────
 Nextest run ID b73d46ab-c910-4e1e-8063-7fc7e332ec15 with nextest profile: ci
    Starting 2 tests across 1 binary (8 tests skipped)
        FAIL [  13.968s] rustfs::connect_perf_drive production_cli_measures_and_signs_exact_binary_provenance
  stdout ───

    running 1 test
    test production_cli_measures_and_signs_exact_binary_provenance ... FAILED

    failures:

    failures:
        production_cli_measures_and_signs_exact_binary_provenance

    test result: FAILED. 0 passed; 1 failed; 0 ignored; 0 measured; 9 filtered out; finished in 13.96s

  stderr ───

    thread 'production_cli_measures_and_signs_exact_binary_provenance' (107015462) panicked at rustfs/tests/connect_perf_drive.rs:527:5:
    assertion `left == right` failed
      left: Number(1001)
     right: 1000
    note: run with `RUST_BACKTRACE=1` environment variable to display a backtrace

        FAIL [   0.015s] rustfs::connect_perf_drive real_local_write_read_is_measured_and_scratch_is_removed
  stdout ───

    running 1 test
    test real_local_write_read_is_measured_and_scratch_is_removed ... FAILED

    failures:

    failures:
        real_local_write_read_is_measured_and_scratch_is_removed

    test result: FAILED. 0 passed; 1 failed; 0 ignored; 0 measured; 9 filtered out; finished in 0.01s

  stderr ───

    thread 'real_local_write_read_is_measured_and_scratch_is_removed' (107016667) panicked at rustfs/tests/connect_perf_drive.rs:111:5:
    assertion `left == right` failed
      left: 1001
     right: 1000
    note: run with `RUST_BACKTRACE=1` environment variable to display a backtrace

────────────
     Summary [  13.985s] 2 tests run: 0 passed, 2 failed, 8 skipped
        FAIL [  13.968s] rustfs::connect_perf_drive production_cli_measures_and_signs_exact_binary_provenance
        FAIL [   0.015s] rustfs::connect_perf_drive real_local_write_read_is_measured_and_scratch_is_removed
error: test run failed
exit_code=100

Transient production mutations were restored byte-for-byte before the complete green rerun. Original/restored SHA256: 8d389e6d8836c041aab8c6a93e56addd65ba3f70b33264e1c74f639d0572a2da.

$ git diff --check

exit_code=0

$ git diff --cached --check

exit_code=0

Independent orchestrator acceptance
Tested code tree: e67c72fb01

Tested tree: 16fbbd2ea4
shellcheck --version && actionlint -> exit 0
cargo fmt --all --check -> exit 0
./scripts/check_unsafe_code_allowances.sh -> exit 0
./scripts/check_layer_dependencies.sh -> exit 0
./scripts/check_architecture_migration_rules.sh -> exit 0
./scripts/check_logging_guardrails.sh -> exit 0
./scripts/check_error_other_format_ratchet.sh -> exit 0
./scripts/check_no_tokio_io_uring.sh -> exit 0
./scripts/check_extension_schema_boundaries.sh -> exit 0
./scripts/check_body_cache_whitelist.sh -> exit 0
./scripts/check_s3s_footprint.sh -> exit 0
./scripts/check_test_count_ratchet.sh -> exit 0
./scripts/check_fips_wording.sh -> exit 0
./scripts/check_embedded_secrets.sh -> exit 0
make script-tests -> exit 0
python3 ./scripts/test_object_generation_protocol_model.py -> exit 0
python3 ./scripts/check_test_wiring.py -> exit 0
./scripts/check_no_planning_docs.sh -> exit 0
./scripts/check_uring_lane_lib_only.sh -> exit 0

$ LANG=en_US.UTF-8 CARGO_BUILD_JOBS=1 CARGO_INCREMENTAL=0 cargo nextest run --profile ci -p rustfs --test connect_perf_drive --no-fail-fast
info: experimental features enabled: setup-scripts
warning: linker stderr: ld: __eh_frame section too large (max 16MB) to encode dwarf unwind offsets in compact unwind table, performance of exception handling might be affected
  |
  = note: `#[warn(linker_messages)]` on by default

warning: `rustfs` (bin "rustfs-cli") generated 1 warning
warning: `rustfs` (bin "rustfs") generated 1 warning (1 duplicate)
    Finished `test` profile [unoptimized + debuginfo] target(s) in 0.69s
────────────
 Nextest run ID cd1edf4c-f1c0-456d-ad1a-76cc00ce05c9 with nextest profile: ci
    Starting 10 tests across 1 binary
        PASS [   0.012s] rustfs::connect_perf_drive cancellation_and_single_collector_leave_no_scratch
        PASS [   0.010s] rustfs::connect_perf_drive cleanup_failure_overrides_cancellation_and_remains_attributed
        PASS [   0.010s] rustfs::connect_perf_drive consent_and_resource_budgets_fail_before_scratch_io
        PASS [   0.009s] rustfs::connect_perf_drive invalid_roots_and_existing_scratch_fail_without_clobbering
        PASS [  20.328s] rustfs::connect_perf_drive production_cli_measures_and_signs_exact_binary_provenance
        PASS [   5.527s] rustfs::connect_perf_drive production_cli_rejects_missing_l1_consent_and_limits_before_artifacts
        PASS [   0.013s] rustfs::connect_perf_drive read_only_root_and_full_drive_errors_are_classified
        PASS [   0.019s] rustfs::connect_perf_drive real_local_write_read_is_measured_and_scratch_is_removed
        PASS [   0.011s] rustfs::connect_perf_drive runtime_timeout_and_full_drive_use_schema_valid_failed_reason
        PASS [   0.049s] rustfs::connect_perf_drive successful_result_has_signed_bounded_private_offline_export
────────────
     Summary [  25.991s] 10 tests run: 10 passed, 0 skipped

exit_code=0

All 54 tracked Cargo.toml files parse without duplicate keys. git diff --check passed. The production source remains unchanged; the two deliberate implementation mutations above were restored before this complete green run.

* fix(test): compare legacy body wire parity as raw bytes

Lossy UTF-8 comparison concealed a changed binary response byte.
Compare the already captured Vec<u8> values directly.

Validation:
- Binary mutant: migrated payload 0x84 vs legacy 0x83; old comparator
  passed 1/1, raw comparator failed 0/1 (exit 100); restored green.
- Non-Swift body-drain mutant: untouched-body delegation test failed
  with empty vs payload (exit 100); restored handler without a diff.
- cargo nextest run --profile ci -p rustfs-s3-types -p rustfs-protocols
  --features rustfs-protocols/swift --no-fail-fast:
  543 tests run, 543 passed, 0 skipped after both mutants restored.
- cargo fmt --all --check and git diff --check passed.

Orchestrator acceptance (LANG=en_US.UTF-8):
cargo nextest run --profile ci -p rustfs-s3-types -p rustfs-protocols --features rustfs-protocols/swift --no-fail-fast
Summary [2.445s] 543 tests run: 543 passed, 0 skipped.

All shared quick checks on the final code tree:
shellcheck --version && actionlint -> exit 0
cargo fmt --all --check -> exit 0
./scripts/check_unsafe_code_allowances.sh -> exit 0
./scripts/check_layer_dependencies.sh -> exit 0
./scripts/check_architecture_migration_rules.sh -> exit 0
./scripts/check_logging_guardrails.sh -> exit 0
./scripts/check_error_other_format_ratchet.sh -> exit 0
./scripts/check_no_tokio_io_uring.sh -> exit 0
./scripts/check_extension_schema_boundaries.sh -> exit 0
./scripts/check_body_cache_whitelist.sh -> exit 0
./scripts/check_s3s_footprint.sh -> exit 0
./scripts/check_test_count_ratchet.sh -> exit 0
./scripts/check_fips_wording.sh -> exit 0
./scripts/check_embedded_secrets.sh -> exit 0
make script-tests -> exit 0
python3 ./scripts/test_object_generation_protocol_model.py -> exit 0
python3 ./scripts/check_test_wiring.py -> exit 0
./scripts/check_no_planning_docs.sh -> exit 0
./scripts/check_uring_lane_lib_only.sh -> exit 0

Final independent acceptance after the drive-test CI repair
Parent: a99d8f4f89
Tested code tree: 93cbd96bf0

Tested tree: 075ae5161c2b62a96812bc1e3c904a7663b4f2f8
shellcheck --version && actionlint -> exit 0
cargo fmt --all --check -> exit 0
./scripts/check_unsafe_code_allowances.sh -> exit 0
./scripts/check_layer_dependencies.sh -> exit 0
./scripts/check_architecture_migration_rules.sh -> exit 0
./scripts/check_logging_guardrails.sh -> exit 0
./scripts/check_error_other_format_ratchet.sh -> exit 0
./scripts/check_no_tokio_io_uring.sh -> exit 0
./scripts/check_extension_schema_boundaries.sh -> exit 0
./scripts/check_body_cache_whitelist.sh -> exit 0
./scripts/check_s3s_footprint.sh -> exit 0
./scripts/check_test_count_ratchet.sh -> exit 0
./scripts/check_fips_wording.sh -> exit 0
./scripts/check_embedded_secrets.sh -> exit 0
make script-tests -> exit 0
python3 ./scripts/test_object_generation_protocol_model.py -> exit 0
python3 ./scripts/check_test_wiring.py -> exit 0
./scripts/check_no_planning_docs.sh -> exit 0
./scripts/check_uring_lane_lib_only.sh -> exit 0

$ LANG=en_US.UTF-8 CARGO_BUILD_JOBS=1 CARGO_INCREMENTAL=0 cargo nextest run --profile ci -p rustfs-s3-types -p rustfs-protocols --features rustfs-protocols/swift --no-fail-fast
    Finished `test` profile [unoptimized + debuginfo] target(s) in 10m 08s
 Nextest run ID 5c7ff102-ec6d-479a-86e0-810859b28bbf with nextest profile: ci
    Starting 543 tests across 8 binaries
     Summary [   6.729s] 543 tests run: 543 passed, 0 skipped
exit_code=0

The only source change is the raw-byte wire comparator. Cargo manifests and the ratchet baselines remain unchanged from the independently validated parent. The invalid-byte and consumed-request mutations remain recorded above; their implementation changes were restored.

* fix: repair integration checks and IPC test readiness

Wait for the RPC/locks test sockets to pass the existing owner and mode
checks inside the original 5s readiness window and 10ms poll interval.

Use the existing admin S3 carrier for the new recovery and quota error
calls. Regenerate the test-count baseline for main's 118 existing tests
across seven crates. Pair the private pagination helper's raw-scan inputs
and remove the redundant Copy clone from the retirement event.

Keep the original assertions, responses, authorization, locks and storage
operations. The pagination body and raw-scan expressions at all five calls
are byte-identical; reinserting only the Copy clone restores the entire
base heal file. All ignored baseline values and s3s counts stay unchanged.

Validation base: c47432e353
Validation tree: f081084bc5
Patch SHA-256: 2dcf0028691bfb2772edfb38e262829b04f4151832dcce823c9dd9f7434b52de
Trace production-prefix SHA-256: fd994f218a21429d62866f4efb8eee1ee57b58f4a51865aa35d05a54ef1ccb45

All commands used LANG=en_US.UTF-8, CARGO_BUILD_JOBS=1,
CARGO_INCREMENTAL=0 and CARGO_NET_OFFLINE=true in one exclusive target.
Every gate retained the same tree and six source hashes before and after.
54 Cargo.toml files parse without duplicate keys.

cargo clippy -p rustfs --all-targets -- -D warnings
    Finished `dev` profile [unoptimized + debuginfo] target(s) in 12m 50s
Exit: 0. Wall time: 770.855s.

The original 19 quick-check steps:
shellcheck --version && actionlint -> exit 0
cargo fmt --all --check -> exit 0
./scripts/check_unsafe_code_allowances.sh -> exit 0
./scripts/check_layer_dependencies.sh -> exit 0
./scripts/check_architecture_migration_rules.sh -> exit 0
./scripts/check_logging_guardrails.sh -> exit 0
./scripts/check_error_other_format_ratchet.sh -> exit 0
./scripts/check_no_tokio_io_uring.sh -> exit 0
./scripts/check_extension_schema_boundaries.sh -> exit 0
./scripts/check_body_cache_whitelist.sh -> exit 0
./scripts/check_s3s_footprint.sh -> exit 0
./scripts/check_test_count_ratchet.sh -> exit 0
./scripts/check_fips_wording.sh -> exit 0
./scripts/check_embedded_secrets.sh -> exit 0
make script-tests -> exit 0
python3 ./scripts/test_object_generation_protocol_model.py -> exit 0
python3 ./scripts/check_test_wiring.py -> exit 0
./scripts/check_no_planning_docs.sh -> exit 0
./scripts/check_uring_lane_lib_only.sh -> exit 0
Wall time: 286.337s.

cargo nextest run --profile ci -p rustfs --no-fail-fast
     Summary [ 217.017s] 5382 tests run: 5382 passed (4 leaky), 40 skipped
Exit: 0. Wall time: 1324.414s.

cargo nextest run --profile ci -p rustfs-s3-types -p rustfs-protocols --features rustfs-protocols/swift --no-fail-fast
     Summary [   2.189s] 543 tests run: 543 passed, 0 skipped
Exit: 0. Wall time: 123.192s.

cargo nextest run --profile ci -p rustfs-ecstore list_objects_paginate --no-fail-fast
     Summary [   0.170s] 8 tests run: 8 passed, 5921 skipped
Exit: 0. Wall time: 261.324s.

cargo nextest run --profile ci -p rustfs-heal durable_mrf_anchor_retires_only_after_exact_bucket_generation_is_deleted --no-fail-fast
     Summary [   0.805s] 1 test run: 1 passed, 835 skipped
Exit: 0. Wall time: 199.801s.

Baseline regeneration:
./scripts/check_test_count_ratchet.sh --update
wrote .config/test-count-baseline.txt: 52 crates, 21340 tests, 164 ignored
Test-only increases: ecstore48, heal3, iam14, io-metrics2, obs3, policy14,
rustfs34. Existing ignored values and all other rows are byte-identical.
s3s footprint OK: files importing s3s is 195 (baseline files=195)
s3s footprint OK: s3s s3_error! lines is 1584 (baseline s3_error_lines=1584)
s3s footprint OK: ecstore files referencing s3s is 36 (baseline ecstore_files=36)

Historical runtime red controls on f622 (same retained two assertions):
cargo nextest run --profile ci -p rustfs --lib -E 'test(=connect::diagnostics::trace_runtime::tests::local_top_rpc_reads_service_http_completions_and_signs_offline) | test(=connect::diagnostics::trace_runtime::tests::local_top_locks_reads_service_manager_and_signs_offline)' --no-fail-fast
Original path readiness with a temporary 500ms bind/seal gap:
     Summary [   0.863s] 2 tests run: 0 passed, 2 failed, 5113 skipped
Exit: 100.
Sealed readiness in that same gap:
     Summary [   5.621s] 2 tests run: 2 passed, 5113 skipped
Exit: 0.
Compiled M1: replace the helper predicate with path.exists():
     Summary [   0.870s] 2 tests run: 0 passed, 2 failed, 5113 skipped
Exit: 100.
Restore the sealed predicate byte-for-byte:
     Summary [   5.640s] 2 tests run: 2 passed, 5113 skipped
Exit: 0.
Both reds observe mode755 with owner_matches=true and fail at the original
RPC/locks unwraps with StateSecurity. M1 failure locations: trace_runtime.rs
2547:29 and 2713:29. No new assertions were introduced.

cargo nextest run --profile ci -p rustfs --lib -E 'test(connect::diagnostics::trace_runtime::tests::)' --no-fail-fast
     Summary [  20.748s] 28 tests run: 28 passed (1 leaky), 5087 skipped
Exit: 0. This security sweep is historical; the full final unit gate above
covers the actual c474 candidate.
The first disk-stopped compile ran no tests and is excluded from red credit.
All temporary seal instrumentation is removed from the final source.

Final unit cleanup markers: 4 leaky, 40 original skips (exit0). The ld
__eh_frame linker warning also appears verbatim in the original f622 run
and the earlier c474 run; there are no new source warnings.

* fix: remove redundant pending closure in disk test

Pass the capture-free pending future factory as a function item in the existing slow-operation test. This fixes the redundant_closure error at disk_store.rs:2914 that stopped strict Clippy before the feature tests could run.

The single-line test edit preserves the factory's result, warning and timeout behavior. Reversing the replacement reproduces the parent file byte for byte. All 220 assertion lines and the production prefix are unchanged; no tests, assertions, guards, profiles or baselines were added or relaxed.

Original Linux CI evidence (run 37888552268 at 53245be9f):
- RIO job113685483992: strict Clippy exit101, redundant closure at disk_store.rs:2914; feature tests skipped.
- Workspace Test and Lint job113685484058: same strict Clippy exit101; nextest and doc tests skipped.
These are lint failures, not runtime mutation kills or known flakes.

Native macOS arm64 validation on tree7a5bf47e7533b29479bc85207416102f1337c566, LANG=en_US.UTF-8, one build job, CARGO_INCREMENTAL=0 and offline dependencies:

shellcheck --version && actionlint -> exit 0
cargo fmt --all --check -> exit 0
./scripts/check_unsafe_code_allowances.sh -> exit 0
./scripts/check_layer_dependencies.sh -> exit 0
./scripts/check_architecture_migration_rules.sh -> exit 0
./scripts/check_logging_guardrails.sh -> exit 0
./scripts/check_error_other_format_ratchet.sh -> exit 0
./scripts/check_no_tokio_io_uring.sh -> exit 0
./scripts/check_extension_schema_boundaries.sh -> exit 0
./scripts/check_body_cache_whitelist.sh -> exit 0
./scripts/check_s3s_footprint.sh -> exit 0
./scripts/check_test_count_ratchet.sh -> exit 0
./scripts/check_fips_wording.sh -> exit 0
./scripts/check_embedded_secrets.sh -> exit 0
make script-tests -> exit 0
python3 ./scripts/test_object_generation_protocol_model.py -> exit 0
python3 ./scripts/check_test_wiring.py -> exit 0
./scripts/check_no_planning_docs.sh -> exit 0
./scripts/check_uring_lane_lib_only.sh -> exit 0
All 19 checks: exit0, 312.161s.

$ cargo clippy -p rustfs -p rustfs-ecstore --all-targets --features rio-v2 -- -D warnings
Finished `dev` profile [unoptimized + debuginfo] target(s) in 20m 11s
exit0, 1213.210s total.

$ cargo nextest run --profile ci -p rustfs-ecstore --features rio-v2 slow_disk_operation_warns_once_while_pending_and_preserves_result --no-fail-fast
Finished `test` profile [unoptimized + debuginfo] target(s) in 4m 34s
PASS [0.026s] rustfs-ecstore disk::disk_store::tests::slow_disk_operation_warns_once_while_pending_and_preserves_result
Summary [   0.170s] 1 test run: 1 passed, 5943 skipped
exit0, 281.920s total; 5943 existing filter skips.

warning: linker stderr: ld: __eh_frame section too large (max 16MB) to encode dwarf unwind offsets in compact unwind table, performance of exception handling might be affected
This linker warning is retained; flags, profiles and allowances are unchanged.

54 tracked Cargo.toml files parsed without duplicate keys. git diff --check passed. No new assertions were introduced, so no new runtime mutation is claimed. Native compilation and the existing helper test do not substitute for actual Linux RIO execution; the new integration CI run still needs to recover before the RPC merge.

Preservation and output hashes:
parent file SHA256: 8e59ee745e3aaaa22848b943d3b10f6353b515956282cde1f3af2c394200c9ac
final file SHA256: b69ba3237061b161448ae8b62b615885d5770b98a5b4a2479bcfbd70f1bd1f17
unchanged production prefix SHA256: 7903a6e288252c420231242d7b58a25fa2a7140ea3dd430cbe7f0dde11369dcb
RIO Linux failure log SHA256:421addb9868fd88e4109a922311e54f91f0dfd70862f8ba7fcaba193482ed90b
Workspace Linux failure log SHA256:e0aea8d0f014c022252a9534968e083c98cfc1fd760da445ecd04ec3f522e26a
19 output SHA256:35a54f28d707956ed1edb29d492c4dfe8a9fe753d103b111b5a2f1b5c41c2b98
RIO Clippy output SHA256:d6046e3868f2b3992fe11cdd04e9e98eb50a87d1234d9a70dbfeb154ff61638d
slow-operation output SHA256:605b7b258d2fd09c6ef2ab2ffc90d433d0509480f1e45125d20c823759e83608

Refs rustfs/backlog#2734.

* test(ecstore): isolate tier-free recovery fixture startup

The fixture constructor can take the production recovery lock and wait for 60 seconds before this test installs its queue and seeds metadata. Disable its background recovery only while creating this fixture, then restore the environment before calling the existing production recovery entrypoint.

Only the existing test setup changes. Its production entrypoint, default lock parameters, original assertions, 30-second timeout, cancellation and cleanup remain intact. No tests or assertions were added. Production and store/init.rs are byte-identical to the parent.

Independent acceptance ran on merge tree 8ec358cb42ba9ebb3bcfeafcef048a3f0366355b; the runner header reports parent HEAD, while before/after index and source hashes confirm this candidate. Worker task branch d8055766b8 has the same tree.

Every command used LANG=en_US.UTF-8. Task compilation also used CARGO_BUILD_JOBS=1, CARGO_INCREMENTAL=0, CARGO_NET_OFFLINE=true, an independent target, and empty RUSTC_WRAPPER/RUSTC_WORKSPACE_WRAPPER to keep compilers in its owned process group without changing repository configuration.

Command:
shellcheck --version && actionlint
Exit 0; observed output excerpt:
version: 0.11.0
Log SHA-256: fc9eec16b4191d058a295813ca9afbe9d2b37b59975da860170726be162a3924

Command:
cargo fmt --all --check
Exit 0; observed output excerpt:
(no stdout or stderr)
Log SHA-256: e3b0c44298fc1c149afbf4c8996fb92427ae41e4649b934ca495991b7852b855

Command:
./scripts/check_unsafe_code_allowances.sh
Exit 0; observed output excerpt:
(no stdout or stderr)
Log SHA-256: e3b0c44298fc1c149afbf4c8996fb92427ae41e4649b934ca495991b7852b855

Command:
./scripts/check_layer_dependencies.sh
Exit 0; observed output excerpt:
Layer dependency guard passed (no new reverse dependencies/cycles).
Log SHA-256: 69c6cb1896c0586b28d1167380d3260d99d8017a4265c8df5a2348f289e93a12

Command:
./scripts/check_architecture_migration_rules.sh
Exit 0; observed output excerpt:
Architecture migration rules passed.
Log SHA-256: 28790e70d070607ed4de0704f292a65030be92ec224ef106d4b72fd8296f896e

Command:
./scripts/check_logging_guardrails.sh
Exit 0; observed output excerpt:
✅ Logging guardrails check passed
Log SHA-256: 7adf41abff8ed8f7617decf86498888ddde78937099dd9ce47f6d359dfc2d317

Command:
./scripts/check_error_other_format_ratchet.sh
Exit 0; observed output excerpt:
error(format!) ratchet passed (total call sites: 579).
Log SHA-256: 66172aef18f150ba0605af122e26ffcf91f6dd86f43e38b05e2e982ce8d783ee

Command:
./scripts/check_no_tokio_io_uring.sh
Exit 0; observed output excerpt:
(no stdout or stderr)
Log SHA-256: e3b0c44298fc1c149afbf4c8996fb92427ae41e4649b934ca495991b7852b855

Command:
./scripts/check_extension_schema_boundaries.sh
Exit 0; observed output excerpt:
Extension schema boundaries passed.
Log SHA-256: f13d0acd0e2ce5d9815241b12b4dab7d01cc11b9b65bcf6dfb7812c878532a51

Command:
./scripts/check_body_cache_whitelist.sh
Exit 0; observed output excerpt:
Body-cache whitelist guard passed.
Log SHA-256: 6b828847b0383950ee6eef9cd846d0c322c945e47efd4c7d1ad881178810d6bb

Command:
./scripts/check_s3s_footprint.sh
Exit 0; observed output excerpt:
s3s footprint OK: files importing s3s is 195 (baseline files=195)
s3s footprint OK: s3s s3_error! lines is 1584 (baseline s3_error_lines=1584)
s3s footprint OK: ecstore files referencing s3s is 36 (baseline ecstore_files=36)
✅ s3s footprint baseline check passed
Log SHA-256: 47f53d04ef645618563fe1f22abc64667f19078f1974feedcca46e4f2c016844

Command:
./scripts/check_test_count_ratchet.sh
Exit 0; observed output excerpt:
test-count ratchet OK: 52 crates, 21340 tests, 164 ignored match .config/test-count-baseline.txt
Log SHA-256: 618a5e330e933de0c6ff484546df401457024a88bb5986d11d588794e33dab60

Command:
./scripts/check_fips_wording.sh
Exit 0; observed output excerpt:
FIPS wording guard passed (README.md and CHANGELOG.md contain no forbidden claims).
KMS wording guard passed (crates/kms claims no Vault KV2 Transit wrapping).
Log SHA-256: 99221cc9979c480ec4f46f50d76c52b18c72fe499f12e15e4f4a22a1b8bfb140

Command:
./scripts/check_embedded_secrets.sh
Exit 0; observed output excerpt:
Embedded secret guard passed (no private key material or provider credential literal in the tree).
Log SHA-256: 437a3ceba32094d2901889965ad191e411aad32cdb18bf0b8659e818011c406b

Command:
make script-tests
Exit 0; observed output excerpt:
test_cli_requires_the_repository_default_branch (__main__.SelfTests.test_cli_requires_the_repository_default_branch) ... ok
test_config_rejects_duplicate_and_invalid_entries (__main__.SelfTests.test_config_rejects_duplicate_and_invalid_entries) ... ok
test_freshness_boundaries (__main__.SelfTests.test_freshness_boundaries) ... ok
test_grace_requires_two_successful_queries_with_no_history (__main__.SelfTests.test_grace_requires_two_successful_queries_with_no_history) ... ok
test_invalid_api_evidence_fails_closed (__main__.SelfTests.test_invalid_api_evidence_fails_closed) ... ok
test_report_retains_every_workflow (__main__.SelfTests.test_report_retains_every_workflow) ... ok
test_requests_filter_schedule_default_branch_and_success_on_server (__main__.SelfTests.test_requests_filter_schedule_default_branch_and_success_on_server) ... ok
Ran 10 tests in 0.042s
OK
Ran 30 tests in 20.627s
OK
Ran 10 tests in 2.268s
OK
Ran 33 tests in 0.127s
OK
Ran 7 tests in 0.002s
OK
Ran 7 tests in 0.001s
OK
Ran 4 tests in 0.108s
OK
Ran 3 tests in 0.564s
OK
PASS: cold stampede script self-test
Log SHA-256: 75fda2d069a373c63667b908baaf99e6591aab9099a739b0c4c22ceb6da8d68c

Command:
python3 ./scripts/test_object_generation_protocol_model.py
Exit 0; observed output excerpt:
test_an_unavailable_decision_quorum_cannot_choose_a_candidate (__main__.ObjectGenerationProtocolModel.test_an_unavailable_decision_quorum_cannot_choose_a_candidate) ... ok
test_counter_allocation_requires_persistence_and_is_consumed_once (__main__.ObjectGenerationProtocolModel.test_counter_allocation_requires_persistence_and_is_consumed_once) ... ok
test_counter_restart_burns_persisted_allocations_before_or_after_issue (__main__.ObjectGenerationProtocolModel.test_counter_restart_burns_persisted_allocations_before_or_after_issue) ... ok
test_crash_after_promise_before_reply_does_not_restore_an_older_ballot (__main__.ObjectGenerationProtocolModel.test_crash_after_promise_before_reply_does_not_restore_an_older_ballot) ... ok
test_duplicate_or_unconfigured_voters_cannot_form_a_quorum (__main__.ObjectGenerationProtocolModel.test_duplicate_or_unconfigured_voters_cannot_form_a_quorum) ... ok
test_empty_or_unobserved_minority_slot_can_retry_without_allocating_another_revision (__main__.ObjectGenerationProtocolModel.test_empty_or_unobserved_minority_slot_can_retry_without_allocating_another_revision) ... ok
test_equal_ballot_cannot_accept_different_bytes_and_lower_ballot_is_rejected (__main__.ObjectGenerationProtocolModel.test_equal_ballot_cannot_accept_different_bytes_and_lower_ballot_is_rejected) ... ok
test_forgetting_tombstone_authority_allows_an_aba_create (__main__.ObjectGenerationProtocolModel.test_forgetting_tombstone_authority_allows_an_aba_create) ... ok
test_issuing_before_persistence_reuses_a_ballot_after_restart (__main__.ObjectGenerationProtocolModel.test_issuing_before_persistence_reuses_a_ballot_after_restart) ... ok
test_lost_ack_and_restart_preserve_the_chosen_operation (__main__.ObjectGenerationProtocolModel.test_lost_ack_and_restart_preserve_the_chosen_operation) ... ok
test_minority_accepted_work_is_adopted_and_finished (__main__.ObjectGenerationProtocolModel.test_minority_accepted_work_is_adopted_and_finished) ... ok
test_omitting_highest_accepted_adoption_has_a_double_choice_counterexample (__main__.ObjectGenerationProtocolModel.test_omitting_highest_accepted_adoption_has_a_double_choice_counterexample) ... ok
test_quorum_choice_does_not_require_an_uncontacted_disk_to_learn_it (__main__.ObjectGenerationProtocolModel.test_quorum_choice_does_not_require_an_uncontacted_disk_to_learn_it) ... ok
test_recovery_failure_or_stale_expected_head_does_not_allocate_a_successor (__main__.ObjectGenerationProtocolModel.test_recovery_failure_or_stale_expected_head_does_not_allocate_a_successor) ... ok
test_replacing_voters_with_empty_state_can_choose_a_conflicting_value (__main__.ObjectGenerationProtocolModel.test_replacing_voters_with_empty_state_can_choose_a_conflicting_value) ... ok
test_skipping_predecessor_recovery_chooses_a_broken_revision_chain (__main__.ObjectGenerationProtocolModel.test_skipping_predecessor_recovery_chooses_a_broken_revision_chain) ... ok
test_stale_publication_rollback_and_gc_cannot_change_the_winner (__main__.ObjectGenerationProtocolModel.test_stale_publication_rollback_and_gc_cannot_change_the_winner) ... ok
test_superseded_operation_identity_cannot_be_reused_for_different_input (__main__.ObjectGenerationProtocolModel.test_superseded_operation_identity_cannot_be_reused_for_different_input) ... ok
test_superseded_operation_retry_returns_its_original_successor (__main__.ObjectGenerationProtocolModel.test_superseded_operation_retry_returns_its_original_successor) ... ok
test_tombstone_and_null_marker_payloads_do_not_reset_authority_revision (__main__.ObjectGenerationProtocolModel.test_tombstone_and_null_marker_payloads_do_not_reset_authority_revision) ... ok
test_tombstone_rejects_a_delayed_never_present_create (__main__.ObjectGenerationProtocolModel.test_tombstone_rejects_a_delayed_never_present_create) ... ok
test_unlearned_predecessor_is_recovered_before_the_second_slot (__main__.ObjectGenerationProtocolModel.test_unlearned_predecessor_is_recovered_before_the_second_slot) ... ok
Ran 24 tests in 0.929s
OK
Log SHA-256: 0b4be926efb0609d497ae58930340153f01957b606206cbbdb3e0d47eb29d3a2

Command:
python3 ./scripts/check_test_wiring.py
Exit 0; observed output excerpt:
OK: e2e modules, runner selection, fuzz matrices, profiles, scheduled alerts, and Quick Checks are wired
Log SHA-256: 21f676f458451525535ab0ef73d6a9c0d9774e16d76a4d19e396ad34936dd39c

Command:
./scripts/check_no_planning_docs.sh
Exit 0; observed output excerpt:
(no stdout or stderr)
Log SHA-256: e3b0c44298fc1c149afbf4c8996fb92427ae41e4649b934ca495991b7852b855

Command:
./scripts/check_uring_lane_lib_only.sh
Exit 0; observed output excerpt:
OK: no 'uring_'-matching test functions under crates/ecstore/tests; --lib is safe
Log SHA-256: 43f3f851e27ec2ab9e34d6b05fefdede38af992699c473904bfe6f53a46d52e3

Command:
cargo nextest run --profile ci -p rustfs-ecstore --lib --features rio-v2 -E 'test(=bucket::lifecycle::bucket_lifecycle_ops::tests::tier_free_version_recovery_production_entrypoint_enqueues_seeded_item)' --no-fail-fast
Exit 0; wall 234.399s.
Observed output:
    Finished `test` profile [unoptimized + debuginfo] target(s) in 3m 49s
    Starting 1 test across 1 binary (5871 tests skipped)
  SETUP PASS [   0.006s] ecstore-base-stack: sh -c 'echo RUST_MIN_STACK=4194304 >> "$NEXTEST_ENV"'
        PASS [   0.440s] rustfs-ecstore bucket::lifecycle::bucket_lifecycle_ops::tests::tier_free_version_recovery_production_entrypoint_enqueues_seeded_item
     Summary [   0.586s] 1 test run: 1 passed, 5871 skipped
Log SHA-256: 34e973f86e8dc2102c1a0e483e0ac802ce7df3d1b40a603932f62c0805c4de27

Command:
cargo nextest run --profile ci -p rustfs-ecstore --lib --features rio-v2 -E 'test(=bucket::lifecycle::bucket_lifecycle_ops::tests::tier_free_version_recovery_eight_coordinators_hold_one_leader_and_take_over) | test(=store::bucket::tests::creation_commit_backfill_runs_under_the_config_write_fence) | test(=store::bucket::tests::orphaned_bucket_recovery_requires_every_volume_absent_and_is_retryable)' --no-fail-fast
Exit 0; wall 4.010s.
Observed output:
    Finished `test` profile [unoptimized + debuginfo] target(s) in 0.43s
    Starting 3 tests across 1 binary (5869 tests skipped)
  SETUP PASS [   0.005s] ecstore-base-stack: sh -c 'echo RUST_MIN_STACK=4194304 >> "$NEXTEST_ENV"'
        PASS [   0.068s] rustfs-ecstore bucket::lifecycle::bucket_lifecycle_ops::tests::tier_free_version_recovery_eight_coordinators_hold_one_leader_and_take_over
        PASS [   0.637s] rustfs-ecstore store::bucket::tests::creation_commit_backfill_runs_under_the_config_write_fence
        PASS [   0.846s] rustfs-ecstore store::bucket::tests::orphaned_bucket_recovery_requires_every_volume_absent_and_is_retryable
     Summary [   0.987s] 3 tests run: 3 passed, 5869 skipped
Log SHA-256: b5da2a927ff18ca9aa217762f7186a6d4371bd9e8feedd7b77940e10b64dffac

Command:
cargo clippy -p rustfs -p rustfs-ecstore --all-targets --features rio-v2 -- -D warnings
Exit 0; wall 1084.508s.
Observed output:
    Finished `dev` profile [unoptimized + debuginfo] target(s) in 18m 03s
Log SHA-256: 28f98be7970faaa71cd118da1da4e6e2e3d23b9dda7e6449eaa7a37e741260b5

Compiled mutation command (unchanged for each red and restored run):
cargo nextest run --profile ci -p rustfs-ecstore --lib --features rio-v2 -E 'test(=bucket::lifecycle::bucket_lifecycle_ops::tests::tier_free_version_recovery_production_entrypoint_enqueues_seeded_item)' --no-fail-fast

M1: reverse only this test-local constructor flag from false to true. The mutant compiled, then failed the original recovered-item queue assertion after 30.481s. Restoring the entire observed candidate passed in 0.460s. Red log 55bf3b1535529ac0bcfb8071f9b56f9f40631007a85c315367e7e93786d65293; restored log 204d5fbe4574a7917f5e740a795d864fe28a7e7fb5e33d27b0a9cf68e2f17f52.

M2: replace only the actual awaited enqueue with false, consuming its original inputs. The mutant compiled, then failed the same original queue assertion after 30.516s. Restoring the entire observed candidate passed in 0.457s. Red log 89186330413215e231e2b06bf013cc31097ea8f4a106d91421a36d6c8986a7fd; restored log e848b42d2302401afa5dbcd9e37bf76d2e56a96ab6de5a488513b997f53c5e9b.

All temporary observation and mutation code was removed before final acceptance. Historical baseline and diagnostic failures remain recorded; they are not mutation credit. The separate Object Lock intent-retry diagnostic still fails, and these native macOS checks do not establish Linux CI recovery.

* refactor(rpc): serve internode rpc bodies with the owned Body (#8441)

* test(rpc): pin the wire bytes of every internode rpc response body

rustfs/backlog#2747 batch 4, step 1 of 2, on the s3s bodies the rpc paths
serve today. Each response body builder in storage/rpc/http_service.rs is
served through hyper's HTTP/1.1 server and the raw bytes the peer reads are
compared with goldens captured from this commit: status, content-length or
chunked framing, chunk boundaries, the terminating chunk, HEAD, and the cut
connection a failure after data produces. The capability responses carry a
per-process proof, so their length header and payload are checked against
the body they encode instead.

The read_file_stream response builder moves into read_file_stream_response
so the test reaches the same code the handler runs; behaviour is unchanged.

* refactor(rpc): serve internode rpc bodies with the owned Body

rustfs/backlog#2747 batch 4, step 2 of 2. The internode rpc HTTP paths
build their responses with rustfs_s3_types::Body: Body::from_stream
replaces Body::from(StreamingBlob::wrap(..)) at the six stream sites, and
the in-memory and empty bodies keep their constructors. InternodeRpcService
converts its responses into the wrapped service's response body through
From<Body>, so the server keeps composing it with the s3s service as
before. The wire-byte goldens from the previous commit pass unchanged;
storage/rpc no longer names s3s.

* fix(rpc): simplify wire test response builder type

Name the boxed response builder used by the capability wire test so
all-target Clippy can check it without a type-complexity warning.

Validation (LANG=en_US.UTF-8, CARGO_INCREMENTAL=0):
- cargo clippy -p rustfs --all-targets -- -D warnings: exit 0.
- cargo nextest run --profile ci -p rustfs --lib -E 'test(/storage::rpc::/)': 279 tests run, 279 passed, 4842 skipped.
- All 19 quick-check commands, including script-tests, both Python scripts and cargo fmt --all --check: exit 0.
- Footprint baseline regenerated: files=194, s3_error_lines=1584, ecstore_files=36; test-count baseline regenerated: 52 crates, 21228 tests, 164 ignored.
- All 54 Cargo.toml files parse without duplicate keys.

No assertion was added or changed by this lint fix. R1-R8 are retained
in the existing PR verification record.

* fix(ecstore): resume fenced Object Lock creation intents

* fix(ecstore): bound creation retry futures on the stack

* fix(ecstore): require native proof for creation retries

* fix(ecstore): bound bucket creation stack usage (#8466)

* fix(ecstore): retain creation fences through rollback

Keep lifecycle, metadata-transaction, and namespace guards around rollback and metadata cleanup. Preserve the original publication error when a guard is lost. Build the timestamp retry fixture through the real interrupted producer so it checks a valid pending creation.

Validated source tree: 4e8e207999.
Two independent source reviews found no blocking regression.

Validation (LANG=en_US.UTF-8):
- bash -c 'shellcheck --version && actionlint' — exit 0
- cargo fmt --all --check — exit 0
- ./scripts/check_unsafe_code_allowances.sh — exit 0
- ./scripts/check_layer_dependencies.sh — exit 0
- ./scripts/check_architecture_migration_rules.sh — exit 0
- ./scripts/check_logging_guardrails.sh — exit 0
- ./scripts/check_error_other_format_ratchet.sh — exit 0
- ./scripts/check_no_tokio_io_uring.sh — exit 0
- ./scripts/check_extension_schema_boundaries.sh — exit 0
- ./scripts/check_body_cache_whitelist.sh — exit 0
- ./scripts/check_s3s_footprint.sh — exit 0
- ./scripts/check_test_count_ratchet.sh — exit 0
- ./scripts/check_fips_wording.sh — exit 0
- ./scripts/check_embedded_secrets.sh — exit 0
- make script-tests — exit 0
- python3 ./scripts/test_object_generation_protocol_model.py — exit 0
- python3 ./scripts/check_test_wiring.py — exit 0
- ./scripts/check_no_planning_docs.sh — exit 0
- ./scripts/check_uring_lane_lib_only.sh — exit 0
- cargo nextest run --profile ci -p rustfs-ecstore --lib --locked --no-fail-fast — 5894 passed, 0 skipped
- cargo check -p rustfs-ecstore --all-targets --locked — exit 0
- cargo nextest run --profile ci -p rustfs --lib --locked --no-fail-fast -E 'test(=admin::handlers::table_catalog::tests::cold_strong_registration_completes_while_other_bucket_backup_waits) | test(=admin::handlers::table_catalog::tests::commit_backend_borrowed_put_preserves_owner_authorization_and_precondition) | test(=admin::handlers::table_catalog::tests::disable_table_bucket_preserves_objects_and_can_be_reenabled) | test(=admin::handlers::table_catalog::tests::load_table_handler_negotiates_vended_credentials_without_breaking_metadata_only_callers) | test(=admin::handlers::table_catalog::tests::namespace_property_handler_updates_object_backed_catalog_and_maps_errors) | test(=app::object::get::tests::execute_get_object_resumes_from_relocated_pool_without_splicing_body) | test(=table_catalog::store::tests::catalog_writer_loss_prevents_recovery_storage_commit)' — 7 passed
- cargo nextest run --profile ci -p rustfs --lib --locked --no-fail-fast -E 'test(unsupported_remote_disk_)' — 2 passed

Additional fault-injection evidence remains incomplete. Timestamp M1 reached its original assertion; its restore run is being collected. M2–M5 and the three planned rollback faults have not run. An automated risk check interrupted the timestamp proof task; no assertions or validation settings were weakened.

* fix(ecstore): box nested bucket rollback state

Keep the guarded rollback future behind a heap allocation so bucket creation does not store its nested state inline. The rollback body, all three fence references and pending-proof revocation order are unchanged.

The same three original Linux tests overflowed on both e611 CI attempts. Local format, ecstore test compilation and all 19 quick-check steps pass; the original Linux feature lanes still need to validate this candidate.

* test(ecstore): isolate rollback guard loss fixtures

Run the three rollback metadata-cleanup cases in separate test processes so reused /proc/self/fd aliases from earlier cases cannot invalidate their creation witnesses. Keep the original fixture, guards, assertions and io_uring test selection, and verify that each child runs the intended case successfully.

Update the generated test-count baseline for the added macro arm.

Validation: cargo check -p rustfs-ecstore --tests; cargo fmt --all --check; all 19 quick-check commands, including make script-tests and the two Python checks. Linux io_uring acceptance and fault controls remain pending.

* refactor(replication): move the replication config onto gateway types (#8443)

* wip: T1.4 replication family

* refactor(replication): move the replication config onto gateway types

The replication configuration family leaves s3s: rustfs-replication no longer depends on s3s, and ECStore bucket metadata, the scanner and the rustfs app/admin layers hold rustfs_gateway_types::persistence::PersistedReplicationConfiguration. The persisted XML goes through the gateway persistence codec behind a new BucketConfigXml trait (the 39 byte goldens pass with unchanged fixtures); the legacy s3s edge converts once in storage/ecfs.rs through storage/s3_compat/replication.rs; the scanner data usage cache keeps its MessagePack bytes through a serde mirror pinned by bytes recorded from s3s 5761ddfe. rustfs-gateway-types 0.32.0 is pinned once (rev 772d3385). Refs rustfs/backlog#2745.

* test: use S3 serializer for list response tests

* fix(ci): restore integration test compilation

Gate the diagnostic test import with its feature, initialize the conditional
MetadataWriteStore fields explicitly, and decode the scanner replication fixture
through its existing cache boundary. Preserve the original fixture and assertions.

Validation was run on tree 544acb80c0 at 6003d4b8.
LANG=en_US.UTF-8; no source changes follow validation.

Quick checks: 19/19 passed.
- shellcheck --version && actionlint: exit 0
- cargo fmt --all --check: exit 0
- ./scripts/check_unsafe_code_allowances.sh: exit 0
- ./scripts/check_layer_dependencies.sh: exit 0
- ./scripts/check_architecture_migration_rules.sh: exit 0
- ./scripts/check_logging_guardrails.sh: exit 0
- ./scripts/check_error_other_format_ratchet.sh: exit 0
- ./scripts/check_no_tokio_io_uring.sh: exit 0
- ./scripts/check_extension_schema_boundaries.sh: exit 0
- ./scripts/check_body_cache_whitelist.sh: exit 0
- ./scripts/check_s3s_footprint.sh: exit 0
- ./scripts/check_test_count_ratchet.sh: exit 0
- ./scripts/check_fips_wording.sh: exit 0
- ./scripts/check_embedded_secrets.sh: exit 0
- make script-tests: exit 0
- python3 ./scripts/test_object_generation_protocol_model.py: exit 0
- python3 ./scripts/check_test_wiring.py: exit 0
- ./scripts/check_no_planning_docs.sh: exit 0
- ./scripts/check_uring_lane_lib_only.sh: exit 0

Darwin validation:
- cargo clippy --locked -p rustfs-ecstore -p rustfs-scanner --all-targets -- -D warnings: exit 0
- cargo nextest run --locked -p rustfs-ecstore -p rustfs-scanner --lib -E 'test(config::com::tests::save_config_preserves_retryable_system_volume_errors) | test(scanner_io::tests::replication_maintenance_refreshes_compacted_stats_across_cycles_and_retries)' --test-threads 1 --no-fail-fast: exit 0, 2/2 passed
- cargo clippy --locked -p rustfs-ecstore -p rustfs-scanner --all-targets --features rustfs-ecstore/diagnostic-read-version-caller,rustfs-scanner/diagnostic-read-version-caller -- -D warnings: exit 101
- cargo nextest run --locked -p rustfs-ecstore -p rustfs-scanner --lib --features rustfs-ecstore/diagnostic-read-version-caller,rustfs-scanner/diagnostic-read-version-caller -E 'test(config::com::tests::save_config_preserves_retryable_system_volume_errors) | test(scanner_io::tests::replication_maintenance_refreshes_compacted_stats_across_cycles_and_retries) | test(cluster::rpc::runtime_sources::tests::) | test(config::com::tests::diagnostic_config_write_retains_origin_and_api_for_results_and_cancellation)' --test-threads 1 --no-fail-fast: exit 0, 14/14 passed

Diagnostic strict Clippy remains red on four unchanged production lints:
runtime_sources.rs if_same_then_else and three redundant_locals reports in
set_disk/core/metadata_read.rs and set_disk/read.rs. Their source ranges and
feature wiring match the base; the base was not compiled separately.

Linux aarch64 validation (Rust/Clippy 1.99.0, UID 501, jobs=1):
- cargo clippy --all-targets -- -D warnings: exit 0; Finished dev profile in 68m 12s
- cargo clippy -p rustfs -p rustfs-ecstore --all-targets --features rio-v2 -- -D warnings: exit 0; Finished dev profile in 49m 45s
Source HEAD/tree and clean status matched before and after both commands.
No compiler or stack overrides were used. protoc 35.1 was installed from its
verified ARM64 release; flatc was absent and not invoked by the workspace build scripts.
The local environment differs from hosted CI.

* fix(ecstore): recover delete batches with missing versions

Treat an explicit non-null version as a no-op when it is absent from both recovery sources. Keep rejecting missing null versions, delete-marker mismatches and conflicting local generations.

Add returning-disk and partial-receipt regressions, extend journal boundary checks, and regenerate the ecstore test-count baseline. Production faults exercise the 24 added assertion sites and 7 semantic checks: 33 faults failed at the expected sites and passed after full restoration.

Validation (LANG=en_US.UTF-8, one build job):

cargo nextest run --locked -p rustfs-ecstore --lib -E 'test(delete_recovery) | test(quorum_delete_cleanup_coordinator_binds_single_and_batch) | test(crash_delete_retry_batch_reclaims_inline_and_reserved_external)' --test-threads 1 --no-fail-fast
Summary [  34.580s] 69 tests run: 69 passed, 6058 skipped; exit 0

cargo clippy --locked -p rustfs-ecstore --all-targets -- -D warnings
Finished `dev` profile [unoptimized + debuginfo] target(s) in 5m 37s; exit 0

All 19 final Quick Checks completed with exit 0:

- shellcheck --version && actionlint
- cargo fmt --all --check
- ./scripts/check_unsafe_code_allowances.sh
- ./scripts/check_layer_dependencies.sh
- ./scripts/check_architecture_migration_rules.sh
- ./scripts/check_logging_guardrails.sh
- ./scripts/check_error_other_format_ratchet.sh
- ./scripts/check_no_tokio_io_uring.sh
- ./scripts/check_extension_schema_boundaries.sh
- ./scripts/check_body_cache_whitelist.sh
- ./scripts/check_s3s_footprint.sh
- ./scripts/check_test_count_ratchet.sh
- ./scripts/check_fips_wording.sh
- ./scripts/check_embedded_secrets.sh
- make script-tests
- python3 ./scripts/test_object_generation_protocol_model.py
- python3 ./scripts/check_test_wiring.py
- ./scripts/check_no_planning_docs.sh
- ./scripts/check_uring_lane_lib_only.sh

test-count ratchet OK: 52 crates, 22009 tests, 165 ignored match .config/test-count-baseline.txt
PASS: cold stampede script self-test
model: states=51023, transitions=167520, terminal=732, chosen_candidates=2
multi-slot round schedules: 640
OK: e2e modules, runner selection, fuzz matrices, profiles, scheduled alerts, and Quick Checks are wired

* fix(ecstore): retry tier commit intent lock contention

Retry transient namespace lock timeouts within the existing three-attempt
CAS budget. Reload and verify the original mutation identity after
contention, and preserve the final typed error when attempts are exhausted.
Cover save/read contention, exhausted retries, replacement identities and
CAS reconciliation without changing the existing timeout or lock policy.

Validation:
- cargo nextest run --locked -p rustfs-ecstore --features test-util --lib -E 'test(tier_mutation)' --test-threads 1 --no-fail-fast: 52 passed.
- cargo nextest run --locked -p rustfs-ecstore --features test-util --lib -E 'test(intent_advance_) | test(=services::tier::tier::tests::caller_cancellation_after_durable_prepare_does_not_hide_the_mutation)' --test-threads 1 --no-fail-fast: 5 passed.
- cargo clippy --locked -p rustfs-ecstore --all-targets -- -D warnings: passed.
- cargo clippy --locked -p rustfs-ecstore --all-targets --features test-util -- -D warnings: passed.
- 27 mutation pairs failed at the intended assertions and passed after restoring the production code.
- bash scripts/check_test_count_ratchet.sh --update, then bash scripts/check_test_count_ratchet.sh: 52 crates, 22013 tests, 165 ignored; ecstore increases from 6581 to 6585.

* refactor(lifecycle): replace s3s domain types with Gateway DTOs

Keep lifecycle rules, restore requests, Object Lock, and scanner cached tags in domain types. Convert at the existing HTTP boundary while preserving legacy XML, scanner cache bytes, date ordering, and tombstones.

Linux test validation before the final import-path-only adjustment:

cargo nextest run -p rustfs-ecstore bucket_config_goldens
39 passed; 6211 skipped by selector.

cargo nextest run --lib -p rustfs -E 'test(/test_zero_rule_lifecycle_tombstone_recognition/) | test(/test_local_lifecycle_staleness_axis_selection/) | test(/test_lifecycle_staleness_preserves_legacy_submillisecond_axis/) | test(/test_merge_incoming_lifecycle/) | test(/test_lifecycle_expiry_statement_matrix/) | test(/test_lifecycle_expiry_subset_xml/) | test(/put_bucket_lifecycle_validation_errors_keep_their_s3_code/) | test(/execute_put_bucket_lifecycle_configuration_rejects_missing_configuration/) | test(/execute_restore_object_rejects_missing_restore_request/) | test(/execute_restore_object_rejects_select_type/) | test(/execute_restore_object_maps_failures_to_typed_s3_errors/) | test(/execute_restore_object_returns_internal_error_when_store_uninitialized/) | test(/restore_request_conversion_preserves_legacy_output_values/) | test(/restore_request_conversion_preserves_presence_and_raw_enums/) | test(/restore_request_conversion_distinguishes_absent_and_empty_containers/)'
25 passed; 5318 skipped by selector.

cargo nextest run --profile ci -p rustfs-ecstore -p rustfs-lifecycle -p rustfs-replication -p rustfs-scanner
7872 passed; 21 skipped under the existing CI profile.

Previously verified mutation controls: 39 production faults, 41 RED/GREEN pairs.

cargo clippy --locked -p rustfs -p rustfs-ecstore -p rustfs-lifecycle -p rustfs-scanner --all-targets -- -D warnings
Passed in 40m 45s before the import-path adjustment, then in 1m 59s on
the final source with warnings denied. The final adjustment imports the
existing helper through the object domain facade; fmt and the architecture
check also passed.

Part of rustfs/backlog#2745. The remaining ecstore configuration families follow separately.

* refactor(admin): move error types to rustfs-s3-types (#8440)

* refactor(protocols): move error types to rustfs-s3-types by codemod

T1.2 batch 3 (rustfs/backlog#2743): s3select-api, s3select-query and
the error items of protocols move off s3s by scripts/codemods, run with
--hold crates/protocols/src/sftp/errors.rs. That file stays on s3s: it
downcasts the backend error, whose runtime type is still s3s::S3Error.

- rustfs/src/protocols/client.rs: the two webdav session methods now
  return rustfs_s3_types::S3Result and convert the s3s error with From.
- codemod: a child module that calls s3_error! through `use super::*`
  gets its own `use rustfs_s3_types::s3_error;` once its parent imports
  that macro by name; otherwise it is listed as GLOB.
- s3-types: a table test pins code, status and the s3s error the legacy
  edge renders for every variant a moved call site names.
- s3_error_lines 1585 -> 1584 (a lifecycle comment reworded).

* refactor(admin): move error types to rustfs-s3-types (WIP, no build)

T1.2 batch 4 (rustfs/backlog#2743), work in progress: rustfs check
still fails with 16 errors (listed in the PR body).

- codemod: --facade PATH treats a crate-local re-export of s3s's error
  items like s3s for the four names (9 new self-tests, 9 mutants red).
- codemod output for rustfs/src/admin and rustfs/src/site_replication
  (--facade crate::storage_api::site_replication::s3); 6 glob children
  get an explicit rustfs_s3_types::s3_error import.
- admin/router.rs: the Operation trait returns rustfs_s3_types::S3Result.
- error.rs: From<ApiError> builds the rustfs-s3-types error; the s3s
  conversion is derived from it.

* fix(admin): convert error types at the s3s boundary

Batch 4 of T1.2 left the admin and site_replication modules returning
rustfs_s3_types::S3Error while their callers and callees on the storage
and auth side still speak s3s::S3Error. Convert at each edge with the
existing From impls in rustfs-s3-types (compat_s3s.rs) instead of
changing any code, message or status:

- admin/auth.rs, admin/handlers/object_zip_download.rs,
  admin/handlers/table_catalog/mod.rs: map the s3s result returned by
  check_key_valid / authorize_request / authorize_internal_object_request
  into the admin error type with .map_err(Into::into).
- storage/ecfs.rs: the s3s-facing side maps the admin site_replication
  gate results the other way.
- admin/handlers/pools.rs: ApiError.code stays s3s::S3ErrorCode, so the
  rustfs code is converted with .into() when the struct is built, in
  production and in the two tests that build or compare it.
- admin/handlers/heal.rs, admin/runtime_sources.rs: the router returns
  s3s errors (S3Route impl); the expected rustfs code is converted with
  .into() before the comparison.
- admin/handlers/table_catalog/mod.rs: iceberg_rest_error takes
  &'static str so the Cow<'static, str> custom code borrows it; every
  caller already passes an ICEBERG_ERROR_* constant or a &'static str.
- admin/handlers/quota.rs: drop a now same-type .into() after s3_error!.
- storage_api.rs, site_replication/mod.rs: drop re-exports and an import
  that no longer have users (clippy -D warnings).

cargo check -p rustfs --all-targets and cargo clippy -p rustfs
--all-targets -- -D warnings are clean; the s3s footprint baseline is
unchanged at files=172, s3_error_lines=303, ecstore_files=28.

Refs rustfs/backlog#2743, rustfs/backlog#2734.

* refactor(ecstore): move bucket configs onto gateway persisted types (#8709)

* refactor(ecstore): move bucket configs onto gateway persisted types

Finish the ECStore batch of T1.4 (rustfs/backlog#2745). The remaining
eleven bucket configuration families (Website, CORS, Tagging, Versioning,
ObjectLock, Encryption, Notification, Logging, Accelerate, RequestPayment,
PublicAccessBlock) now read and write their stored XML through the
rustfs-gateway-types persistence and cors_tagging codecs instead of the
s3s codec, so the legacy s3s bucket-config codec and its
s3gate-metadata-xml compat marker are removed.

- ecstore: BucketConfigXml is implemented by the gateway codec for every
  family; metadata, metadata_sys, versioning, object lock, replication and
  lifecycle consumers take the persisted types. Region moves to
  rustfs_s3_types::Region, and the ETag conversion to s3s moves to the
  rustfs edge with its tests. Only warm_backend.rs and warm_backend_s3.rs
  still reference s3s, held for T1.5 (rustfs/backlog#2746).
- scanner: object lock cache, item actions and data usage use the
  persisted types.
- rustfs: the legacy S3 edge converts each family between s3s DTOs and the
  persisted types in storage/storage_api.rs by writing one codec's XML and
  reading it with the other; Tagging converts field by field.
- Bump the rustfs-gateway-types pin to 21283e87.

The 39 persisted-byte goldens are unchanged and pass, so a document the
old codec wrote reads back and re-encodes to the same bytes. s3s footprint:
files 181 -> 146, ecstore_files 28 -> 2, s3_error_lines unchanged.

* test(app): prepare the PutBucketEncryption config from the request

The uninitialized-store test passed an empty persisted configuration, which the use case now validates before it reaches the store check, so it answered MalformedXML instead of InternalError. Convert the request's configuration the way the S3 edge does.

* style(app): format the PutBucketEncryption test

* refactor(s3-client): move remote error codes and DTO enums off s3s (#8783)

Part of T1.5 (rustfs/backlog#2746), batch 1.

* refactor(s3select): run queries on gateway DTO request shapes (#8794)

* refactor(s3select): run queries on an engine request built from gateway DTO shapes

Part of T1.5 (rustfs/backlog#2746), batch 2.

* fix(s3select): keep the select request conversion in the app layer

The layer guard forbids app -> storage; the conversion moves from storage/s3_compat/select.rs to app/select_request.rs unchanged.

* fix(heal): retry transient metadata conflicts (#8812)

* fix(heal): retry transient metadata conflicts

* chore: update heal test count baseline

* test: reserve nextest capacity for progress probes (#8818)

* test(e2e): use a non-default region for the differential negative control

Since main's region-aware CreateBucket and GetBucketLocation (#8739, #8743), us-east-1 answers an empty LocationConstraint and 200 for an owned re-create, so the control no longer reported its own region. Also refresh the e2e-full linux selection from the CI listing (682 tests).

* refactor(protocols): move StorageBackend to gateway types (#8825)

* refactor(protocols): move storage backend to gateway types

* test: update e2e test count baseline

* fix: preserve copy source keys and unknown lengths

* test: update protocols test count baseline

* fix(protocols): preserve typed SFTP backend errors

* test(protocols): use gateway list input in policy tests

* fix(protocols): clear clippy failures after T1.5 merge (#8831)

fix(protocols): clear clippy failures after merge

* fix(ecstore): finalize restore metadata after local copy-back (#8836)

* fix(ecstore): finalize restore metadata after local copy-back

* test(ecstore): ratchet added restore finalization cases

* fix(ecstore): keep restore copyback compatible with older peers

* fix(ecstore): gate restore copyback on fleet compatibility

* fix(ecstore): retain transition identity during restore copyback

* fix(ecstore): route committed restore copyback locally

* fix restore copyback cleanup

* fix(ecstore): honor restore probe test fallback

* refactor(storage): migrate option errors to domain types (#8839)

* refactor(s3): migrate private multipart validation errors (#8844)

* refactor(auth): derive policy conditions from AuthzFacts (#8442)

* refactor(auth): extract authorization facts

* test(e2e): update smoke selection baseline

* test(e2e): update full selection baseline

* refactor(admin): migrate health and prefix usage to owned requests (#8887)

* refactor(admin): add owned health router boundary

* refactor(admin): migrate prefix usage to owned request

* refactor(admin): complete the owned admin request boundary (#8949)

* refactor(admin): complete the owned admin request boundary

Every type implementing the owned admin `Operation` now serves the legacy
router through one blanket impl that converts the s3s request and response
at that single edge, so handlers register with `AdminOperation(&Handler {})`
and need no adapter statics; `LegacyAdminAdapter` is removed.

`AdminRequest` gains the accessors handlers read today (`remote_addr`,
`client_info`, `request_context`, `req_info`, `req_info_mut`,
`install_req_info`, `envelope_mut`, `take_body`), the request helpers gain
envelope forms (`authenticate_envelope_credentials`,
`object_store_from_envelope`, the table-catalog `*_from_envelope` helpers,
`KmsAdminAudit::from_envelope`, `AccountAuditContext::from_envelope`,
`CallerIdentity::resolve_envelope`), `read_compatible_admin_body` accepts
either body type, and the owned `Body` gains `store_all_limited` and
`Default`. `admin::operation::test_support::TestRequest` builds owned
requests for handler tests. Unused accessors carry a per-method dead-code
expectation that each migrating handler batch removes.

No handler behavior changes; the remaining migrations follow in batches.

* chore(admin): add the T1.12 admin handler codemod

The script rewrites an admin handler file from the s3s request and response
types to the owned boundary (imports, signatures, request field and helper
calls, body access, response construction, and test request literals) and
leaves a TODO comment at every site it cannot rewrite mechanically. It is
idempotent and is removed once T1.12 completes.

* refactor(admin): migrate replication and KMS handlers to owned requests (#8951)

Move the T1.12 batch 2 admin handlers (site_replication, kms_keys, event,
replication, diagnostics, on_demand_migration, audit, account, kms_rekey,
kms_key_lifecycle, object_data_cache, tls_debug, integrity) from the s3s
request/response types to the owned AdminRequest/AdminResponse boundary.
Handler logic, authorization order, error codes and response bytes are
unchanged; the request/response plumbing now goes through the envelope
accessors and helpers introduced by the owned boundary.

The shared response helpers (utils::json_response, utils::empty_response,
handlers::admin_json_response, target_descriptor::build_json_response,
profile::authorize_profile_request) and the site replication repair
executor still answer with the s3s types, so the migrated files carry
owned-boundary forms of those helpers and convert the repair response at
its call edge. The streaming console-log body and the two devnull drains
move from the s3s byte stream to the owned Body stream/frame API.

Drop the dead_code expectations for the owned accessors and envelope
helpers this batch starts using, and lower the s3s footprint baseline to
the measured 99 files.

Part of rustfs/backlog#2788 (rustfs/backlog#2734 T1.12).

* refactor(admin): add owned forms of the shared admin response helpers

Handler batches were copying `json_response`, `empty_response`,
`admin_json_response`, and `build_json_response` into each migrated file
because the shared helpers still return the s3s response. Provide owned
forms next to each legacy helper (`owned_json_response`,
`owned_empty_response`, `owned_admin_json_response`,
`owned_build_json_response`, `authorize_profile_envelope`) with parity
tests against the legacy shape, derive `Debug` for `AdminResponse` so
tests can use `expect_err`, and fix three codemod rules: strip owned
string conversions in test credentials, annotate non-literal URI parses,
and leave a rebound request parameter immutable.

* refactor(admin): use the shared owned response helpers in batch 2 (#8954)

Replace the per-file owned-boundary copies of json_response,
empty_response, admin_json_response, build_json_response and
authorize_profile_request in the batch 2 admin handlers with the shared
owned_* helpers and authorize_profile_envelope that the boundary branch
now provides, and restore expect_err in the handler tests now that
AdminResponse implements Debug; the `.err().expect(..)` stand-ins were
failing the base branch's clippy step (clippy::err_expect under
-D warnings). The site replication repair call edge
keeps its conversion because the repair executor still answers with the
s3s response type.

Drop the dead_code expectations on the shared owned helpers this batch
now uses in non-test code.

Part of rustfs/backlog#2788 (rustfs/backlog#2734 T1.12).

* fix(admin): preserve legacy body limit error responses

* refactor(admin): move policy, ILM and batch 3 handlers to owned requests (#8956)

Batch 3 of T1.12 (rustfs/backlog#2788, parent rustfs/backlog#2734) moves
14 admin handler files to AdminRequest/AdminResponse and the owned
Operation trait: ilm_transition, policies, idp_compat, config_admin,
plugins_instances, profile_admin, group, heal, object_zip_download,
kms_key_metadata, durability, inspect_archive, sts and
gateway_key_inventory. Handler logic, authorization order, error codes
and response bytes are unchanged; test request literals use
test_support::TestRequest.

The handlers use the shared owned helpers (owned_json_response,
owned_admin_json_response, owned_build_json_response,
authorize_profile_envelope). The dead_code expectations on the
AdminRequest accessors and RequestEnvelope::uri_mut they now exercise
are removed. The legacy target_descriptor::build_json_response keeps
only its equivalence test caller and carries the matching expectation
until it retires.

object_zip_download builds the s3s request the S3 gate still reads from
the owned request facts; heal authenticates through
authenticate_envelope_credentials; the trace, inspect-archive and zip
bodies use Body::from_stream. extract_plugin_instance_filters takes the
URI so extensions.rs keeps calling it, and the two idp_compat
delegations stay on the legacy trait until their delegates migrate.

The s3s footprint baseline drops from 99 to 90 files.

* test(admin): anchor the replication diff body contract on the call

The source-contract test looked for `let body = read_compatible_admin_body(`
on one line; rustfmt wrapped that binding after the handler moved to the
owned request boundary, so the marker was no longer found. Anchor on the
call itself so formatting cannot change what the contract checks.

* refactor(admin): migrate table catalog and batch 1 to owned requests (#8955)

* fix(admin): preserve catalog body limit response bytes

* refactor(admin): migrate identity and cluster handlers to owned requests (#8966)

* refactor(admin): migrate identity and cluster handlers to owned requests

Move the T1.12 batch 4 admin handlers (user, service_account, oidc, tier,
system, pools, quota, kms_dynamic, kms_backup, rebalance, module_switch,
bucket_meta, cluster_snapshot, realtime, bucket_recovery, profile) from the
s3s request/response types to the owned AdminRequest/AdminResponse
boundary. All 90 impl Operation blocks now take AdminRequest and return
AdminResponse; handler logic, authorization order, error codes and response
bytes are unchanged. The handlers use the shared owned helpers
(owned_json_response, owned_admin_json_response, authorize_profile_envelope,
the envelope-form runtime sources and audit constructors) instead of local
copies.

profile.rs drops the s3s forms of authorize_profile_request and
profile_not_implemented_response now that profile_admin.rs is on the
envelope gate too; profile_not_implemented_response answers with an
AdminResponse. The realtime metrics stream moves from the s3s ByteStream
adapter to Body::from_stream, and the rebalance idle-404 document still
renders through s3s::S3Error::to_http_response with its body converted at
the edge.

The idp_compat delegators ImportIamV2 and AddServiceAccountLdap move to the
owned trait as well: with their targets on both traits, the method-call
delegation became ambiguous between the owned impl and the router blanket
impl, so they now take the owned request directly.

KmsAdminAudit::from_request loses its last production caller with this batch
and is deleted; its comparison test now asserts the envelope facts directly.

Lower the s3s footprint baseline to the measured 68 files.

Part of rustfs/backlog#2788 (rustfs/backlog#2734 T1.12).

* refactor(admin): remove the legacy admin helpers batch 4 superseded

Batch 4 moved the last production callers of the s3s-typed admin helpers
to their owned forms, which left `admin_json_response`, `json_response`,
`authenticate_request`, `authorize_admin_request`, `object_store_from_req`
and `object_store_from_extensions` unused, along with the `Body`,
`S3Request` and `S3Response` re-exports of the admin `s3` facade. Remove
them. The owned forms keep their documentation; the behavior tests of the
removed `json_response` now exercise `owned_json_response`, and the
compat-prefix test asserts the owned response directly.

Two tests existed only to compare a removed legacy helper with its owned
twin (`owned_json_response_matches_the_legacy_shape`) or to cover the
removed legacy gate (`authorize_admin_request_without_credentials_is_rejected`);
they go with the helpers, and the owned equivalents stay. The rustfs
test-count row drops from 5957 to 5955 and the s3s footprint from 68 to 66
files.

* refactor(admin): drop the legacy target descriptor JSON response helper (#8968)

`target_descriptor::build_json_response` returned the s3s response and
had no production caller left once the event, audit and plugin instance
handlers moved to `owned_build_json_response`; it survived only for the
legacy/owned equivalence test. Remove it, give the owned helper its own
documentation, and make the test assert the owned response directly:
status, `application/json`, the echoed `x-request-id` when present and
only then, and the body bytes. The file no longer imports s3s, so the
footprint baseline drops from 66 to 65 files.

* test(ecstore): pin native history on cancellation under the cancellable queue

main moved the local write path onto the disk blocking queue, which skips an
operation once its waiter is dropped, so a borrowed write cancelled while it is
queued no longer creates its file. The integration-only native publication
test still expected the queued open to run. Assert the new execution model
instead (the open never runs) while keeping its safety claims: the Started
record is taken before queueing and the cancelled publication never regrants
native history.

* chore(gateway): raise test-count rows for tests merged from main

* test(s3select): use the owned JSON input types in merged planner tests

* fix(ecstore): scope the merged direct-writer test hook to test builds

* fix(ecstore): create parents before admitted truncate opens

Align the heal tests with removal-disabled preservation and local-only outcome
fields, and refresh the e2e-full selection for the synced tests.

* chore(gateway): update test-count baseline after main sync

Main #9241 removed the node pool-metadata readiness cache with 8 tests
and added 6, so the rustfs row drops from 6440 to 6438. The ecstore and
s3select rows rise for tests added by #9237, #9239 and #9241.

* feat(gateway-adapter): add IAM-backed CredentialProvider

Add the ring-2 crate rustfs-gateway-adapter with IamCredentials, the
gateway CredentialProvider answered by RustFS IAM (rustfs/backlog#2760,
credentials half; PolicySource is deferred).

- Pin rustfs-gateway at the existing gateway rev with
  default-features = false so the server crate stays out of the build.
- IamKeyLookup is a narrow IAM trait implemented for IamSys so the
  adapter is testable without an in-memory IAM backend.
- Root credentials are injected at construction and resolve before IAM
  is ready.
- Mapping: keystone: prefix -> NotFound (deliberate change from the
  legacy empty secret); IAM not ready or not initialized -> Unavailable;
  unknown key or NoSuch{User,ServiceAccount,TempAccount,Account} ->
  NotFound; other IAM errors -> Backend; invalid or "off" principals
  resolve disabled; STS credentials carry a SessionBinding, and an STS
  credential without an expiry is a Backend fault rather than a
  never-ending session.
- No authorization in lookup; no Debug on secret-holding types; IAM
  error text is dropped after classification.

* feat(scripts): add junit_diff.py for dual-stack JUnit comparison (#9275)

* feat(gateway-adapter): add IAM-backed PolicySource

Add IamPolicySource, the gateway PolicySource answered by RustFS IAM
(rustfs/backlog#2760, policy half). One snapshot per request carries a
RustfsPrincipal: principal kind, account, effective username, parent
user, credential groups, live group memberships, token claims and
is_owner.

- Anonymous callers get an empty snapshot without asking IAM.
- IAM not ready, not initialized or failing is PolicyError::unavailable,
  never an empty snapshot.
- Unknown, disabled and session-kind-mismatched principals are refused
  with constant labels; unreadable tokens are malformed.
- Claims are read from the stored token (root secret for STS, the
  account's own secret for service accounts), and is_owner mirrors the
  legacy has_root_access rule.
- RustfsPrincipal holds no secret or session token and has no Debug.

* ci(gateway): make the test-count ratchet a floor, not exact (#9278)

ci(gateway): make the test-count ratchet a floor instead of an exact baseline

Tests below a row still fail; tests above it, a member without a row and ignores below the row now pass with a note. Ignores above a row still fail. Adds contract tests wired into make script-tests.

* chore(gateway): record ignored microbenchmarks merged from main (#9290)

chore(gateway): record the ignored microbenchmarks merged from main

main added three throughput microbenchmarks that are ignored by design
(#9217): one each in rustfs-checksums, rustfs-rio and rustfs-utils. The
test-count ratchet treats a higher ignored count as a failure until the
row is raised, so raise those three rows to 1 and tighten the test rows
to the merged tree with --update.

* fix(gateway-adapter): share live root and refuse short secrets

Address the security review of the T2.1 adapters (rustfs/backlog#2760).

- Add RootCredentials (validated: root secret and STS signing key at
  least MIN_SECRET_KEY_LEN bytes), the RootSource trait and the settable
  RootCell. IamAdapters::new is now the only constructor of
  IamCredentials and IamPolicySource, so both read one live root source
  per request; no root source answers unavailable.
- A looked-up identity whose secret is shorter than 8 bytes is NotFound
  in the credential provider and DISABLED_PRINCIPAL in the policy
  source, whatever the lookup reports as valid.
- STS claims are read with the STS signing key from the root source
  instead of assuming it equals the root secret.
- Cover the off-status-alone branch, the OIDC owner exclusion, service
  accounts of root, the name live groups are read under, and another
  session's token.

* ci(s3-tests): add reference parity lane against pinned 1.0.5 (#9288)

* fix(gateway-adapter): effective secret length and unsplittable pair

Address the confirmation review of the T2.1 adapters
(rustfs/backlog#2760).

- One rule for every secret: its length after trailing NUL bytes are
  removed must be at least MIN_SECRET_KEY_LEN. HMAC zero-pads its key,
  so trailing NULs add nothing; interior NULs count. Applied to the root
  secret, the STS signing key and every looked-up credential.
- IamAdapters keeps both halves private and hands them to a builder
  together through the consuming install(builder, make_authenticator);
  compile-fail doctests show a mixed pair cannot be assembled.
- Any Fn() -> Option<Arc<RootCredentials>> is a RootSource, so the
  binary reads its live root accessor directly; RootCell is now a
  test-only helper.

* docs(gateway-adapter): state what install guarantees about the provider handle

---------

Co-authored-by: rustfs-cla-assistant[bot] <273121464+rustfs-cla-assistant[bot]@users.noreply.github.com>
2026-11-11 04:29:40 -05:00
Chris An 04bcb57e6a fix(ecstore): pin multipart index markers while their root is published (#9318)
A multipart index marker is recorded before its staging root is published, and a listing prunes markers whose root is gone. The window in between was covered by a five-minute age check, which cannot prove that the publisher's filesystem work has ended: a publisher stalled past it lost its marker, and the upload it then published stayed out of ListMultipartUploads on that drive until restart.

The publisher now holds an in-memory publication pin from before it records the marker until the filesystem work that publishes the root has returned. The pin travels with that work (the blocking closure or the lease owner chain), so it also covers work that outlives a cancelled caller. Pruning reads the pin first and the staging root second, under the marker lease. The age check and the per-write marker touch are removed.
2026-11-11 03:22:46 -05:00
Chris An 31e1e42ad7 refactor: land the s3s decoupling groundwork on main (#9276)
Lands gateway/integration on main: the guard rails and the domain
decoupling for replacing s3s with rustfs/gateway (rustfs/backlog#2734,
Phase 0 and Phase 1). The S3 edge is still s3s and no wire behaviour
change is intended.

- Guard rails: lower-only s3s footprint baseline and edge allow-list,
  per-crate test-count floor, 39 byte-level goldens for the persisted
  bucket-config XML families, a two-server differential harness over
  all 73 S3 operations, a JUnit differ, and a reference-parity lane
  against the sha256-pinned 1.0.5 binary.
- Owned types: rustfs-s3-types (S3Error, S3ErrorCode, Body),
  RequestEnvelope, AuthzFacts.
- Engine crates off s3s: ecstore, lifecycle, replication, scanner,
  protocols, s3select, s3-client, rpc.
- Admin: error types moved; 300 of 314 Operation impls take the owned
  AdminRequest.
- Server: the s3s-compat tower layers grouped under
  server/legacy_compat; RUSTFS_S3_STACK=legacy|gateway selector, where
  gateway refuses to start until the gateway edge exists.

Fixes made on the branch that change ecstore and heal behaviour are
listed in the pull request description. The per-commit history stays
on gateway/integration.
2026-11-11 02:39:52 -05:00
Chris An 0e402b2534 fix(scanner): retain child objects beneath object prefixes (#9277) 2026-11-11 14:36:53 +08:00
Chris An db39cd5634 fix(ecstore): let missing-key deletes complete on UUID-named folders (#9280)
* fix(ecstore): let missing-key deletes complete on UUID-named folders

Deleting a key that does not exist probes the key's directory for
interrupted delete transactions. The probe was bounded at 4096 entries
per drive and charged every entry inside a UUID-named child, so a key
naming a folder of roughly 1,400 or more UUID-named child keys answered
503 SlowDown on every retry.

Inside a UUID-named child only journal files and data-directory markers
can be evidence, so every other name (a child key's own xl.meta, data
directories or children) is now skipped unread at 1/64 like plain
children of the folder. The per-call allowance of the bounded protocol
rises to 262,144 entries per drive; the resumable background step keeps
4096 and the request budget keeps its sizing, so the per-request total
is unchanged. The probe runs on the durable-mutation lane instead of
the one-worker foreground lane, and stops incomplete when its caller
goes away or after a third of the drive timeout (at most 10 s), so a
slow walk answers SlowDown instead of timing out the drive.

* test(ecstore): keep discovery fixtures meaningful under 1/64 charging

Part files inside a data directory are inert to object-scoped discovery
now, so reclaim_orphan_data_dirs_drains_empty_discovery_pages fills its
first step with the referenced data directories of many versions, which
still cost a full entry each. The charging test drives the walk directly
instead of through the wall-clock-limited disk entry point, so a loaded
machine cannot turn it into an incomplete probe.
2026-11-10 23:24:50 -05:00
Chris An 55d931353f ci(s3-tests): add reference parity lane against pinned 1.0.5 (#9288) 2026-11-10 22:50:17 -05:00
overtrue 9d513cffe9 test(e2e): update smoke selection for reference-binary unit tests 2026-11-11 11:25:30 +08:00
overtrue c5afd3738c chore: merge gateway/integration into gw/T3-reference-binary 2026-11-11 10:33:04 +08:00
Chris An fc09148f08 chore(gateway): record ignored microbenchmarks merged from main (#9290)
chore(gateway): record the ignored microbenchmarks merged from main

main added three throughput microbenchmarks that are ignored by design
(#9217): one each in rustfs-checksums, rustfs-rio and rustfs-utils. The
test-count ratchet treats a higher ignored count as a failure until the
row is raised, so raise those three rows to 1 and tighten the test rows
to the merged tree with --update.
2026-11-10 21:30:34 -05:00
rustfs-cla-assistant[bot] c9e1464a2a Merge remote-tracking branch 'origin/main' into gateway/integration 2026-11-11 02:06:08 +00:00
Chris An 0e3028d29e ci(gateway): make the test-count ratchet a floor, not exact (#9278)
ci(gateway): make the test-count ratchet a floor instead of an exact baseline

Tests below a row still fail; tests above it, a member without a row and ignores below the row now pass with a note. Ignores above a row still fail. Adds contract tests wired into make script-tests.
2026-11-10 20:53:06 -05:00
overtrue 6b44869b4f test(e2e): allow a pinned release binary as differential reference side
RUSTFS_E2E_REFERENCE_BINARY makes side A of the dual-stack differential a released rustfs binary, started without RUSTFS_S3_STACK. A missing, non-file or non-executable path fails the run instead of falling back to the same-binary mode. The report records which side is the reference and each side's --version output.
2026-11-11 09:44:12 +08:00
Chris An 365014a385 feat(scripts): add junit_diff.py for dual-stack JUnit comparison (#9275) 2026-11-10 19:55:24 -05:00
Ramakrishna Chilaka 89d01aef4f perf(ecstore): reduce syscalls and directory re-opens in PUT and DELETE (#9264)
Co-authored-by: RustFS <hello@rustfs.com>
Co-authored-by: Hauser Zhi <housemecn@gmail.com>
2026-11-11 00:06:11 +00:00
dependabot[bot] 86c3395d07 chore(deps): bump the dependencies group across 1 directory with 12 updates (#9251)
Bumps the dependencies group with 12 updates in the / directory:

| Package | From | To |
| --- | --- | --- |
| [async-nats](https://github.com/nats-io/nats.rs) | `0.51.0` | `0.52.0` |
| [tar-codec](https://github.com/astral-sh/astral-codec) | `0.0.15` | `0.0.16` |
| [tar-framing](https://github.com/astral-sh/astral-codec) | `0.0.15` | `0.0.16` |
| [flate2](https://github.com/rust-lang/flate2-rs) | `1.1.10` | `1.1.11` |
| [smallvec](https://github.com/servo/rust-smallvec) | `1.16.4` | `1.16.5` |
| [starshard](https://github.com/houseme/starshard) | `2.4.0` | `2.4.1` |
| [zip](https://github.com/zip-rs/zip2) | `9.3.0` | `9.4.1` |
| [dial9](https://github.com/dial9-rs/dial9) | `0.5.7` | `0.5.8` |
| [rustfs-mimalloc](https://github.com/houseme/rustfs-mimalloc) | `0.7.3` | `0.7.4` |
| [hotpath](https://github.com/hotpath-rs/hotpath) | `0.30.3` | `0.31.1` |
| [rustfs-mimalloc-sys](https://github.com/houseme/rustfs-mimalloc) | `0.7.3` | `0.7.4` |
| [toml](https://github.com/toml-rs/toml) | `1.2.0+spec-1.1.0` | `1.2.1+spec-1.1.0` |



Updates `async-nats` from 0.51.0 to 0.52.0
- [Release notes](https://github.com/nats-io/nats.rs/releases)
- [Commits](https://github.com/nats-io/nats.rs/compare/async-nats/v0.51.0...async-nats/v0.52.0)

Updates `tar-codec` from 0.0.15 to 0.0.16
- [Release notes](https://github.com/astral-sh/astral-codec/releases)
- [Commits](https://github.com/astral-sh/astral-codec/compare/v0.0.15...v0.0.16)

Updates `tar-framing` from 0.0.15 to 0.0.16
- [Release notes](https://github.com/astral-sh/astral-codec/releases)
- [Commits](https://github.com/astral-sh/astral-codec/compare/v0.0.15...v0.0.16)

Updates `flate2` from 1.1.10 to 1.1.11
- [Release notes](https://github.com/rust-lang/flate2-rs/releases)
- [Changelog](https://github.com/rust-lang/flate2-rs/blob/main/CHANGELOG.md)
- [Commits](https://github.com/rust-lang/flate2-rs/compare/1.1.10...v1.1.11)

Updates `smallvec` from 1.16.4 to 1.16.5
- [Release notes](https://github.com/servo/rust-smallvec/releases)
- [Commits](https://github.com/servo/rust-smallvec/compare/v1.16.4...v1.16.5)

Updates `starshard` from 2.4.0 to 2.4.1
- [Changelog](https://github.com/houseme/starshard/blob/main/CHANGELOG.md)
- [Commits](https://github.com/houseme/starshard/compare/2.4.0...2.4.1)

Updates `zip` from 9.3.0 to 9.4.1
- [Release notes](https://github.com/zip-rs/zip2/releases)
- [Changelog](https://github.com/zip-rs/zip2/blob/master/CHANGELOG.md)
- [Commits](https://github.com/zip-rs/zip2/compare/v9.3.0...v9.4.1)

Updates `dial9` from 0.5.7 to 0.5.8
- [Release notes](https://github.com/dial9-rs/dial9/releases)
- [Changelog](https://github.com/dial9-rs/dial9/blob/main/CHANGELOG.md)
- [Commits](https://github.com/dial9-rs/dial9/compare/dial9-v0.5.7...dial9-v0.5.8)

Updates `rustfs-mimalloc` from 0.7.3 to 0.7.4
- [Release notes](https://github.com/houseme/rustfs-mimalloc/releases)
- [Changelog](https://github.com/houseme/rustfs-mimalloc/blob/main/CHANGELOG.md)
- [Commits](https://github.com/houseme/rustfs-mimalloc/compare/v0.7.3...v0.7.4)

Updates `hotpath` from 0.30.3 to 0.31.1
- [Release notes](https://github.com/hotpath-rs/hotpath/releases)
- [Changelog](https://github.com/hotpath-rs/hotpath/blob/main/CHANGELOG.md)
- [Commits](https://github.com/hotpath-rs/hotpath/compare/v0.30.3...v0.31.1)

Updates `rustfs-mimalloc-sys` from 0.7.3 to 0.7.4
- [Release notes](https://github.com/houseme/rustfs-mimalloc/releases)
- [Changelog](https://github.com/houseme/rustfs-mimalloc/blob/main/CHANGELOG.md)
- [Commits](https://github.com/houseme/rustfs-mimalloc/compare/v0.7.3...v0.7.4)

Updates `toml` from 1.2.0+spec-1.1.0 to 1.2.1+spec-1.1.0
- [Commits](https://github.com/toml-rs/toml/compare/toml-v1.2.0...toml-v1.2.1)

---
updated-dependencies:
- dependency-name: async-nats
  dependency-version: 0.52.0
  dependency-type: direct:production
  update-type: version-update:semver-minor
  dependency-group: dependencies
- dependency-name: tar-codec
  dependency-version: 0.0.16
  dependency-type: direct:production
  update-type: version-update:semver-patch
  dependency-group: dependencies
- dependency-name: tar-framing
  dependency-version: 0.0.16
  dependency-type: direct:production
  update-type: version-update:semver-patch
  dependency-group: dependencies
- dependency-name: flate2
  dependency-version: 1.1.11
  dependency-type: direct:production
  update-type: version-update:semver-patch
  dependency-group: dependencies
- dependency-name: smallvec
  dependency-version: 1.16.5
  dependency-type: direct:production
  update-type: version-update:semver-patch
  dependency-group: dependencies
- dependency-name: starshard
  dependency-version: 2.4.1
  dependency-type: direct:production
  update-type: version-update:semver-patch
  dependency-group: dependencies
- dependency-name: zip
  dependency-version: 9.4.1
  dependency-type: direct:production
  update-type: version-update:semver-minor
  dependency-group: dependencies
- dependency-name: dial9
  dependency-version: 0.5.8
  dependency-type: direct:production
  update-type: version-update:semver-patch
  dependency-group: dependencies
- dependency-name: rustfs-mimalloc
  dependency-version: 0.7.4
  dependency-type: direct:production
  update-type: version-update:semver-patch
  dependency-group: dependencies
- dependency-name: hotpath
  dependency-version: 0.31.1
  dependency-type: direct:production
  update-type: version-update:semver-minor
  dependency-group: dependencies
- dependency-name: rustfs-mimalloc-sys
  dependency-version: 0.7.4
  dependency-type: direct:production
  update-type: version-update:semver-patch
  dependency-group: dependencies
- dependency-name: toml
  dependency-version: 1.2.1+spec-1.1.0
  dependency-type: direct:production
  update-type: version-update:semver-patch
  dependency-group: dependencies
...

Signed-off-by: dependabot[bot] <support@github.com>
Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
Co-authored-by: Hauser Zhi <housemecn@gmail.com>
2026-11-11 00:04:47 +00:00
cxymds c9743f166e fix(s3): normalize leading-slash keys in bulk deletes (#9258)
* fix(s3): normalize leading-slash keys in bulk deletes

* test(s3): align bulk delete reserved-prefix guard expectations
2026-11-10 17:56:00 -05:00
Boyi (Elliot) Li 50707a610c test(utils,rio,checksums): add checksum SIMD equivalence and bench UTs (#9217)
Add unaligned-boundary SIMD vs scalar equivalence unit tests and on-demand (#[ignore]) utility-level throughput microbenchmarks across rustfs-utils, rustfs-rio, and rustfs-checksums:
- crates/utils/src/hash.rs: verify HighwayHasher SIMD (AVX2/SSE4.1/NEON) against highway::PortableHash scalar reference and contiguous hash_encode vs odd-chunked hash_encode_slices across unaligned tail boundaries; provide on-demand throughput microbenchmark.
- crates/rio/src/checksum.rs: verify SIMD-accelerated CRC32, CRC32C, and CRC64_NVME against bitwise scalar reference implementations across vector tail boundaries and measure utility-level throughput across all 10 BASE_CHECKSUM_TYPES.
- crates/checksums/src/lib.rs: verify streaming chunked update + finalize + headers() against one-shot digests across all 9 ChecksumAlgorithm variants for unaligned chunk and payload sizes, plus on-demand throughput microbenchmark.
2026-11-10 17:54:11 -05:00
Patrick Wicki ba3e0e1546 fix(admin): keep measured bucket usage when the overlay is unconverged (#9243)
The single-process usage overlay clears usage_snapshot_complete when it
holds counts it cannot reconcile with the persisted snapshot, for
example writes into a bucket that has no verified baseline. The admin
datausageinfo path then judged the overlaid document as structurally
incomplete and replaced it with an empty one, hiding every bucket the
scanner had measured. Because each write restarts that bucket's scanner
fence, a continuously written bucket kept the whole response empty
indefinitely.

Decide whether the persisted snapshot is usable before the overlay is
applied, and report overlay-induced incompleteness as a non-converged
snapshot instead, as is already done for buckets the scan never reached
(rustfs#5806).
2026-11-10 17:52:06 -05:00
Chris An 95e128336d ci: narrow checks for stable installation metadata (#9269)
* ci: narrow checks for stable installation metadata

* fix: reject release metadata marker collisions
2026-11-10 16:10:04 -05:00
Chris An 8b7bbdb17d chore(release): align installation references for 1.0.5 (#9268) 2026-11-10 14:48:02 -05:00
Chris An 2780bba67b ci: stage release builds and promote verified artifacts (#9265)
* ci: stage release builds and promote verified artifacts

* fix: preserve release package and channel retry contracts

* fix: align installation chart versions with sealed candidates
2026-11-10 13:15:05 -05:00
overtrue 443e8afee1 chore(gateway): merge main into integration
# Conflicts:
#	crates/ecstore/src/disk/local/delete_recovery.rs
#	crates/ecstore/src/store/bucket.rs
2026-11-10 21:40:26 +08:00
hector 3a23122104 ci: run macOS release builds on macos-latest-xlarge (#9260)
Use the larger GitHub-hosted macOS runner for the macos-aarch64
release target. The target id, triple, and other platforms stay the same.

Co-authored-by: hector <majinghe@users.noreply.github.com>
2026-11-10 18:55:19 +08:00
Ramakrishna Chilaka 0b1ebeffec perf(ecstore): reduce syscalls in multipart copy, commit, and delete (#9247) 2026-11-10 12:27:10 +08:00
Chris An 0f9258594c fix: preserve scanner backlog authority after pool replacement (#9252)
* fix(ecstore): allow bucket creation after failed decommission

* fix(scanner): preserve backlog membership across pool reindexing

* test(scanner): cover partial publication after pool reindexing
2026-11-09 20:46:20 -05:00
overtrue 782e3e5209 chore(gateway): merge main into integration 2026-11-10 08:28:05 +08:00
Chris An 33fc735ee2 chore(release): prepare 1.0.5 (#9249) 2026-11-09 17:19:35 -05:00
Chris An 509869cfc5 fix(tier): recover orphaned prepared mutations (#9246) 2026-11-09 22:18:14 +08:00
overtrue cfffb1700e chore(gateway): update test-count baseline after main sync
Main #9241 removed the node pool-metadata readiness cache with 8 tests
and added 6, so the rustfs row drops from 6440 to 6438. The ecstore and
s3select rows rise for tests added by #9237, #9239 and #9241.
2026-11-09 10:21:53 +08:00
rustfs-cla-assistant[bot] cd62f3ba4f Merge remote-tracking branch 'origin/main' into gateway/integration 2026-11-09 02:07:05 +00:00
142 changed files with 13797 additions and 4397 deletions
+21 -26
View File
@@ -1,43 +1,38 @@
---
name: rustfs-release-publish
description: Publish a RustFS version through Console, release-branch CI, preview acceptance, human confirmation, and final publication. Use only for an explicit release request.
description: Publish a RustFS version through frozen Linux candidate acceptance, desktop completion, manifest approval, byte-preserving publication and post-release follow-up. Use only for an explicit release request.
---
# RustFS Release
Require the exact final SemVer target before version edits or publication. If unspecified, inspect current releases/tags and ask with concrete candidates; do not guess the channel or bump. A named bump type resolves to an exact version that still needs confirmation. Report the confirmed target verbatim; resolve any later mismatch before proceeding. Independent read-only preflight may continue.
Require the exact final SemVer target before version edits or publication. If unspecified, inspect releases/tags and ask with concrete candidates; do not guess the channel or bump. Reuse the user's confirmed target and authorized delivery scope. Independent read-only preflight may continue.
## Invariants
- Cargo stores `<target>`, never `-preview.N`. Before publication, installation defaults retain the previous available deliverable. Final and preview binaries embed their own tag names, so version/asset names differ despite identical source.
- Use annotated tags without `v`. Preview names are exactly `<target>-preview.<digits>`; choose the next unused iteration after fetching tags. Preview Releases are prereleases, never Latest; publish only versioned assets/exact Docker tags, never latest assets, R2, Helm, moving Docker channels, or installation/banner updates.
- `release/<target>` owns the candidate. Retain its history, backport only reviewed fixes, and never merge main wholesale or force-push it. Pin `PREVIEW_HASH` to its exact commit with complete passing CI. Preview and final tags both point there; never tag a later main/branch HEAD or insert another bump.
- Phase failures block dependents. Keep preview Releases until final publication and notes generation finish; CI removes Releases while retaining tags. Never move an existing preview or final tag.
- Cargo and every candidate executable contain the final product version. `<target>-preview.<run_number>` is an immutable candidate identifier, not a public tag or displayed binary version.
- Freeze the exact source, reviewed workflow recipe, toolchain, Console, CLI, dependency lock, previous stable version and validation plan. Newer unrelated main or Console releases do not change those inputs.
- Retain `release/<target>` history when a release branch is used; backport only reviewed fixes. Pin `CANDIDATE_SHA` to its exact CI-validated commit. Never force-push a release branch or move a public tag.
- Build Linux first. Complete field acceptance before macOS/Windows and packaging. Full certification and final acceptance are required before publication; a partial Linux PASS grants no release authority.
- Promote the accepted bytes. Final publication never recompiles binaries or rebuilds packages/images. Infrastructure retries reuse successful artifacts; changed source or inputs require a new candidate and affected acceptance.
- Phase failures block dependents. No latest/default-channel update before versioned delivery. Use the [pipeline runbook](../../../docs/operations/release-pipeline.md) for executable entry points, locks and recovery.
## Read by Phase
Read only the current phase's procedure; retain completed evidence across turns. Follow root verification, Git/PR, and authorization rules.
## Read by phase
| Phase | Procedure |
|---|---|
| 0 | Clean tree, fetch main/tags, verify GitHub access, then complete the mandatory [Console release gate](references/console-gate.md). A green Console build alone is insufficient. |
| 1–2 | [Prepare/reuse the release candidate and run exact-SHA branch CI](references/release-candidate.md). Source version changes use `rustfs-release-version-bump` with authorized delivery scope. |
| 3 | [Verify the preview build, assets, channels, and notes](references/preview-release.md). |
| 4–5 | [Run the downloaded binary, Console CRUD, and latest-rc matrix](references/preview-acceptance.md). Every check must pass. |
| Confirmation | Complete the gate below and end the turn. |
| 6 | After fresh confirmation, [publish and verify the final tag](references/final-release.md) at `PREVIEW_HASH`. |
| 7 | [Reconcile Security Advisories, milestones, installation references, and the website; reconcile main and remove the completed release branch](references/post-release-updates.md) only after complete publication. |
| 0 | Clean tree, fetch main/tags, verify access, then complete the mandatory [Console gate](references/console-gate.md). |
| 1–2 | [Prepare the exact source and run branch CI](references/release-candidate.md). Version edits use `rustfs-release-version-bump`. |
| 3 | [Build and verify the Linux candidate](references/preview-release.md). |
| 4–5 | [Run artifact, field, Console and rc acceptance](references/preview-acceptance.md), then complete desktop/package/image/chart validation. |
| Confirmation | Report the complete manifest and obtain the confirmation below. |
| 6 | [Publish the accepted manifest](references/final-release.md). |
| 7 | [Reconcile advisories, milestones, installation references, website and release branch](references/post-release-updates.md). |
## Human Confirmation
## Human confirmation
After Phases 3–5 pass, recheck that the release branch still equals `PREVIEW_HASH` and that the current Console asset tag/digest match the preview's recorded build inputs. Report the target, branch, preview tag/hash, branch CI run, preview Release URL, accepted Console asset identity/result, and rc matrix. Explicitly ask whether to publish the final tag and end the turn without creating/pushing it.
After all acceptance passes, report the target, source SHA, source CI run, Linux and completion run URLs, final manifest SHA-256, accepted Console/CLI identities, compatibility/performance results and backlog report. Explicitly ask whether to publish this exact manifest and end the turn without dispatching publication.
Only a new affirmative user reply to that exact report authorizes Phase 6. The original release request, earlier confirmation, silence, or automated follow-up does not count. Confirmation binds the target, preview tag, hash, and accepted build inputs; a changed value or repeated/failed acceptance cycle invalidates it. Revalidate and obtain fresh confirmation before continuing.
Only a new affirmative reply to that exact report authorizes Phase 6. An earlier general release request, silence or automatic continuation does not count. Reuse confirmation for retries of that same manifest; do not ask again merely because a destination failed. A changed source, input, manifest or acceptance result invalidates it and requires renewed acceptance/confirmation.
## Recovery
## Recovery and reporting
- Before the final tag exists, land fixes on main, backport to the candidate, verify target versions, and restart branch CI/acceptance. Use a new preview iteration once a preview was tagged; main may advance independently.
- If only an external build input changed, record it, create a new preview iteration even when the source SHA is unchanged, and repeat the affected build and artifact acceptance before fresh confirmation. Reuse source checks whose inputs remain valid.
- Once the final tag exists, retry publication jobs for that exact tag. A source fix requires a newly confirmed target, not retagging or a newer preview. Phase 7 retries recheck artifact availability and superseding releases.
- If abandoned after the bump merged, report the unpublished version on main and record whether to revert or let the next release overwrite it. Deleting unpublished candidate branches or preview tags requires an explicit decision. Completed release branches follow Phase 7 cleanup; keep release tags.
Report completed phase results and incomplete work with evidence: Console previous/latest tags, release need, hash and run/Release URLs; target, branch CI/SHA, previews and hash, confirmation state; preview/final assets, latest-channel state, Console/rc results and cleanup; advisory reconciliation results and pending disclosure decisions; both milestone states, installation PR checks/merge state, and website text/link/PR/deployment state. Keep artifact publication separate from follow-up delivery; do not call an open PR a live update. Report deviations and their authorization.
Resume failed infrastructure jobs without rebuilding successful units. Fix product defects on main, backport when needed, and start a new candidate. After a final tag exists, retry the same bytes or use a new patch version; never move the tag. Do not conflate implemented, CI-passed, field-accepted, approved, published and post-release-complete states. Report incomplete work and deviations with evidence; a tag or green build alone is not a completed release.
@@ -4,7 +4,7 @@ Read during Phase 0, before changing RustFS version files or tags. Follow the pa
### Console release gate
Complete this gate before changing any RustFS version file or creating any RustFS tag. RustFS `build.yml` downloads the asset returned by `repos/rustfs/console/releases/latest`, so a successful Console build alone is insufficient.
Complete this gate before changing any RustFS version file or creating any RustFS tag. Candidate creation resolves `repos/rustfs/console/releases/latest` once and freezes the exact asset for every build, so a successful Console build alone is insufficient. After freezing, later Console releases do not silently change or invalidate the accepted inputs.
1. Read the latest published Console tag and compare it with Console `main`:
@@ -1,22 +1,13 @@
# Final Release Publication
# Final release publication
Read only after fresh user confirmation of the preview acceptance report. The [parent confirmation gate](../SKILL.md#human-confirmation) applies.
Read only after fresh confirmation of the complete candidate report. The [parent confirmation gate](../SKILL.md#human-confirmation) applies.
## Phase 6 — Publish the final tag on the validated commit
Dispatch `release-publish.yml` on main with the successful completion run ID and approved `release-manifest.json` SHA-256. Do not create the tag manually. The workflow revalidates source, bytes and trusted certification, then creates the annotated final tag at `CANDIDATE_SHA`. A recipe change fails closed and requires a new candidate; unrelated main changes do not require a rebuild.
No second source version bump. Before creating the final tag, recheck the remote `release/<target>` HEAD equals `PREVIEW_HASH` and the current Console release tag/asset digest still match the accepted preview inputs. If either changed after confirmation, the approval no longer applies; validate the changed inputs with a new preview and obtain fresh confirmation. The final tag goes on the exact commit the preview validated:
Require both versioned publication and channel publication jobs to succeed. Verify the final tag's commit and manifest identity; all six platform ZIPs and compatibility aliases; DEB/RPM, checksums, SBOM, provenance and evidence bundle; both multi-architecture image variants in Docker Hub, GHCR and Quay; and the accepted Helm chart in the chart repository. Download a host-compatible final artifact and confirm its digest is exactly the accepted candidate digest and its version/source output agrees.
```bash
git fetch origin --tags
git rev-parse "<preview-tag>^{commit}" # must equal PREVIEW_HASH — abort if not
git tag -a "<target>" -m "Release <target>" "$PREVIEW_HASH"
git push origin "<target>"
```
For stable releases verify GitHub Latest, R2 download aliases and `latest.json` unless a newer stable release already owns those channels. Prereleases must remain GitHub prereleases and leave stable pointers unchanged. Verify generated release notes compare against the frozen previous stable release, excluding internal previews.
- CI rebuilds from the same source with a different tag name, so the binary self-reports `<target>`. Other inputs can change: `build.yml` reads Console `releases/latest` during each build. Compare the actual Console tag, URL, and verified digest in every final build job with the accepted preview record, and inspect available build provenance for other relevant changes. A pre-tag check cannot prevent a later input change. On a mismatch or missing evidence, repeat affected acceptance against the final artifacts and report the limitation; failed acceptance leaves release validation incomplete even if publication already succeeded. Never move the final tag to conceal a failure.
- Verify the final tag's complete publication path: all matrix and release jobs green; `gh release view "<target>"` shows the full versioned and `-latest` asset set plus checksums, SBOM, and provenance; Docker and Helm workflows succeed; `latest.json` points to `<target>`. A stable target must have `isPrerelease=false` and `isLatest=true`. An alpha/beta/rc target must have `isPrerelease=true`; GitHub does not permit prereleases to be Latest, but the project `latest.json` still advances to the final non-preview target.
- Verify the final Release body contains `## What's Changed` and a Full Changelog link. When `PREVIOUS_DELIVERABLE` exists, the link MUST be `https://github.com/rustfs/rustfs/compare/<PREVIOUS_DELIVERABLE>...<target>` and the baseline MUST equal the preview Release baseline; for example, both `1.0.0-beta.12-preview.1` and `1.0.0-beta.12` compare from `1.0.0-beta.11`.
- Verify the preview cleanup: `cleanup-preview-releases` must succeed and `gh release view "<preview-tag>"` must then report `release not found` for every preview iteration of this target. Each surviving preview tag must still resolve to its own recorded commit; the accepted preview resolves to `PREVIEW_HASH`. If the job failed, delete the leftover Releases manually with `gh release delete "<preview-tag>" --yes` and report it.
- Verify a host-compatible final artifact's checksum and `./rustfs --version`; it must report `<target>` and the expected source SHA. This identity check does not replace affected acceptance when build inputs changed.
On infrastructure failure retry the same workflow/seal; matching destinations are reused. Never clobber an asset, force a tag, rebuild the image or silently accept a changed digest. A source fix after the final tag requires a new patch version. Cross-registry writes are not atomic: report the unfinished destination rather than claiming publication complete.
Never pass `--cleanup-tag` when removing preview Releases. Remove them manually only if `cleanup-preview-releases` failed, after final publication and notes generation. Record deleted Releases and surviving tags.
New candidates create no preview Releases to clean up. For a legacy preview release, preserve its tag and remove its Release only after final delivery and notes are verified. Do not pass `--cleanup-tag`. Complete the [post-release follow-up](post-release-updates.md) separately.
@@ -24,7 +24,7 @@ Reconcile advisories after every completed final release, including Console-only
1. Re-read the published deliverable state before edits or a retry. If a newer release has superseded this target, do not downgrade installation references or the website announcement. Report the superseding release and leave those defaults intact.
2. Invoke `rustfs-release-version-bump` with stage `post-release`, the exact target, and the authorized delivery scope. Prepare the follow-up on current main; do not edit or move either validated release tag. Cargo versions may already be preparing the next target and must remain untouched.
3. Verify the exact Docker image manifest, Release download links, RPM source archive, and Helm repository entry before publishing the installation PR. The chart workflow overrides chart/app versions from the final tag, pins the source checkout to that release, and waits for its default Docker image before packaging.
3. Verify the exact Docker image manifest, Release download links, RPM source archive, and Helm repository entry before publishing the installation PR. The chart is packaged and tested before sealing; publication uploads those same bytes only after versioned Docker images are available.
4. Report the installation PR and its merge state separately from the completed release. Reuse existing merge authorization and repository checks; do not infer new merge authority from this reference.
## rustfs.com top banner
@@ -54,4 +54,4 @@ Use the existing announcement configuration from [website PR #112](https://githu
1. Compare the exact published tag with current main. Audit every release-side commit since their merge base, including inherited backports. An ahead count is a history difference, not proof of missing changes. Use ancestry and patch equivalence, then inspect current code and regression evidence for squashed, adapted, superseded, or reverted changes. Record each commit as present, superseded, release-only, or missing; a positive `git cherry` result alone does not prove a missing fix.
2. Bring missing shared changes to main through a reviewed PR and the required checks before closing the follow-up. Preserve main's later changes; decide version bumps and pinned Console configuration separately. If a history merge is needed, use a merge commit and verify the release commit is an ancestor of main; squash does not establish that ancestry. Do not merge main into the published release branch or move its tags. Report unresolved changes instead of claiming reconciliation is complete.
3. After publication and authorized follow-ups are complete, delete the remote `release/<target>` branch. Immediately before deletion, verify its HEAD still equals the published tag's commit and no open PR targets or originates from it. Preserve a branch with newer work or open PRs and report the blocker. Use a deletion with an expected-old-SHA check to avoid removing a concurrently advanced branch.
4. Verify the remote branch is absent and the final and preview tags still resolve to their recorded commits. Remove a local release branch only if it is task-owned, unchecked out, and has no unpreserved work. Record cleanup separately from artifact publication; an explicitly authorized deletion before reconciliation does not make unresolved main changes complete.
4. Verify the remote branch is absent and the final tag and any legacy preview tags still resolve to their recorded commits. Remove a local release branch only if it is task-owned, unchecked out, and has no unpreserved work. Record cleanup separately from artifact publication; an explicitly authorized deletion before reconciliation does not make unresolved main changes complete.
@@ -1,10 +1,10 @@
# Preview Artifact Acceptance
# Candidate Artifact Acceptance
Read after Phase 3 succeeds. Complete every check below before the parent skill's manual confirmation gate. These checks cover the downloaded artifact, embedded Console, and latest rc client.
Read after Phase 3 succeeds. Complete every check below before the parent skill's manual confirmation gate. These checks cover the downloaded artifact, embedded Console, and the rc release frozen in the candidate inputs.
## Phase 4 — Run the artifact locally, verify the console
Use a unique task-owned scratch directory. Select the versioned archive matching the host OS, architecture, and runtime, download its checksum file, and verify the archive before extraction. Do not substitute a different build when a matching artifact is unavailable; report that acceptance gap. The shell example below assumes the verified executable is at `RUSTFS_BINARY` (`rustfs.exe` on Windows), `SCRATCH` is the task directory, and `S3_PORT` and `CONSOLE_PORT` are distinct available ports. Use equivalent native commands when the shell differs.
Use a unique task-owned scratch directory. Select the versioned archive matching the acceptance host, and verify it against the sealed manifest before extraction. Before desktop completion, use the Linux acceptance environment rather than rebuilding for the local desktop. Do not substitute a different build when a matching artifact is unavailable; report that acceptance gap. The shell example below assumes the verified executable is at `RUSTFS_BINARY` (`rustfs.exe` on Windows), `SCRATCH` is the task directory, and `S3_PORT` and `CONSOLE_PORT` are distinct available ports. Use equivalent native commands when the shell differs.
Bind both listeners to loopback. Choose task-only credentials, set loopback proxy exclusions for the server/client processes, and record the server PID. A port probe does not reserve a port; retry startup with fresh ports on a bind conflict rather than connecting to an existing service.
@@ -24,16 +24,16 @@ PREVIEW_SERVER_PID=$!
Checks (all must pass):
- `"$RUSTFS_BINARY" --version` reports the preview tag name and the short SHA of `PREVIEW_HASH`. Reporting `<target>` without the `-preview.N` suffix means the build did not embed the tag — treat as FAIL and investigate before proceeding.
- `"$RUSTFS_BINARY" --version` reports the exact final `<target>` and full `CANDIDATE_SHA`. A preview suffix, another version or another source is a failure.
- Wait with a bounded readiness deadline while checking the recorded server process remains alive; `curl --noproxy '*' -fsS "$S3_ENDPOINT/health/ready"` must return ready from this process's listener.
- Startup log shows the embedded console being served (this was the regression that `fix(release): require embedded console assets` guards).
- Open `CONSOLE_ENDPOINT` in the browser and log in with the task credentials. The dashboard must render without JS console errors; create a bucket, upload a file, download it back (byte-identical), and delete the object and bucket. Keep the server running for Phase 5.
## Phase 5 — Validate with the latest rc client
## Phase 5 — Validate with the pinned rc client
`rc` is the RustFS CLI client from <https://github.com/rustfs/cli>.
- Resolve the latest release with `gh api repos/rustfs/cli/releases/latest --jq .tag_name`; verify the local client's version or download and verify a host-compatible client into the task directory. Set `RC_BIN` to that executable and record its version/digest. Avoid changing the user's installed client for this check.
- Read the CLI release and asset digests from `inputs.json`; verify the local client's version or download a host-compatible asset from that exact release into the task directory. If the host is a desktop, resolve its asset within the same pinned CLI release and record the digest; do not independently follow `latest`. Set `RC_BIN` to that executable and record its version/digest. Avoid changing the user's installed client for this check.
- Inspect the client's current help for an isolated configuration option and use it when supported. Otherwise choose a unique `PREVIEW_ALIAS` and confirm it is absent before creating it; never overwrite an existing alias. Use task-owned bucket/user names and input/output files. The names below are safe only inside the newly started isolated server.
- Point the client at `S3_ENDPOINT` with loopback proxy exclusions and run the command matrix, recording PASS/FAIL per command:
@@ -60,3 +60,9 @@ cmp <local-file> "$SCRATCH/out"
```
- Any FAIL blocks acceptance. On success or failure, stop only the recorded task process and remove only task-created aliases/data after retaining the needed failure evidence. Do not erase an existing client configuration or unrelated server data.
## Field acceptance and desktop completion
Before accepting the Linux manifest, record Scanner/Heal, multi-pool, memory, major features, high-concurrency 503, performance and previous-version integrity/upgrade results in backlog. Preserve the release-specific agreed limits and baseline; a PASS label without artifact identities or test results does not satisfy acceptance.
Dispatch `release-complete.yml` only after that report passes, using the exact Linux manifest digest and backlog URL. Require native macOS/Windows checks, package extraction, image startup/scans, chart tests and complete certification to pass. Complete any desktop Console/client checks applicable to the release against these same archives. Record the completion run and final manifest SHA-256 before entering the parent confirmation gate. Partial Linux acceptance does not authorize final publication.
@@ -1,25 +1,9 @@
# Preview Release Verification
# Linux candidate verification
Read for Phase 3 after the candidate passed exact-SHA CI. The [parent release invariants](../SKILL.md) apply.
Read for Phase 3 after the source passed exact-SHA CI. The [parent invariants](../SKILL.md) apply; commands and recovery live in the [pipeline runbook](../../../../docs/operations/release-pipeline.md).
## Preview channels
Dispatch `release-candidate.yml` on main with the confirmed final version and `CANDIDATE_SHA`. Record the run ID, frozen `inputs.json`, every Linux archive digest and `linux-manifest.json` SHA-256. Require the four Linux builds, native startup/S3 checks, previous-release upgrade matrix and Linux seal to succeed. Missing, skipped, cancelled or failed required work is incomplete acceptance.
- Use `<target>-preview.N` for every target, e.g. `1.0.0-beta.10-preview.3` or `1.1.0-preview.1`.
- The canonical suffix is exactly `-preview.<digits>`. `build.yml` recognizes it before alpha/beta/rc classification and routes it to the preview-only path; any other tag containing `-preview` fails closed instead of being treated as a release.
- A preview Release MUST be published with `isPrerelease=true` and `isLatest=false`. Any `*-latest` preview asset or preview-triggered `latest.json`, R2, Helm, or moving Docker channel tag is a pipeline failure. Docker images may use the exact preview version tag, including its existing variant suffixes.
- Preview Releases are cleaned up by the `cleanup-preview-releases` job after `publish-release` succeeds for the deliverable tag. It deletes every Release whose tag is exactly `<target>-preview.<digits>` and never passes `--cleanup-tag`, so the tags survive.
Verify the frozen Console identity equals the one accepted in Phase 0. Every matrix job consumes that same digest and exact toolchain. The binary must report the final product version and source commit; its candidate ID is separate. No public tag, Release, registry tag, R2 release path, Helm repository or latest pointer should change during this stage.
## Phase 3 — CI and preview Release verification
- Find and watch the tag build: `gh run list --workflow build.yml --branch "<preview-tag>" --limit 1` then `gh run watch <run-id>`. Every build matrix target must succeed (linux x86_64/aarch64 × musl/gnu, macos-aarch64, windows-x86_64).
- Confirm the preview tag resolves to the Phase 2 CI-validated `PREVIEW_HASH`. If the release branch advances during acceptance, repeat Phases 2–5 with a new preview tag.
- Confirm the Release publication jobs (`create-release`, `upload-release-assets`, and `publish-release`) succeed while `update-latest-version` is skipped.
- Record the inputs actually consumed by each build job: source SHA and the embedded Console release tag, asset URL, and verified SHA256 from the download logs. The workflow reads Console `releases/latest` separately for each build; confirm the matrix used the Console asset accepted in Phase 0. A missing identity or differing asset blocks acceptance until resolved. Retain available toolchain and build provenance with the artifact checksums; the same source SHA alone does not establish equivalent builds.
- Verify `gh release view "<preview-tag>" --json isPrerelease,assets,url`: `isPrerelease` must be `true`, and the Release must contain all 6 versioned platform zips, checksums, SBOM, and provenance with no `-latest` assets. Confirm `gh api repos/{owner}/{repo}/releases/latest --jq .tag_name` does not return `<preview-tag>`.
- Record `PREVIOUS_DELIVERABLE`, selected from published Releases by `publishedAt` after excluding the current tag and every `-preview.N` tag. Verify `gh release view "<preview-tag>" --json body --jq .body` contains `## What's Changed` and, when `PREVIOUS_DELIVERABLE` exists, `**Full Changelog**: https://github.com/rustfs/rustfs/compare/<PREVIOUS_DELIVERABLE>...<preview-tag>`. For a repository with no previous deliverable, verify a Full Changelog link exists and record the GitHub baseline fallback.
- Confirm Helm is skipped. If preview Docker images are published, confirm they use only exact preview tags (and variant suffixes), with no `latest`, `alpha`, `beta`, or `rc` channel updates. Preview validation covers the built RustFS binaries, embedded console, and rc compatibility; it does not authorize advancing any installation default.
## Release notes
- When a previous deliverable exists, GitHub Release notes for the preview and final tags MUST use it as their shared comparison baseline: the most recently published non-preview Release before the target. Internal `-preview.N` Releases are explicitly excluded from that selection, even when they point at the same commit as the final tag — cleanup runs after the notes are generated, so the preview Release is still present and would otherwise be picked as the baseline. If no previous deliverable exists, omit `previous_tag_name` and record that GitHub's default baseline fallback was used.
- Generated Release notes carry a workflow-management marker so retries can repair them. Before manually curating a generated body, remove that marker; unmarked non-placeholder notes are preserved by later workflow runs.
A green Linux run is ready for field acceptance, not publication. The partial certification must have `publish_allowed: false`. Retain the exact Linux manifest for the next stage; a new source/build-input candidate must not inherit the old approval. A newer unrelated main commit does not invalidate this evidence.
@@ -16,22 +16,13 @@ Read for Phases 1–2. The [parent release invariants](../SKILL.md) and Console
- Only when neither the release branch, an unfinished preview tag, nor current main already has `<target>`: if main is at a later version, select and verify a historical commit containing `<target>` rather than downgrading main; block if none exists. Otherwise invoke the `rustfs-release-version-bump` skill with stage `prepare`, the final `<target>` (NOT a preview version), and full GitHub flow (commit/push/PR).
- If a bump PR was needed, get it merged into main and record its merge commit. For a new release branch when main already has `<target>`, identify the main commit that introduced those versions. Phase 2 chooses a release candidate containing that commit; an existing release branch keeps its own history and a newer main HEAD is not required.
Do not wait for the moving latest main HEAD to become green. The release-branch CI in Phase 2 validates the chosen commit before a preview tag is created.
Do not wait for the moving latest main HEAD to become green. The release-branch CI in Phase 2 validates the chosen commit before a candidate build is dispatched.
## Phase 2 — Pin and validate the release candidate, then publish the preview tag
## Phase 2 — Pin and validate the release source
- Use `release/<target>`. On the first attempt, create it from a selected main commit containing the version bump and intended changes; it need not be the latest main HEAD. On a restart, retain the existing branch and backport only reviewed fixes from main. Verify the branch's Cargo versions still equal `<target>`. If adopting an existing preview, run this branch CI before continuing its acceptance; reuse the tag only when it points to the validated SHA.
- The current `ci.yml` runs automatically on main pushes, not release-branch pushes. Dispatch it explicitly with `gh workflow run ci.yml --ref "release/<target>"`. Locate that dispatch run, verify its `head_sha` equals the release branch's remote HEAD, and require the full expected CI job set to pass, including `Quick Checks`, `Test and Lint`, and `End-to-End Tests (full merge gate)`. Treat a missing or skipped required lane as incomplete. A successful PR check or CI run for a different SHA does not qualify. If the branch advances or a run is cancelled, validate the new SHA again. Do not tag a candidate with failing or incomplete CI.
- Use `release/<target>`. On the first attempt, create it from a selected main commit containing the version bump and intended changes; it need not be the latest main HEAD. On a restart, retain the existing branch and backport only reviewed fixes from main. Verify the branch's Cargo versions still equal `<target>`. A legacy preview can anchor source selection, but its tag and old artifacts do not constitute a candidate in the new protocol; freeze and verify a new candidate after the release recipe is aligned.
- The current `ci.yml` runs automatically on main pushes, not release-branch pushes. Dispatch it explicitly with `gh workflow run ci.yml --ref "release/<target>"`. Locate that dispatch run, verify its `head_sha` equals the release branch's remote HEAD, and require the full expected CI job set to pass, including `Quick Checks`, `Test and Lint`, and `End-to-End Tests (full merge gate)`. Treat a missing or skipped required lane as incomplete. A successful PR check or CI run for a different SHA does not qualify. If the branch advances or a run is cancelled, validate the new SHA again. Do not dispatch a candidate with failing or incomplete CI.
- Prefer protecting `release/*` against force pushes and deletion and requiring reviewed backport PRs. Check live rulesets rather than assuming the main ruleset covers release branches; report a missing release-branch rule as a gap.
- Set `PREVIEW_HASH` to the CI-validated release-branch commit and recheck the remote branch HEAD immediately before tagging. Report the branch, commit, and CI run URL; do not derive `PREVIEW_HASH` from a later `origin/main` fetch. When adopting an existing preview tag at this SHA, skip tag creation and continue Phase 3.
```bash
git tag -a "<preview-tag>" -m "Release <preview-tag>" "$PREVIEW_HASH"
git push origin "<preview-tag>"
```
Pushing the tag triggers `.github/workflows/build.yml` ("Build and Release"); `docker.yml` chains off it via `workflow_run`.
The preview run builds versioned artifacts and publishes them in a GitHub prerelease. Docker may publish exact preview image tags. Latest-channel, R2, and Helm publication must be skipped; installation references and the website announcement stay unchanged.
On a restart after a backport, re-run release-branch CI before setting the new `PREVIEW_HASH`. Use the next unused preview iteration when the validated commit differs from an existing preview tag; never move an existing tag.
- Set `CANDIDATE_SHA` to the CI-validated release-branch commit. Record the branch, exact commit and CI run URL. The new candidate protocol starts from a commit whose release recipe matches the reviewed main dispatcher; backport the recipe before freezing an older release branch if needed.
- Dispatch the Linux candidate as described in [candidate verification](preview-release.md). Do not create a preview or final tag here. The final executable already contains `<target>`; publication later promotes those same bytes.
- After a source or build-input change, run the affected source checks and create a new candidate. Reuse the frozen candidate after an infrastructure failure. Do not restart acceptance for unrelated main advances.
@@ -4,18 +4,18 @@ description: Prepare an exact RustFS Cargo version or align installation referen
---
# Release Version Files
Require an exact target; ask if missing or ambiguous. Reject targets containing `-preview`: those are tag-only. Infer `post-release` only from an explicit request or parent workflow; otherwise use `prepare`. Delivery defaults to local edit/verify; commit, push, PR, and merge follow existing authorization and [Git rules](../../references/pull-requests.md).
Require an exact target; ask if missing or ambiguous. Reject targets containing `-preview`: those identify candidates, not product versions or new public tags. Infer `post-release` only from an explicit request or parent workflow; otherwise use `prepare`. Delivery defaults to local edit/verify; commit, push, PR, and merge follow existing authorization and [Git rules](../../references/pull-requests.md).
| Stage | Files and constraints |
|---|---|
| `prepare` | Only `Cargo.toml` and `Cargo.lock`: workspace package, internal dependency, and workspace member versions become `<target>`. External dependencies stay unchanged. |
| `post-release` | Only `README.md`, `README_ZH.md`, `flake.nix`, `helm/rustfs/Chart.yaml`, and `rustfs.spec`. Cargo may already target the next release; leave it untouched. |
Before installation edits, verify the exact non-preview Release, downloadable assets/source archive, pullable `rustfs/rustfs:<target>` image manifest, and published Helm chart. Block on missing artifacts. Do not downgrade defaults if a newer deliverable superseded the target. Preview preparation retains the previous published installation references; follow-up commits never replace either validated tag.
Before installation edits, verify the exact non-preview Release, downloadable assets/source archive, pullable `rustfs/rustfs:<target>` image manifest, and published Helm chart. Block on missing artifacts. Do not downgrade defaults if a newer deliverable superseded the target. Candidate preparation retains the previous published installation references; follow-up commits never replace validated tags or sealed artifacts.
Packaging policy:
- Docker image tags have no `v` prefix. Derive chart/app versions with `scripts/helm_chart_version.sh <target>` (`beta.N -> 0.N.0`; other versions unchanged). Helm CI derives versions from the final tag, so no early chart bump is required.
- Docker image tags have no `v` prefix. Derive chart/app versions with `scripts/helm_chart_version.sh <target>`; both equal the final product SemVer, including prerelease suffixes. Helm CI uses the frozen candidate version before sealing, so no early installation chart bump is required. Previously published chart versions remain unchanged.
- RPM `Version` is numeric; `Release` is the prerelease suffix or `1` for stable. Expanded `Source0` and the unpack directory must resolve to the exact target including its suffix. Add the changelog entry using current git identity, date, and target; never copy an example identity/date.
- Changing these mappings requires explicit confirmation.
+1 -1
View File
@@ -1 +1 @@
sha256=fbabc660e6e01fd8ff6b9d1de7d68e58242d60c942f85aed5074b0ec2918c154
sha256=21e9a1561ac20382503c38353bbb9b830658534a4f277c254a42d0afc6ebed8b
+1 -1
View File
@@ -71,7 +71,7 @@ s3s-footprint-check: ## Check the s3s dependency footprint ratchet stays frozen
./scripts/check_s3s_footprint.sh
.PHONY: test-count-ratchet-check
test-count-ratchet-check: ## Check no workspace crate lost tests or gained #[ignore] against the committed baseline
test-count-ratchet-check: ## Check no workspace crate fell below its test floor or rose above its #[ignore] ceiling in the committed baseline
@echo "🧪 Checking per-crate test-count ratchet..."
./scripts/check_test_count_ratchet.sh
+13
View File
@@ -22,6 +22,14 @@ TEST_THREADS ?= 1
# differ from CI and .config/nextest.toml test-groups will NOT apply.
.PHONY: script-tests
.PHONY: release-package-tests
release-package-tests: ## Run release package integration on Linux with fpm, dpkg and rpm installed
$(RUSTFS_PYTHON_BIN) ./scripts/test_release_packages.py
.PHONY: release-image-tests
release-image-tests: ## Run OCI integration with Docker, Go and skopeo; set RELEASE_IMAGE_VARIANT and RELEASE_IMAGE_OUTPUT
$(RUSTFS_PYTHON_BIN) ./scripts/test_release_images.py
script-tests: ## Run shell script tests
@echo "Running script tests..."
trap 'status=$$?; printf "ERROR: script-tests failed (exit %s): %s\n" "$$status" "$$BASH_COMMAND" >&2; exit "$$status"' ERR
@@ -53,12 +61,15 @@ script-tests: ## Run shell script tests
$(RUSTFS_PYTHON_BIN) ./scripts/test_check_doc_paths.py
./scripts/check_embedded_secrets.sh --self-test
./scripts/check_s3s_footprint.sh --self-test
./scripts/test_check_test_count_ratchet.sh
$(RUSTFS_PYTHON_BIN) ./scripts/check_test_wiring.py --self-test
$(RUSTFS_PYTHON_BIN) ./scripts/codemods/s3_error_to_s3_types.py --self-test
$(RUSTFS_PYTHON_BIN) ./scripts/test_e2e_binary.py
$(RUSTFS_PYTHON_BIN) ./scripts/test_download_release_asset.py
$(RUSTFS_PYTHON_BIN) ./scripts/test_reference_binary.py
$(RUSTFS_PYTHON_BIN) ./scripts/test_inspect_btf_prototypes.py
$(RUSTFS_PYTHON_BIN) ./scripts/test_release_build_workflow.py
$(RUSTFS_PYTHON_BIN) -m unittest discover -s scripts/release -p "test_*.py"
$(RUSTFS_PYTHON_BIN) ./scripts/test_release_glibc.py
$(RUSTFS_PYTHON_BIN) ./scripts/test_upgrade_workflow.py
$(RUSTFS_PYTHON_BIN) ./scripts/test_migration_gate_evidence.py
@@ -79,8 +90,10 @@ script-tests: ## Run shell script tests
$(RUSTFS_PYTHON_BIN) ./scripts/test_inspect_validation_runner.py
$(RUSTFS_PYTHON_BIN) ./scripts/test_retain_validation_evidence.py
$(RUSTFS_PYTHON_BIN) ./scripts/test_ci_timing_report.py
$(RUSTFS_PYTHON_BIN) ./scripts/test_junit_diff.py
$(RUSTFS_PYTHON_BIN) ./scripts/s3-tests/test_report_compat.py
$(RUSTFS_PYTHON_BIN) ./scripts/s3-tests/test_runner_tools.py
$(RUSTFS_PYTHON_BIN) ./scripts/s3-tests/test_reference_parity.py
bash -n ./scripts/validate_object_data_cache_cold_stampede.sh
$(RUSTFS_PYTHON_BIN) ./scripts/check_object_data_cache_follower_samples.py --self-test
./scripts/validate_object_data_cache_cold_stampede.sh --self-test
+20
View File
@@ -0,0 +1,20 @@
# Pinned RustFS release used as the "reference" target of parity lanes
# (.github/workflows/reference-parity.yml). This is the single place that names
# the reference release; .github/actions/reference-binary resolves, downloads and
# verifies it, and fails on a missing pin, a missing asset or a digest mismatch.
#
# Bumping the reference is a reviewed event: every per-test difference between
# the candidate and the reference is measured against this exact build.
#
# Each sha256 is the value published in the release's SHA256SUMS asset, which
# matches the digest GitHub reports for the same asset. The action checks both.
schema = 1
repository = "rustfs/rustfs"
version = "1.0.5"
tag = "1.0.5"
[platforms.linux-x86_64-gnu]
asset = "rustfs-linux-x86_64-gnu-v1.0.5.zip"
sha256 = "5addae2f74756bf80104d0975e6347cda843bc2daffdfcd1f0cb80118b95cbd0"
executable = "rustfs"
+24
View File
@@ -0,0 +1,24 @@
# Targets of the s3-tests reference parity lane (.github/workflows/reference-parity.yml).
#
# Exactly one target uses `binary = "reference"` (the release pinned in
# .config/reference-binary.toml). Every other target uses `binary = "candidate"`
# (the binary built from the branch under test) and is compared per node id
# against the reference. All targets run the same harness, test list and server
# configuration; `env` adds only the target's own RUSTFS_* settings.
#
# Adding a target is one more [[targets]] table, for example:
#
# [[targets]]
# name = "candidate-gateway"
# binary = "candidate"
# env = { RUSTFS_S3_STACK = "gateway" }
schema = 1
[[targets]]
name = "reference"
binary = "reference"
[[targets]]
name = "candidate-legacy"
binary = "candidate"
+12
View File
@@ -0,0 +1,12 @@
# Tolerated per-test differences of the s3-tests reference parity lane.
#
# One entry per line, four fields separated by " | ":
#
# <target> | <pytest node id> | <tracking issue URL> | <reason>
#
# <target> is a candidate name from .config/reference-parity-targets.toml, the
# node id is exactly as the lane prints it (s3tests/functional/test_s3.py::...),
# and the issue URL is https://github.com/rustfs/<repo>/issues/<number>.
#
# Same semantics as scripts/junit_diff.py --allow: an entry whose difference no
# longer occurs is stale and fails the lane, so remove it when it is fixed.
+82 -24
View File
@@ -1,44 +1,93 @@
{
"schema_version": 1,
"plan_id": "rustfs-release-validation",
"description": "Versioned certification plan for every public RustFS release exit (rustfs/backlog#2662, R01). preview, prerelease and stable all require this explicit plan. Scripts/release/release_gate.py fail-closes publication on any missing, duplicate, stale, substituted or non-PASS required item.",
"release_types": ["preview", "prerelease", "stable"],
"description": "Certification for immutable candidates. All shipped platforms require native identity and startup; stable and prerelease require the pinned upgrade matrix. Linux-only acceptance never authorizes publication.",
"release_types": [
"preview",
"prerelease",
"stable"
],
"targets": [
{"os": "linux", "arch": "x86_64", "abi": "musl"},
{"os": "linux", "arch": "aarch64", "abi": "musl"},
{"os": "linux", "arch": "x86_64", "abi": "gnu"},
{"os": "linux", "arch": "aarch64", "abi": "gnu"},
{"os": "macos", "arch": "aarch64", "abi": "none"},
{"os": "windows", "arch": "x86_64", "abi": "msvc"}
{
"os": "linux",
"arch": "x86_64",
"abi": "musl"
},
{
"os": "linux",
"arch": "aarch64",
"abi": "musl"
},
{
"os": "linux",
"arch": "x86_64",
"abi": "gnu"
},
{
"os": "linux",
"arch": "aarch64",
"abi": "gnu"
},
{
"os": "macos",
"arch": "aarch64",
"abi": "none"
},
{
"os": "windows",
"arch": "x86_64",
"abi": "msvc"
}
],
"targets_basis": "Platform matrix of .github/workflows/build.yml (prepare-platform-matrix). gnu and macOS targets build with the pyroscope feature; musl and Windows targets build with no extra features.",
"features": ["pyroscope"],
"trusted_workflows": [
{"repository": "rustfs/s3chaos", "workflow_id": "release-gate.yml"}
"features": [
"pyroscope"
],
"trusted_workflows_basis": "The shared evidence envelope producer lives in rustfs/s3chaos (G01, fault/release_gate.rs) and its plan/live split workflow is .github/workflows/release-gate.yml there. Only results from these identities may enter a release certification; PR-uploaded JSON never gains release authority. R02 must extend this list when additional verifier workflows land, never widen it to untrusted sources.",
"trusted_workflows": [
{
"repository": "rustfs/rustfs",
"workflow_id": "release-candidate.yml"
},
{
"repository": "rustfs/rustfs",
"workflow_id": "release-complete.yml"
}
],
"trusted_workflows_basis": "Receipts are produced by the reviewed main workflows and fetched by exact run ID with GitHub artifact digest verification. Candidate inputs freeze the harness commit separately from product source.",
"lanes": [
{
"lane": "identity",
"harness_repository": "rustfs/s3chaos",
"harness_sha": "9a48f55c1d6d58ce63a8fb1ea7317e024f32f11d",
"harness_basis": "Fixed G01 evidence-envelope producer revision (rustfs/s3chaos PR #102 head recorded in rustfs/backlog#2658). R02 re-pins this when the live verifier ships.",
"harness_repository": "rustfs/rustfs",
"harness_sha": "candidate.workflow_sha",
"harness_basis": "The exact workflow commit is frozen in candidate inputs; no mutable branch is accepted as a harness identity.",
"timeout_seconds": 1800,
"environment_capabilities": ["process-exec"],
"cases": ["artifact-identity", "startup"],
"environment_capabilities": [
"process-exec"
],
"cases": [
"artifact-identity",
"startup"
],
"applies_to": {
"release_types": ["preview", "prerelease", "stable"],
"release_types": [
"preview",
"prerelease",
"stable"
],
"platforms": "all"
},
"applies_basis": "Artifact identity and startup checks are mandatory for every release type, preview included, on every shipped target (rustfs/backlog#2662 frozen design item 5)."
},
{
"lane": "upgrade",
"harness_repository": "rustfs/s3chaos",
"harness_sha": "9a48f55c1d6d58ce63a8fb1ea7317e024f32f11d",
"harness_basis": "Same fixed envelope producer as the identity lane; the workflow SHA and the product build SHA are bound separately and need not be equal.",
"harness_repository": "rustfs/rustfs",
"harness_sha": "candidate.workflow_sha",
"harness_basis": "The exact workflow commit is frozen in candidate inputs; no mutable branch is accepted as a harness identity.",
"timeout_seconds": 5400,
"environment_capabilities": ["process-exec", "previous-release-artifact"],
"environment_capabilities": [
"process-exec",
"previous-release-artifact"
],
"cases": [
"direct-upgrade",
"mixed-version-upgrade",
@@ -49,8 +98,17 @@
],
"cases_basis": "Case IDs of the pinned previous-release matrix in .github/workflows/e2e-upgrade.yml (source.version/asset/sha256 and the case list mapped there).",
"applies_to": {
"release_types": ["prerelease", "stable"],
"platforms": [{"os": "linux", "arch": "x86_64", "abi": "gnu"}]
"release_types": [
"prerelease",
"stable"
],
"platforms": [
{
"os": "linux",
"arch": "x86_64",
"abi": "gnu"
}
]
},
"applies_basis": "preview releases use the pre-declared smaller set (identity lane only). The upgrade lane is only exercised on linux-x86_64-gnu, the platform of the pinned source assets."
}
+13 -10
View File
@@ -4,15 +4,17 @@
# One row per workspace member: <package>, tests, ignored, tab-separated.
# tests counts #[test], #[tokio::test], #[tokio::test(...)] and #[test_case(...)]
# attribute lines under the package directory; ignored counts #[ignore] and
# #[ignore = "..."]. Both columns are exact: CI fails when a crate's tests fall
# below or rise above its row, or when its ignored count changes. Raise a row in
# the PR that adds tests; lowering one is a reviewed statement that tests were
# intentionally removed. Regenerate with:
# #[ignore = "..."]. The tests column is a floor and the ignored column is a
# ceiling: CI fails when a crate's tests fall below its row or its ignored count
# rises above it. Extra tests and fewer ignores pass; --update tightens the rows.
# Lowering a tests row or raising an ignored row is a reviewed statement that
# tests were intentionally removed or ignored; justify it in the PR description.
# Regenerate with:
# scripts/check_test_count_ratchet.sh --update
e2e_test 1026 20
rustfs 6438 64
rustfs 6440 64
rustfs-audit 70 0
rustfs-checksums 26 0
rustfs-checksums 28 1
rustfs-common 36 0
rustfs-concurrency 87 0
rustfs-config 43 0
@@ -20,9 +22,10 @@ rustfs-connect-symbol-catalog 6 0
rustfs-credentials 24 0
rustfs-crypto 67 0
rustfs-data-usage 58 0
rustfs-ecstore 7850 2
rustfs-ecstore 7858 2
rustfs-extension-schema 23 0
rustfs-filemeta 361 0
rustfs-gateway-adapter 74 0
rustfs-heal 1128 3
rustfs-heal-contracts 8 0
rustfs-iam 415 0
@@ -43,14 +46,14 @@ rustfs-policy 758 0
rustfs-protocols 770 1
rustfs-protos 75 0
rustfs-replication 209 0
rustfs-rio 223 0
rustfs-rio 225 1
rustfs-rio-v2 34 6
rustfs-s3-client 71 0
rustfs-s3-ops 8 0
rustfs-s3-types 105 0
rustfs-s3select-api 232 0
rustfs-s3select-query 235 2
rustfs-scanner 1386 20
rustfs-scanner 1392 20
rustfs-scanner-metrics 90 0
rustfs-security-governance 20 0
rustfs-signer 34 0
@@ -59,6 +62,6 @@ rustfs-targets 612 26
rustfs-test-utils 1 0
rustfs-tls-runtime 44 0
rustfs-trusted-proxies 72 0
rustfs-utils 251 0
rustfs-utils 253 1
rustfs-zip 30 0
zstd 11 0
@@ -0,0 +1,96 @@
# Copyright 2026 RustFS Team
#
# Licensed under the Apache License, Version 2.0 (the "License");
# you may not use this file except in compliance with the License.
# You may obtain a copy of the License at
#
# http://www.apache.org/licenses/LICENSE-2.0
#
# Unless required by applicable law or agreed to in writing, software
# distributed under the License is distributed on an "AS IS" BASIS,
# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
# See the License for the specific language governing permissions and
# limitations under the License.
name: "Reference RustFS binary"
description: "Download the pinned reference RustFS release, verify its sha256 and expose the executable path. A missing pin, missing asset or digest mismatch fails the step."
inputs:
pin-file:
description: "Pin file naming the reference release (repository-relative)"
required: false
default: ".config/reference-binary.toml"
platform:
description: "Platform table in the pin file"
required: false
default: "linux-x86_64-gnu"
destination:
description: "Directory that receives the executable; defaults to $RUNNER_TEMP/reference-binary"
required: false
default: ""
outputs:
path:
description: "Absolute path of the verified executable"
value: ${{ steps.unpack.outputs.path }}
version:
description: "Pinned reference version"
value: ${{ steps.pin.outputs.version }}
sha256:
description: "Pinned and verified archive sha256"
value: ${{ steps.pin.outputs.sha256 }}
runs:
using: "composite"
steps:
- name: Resolve reference pin
id: pin
shell: bash
env:
PIN_TOOL: ${{ github.action_path }}/reference_binary.py
PIN_FILE: ${{ inputs.pin-file }}
PIN_PLATFORM: ${{ inputs.platform }}
run: |
set -euo pipefail
resolved="$(python3 "$PIN_TOOL" resolve --pin "$PIN_FILE" --platform "$PIN_PLATFORM")"
printf '%s\n' "$resolved"
printf '%s\n' "$resolved" >> "$GITHUB_OUTPUT"
- name: Download and verify reference archive
id: download
shell: bash
env:
GH_TOKEN: ${{ github.token }}
ASSET_DOWNLOADER: ${{ github.action_path }}/../setup/download_release_asset.py
REF_REPOSITORY: ${{ steps.pin.outputs.repository }}
REF_TAG: ${{ steps.pin.outputs.tag }}
REF_ASSET: ${{ steps.pin.outputs.asset }}
REF_SHA256: ${{ steps.pin.outputs.sha256 }}
run: |
set -euo pipefail
archive="$RUNNER_TEMP/reference-archive/$REF_ASSET"
# Fails unless the release lists exactly one asset of this name whose
# published digest equals the pin and whose downloaded bytes match it.
python3 "$ASSET_DOWNLOADER" --repository "$REF_REPOSITORY" --tag "$REF_TAG" \
--asset "$REF_ASSET" --output "$archive" --sha256 "$REF_SHA256"
echo "archive=$archive" >> "$GITHUB_OUTPUT"
- name: Unpack reference executable
id: unpack
shell: bash
env:
PIN_TOOL: ${{ github.action_path }}/reference_binary.py
REF_ARCHIVE: ${{ steps.download.outputs.archive }}
REF_SHA256: ${{ steps.pin.outputs.sha256 }}
REF_EXECUTABLE: ${{ steps.pin.outputs.executable }}
REF_DESTINATION: ${{ inputs.destination }}
run: |
set -euo pipefail
destination="${REF_DESTINATION:-$RUNNER_TEMP/reference-binary}"
unpacked="$(python3 "$PIN_TOOL" unpack --archive "$REF_ARCHIVE" --sha256 "$REF_SHA256" \
--executable "$REF_EXECUTABLE" --destination "$destination")"
path="${unpacked#path=}"
test -x "$path"
echo "Reference executable: $path"
"$path" --version
echo "path=$path" >> "$GITHUB_OUTPUT"
+144
View File
@@ -0,0 +1,144 @@
#!/usr/bin/env python3
# Copyright 2026 RustFS Team
#
# Licensed under the Apache License, Version 2.0 (the "License");
# you may not use this file except in compliance with the License.
# You may obtain a copy of the License at
#
# http://www.apache.org/licenses/LICENSE-2.0
#
# Unless required by applicable law or agreed to in writing, software
# distributed under the License is distributed on an "AS IS" BASIS,
# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
# See the License for the specific language governing permissions and
# limitations under the License.
"""Resolve and unpack the pinned reference RustFS release binary.
`resolve` reads the pin file and prints the selected platform's identity as
`key=value` lines (GitHub step-output format). `unpack` re-verifies the
downloaded archive digest against the pin and extracts exactly one executable.
Every missing or malformed input is a failure; nothing is skipped.
"""
import argparse
import hashlib
from pathlib import Path
import re
import sys
import tomllib
import zipfile
REPOSITORY = re.compile(r"[A-Za-z0-9][A-Za-z0-9_.-]*/[A-Za-z0-9][A-Za-z0-9_.-]*")
TAG = re.compile(r"[A-Za-z0-9][A-Za-z0-9_.+-]*")
PLATFORM = re.compile(r"[a-z0-9][a-z0-9_-]*")
ASSET = re.compile(r"[A-Za-z0-9][A-Za-z0-9_.+-]*\.zip")
SHA256 = re.compile(r"[0-9a-f]{64}")
EXECUTABLE = re.compile(r"[A-Za-z0-9][A-Za-z0-9_.-]*")
MAX_EXECUTABLE_BYTES = 1024 * 1024 * 1024
class PinError(Exception):
"""A pin, archive or argument that cannot be trusted."""
def field(table, key, pattern, where):
value = table.get(key)
if not isinstance(value, str) or not pattern.fullmatch(value):
raise PinError(f"{where}: missing or invalid '{key}'")
return value
def resolve(pin_path, platform):
if not PLATFORM.fullmatch(platform or ""):
raise PinError("invalid platform name")
try:
data = tomllib.loads(Path(pin_path).read_text(encoding="utf-8"))
except FileNotFoundError as exc:
raise PinError(f"{pin_path}: reference pin file is missing") from exc
except (OSError, UnicodeDecodeError, tomllib.TOMLDecodeError) as exc:
raise PinError(f"{pin_path}: cannot read reference pin: {exc}") from exc
if data.get("schema") != 1 or type(data.get("schema")) is not int:
raise PinError(f"{pin_path}: unsupported pin schema")
identity = {
"repository": field(data, "repository", REPOSITORY, pin_path),
"version": field(data, "version", TAG, pin_path),
"tag": field(data, "tag", TAG, pin_path),
}
platforms = data.get("platforms")
if not isinstance(platforms, dict) or not platforms:
raise PinError(f"{pin_path}: no pinned platforms")
entry = platforms.get(platform)
if not isinstance(entry, dict):
raise PinError(f"{pin_path}: no pin for platform '{platform}'")
where = f"{pin_path} [platforms.{platform}]"
identity["asset"] = field(entry, "asset", ASSET, where)
identity["sha256"] = field(entry, "sha256", SHA256, where)
identity["executable"] = field(entry, "executable", EXECUTABLE, where)
if identity["version"] not in identity["asset"]:
raise PinError(f"{where}: asset name does not carry version {identity['version']}")
unknown = set(entry) - {"asset", "sha256", "executable"}
if unknown:
raise PinError(f"{where}: unknown keys {sorted(unknown)}")
return identity
def file_sha256(path):
with open(path, "rb") as stream:
return hashlib.file_digest(stream, "sha256").hexdigest()
def unpack(archive, sha256, executable, destination):
if not SHA256.fullmatch(sha256 or "") or not EXECUTABLE.fullmatch(executable or ""):
raise PinError("invalid expected digest or executable name")
try:
actual = file_sha256(archive)
except OSError as exc:
raise PinError(f"{archive}: reference archive is missing or unreadable") from exc
if actual != sha256:
raise PinError(f"{archive}: sha256 {actual} does not match the pinned digest")
try:
with zipfile.ZipFile(archive) as bundle:
matches = [info for info in bundle.infolist() if info.filename == executable]
if len(matches) != 1:
raise PinError(f"{archive}: must contain exactly one top-level '{executable}'")
info = matches[0]
if info.is_dir() or not 0 < info.file_size <= MAX_EXECUTABLE_BYTES:
raise PinError(f"{archive}: '{executable}' is not a plausible executable")
destination = Path(destination)
destination.mkdir(parents=True, exist_ok=True)
target = destination / executable
with bundle.open(info) as source, open(target, "wb") as sink:
while chunk := source.read(1024 * 1024):
sink.write(chunk)
except zipfile.BadZipFile as exc:
raise PinError(f"{archive}: not a valid zip archive") from exc
target.chmod(0o755)
return target
def main(argv=None):
parser = argparse.ArgumentParser(description=__doc__.splitlines()[0])
sub = parser.add_subparsers(dest="mode", required=True)
res = sub.add_parser("resolve")
res.add_argument("--pin", required=True)
res.add_argument("--platform", required=True)
unp = sub.add_parser("unpack")
unp.add_argument("--archive", required=True)
unp.add_argument("--sha256", required=True)
unp.add_argument("--executable", required=True)
unp.add_argument("--destination", required=True)
args = parser.parse_args(argv)
try:
if args.mode == "resolve":
for key, value in resolve(args.pin, args.platform).items():
print(f"{key}={value}")
else:
print(f"path={unpack(args.archive, args.sha256, args.executable, args.destination)}")
except PinError as exc:
print(f"reference binary: {exc}", file=sys.stderr)
return 1
return 0
if __name__ == "__main__":
sys.exit(main())
+128 -21
View File
@@ -350,10 +350,10 @@
"roles": [
"periodic",
"manual",
"release"
"reusable"
],
"events": [
"push",
"workflow_call",
"schedule",
"workflow_dispatch"
],
@@ -367,7 +367,7 @@
"monitor": "workflow",
"max_age_hours": 36,
"result_owner": "build.yml",
"disposition": "retained independent entry",
"disposition": "Builds artifacts only; public release writes belong to release-publish.yml.",
"failure_owner": "workflow",
"origin": "baseline",
"alert_contract": "scheduled-validation"
@@ -745,14 +745,13 @@
},
{
"workflow": ".github/workflows/docker.yml",
"name": "Docker Images",
"name": "Build candidate images",
"roles": [
"manual",
"release"
"release",
"reusable"
],
"events": [
"workflow_run",
"workflow_dispatch"
"workflow_call"
],
"cadence": "event",
"timezone": "UTC",
@@ -761,7 +760,7 @@
"policy": "active",
"monitor": "none",
"result_owner": "docker.yml",
"disposition": "retained independent entry",
"disposition": "Builds artifacts only; public release writes belong to release-publish.yml.",
"monitor_reason": "Event-driven or operator-requested work has no periodic freshness obligation.",
"origin": "baseline"
},
@@ -870,6 +869,33 @@
"failure_owner": "watchdog",
"origin": "baseline"
},
{
"workflow": ".github/workflows/reference-parity.yml",
"name": "Reference Parity",
"roles": [
"pr",
"periodic",
"manual"
],
"events": [
"pull_request",
"schedule",
"workflow_dispatch"
],
"cadence": "nightly",
"timezone": "UTC",
"schedule": [
"37 1 * * *"
],
"owner": "@overtrue",
"policy": "active",
"monitor": "workflow",
"max_age_hours": 36,
"result_owner": "reference-parity.yml",
"disposition": "s3-tests per-node-id parity of branch candidates against the pinned reference release",
"failure_owner": "watchdog",
"origin": "added"
},
{
"workflow": ".github/workflows/functional-chain-health.yml",
"name": "Functional chain health",
@@ -954,14 +980,13 @@
},
{
"workflow": ".github/workflows/helm-package.yml",
"name": "Publish helm chart to artifacthub",
"name": "Build candidate Helm chart",
"roles": [
"manual",
"release"
"release",
"reusable"
],
"events": [
"workflow_run",
"workflow_dispatch"
"workflow_call"
],
"cadence": "event",
"timezone": "UTC",
@@ -970,7 +995,7 @@
"policy": "active",
"monitor": "none",
"result_owner": "helm-package.yml",
"disposition": "retained independent entry",
"disposition": "Builds artifacts only; public release writes belong to release-publish.yml.",
"monitor_reason": "Event-driven or operator-requested work has no periodic freshness obligation.",
"origin": "baseline"
},
@@ -996,6 +1021,26 @@
"policy_reason": "Preserve operator-disabled state; enabling requires a separate policy decision.",
"origin": "baseline"
},
{
"workflow": ".github/workflows/manual-main-packages.yml",
"name": "Manual main packages",
"roles": [
"manual"
],
"events": [
"workflow_dispatch"
],
"cadence": "event",
"timezone": "UTC",
"schedule": [],
"owner": "@overtrue",
"policy": "active",
"monitor": "none",
"result_owner": "manual-main-packages.yml",
"disposition": "Packages the origin/main tip as workflow artifacts only. No GitHub Release or download-channel writes.",
"monitor_reason": "Operator-requested packaging has no periodic freshness obligation.",
"origin": "manual-main-packages"
},
{
"workflow": ".github/workflows/minio-interop.yml",
"name": "minio-interop",
@@ -1182,14 +1227,13 @@
},
{
"workflow": ".github/workflows/package.yml",
"name": "Package DEB/RPM",
"name": "Build candidate packages",
"roles": [
"manual",
"release"
"release",
"reusable"
],
"events": [
"workflow_run",
"workflow_dispatch"
"workflow_call"
],
"cadence": "event",
"timezone": "UTC",
@@ -1198,7 +1242,7 @@
"policy": "active",
"monitor": "none",
"result_owner": "package.yml",
"disposition": "retained independent entry",
"disposition": "Builds artifacts only; public release writes belong to release-publish.yml.",
"monitor_reason": "Event-driven or operator-requested work has no periodic freshness obligation.",
"origin": "baseline"
},
@@ -1727,6 +1771,69 @@
"disposition": "retained independent entry",
"monitor_reason": "Event-driven or operator-requested work has no periodic freshness obligation.",
"origin": "baseline"
},
{
"workflow": ".github/workflows/release-candidate.yml",
"name": "Release candidate (Linux)",
"roles": [
"manual",
"release"
],
"events": [
"workflow_dispatch"
],
"cadence": "event",
"timezone": "UTC",
"schedule": [],
"owner": "@overtrue",
"policy": "active",
"monitor": "none",
"result_owner": "release-candidate.yml",
"disposition": "Explicit immutable candidate stage",
"monitor_reason": "Operator-requested release stage has no periodic freshness obligation.",
"origin": "release-candidate-migration"
},
{
"workflow": ".github/workflows/release-complete.yml",
"name": "Release candidate (complete)",
"roles": [
"manual",
"release"
],
"events": [
"workflow_dispatch"
],
"cadence": "event",
"timezone": "UTC",
"schedule": [],
"owner": "@overtrue",
"policy": "active",
"monitor": "none",
"result_owner": "release-complete.yml",
"disposition": "Explicit immutable candidate stage",
"monitor_reason": "Operator-requested release stage has no periodic freshness obligation.",
"origin": "release-candidate-migration"
},
{
"workflow": ".github/workflows/release-publish.yml",
"name": "Publish accepted release",
"roles": [
"manual",
"release"
],
"events": [
"workflow_dispatch"
],
"cadence": "event",
"timezone": "UTC",
"schedule": [],
"owner": "@overtrue",
"policy": "active",
"monitor": "none",
"result_owner": "release-publish.yml",
"disposition": "Explicit immutable candidate stage",
"monitor_reason": "Operator-requested release stage has no periodic freshness obligation.",
"origin": "release-candidate-migration"
}
]
}
+11 -4
View File
@@ -5,7 +5,7 @@
## Billing facts
This repository is public. For a public repository, GitHub's standard hosted runners (`ubuntu-latest`, `windows-latest`, `macos-latest`) cost nothing and have no minute cap, whatever plan the organisation is on. GitHub bills only two things: larger hosted runners, which have custom labels and are not used here, and standard runners in private repositories.
This repository is public. For a public repository, GitHub's standard hosted runners (`ubuntu-latest`, `windows-latest`, `macos-latest`) cost nothing and have no minute cap, whatever plan the organisation is on. GitHub bills only two things: larger hosted runners, including the `macos-latest-xlarge` release lane used here, and standard runners in private repositories.
Every other label in this repository is capacity RustFS pays for. ARC pods and lab machines accrue cost while the machine exists. A Ubicloud label is metered for the minutes a job holds its VM:
@@ -13,6 +13,7 @@ Every other label in this repository is capacity RustFS pays for. ARC pods and l
|---|---|---|---|
| `ubuntu-latest` | nobody | GitHub-hosted, Ubuntu 24.04, 4 vCPU, 16 GB RAM, about 14 GB of free disk. Ships a C toolchain, `docker`, `gh`, `curl`, `wget`, `jq`, Python. Can `mount` tmpfs and loop devices; io_uring works | the default for every job |
| `windows-latest`, `macos-latest` | nobody | GitHub-hosted standard runners | Windows and macOS targets |
| `macos-latest-xlarge` | RustFS | GitHub-hosted larger Apple Silicon runner | macOS release builds, deferred until Linux acceptance for candidates |
| `sm-standard-2` | RustFS | ARC pod, Ubuntu 24.04, minimal image: no `gh`, no C toolchain; Docker only on the `dind-` label | only a job that a hosted runner cannot run (see the placement rule) |
| `sm-standard-4` | RustFS | ARC pod, Ubuntu 24.04, 14 CPUs and 28 GB (the label is not the core count), large disk | workspace-scale Cargo builds |
| `ubicloud-standard-16-arm` | RustFS | Ubicloud VM, alias of `ubicloud-standard-16-arm-ubuntu-2404`: Ubuntu 24.04 arm64, 16 vCPU, 48 GB, 300 GB disk. Reached through the `build.yml` matrix, so `actionlint.yaml` does not list it | native aarch64 release builds (`linux-aarch64-musl`, `linux-aarch64-gnu`) |
@@ -32,7 +33,7 @@ A job runs on `ubuntu-latest` unless it has one of these reasons, and the reason
Jobs that do only the following have no reason to be self-hosted: shell or Python scripts, `cargo fmt`, `cargo deny`, typos, actionlint, API calls (`gh`, dependency review, issue and stale bots, CLA checks), downloading an artifact and running a prebuilt binary, or a `cargo build -p rustfs` with its own cache key. `e2e-distributed.yml` and `e2e-upgrade.yml` build the server and the `e2e_test` harness on `ubuntu-latest` inside their budgets, so "it compiles something" alone is not a reason.
The functional chain's `prepare` and `complete-chain` jobs, the performance runner availability check, and runner hygiene use `ubuntu-latest`: they inspect APIs or artifacts, not the lab hosts. The actual functional and performance suites remain on their lab runners. Functional-chain health collection uses `ubuntu-latest` so a lab outage cannot also stop its independent monitor. Docker image packaging also uses `ubuntu-latest` for both musl and glibc variants; it downloads release binaries rather than compiling the workspace, and still sets up QEMU and Buildx explicitly.
The functional chain's `prepare` and `complete-chain` jobs, the performance runner availability check, and runner hygiene use `ubuntu-latest`: they inspect APIs or artifacts, not the lab hosts. The actual functional and performance suites remain on their lab runners. Functional-chain health collection uses `ubuntu-latest` so a lab outage cannot also stop its independent monitor. Docker image packaging also uses `ubuntu-latest` for both musl and glibc variants; it wraps verified candidate binaries rather than compiling the workspace, and still sets up QEMU and Buildx explicitly.
Moving these jobs off AKS reduces capacity demand, not necessarily the Azure bill immediately. VM costs fall only when nodes scale down or smaller capacity replaces them.
@@ -52,7 +53,7 @@ A hosted runner is also the only choice for a few things the pods cannot do: `mo
These are the patterns that have cost real money here without validating anything more:
- An `if: always()` summary or alert job on a self-hosted label inside a `workflow_run`-triggered workflow. The upstream workflow completes many times a day, the real jobs skip, and the summary still starts a pod each time. Gate it on the upstream job having run, and keep it on `ubuntu-latest`.
- Per-push workflows on `main`. A merge triggers the full `ci.yml` matrix, `cache-warm.yml` and the `build.yml` development build; with tens of merges a day that is as much self-hosted time as all pull requests together. Anything that does not verify the merged tree itself belongs on a schedule or behind a `paths` filter.
- Per-push workflows on `main`. A merge triggers the applicable `ci.yml` matrix and cache warming; `build.yml` development builds are scheduled or manual; with tens of merges a day that is as much self-hosted time as all pull requests together. Anything that does not verify the merged tree itself belongs on a schedule or behind a `paths` filter.
- `apt-get install` of tools the hosted image already ships. It is only needed on the minimal pod images.
- A throttled `CARGO_BUILD_JOBS` on a 14-CPU pod. The limit exists for link-storm reasons documented in `ci.yml`; measure before raising it, and measure before lowering it further.
- Tiny jobs on ARC pods. Each one pays roughly a minute of pod start-up before its first step; a compile-free job that takes ten seconds costs more to schedule than to run.
@@ -66,7 +67,7 @@ gh api "repos/rustfs/rustfs/actions/runs/<run_id>/jobs?per_page=100" \
--jq '.jobs[] | "\(.conclusion)\t\(.labels | join(","))\t\(.started_at)\t\(.completed_at)\t\(.name)"'
```
Only rows whose label starts with `sm-`, `dind-`, `ubicloud-`, `smoke-` or `pf-` cost anything. At the October 2026 review, a full pull-request run of `ci.yml` spent about 150 self-hosted minutes, a third of it in jobs that never compiled; treat that as the scale of what a placement mistake costs, not as a current figure.
Rows using `macos-latest-xlarge` or a label starting with `sm-`, `dind-`, `ubicloud-`, `smoke-` or `pf-` use paid capacity. At the October 2026 review, a full pull-request run of `ci.yml` spent about 150 self-hosted minutes, a third of it in jobs that never compiled; treat that as the scale of what a placement mistake costs, not as a current figure.
## Checklist for a new or moved job
@@ -76,3 +77,9 @@ Only rows whose label starts with `sm-`, `dind-`, `ubicloud-`, `smoke-` or `pf-`
- A self-hosted `runs-on` carries a comment naming its reason from the placement rule.
- Every `./.github/actions/setup` call states `cache-save-if`, and a key read from one pool is not written from the other.
- Moving a job does not change its check name, so the `main` ruleset and `docs/testing/ci-gates.md` stay as they are; renaming it does, and then that document's change checklist applies.
## Release stages
The [candidate release pipeline](../../docs/operations/release-pipeline.md) builds Linux first and defers paid desktop capacity until acceptance. Daily development builds also select Linux; use manual platform selection for desktop development artifacts. All six platforms remain required before publication. The pipeline records and promotes the exact tested bytes. Builder workflows have no public-release credentials. Keep periodic development builds, PR verification and field-test chains separate when their contracts differ; do not re-enable disabled workflows as part of a source-file migration.
Larger runners remain billed for public repositories; see [GitHub runner billing](https://docs.github.com/en/actions/concepts/runners/larger-runners#billing). Actual spending comes from organization billing, not runner labels or an estimated build duration.
+149 -8
View File
@@ -23,9 +23,16 @@ on:
- 'deny.toml'
- '.github/actions/**'
- '.github/workflows/**'
- 'scripts/release/create_or_update_release.sh'
- 'scripts/release/package_service_scripts.sh'
- 'scripts/release/package_versions.sh'
- 'scripts/release/**'
- 'scripts/security/generate_release_supply_chain_assets.py'
- '.config/release-validation-plan.json'
- '.github/scheduled-validations.json'
- 'scripts/test_helm_release_workflow.py'
- 'scripts/test_release_packages.py'
- 'scripts/test_release_images.py'
- 'Dockerfile*'
- 'entrypoint.sh'
- 'bootstrap/**'
- 'scripts/test_package_service_scripts.sh'
- 'scripts/test_package_versions.sh'
- 'scripts/test_docker_workflow.py'
@@ -41,9 +48,16 @@ on:
- 'deny.toml'
- '.github/actions/**'
- '.github/workflows/**'
- 'scripts/release/create_or_update_release.sh'
- 'scripts/release/package_service_scripts.sh'
- 'scripts/release/package_versions.sh'
- 'scripts/release/**'
- 'scripts/security/generate_release_supply_chain_assets.py'
- '.config/release-validation-plan.json'
- '.github/scheduled-validations.json'
- 'scripts/test_helm_release_workflow.py'
- 'scripts/test_release_packages.py'
- 'scripts/test_release_images.py'
- 'Dockerfile*'
- 'entrypoint.sh'
- 'bootstrap/**'
- 'scripts/test_package_service_scripts.sh'
- 'scripts/test_package_versions.sh'
- 'scripts/test_docker_workflow.py'
@@ -132,12 +146,38 @@ jobs:
if: github.repository == 'rustfs/rustfs' && (github.event_name != 'pull_request' || github.event.action != 'closed')
runs-on: ubuntu-latest
timeout-minutes: 5
permissions:
contents: read
actions: read
outputs:
release_changed: ${{ steps.scope.outputs.release_changed }}
steps:
- name: Checkout repository
uses: actions/checkout@f548e57e544e1ff5a4c46bf1e1b8685f8e4a348a # v7
with:
fetch-depth: 0
persist-credentials: false
- name: Scope native packaging checks to release changes
id: scope
env:
BASE_SHA: ${{ github.event.pull_request.base.sha || github.event.before }}
run: |
python3 - <<'PY'
import os, re, subprocess
event, base = os.environ['GITHUB_EVENT_NAME'], os.environ.get('BASE_SHA', '')
changed = event == 'workflow_dispatch'
if event in ('push', 'pull_request'):
if not re.fullmatch(r'[0-9a-f]{40}', base) or base == '0' * 40:
changed = True
else:
paths = subprocess.check_output(['git', 'diff', '--name-only', base, os.environ['GITHUB_SHA']], text=True).splitlines()
pattern = r'^(scripts/release/|scripts/test_release_|Dockerfile|entrypoint.sh$|bootstrap/|\.dockerignore$|\.config/release-validation-plan.json$|\.github/workflows/(audit|build|docker|package|helm-package|release-[^.]+)\.yml$)'
changed = any(re.match(pattern, path) for path in paths)
with open(os.environ['GITHUB_OUTPUT'], 'a') as output:
output.write(f'release_changed={str(changed).lower()}\n')
PY
- name: Report unpinned GitHub Actions
run: ./scripts/security/check_workflow_pins.sh --enforce
@@ -150,8 +190,35 @@ jobs:
- name: Check checkouts clear their credentials
run: ./scripts/security/check_persist_credentials.sh
- name: Check preview release workflow policy
run: ./scripts/security/check_preview_release_workflow.sh
- name: Check release candidate and promotion contracts
run: python3 -m unittest discover -s scripts/release -p 'test_*.py'
- name: Prepare artifact transport rehearsal
run: |
mkdir -p "$RUNNER_TEMP/release-transport"
printf '{"purpose":"transport contract, not release evidence"}\n' > "$RUNNER_TEMP/release-transport/transport.json"
- uses: actions/upload-artifact@043fb46d1a93c77aae656e7c1c64a875d1fc6a0a # v7.0.1
with:
name: release-transport-contract-${{ github.run_attempt }}
path: ${{ runner.temp }}/release-transport/transport.json
if-no-files-found: error
retention-days: 1
- name: Verify actual GitHub artifact metadata and downloaded bytes
env:
GH_TOKEN: ${{ github.token }}
run: |
python3 - <<'PY'
import os, sys
from pathlib import Path
sys.path.insert(0, 'scripts/release')
import candidate as c
run = c.github(f"repos/{c.REPOSITORY}/actions/runs/{os.environ['GITHUB_RUN_ID']}")
destination = Path(os.environ['RUNNER_TEMP']) / 'release-transport-downloaded'
c.download_artifact(run, f"release-transport-contract-{os.environ['GITHUB_RUN_ATTEMPT']}", destination)
c.require(c.read_json(destination / 'transport.json') == {'purpose': 'transport contract, not release evidence'}, 'artifact round trip changed')
PY
- name: Check tier evidence workflow isolation
run: ./scripts/security/check_tier_artifact_workflow.sh
@@ -162,6 +229,80 @@ jobs:
- name: Check package service lifecycle contract
run: ./scripts/test_package_service_scripts.sh
release-package-contract:
name: Release package contract
needs: workflow-pin-report
if: needs.workflow-pin-report.outputs.release_changed == 'true'
runs-on: ubuntu-latest
timeout-minutes: 15
steps:
- uses: actions/checkout@f548e57e544e1ff5a4c46bf1e1b8685f8e4a348a # v7
with:
persist-credentials: false
- name: Install real package managers
run: |
sudo apt-get update
sudo apt-get install -y ruby ruby-dev build-essential rpm fakeroot cpio
sudo gem install fpm --version 1.17.0 --no-document
- name: Exercise accepted and substituted payloads through both recipes
run: python3 scripts/test_release_packages.py
release-image-contract:
name: Release image contract (${{ matrix.variant }})
needs: workflow-pin-report
if: needs.workflow-pin-report.outputs.release_changed == 'true'
runs-on: ubuntu-latest
timeout-minutes: 35
strategy:
fail-fast: false
matrix:
variant: [musl, gnu]
env:
RELEASE_IMAGE_VARIANT: ${{ matrix.variant }}
NO_PROXY: 127.0.0.1,localhost,::1
steps:
- uses: actions/checkout@f548e57e544e1ff5a4c46bf1e1b8685f8e4a348a # v7
with:
persist-credentials: false
- name: Install OCI tools
run: |
echo "RELEASE_IMAGE_OUTPUT=$RUNNER_TEMP/image-contract" >> "$GITHUB_ENV"
echo 'no_proxy=127.0.0.1,localhost,::1' >> "$GITHUB_ENV"
sudo apt-get update
sudo apt-get install -y skopeo
go version
- uses: docker/setup-qemu-action@c7c53464625b32c7a7e944ae62b3e17d2b600130 # v3
- uses: docker/setup-buildx-action@8d2750c68a42422c14e847fe6c8ac0403b4cbd6f # v3
- name: Exercise real OCI packaging, startup, registry retry and conflicts
run: python3 scripts/test_release_images.py
- name: Exercise amd64 OCI archive scanning
uses: aquasecurity/trivy-action@ed142fd0673e97e23eac54620cfb913e5ce36c25
env:
TRIVY_PLATFORM: linux/amd64
with:
input: ${{ runner.temp }}/image-contract-scan/image-${{ matrix.variant }}-amd64.tar
format: sarif
output: ${{ runner.temp }}/image-contract/scan-${{ matrix.variant }}-amd64.sarif
exit-code: "0"
- name: Exercise arm64 OCI archive scanning
uses: aquasecurity/trivy-action@ed142fd0673e97e23eac54620cfb913e5ce36c25
env:
TRIVY_PLATFORM: linux/arm64
with:
input: ${{ runner.temp }}/image-contract-scan/image-${{ matrix.variant }}-arm64.tar
format: sarif
output: ${{ runner.temp }}/image-contract/scan-${{ matrix.variant }}-arm64.sarif
exit-code: "0"
- name: Verify scan receipts before recording the archive
run: |
python3 - <<'PY'
import os, sys
from pathlib import Path
sys.path.insert(0, 'scripts/release')
import deliver, test_candidate
deliver.image_record(test_candidate.inputs(), os.environ['RELEASE_IMAGE_VARIANT'], Path(os.environ['RELEASE_IMAGE_OUTPUT']))
PY
dependency-review:
name: Dependency Review
runs-on: ubuntu-latest
File diff suppressed because it is too large Load Diff
+71 -751
View File
@@ -1,779 +1,99 @@
# Copyright 2024 RustFS Team
#
# Licensed under the Apache License, Version 2.0 (the "License");
# you may not use this file except in compliance with the License.
# You may obtain a copy of the License at
#
# http://www.apache.org/licenses/LICENSE-2.0
#
# Unless required by applicable law or agreed to in writing, software
# distributed under the License is distributed on an "AS IS" BASIS,
# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
# See the License for the specific language governing permissions and
# limitations under the License.
# Docker Images Workflow
#
# This workflow builds Docker images using pre-built binaries from the build workflow.
#
# Trigger Types:
# 1. workflow_run: Automatically triggered when "Build and Release" workflow completes
# 2. workflow_dispatch: Manual trigger for standalone Docker builds
#
# Key Features:
# - Only triggers when Linux builds (x86_64 + aarch64) are successful
# - Independent of macOS/Windows build status
# - Uses workflow_run event for precise control
# - Only builds Docker images for releases and prereleases (development builds are skipped)
name: Docker Images
# Permissions needed for workflow_run event and Docker registry access
# Copyright 2026 RustFS Team
# SPDX-License-Identifier: Apache-2.0
name: Build candidate images
on:
workflow_call:
inputs:
candidate_manifest:
type: string
required: true
permissions:
contents: read
packages: write
on:
# Automatically triggered when build workflow completes
workflow_run:
workflows: [ "Build and Release" ]
types: [ completed ]
# Manual trigger with same parameters for consistency
workflow_dispatch:
inputs:
push_images:
description: "Push images to registries"
required: false
default: true
type: boolean
version:
description: "Version to build (latest, v1.0.0, v1.0.0-alpha1, or v1.0.1-preview.1)"
required: false
default: "latest"
type: string
force_rebuild:
description: "Force rebuild even if binary exists (useful for testing)"
required: false
default: false
type: boolean
env:
CONCLUSION: ${{ github.event.workflow_run.conclusion }}
HEAD_BRANCH: ${{ github.event.workflow_run.head_branch }}
HEAD_SHA: ${{ github.event.workflow_run.head_sha }}
TRIGGERING_EVENT: ${{ github.event.workflow_run.event }}
DOCKERHUB_USERNAME: rustfs
CARGO_TERM_COLOR: always
REGISTRY_DOCKERHUB: rustfs/rustfs
REGISTRY_GHCR: ghcr.io/${{ github.repository }}
REGISTRY_QUAY: quay.io/rustfs/rustfs
DOCKER_PLATFORMS: linux/amd64,linux/arm64
actions: read
jobs:
# Check if we should build Docker images.
# Short-circuit at job level so per-merge development builds (workflow_run
# from a push to main) skip the whole run without spawning a checkout job:
# images are only published for tag builds — build.yml push triggers cover
# only main and tags, so a non-main head_branch is a tag name — and for
# manual dispatch.
build-check:
name: Docker Build Check
if: >-
github.event_name == 'workflow_dispatch' ||
(github.event.workflow_run.conclusion == 'success' &&
github.event.workflow_run.event == 'push' &&
github.event.workflow_run.head_branch != 'main')
runs-on: ubuntu-latest
timeout-minutes: 30
outputs:
should_build: ${{ steps.check.outputs.should_build }}
should_push: ${{ steps.check.outputs.should_push }}
build_type: ${{ steps.check.outputs.build_type }}
version: ${{ steps.check.outputs.version }}
short_sha: ${{ steps.check.outputs.short_sha }}
is_prerelease: ${{ steps.check.outputs.is_prerelease }}
create_latest: ${{ steps.check.outputs.create_latest }}
source_ref: ${{ steps.check.outputs.source_ref }}
steps:
- name: Checkout repository
uses: actions/checkout@f548e57e544e1ff5a4c46bf1e1b8685f8e4a348a # v7
with:
persist-credentials: false
# For workflow_run events, checkout the specific commit that triggered the workflow
ref: ${{ github.event.workflow_run.head_sha || github.sha }}
- name: Check build conditions
id: check
env:
# dispatch inputs via env, not `${{ }}` interpolation: they are
# free-form strings and would otherwise be evaluated by bash.
INPUT_VERSION: ${{ github.event.inputs.version }}
INPUT_PUSH_IMAGES: ${{ github.event.inputs.push_images }}
INPUT_FORCE_REBUILD: ${{ github.event.inputs.force_rebuild }}
run: |
should_build=false
should_push=false
build_type="none"
version=""
short_sha=""
is_prerelease=false
create_latest=false
source_ref="$GITHUB_SHA"
# Pre-GA policy: until the first stable (vX.Y.Z) tag exists, every
# prerelease (alpha/beta/rc) also moves `latest`, so users pulling
# `latest` get the newest test build. Once a stable tag is published
# this returns false and `latest` follows stable releases only.
prerelease_moves_latest() {
local stable_tags
stable_tags=$(git ls-remote --tags --refs origin 2>/dev/null \
| awk '{print $2}' \
| grep -E '^refs/tags/v?[0-9]+\.[0-9]+\.[0-9]+$' || true)
if [[ -z "$stable_tags" ]]; then
return 0
fi
echo "ℹ️ Stable release tag(s) already exist; prereleases no longer update latest"
return 1
}
if [[ "${{ github.event_name }}" == "workflow_run" ]]; then
# Triggered by build workflow completion
echo "🔗 Triggered by build workflow completion"
# Check if the triggering workflow was successful
# If the workflow succeeded, it means ALL builds (including Linux x86_64 and aarch64) succeeded
if [[ "$CONCLUSION" == "success" ]]; then
echo "✅ Build workflow succeeded, all builds including Linux are successful"
should_build=true
should_push=true
else
echo "❌ Build workflow failed (conclusion: $CONCLUSION), skipping Docker build"
should_build=false
fi
# Extract version info from commit message or use commit SHA
# Use Git to generate consistent short SHA (ensures uniqueness like build.yml)
short_sha=$(git rev-parse --short "$HEAD_SHA")
source_ref="$HEAD_SHA"
# Determine build type based on triggering workflow event and ref
triggering_event="$TRIGGERING_EVENT"
head_branch="$HEAD_BRANCH"
echo "🔍 Analyzing triggering workflow:"
echo " 📋 Event: $triggering_event"
echo " 🌿 Head branch: $head_branch"
echo " 📎 Head SHA: $HEAD_SHA"
# Check if this was triggered by a tag push
if [[ "$triggering_event" == "push" ]]; then
# For tag pushes, head_branch will be like "refs/tags/v1.0.0" or just "v1.0.0"
if [[ "$head_branch" == refs/tags/* ]]; then
# Extract tag name from refs/tags/TAG_NAME
tag_name="${head_branch#refs/tags/}"
version="$tag_name"
elif [[ "$head_branch" =~ ^v?[0-9]+\.[0-9]+\.[0-9]+ ]]; then
# Direct tag name like "v1.0.0" or "1.0.0-alpha.1"
version="$head_branch"
elif [[ "$head_branch" == "main" ]]; then
# Regular branch push to main
build_type="development"
version="dev-${short_sha}"
should_build=false
echo "⏭️ Skipping Docker build for development version (main branch push)"
else
# Other branch push
build_type="development"
version="dev-${short_sha}"
should_build=false
echo "⏭️ Skipping Docker build for development version (branch: $head_branch)"
fi
# If we extracted a version (tag), determine release type
if [[ -n "$version" ]] && [[ "$version" != "dev-${short_sha}" ]]; then
# Remove 'v' prefix if present for consistent version format
if [[ "$version" == v* ]]; then
version="${version#v}"
fi
if [[ "$version" =~ -preview\.[0-9]+$ ]]; then
build_type="preview"
is_prerelease=true
echo "🔍 Building Docker image for preview: $version (version tags only)"
elif [[ "$version" == *"-preview"* ]]; then
echo "❌ Invalid preview tag: $version (expected suffix: -preview.<number>)" >&2
exit 1
elif [[ "$version" == *"alpha"* ]] || [[ "$version" == *"beta"* ]] || [[ "$version" == *"rc"* ]]; then
build_type="prerelease"
is_prerelease=true
# Pre-GA policy: prereleases update latest until the first stable tag exists.
if prerelease_moves_latest; then
create_latest=true
echo "🧪 Building Docker image for prerelease: $version (creating latest tag)"
else
echo "🧪 Building Docker image for prerelease: $version"
fi
else
build_type="release"
create_latest=true
echo "🚀 Building Docker image for release: $version"
fi
fi
else
# Non-push events
build_type="development"
version="dev-${short_sha}"
should_build=false
echo "⏭️ Skipping Docker build for development version (event: $triggering_event)"
fi
echo "🔄 Build triggered by workflow_run:"
echo " 📋 Conclusion: $CONCLUSION"
echo " 🌿 Branch: $HEAD_BRANCH"
echo " 📎 SHA: $HEAD_SHA"
echo " 🎯 Event: $TRIGGERING_EVENT"
elif [[ "${{ github.event_name }}" == "workflow_dispatch" ]]; then
# Manual trigger
input_version="$INPUT_VERSION"
version="${input_version}"
should_push="$INPUT_PUSH_IMAGES"
should_build=true
# Get short SHA
short_sha=$(git rev-parse --short HEAD)
echo "🎯 Manual Docker build triggered:"
echo " 📋 Requested version: $input_version"
echo " 🔧 Force rebuild: $INPUT_FORCE_REBUILD"
echo " 🚀 Push images: $should_push"
case "$input_version" in
"latest")
build_type="release"
create_latest=true
echo "🚀 Building with latest stable release version"
;;
*-preview*)
if [[ ! "$input_version" =~ ^v?[0-9]+\.[0-9]+\.[0-9]+(-[0-9A-Za-z.-]+)?-preview\.[0-9]+$ ]]; then
echo "❌ Invalid preview version: $input_version" >&2
exit 1
fi
build_type="preview"
is_prerelease=true
version="${input_version#v}"
echo "🔍 Building with preview version: $version (version tags only)"
;;
# Prerelease versions (must match first, more specific)
v*alpha*|v*beta*|v*rc*|*alpha*|*beta*|*rc*)
build_type="prerelease"
is_prerelease=true
# Pre-GA policy: prereleases update latest until the first stable tag exists.
if prerelease_moves_latest; then
create_latest=true
echo "🧪 Building with prerelease version: $input_version (creating latest tag)"
else
echo "🧪 Building with prerelease version: $input_version"
fi
;;
# Release versions (match after prereleases, more general)
v[0-9]*|[0-9]*.*.*)
build_type="release"
create_latest=true
echo "📦 Building with specific release version: $input_version"
;;
*)
# Invalid version for Docker build
should_build=false
echo "❌ Invalid version for Docker build: $input_version"
echo "⚠️ Only release versions (latest, v1.0.0, 1.0.0) and prereleases (v1.0.0-alpha1, 1.0.0-beta2) are supported"
;;
esac
if [[ "$should_build" == true && "$input_version" != "latest" ]]; then
tag_ref="refs/tags/$input_version"
if ! git ls-remote --exit-code origin "$tag_ref" >/dev/null 2>&1; then
if [[ "$input_version" == v* ]]; then
tag_ref="refs/tags/${input_version#v}"
else
tag_ref="refs/tags/v$input_version"
fi
fi
if ! git ls-remote --exit-code origin "$tag_ref" >/dev/null 2>&1; then
echo "❌ Release tag not found for Docker build: $input_version"
exit 1
fi
source_ref="$tag_ref"
fi
fi
{
echo "should_build=$should_build"
echo "should_push=$should_push"
echo "build_type=$build_type"
echo "version=$version"
echo "short_sha=$short_sha"
echo "is_prerelease=$is_prerelease"
echo "create_latest=$create_latest"
echo "source_ref=$source_ref"
} >> "$GITHUB_OUTPUT"
echo "🐳 Docker Build Summary:"
echo " - Should build: $should_build"
echo " - Should push: $should_push"
echo " - Build type: $build_type"
echo " - Version: $version"
echo " - Short SHA: $short_sha"
echo " - Is prerelease: $is_prerelease"
echo " - Create latest: $create_latest"
echo " - Source ref: $source_ref"
# Consume the trusted release certification for the exact version this
# workflow wraps and verify the certified input digests. A manual dispatch
# without that certification fails here and pushes no official tag; only
# results from the trusted verification sources carry release authority.
release-certification:
name: Release Certification
needs: build-check
if: needs.build-check.outputs.should_build == 'true'
runs-on: ubuntu-latest
timeout-minutes: 30
permissions:
contents: read
actions: read
steps:
- name: Checkout repository
uses: actions/checkout@f548e57e544e1ff5a4c46bf1e1b8685f8e4a348a # v7
with:
persist-credentials: false
ref: ${{ needs.build-check.outputs.source_ref }}
- name: Download and verify release certification
env:
GH_TOKEN: ${{ secrets.RELEASE_VALIDATION_TOKEN || github.token }}
TRIGGERING_RUN_ID: ${{ github.event.workflow_run.id }}
VERSION: ${{ needs.build-check.outputs.version }}
shell: bash
run: |
set -euo pipefail
mkdir -p certification
run_id="$TRIGGERING_RUN_ID"
if [[ -z "$run_id" ]]; then
run_id="$(gh api --method GET \
"repos/${GITHUB_REPOSITORY}/actions/workflows/build.yml/runs" \
-f branch="$VERSION" -f status=success -F per_page=1 \
--jq '.workflow_runs[0].id // empty' || true)"
fi
if [[ -z "$run_id" ]]; then
echo "No certified Build and Release run found for ${VERSION}; failing closed" >&2
exit 1
fi
if ! gh run download "$run_id" -n release-certification -D certification; then
echo "Build run ${run_id} published no release-certification artifact; failing closed" >&2
exit 1
fi
python3 scripts/release/release_gate.py verify-promotion \
--plan .config/release-validation-plan.json \
--candidate certification/candidate.json \
--summary certification/release-gate-output \
--artifact-sha256 "$(jq -r '.deliverables[0].sha256' certification/candidate.json)" \
--artifact-name "$(jq -r '.deliverables[0].name' certification/candidate.json)"
# Input digest binding: every zip this image wraps must be the
# certified byte-for-byte content, never a mutable latest pointer.
while IFS=$'\t' read -r name location expected; do
echo "Verifying certified input ${name}"
actual="$(curl -fsSL "$location" | sha256sum | awk '{print $1}')"
if [[ "$actual" != "$expected" ]]; then
echo "Certified input ${name} does not match its recorded digest" >&2
exit 1
fi
done < <(jq -r '.deliverables[] | select(.name | contains("linux")) | [.name, .location, .sha256] | @tsv' certification/candidate.json)
# Build multi-arch Docker images
# Strategy: Build images using pre-built binaries from dl.rustfs.com
# Supports both release and dev channel binaries based on build context
# Only runs when should_build is true (which includes workflow success check)
build-docker:
name: Build Docker Images
needs: [ build-check, release-certification ]
if: needs.build-check.outputs.should_build == 'true'
runs-on: ubuntu-latest
timeout-minutes: 60
timeout-minutes: 90
strategy:
fail-fast: false
matrix:
include:
- variant: musl
file: Dockerfile
suffix: ""
- variant: glibc
- variant: gnu
file: Dockerfile.glibc
suffix: "-glibc"
env:
CANDIDATE_MANIFEST: ${{ inputs.candidate_manifest }}
VARIANT: ${{ matrix.variant }}
steps:
- name: Checkout repository
uses: actions/checkout@f548e57e544e1ff5a4c46bf1e1b8685f8e4a348a # v7
- uses: actions/checkout@f548e57e544e1ff5a4c46bf1e1b8685f8e4a348a # v7
with:
ref: ${{ fromJson(inputs.candidate_manifest).source_sha }}
persist-credentials: false
ref: ${{ needs.build-check.outputs.source_ref }}
- name: Login to Docker Hub
uses: docker/login-action@c94ce9fb468520275223c153574b00df6fe4bcc9 # v3
with:
username: ${{ env.DOCKERHUB_USERNAME }}
password: ${{ secrets.DOCKERHUB_TOKEN }}
- name: Login to GitHub Container Registry
uses: docker/login-action@c94ce9fb468520275223c153574b00df6fe4bcc9 # v3
with:
registry: ghcr.io
username: ${{ secrets.GHCR_USERNAME }}
password: ${{ secrets.GHCR_PASSWORD }}
- name: Login to Quay.io
uses: docker/login-action@c94ce9fb468520275223c153574b00df6fe4bcc9 # v3
with:
registry: quay.io
username: ${{ secrets.QUAY_USERNAME }}
password: ${{ secrets.QUAY_PASSWORD }}
- name: Set up QEMU
uses: docker/setup-qemu-action@c7c53464625b32c7a7e944ae62b3e17d2b600130 # v3
- name: Set up Docker Buildx
uses: docker/setup-buildx-action@8d2750c68a42422c14e847fe6c8ac0403b4cbd6f # v3
- name: Extract metadata and generate tags
id: meta
- name: Reuse completed image
id: reuse
env:
GH_TOKEN: ${{ github.token }}
run: python3 scripts/release/deliver.py reuse --kind image --target "$VARIANT"
- name: Install OCI tools
if: steps.reuse.outputs.reused != 'true'
run: |
BUILD_TYPE="${{ needs.build-check.outputs.build_type }}"
VERSION="${{ needs.build-check.outputs.version }}"
CREATE_LATEST="${{ needs.build-check.outputs.create_latest }}"
VARIANT_SUFFIX="${{ matrix.suffix }}"
# Convert version format for Dockerfile compatibility. The former
# DOCKER_CHANNEL was "release" down every branch and was passed as a
# build-arg no Dockerfile declares, so it is gone.
case "$VERSION" in
"latest")
DOCKER_RELEASE="latest"
;;
v*)
# For versioned releases (v1.0.0), remove 'v' prefix for Dockerfile
DOCKER_RELEASE="${VERSION#v}"
;;
*)
# For other versions, pass as-is
DOCKER_RELEASE="${VERSION}"
;;
esac
echo "docker_release=$DOCKER_RELEASE" >> "$GITHUB_OUTPUT"
echo "🐳 Docker build parameters:"
echo " - Original version: $VERSION"
echo " - Docker RELEASE: $DOCKER_RELEASE"
# Generate tags based on build type
# Only support release and prerelease builds (no development builds)
TAG_BASE="${VERSION}${VARIANT_SUFFIX}"
TAGS="${{ env.REGISTRY_DOCKERHUB }}:$TAG_BASE,${{ env.REGISTRY_GHCR }}:$TAG_BASE,${{ env.REGISTRY_QUAY }}:$TAG_BASE"
# Add latest when requested (stable releases, and prereleases before GA)
if [[ "$CREATE_LATEST" == "true" ]]; then
TAGS="$TAGS,${{ env.REGISTRY_DOCKERHUB }}:latest${VARIANT_SUFFIX},${{ env.REGISTRY_GHCR }}:latest${VARIANT_SUFFIX},${{ env.REGISTRY_QUAY }}:latest${VARIANT_SUFFIX}"
fi
# Always add the channel tag for prereleases, independent of latest
if [[ "$BUILD_TYPE" == "prerelease" ]]; then
# Prerelease channel tags (alpha, beta, rc)
if [[ "$VERSION" == *"alpha"* ]]; then
CHANNEL="alpha"
elif [[ "$VERSION" == *"beta"* ]]; then
CHANNEL="beta"
elif [[ "$VERSION" == *"rc"* ]]; then
CHANNEL="rc"
fi
if [[ -n "$CHANNEL" ]]; then
TAGS="$TAGS,${{ env.REGISTRY_DOCKERHUB }}:${CHANNEL}${VARIANT_SUFFIX},${{ env.REGISTRY_GHCR }}:${CHANNEL}${VARIANT_SUFFIX},${{ env.REGISTRY_QUAY }}:${CHANNEL}${VARIANT_SUFFIX}"
fi
fi
# Output tags
echo "tags=$TAGS" >> "$GITHUB_OUTPUT"
# Staging identity for this run's promotion: official tags are only
# created after the staged image verifies against the certification,
# and they point at this same digest (no rebuild-after-verify).
STAGING_TAG="staging-${GITHUB_RUN_ID}-${GITHUB_RUN_ATTEMPT}${VARIANT_SUFFIX}"
STAGING_TAGS="${{ env.REGISTRY_DOCKERHUB }}:${STAGING_TAG},${{ env.REGISTRY_GHCR }}:${STAGING_TAG},${{ env.REGISTRY_QUAY }}:${STAGING_TAG}"
echo "staging_tags=$STAGING_TAGS" >> "$GITHUB_OUTPUT"
echo "staging_tag=$STAGING_TAG" >> "$GITHUB_OUTPUT"
# Generate labels
LABELS="org.opencontainers.image.title=RustFS"
LABELS="$LABELS,org.opencontainers.image.description=RustFS distributed object storage system"
LABELS="$LABELS,org.opencontainers.image.version=$VERSION"
SOURCE_REVISION="$(git rev-parse HEAD)"
LABELS="$LABELS,org.opencontainers.image.revision=$SOURCE_REVISION"
LABELS="$LABELS,org.opencontainers.image.source=${{ github.server_url }}/${{ github.repository }}"
LABELS="$LABELS,org.opencontainers.image.created=$(date -u +'%Y-%m-%dT%H:%M:%SZ')"
LABELS="$LABELS,org.opencontainers.image.build-type=$BUILD_TYPE"
echo "labels=$LABELS" >> "$GITHUB_OUTPUT"
echo "🐳 Generated Docker tags:"
echo "$TAGS" | tr ',' '\n' | sed 's/^/ - /'
echo "📋 Build type: $BUILD_TYPE"
echo "🔖 Version: $VERSION"
- name: Build and push Docker image
sudo apt-get update
sudo apt-get install -y skopeo
- uses: docker/setup-qemu-action@c7c53464625b32c7a7e944ae62b3e17d2b600130 # v3
if: steps.reuse.outputs.reused != 'true'
- uses: docker/setup-buildx-action@8d2750c68a42422c14e847fe6c8ac0403b4cbd6f # v3
if: steps.reuse.outputs.reused != 'true'
- name: Supply accepted binaries and pinned CLI
if: steps.reuse.outputs.reused != 'true'
env:
GH_TOKEN: ${{ github.token }}
run: python3 scripts/release/deliver.py image-prepare --target "$VARIANT"
- name: Build OCI archive without registry writes
if: steps.reuse.outputs.reused != 'true'
uses: docker/build-push-action@10e90e3645eae34f1e60eeb005ba3a3d33f178e8 # v6
with:
context: .
file: ${{ matrix.file }}
platforms: ${{ env.DOCKER_PLATFORMS }}
push: ${{ needs.build-check.outputs.should_push == 'true' }}
# Push to staging tags only: official tags are created by the
# certification-gated promotion step below from this same digest.
tags: ${{ steps.meta.outputs.staging_tags }}
labels: ${{ steps.meta.outputs.labels }}
# No layer cache. This build compiles nothing — it downloads a
# release zip and runs apk/apt — so the cache could only save the
# minute or two those take, while creating a correctness problem: with
# RELEASE=latest the binary URL is resolved by curl *inside* a RUN
# layer, and the layer key does not include what that resolved to. A
# rebuild at the same RELEASE value (dispatch with version=latest, or
# a re-run of the same version) would hit the old layer and ship the
# previous release's binary. mode=max also consumed the same 10GB
# Actions cache quota the Rust lanes are fighting over.
#
# Only RELEASE is passed: it is the sole build-arg the Dockerfiles
# declare besides TARGETARCH. BUILDTIME, VERSION, BUILD_TYPE, REVISION
# and CHANNEL were never read by any stage (and BUILDTIME's $(date ...)
# was a literal here, not a shell substitution). BUILD_DATE and VCS_REF
# are declared by the Dockerfiles but deliberately left unset —
# supplying them would change the published image labels.
platforms: linux/amd64,linux/arm64
push: false
provenance: false
outputs: type=oci,dest=delivery/image-${{ matrix.variant }}.tar
build-args: |
RELEASE=${{ steps.meta.outputs.docker_release }}
provenance: true
sbom: true
# Add retry mechanism by splitting the build process
no-cache: false
pull: true
# A later-built wrapper: the staged image must complete this platform's
# own binary identity verification before any official tag is written.
# It may not inherit an unverified package's conclusion.
- name: Verify staged image against certification
if: needs.build-check.outputs.should_push == 'true'
env:
GH_TOKEN: ${{ secrets.RELEASE_VALIDATION_TOKEN || github.token }}
TRIGGERING_RUN_ID: ${{ github.event.workflow_run.id }}
VERSION: ${{ needs.build-check.outputs.version }}
shell: bash
run: |
set -euo pipefail
mkdir -p certification
run_id="$TRIGGERING_RUN_ID"
if [[ -z "$run_id" ]]; then
run_id="$(gh api --method GET \
"repos/${GITHUB_REPOSITORY}/actions/workflows/build.yml/runs" \
-f branch="$VERSION" -f status=success -F per_page=1 \
--jq '.workflow_runs[0].id // empty' || true)"
fi
gh run download "$run_id" -n release-certification -D certification
STAGING_REF="${{ env.REGISTRY_DOCKERHUB }}:staging-${GITHUB_RUN_ID}-${GITHUB_RUN_ATTEMPT}${{ matrix.suffix }}"
case "${{ matrix.variant }}" in
musl) libc="musl" ;;
glibc) libc="gnu" ;;
esac
for platform in linux/amd64 linux/arm64; do
case "$platform" in
linux/amd64) arch_sub="x86_64" ;;
linux/arm64) arch_sub="aarch64" ;;
esac
docker pull --platform "$platform" "$STAGING_REF"
container="$(docker create --platform "$platform" "$STAGING_REF")"
docker cp "$container:/usr/bin/rustfs" "$RUNNER_TEMP/staged-rustfs"
docker rm "$container" >/dev/null
actual="$(sha256sum "$RUNNER_TEMP/staged-rustfs" | awk '{print $1}')"
expected="$(jq -r --arg needle "linux-${arch_sub}-${libc}" \
'[.deliverables[] | select(.name | contains($needle)) | .binary_sha256][0] // empty' \
certification/candidate.json)"
if [[ -z "$expected" || "$actual" != "$expected" ]]; then
echo "Staged image binary for ${platform} does not match the certified binary digest" >&2
exit 1
fi
python3 scripts/release/release_gate.py verify-promotion \
--plan .config/release-validation-plan.json \
--candidate certification/candidate.json \
--summary certification/release-gate-output \
--artifact "$RUNNER_TEMP/staged-rustfs" \
--artifact-role binary
done
- name: Promote staged image to official tags
if: needs.build-check.outputs.should_push == 'true'
shell: bash
run: |
set -euo pipefail
STAGING_REF="${{ env.REGISTRY_DOCKERHUB }}:staging-${GITHUB_RUN_ID}-${GITHUB_RUN_ATTEMPT}${{ matrix.suffix }}"
IFS=',' read -r -a official_tags <<< "${{ steps.meta.outputs.tags }}"
for tag in "${official_tags[@]}"; do
docker buildx imagetools create -t "$tag" "$STAGING_REF"
echo "Promoted $tag -> $STAGING_REF"
done
# Downstream consumers (Helm) verify this record instead of inheriting
# an unverified wrapper's conclusion.
- name: Record image promotion
if: needs.build-check.outputs.should_push == 'true'
shell: bash
env:
GH_TOKEN: ${{ secrets.RELEASE_VALIDATION_TOKEN || github.token }}
TRIGGERING_RUN_ID: ${{ github.event.workflow_run.id }}
VERSION: ${{ needs.build-check.outputs.version }}
run: |
set -euo pipefail
mkdir -p certification
run_id="$TRIGGERING_RUN_ID"
if [[ -z "$run_id" ]]; then
run_id="$(gh api --method GET \
"repos/${GITHUB_REPOSITORY}/actions/workflows/build.yml/runs" \
-f branch="$VERSION" -f status=success -F per_page=1 \
--jq '.workflow_runs[0].id // empty' || true)"
fi
gh run download "$run_id" -n release-certification -D certification
STAGING_REF="${{ env.REGISTRY_DOCKERHUB }}:staging-${GITHUB_RUN_ID}-${GITHUB_RUN_ATTEMPT}${{ matrix.suffix }}"
digest="$(docker buildx imagetools inspect "$STAGING_REF" --format '{{json .Manifest}}' | jq -r '.digest')"
candidate_sha="$(python3 -c "import json,hashlib,sys; sys.path.insert(0,'scripts/release'); import release_gate; print(release_gate.canonical_sha256(json.load(open('certification/candidate.json'))))")"
jq -n \
--arg version "$VERSION" \
--arg digest "$digest" \
--arg staging_ref "$STAGING_REF" \
--arg candidate_sha256 "$candidate_sha" \
--arg tags "${{ steps.meta.outputs.tags }}" \
--slurpfile candidate certification/candidate.json \
'{schema_version: 1, version: $version, image_digest: $digest, staging_ref: $staging_ref,
promoted_tags: ($tags | split(",")), candidate_sha256: $candidate_sha256,
binary_sha256s: ($candidate[0].deliverables | map(.binary_sha256) | unique)}' \
> image-promotion-record.json
- name: Upload image promotion record
if: needs.build-check.outputs.should_push == 'true'
uses: actions/upload-artifact@043fb46d1a93c77aae656e7c1c64a875d1fc6a0a # v7.0.1
with:
name: image-promotion-record${{ matrix.suffix }}
path: image-promotion-record.json
if-no-files-found: error
retention-days: 30
# Note: Manifest creation is no longer needed as we only build one variant
# Multi-arch manifests are automatically created by docker/build-push-action
# Report-only container image vulnerability scan
scan-docker-image:
name: Scan Docker Images
needs: [ build-check, build-docker ]
if: needs.build-check.outputs.should_build == 'true' && needs.build-check.outputs.should_push == 'true'
runs-on: ubuntu-latest
timeout-minutes: 30
permissions:
contents: read
security-events: write
strategy:
fail-fast: false
matrix:
include:
- variant: musl
suffix: ""
- variant: glibc
suffix: "-glibc"
steps:
- name: Login to GitHub Container Registry
# docker/login-action v3
uses: docker/login-action@c94ce9fb468520275223c153574b00df6fe4bcc9
with:
registry: ghcr.io
username: ${{ secrets.GHCR_USERNAME }}
password: ${{ secrets.GHCR_PASSWORD }}
- name: Scan image with Trivy
# aquasecurity/trivy-action v0.36.0
BINARY_SOURCE=supplied
RELEASE=${{ fromJson(inputs.candidate_manifest).version }}
VCS_REF=${{ fromJson(inputs.candidate_manifest).source_sha }}
- name: Verify wrapped binaries and startup on both architectures
if: steps.reuse.outputs.reused != 'true'
run: python3 scripts/release/deliver.py image-smoke --target "$VARIANT"
- name: Scan amd64 image before sealing
if: steps.reuse.outputs.reused != 'true'
uses: aquasecurity/trivy-action@ed142fd0673e97e23eac54620cfb913e5ce36c25
env:
TRIVY_PLATFORM: linux/amd64
with:
image-ref: ${{ env.REGISTRY_GHCR }}:${{ needs.build-check.outputs.version }}${{ matrix.suffix }}
input: delivery-scan/image-${{ matrix.variant }}-amd64.tar
format: sarif
output: trivy-${{ matrix.variant }}.sarif
ignore-unfixed: true
vuln-type: os,library
severity: CRITICAL,HIGH
output: delivery/scan-${{ matrix.variant }}-amd64.sarif
exit-code: "0" # Keep the repository's report-only vulnerability policy.
- name: Scan arm64 image before sealing
if: steps.reuse.outputs.reused != 'true'
uses: aquasecurity/trivy-action@ed142fd0673e97e23eac54620cfb913e5ce36c25
env:
TRIVY_PLATFORM: linux/arm64
with:
input: delivery-scan/image-${{ matrix.variant }}-arm64.tar
format: sarif
output: delivery/scan-${{ matrix.variant }}-arm64.sarif
exit-code: "0"
# Surface findings in the Security tab; an artifact alone is write-only
# reporting nobody reviews.
- name: Upload scan results to code scanning
uses: github/codeql-action/upload-sarif@99df26d4f13ea111d4ec1a7dddef6063f76b97e9 # v4.37.0
- name: Record verified OCI identity
if: steps.reuse.outputs.reused != 'true'
run: python3 scripts/release/deliver.py image-record --target "$VARIANT"
- uses: actions/upload-artifact@043fb46d1a93c77aae656e7c1c64a875d1fc6a0a # v7.0.1
if: steps.reuse.outputs.reused != 'true'
with:
sarif_file: trivy-${{ matrix.variant }}.sarif
category: container-image-${{ matrix.variant }}
- name: Upload container scan report
uses: actions/upload-artifact@043fb46d1a93c77aae656e7c1c64a875d1fc6a0a # v7.0.1
with:
name: container-image-scan-${{ matrix.variant }}
path: trivy-${{ matrix.variant }}.sarif
name: image-${{ matrix.variant }}
path: delivery/*
if-no-files-found: error
retention-days: 30
# Docker build summary
docker-summary:
name: Docker Build Summary
needs: [ build-check, build-docker ]
if: always() && needs.build-check.outputs.should_build == 'true'
runs-on: ubuntu-latest
timeout-minutes: 10
steps:
- name: Docker build completion summary
run: |
BUILD_TYPE="${{ needs.build-check.outputs.build_type }}"
VERSION="${{ needs.build-check.outputs.version }}"
CREATE_LATEST="${{ needs.build-check.outputs.create_latest }}"
echo "🐳 Docker build completed successfully!"
echo "📦 Build type: $BUILD_TYPE"
echo "🔢 Version: $VERSION"
echo "🚀 Strategy: Images using pre-built binaries (release channel only)"
echo ""
case "$BUILD_TYPE" in
"preview")
echo "🔍 Preview Docker image has been built with ${VERSION} tags"
echo "⏭️ Preview images do not update latest or prerelease channels"
;;
"release")
echo "🚀 Release Docker image has been built with ${VERSION} tags"
echo "✅ This image is ready for production use"
if [[ "$CREATE_LATEST" == "true" ]]; then
echo "🏷️ Latest tag has been created for stable release"
fi
;;
"prerelease")
echo "🧪 Prerelease Docker image has been built with ${VERSION} tags"
echo "⚠️ This is a prerelease image - use with caution"
# Prereleases move latest until the first stable tag exists (pre-GA policy).
if [[ "$CREATE_LATEST" == "true" ]]; then
echo "🏷️ Latest tag has been created for prerelease: $VERSION"
else
echo "🚫 Latest tag NOT created for prerelease"
fi
;;
*)
echo "❌ Unexpected build type: $BUILD_TYPE"
;;
esac
compression-level: 0
+51 -277
View File
@@ -1,229 +1,26 @@
# Copyright 2024 RustFS Team
#
# Licensed under the Apache License, Version 2.0 (the "License");
# you may not use this file except in compliance with the License.
# You may obtain a copy of the License at
#
# http://www.apache.org/licenses/LICENSE-2.0
#
# Unless required by applicable law or agreed to in writing, software
# distributed under the License is distributed on an "AS IS" BASIS,
# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
# See the License for the specific language governing permissions and
# limitations under the License.
name: Publish helm chart to artifacthub
# Copyright 2026 RustFS Team
# SPDX-License-Identifier: Apache-2.0
name: Build candidate Helm chart
on:
workflow_run:
workflows: [ "Build and Release" ]
types: [ completed ]
workflow_dispatch:
workflow_call:
inputs:
version:
description: "Release version to publish, e.g. 1.0.0-beta.1 or v1.0.0-beta.1"
required: true
default: "1.0.0-beta.1"
candidate_manifest:
type: string
required: true
permissions:
contents: read
actions: read
jobs:
# Consume the same trusted release certification as every other publication
# exit and verify the image promotion record of the Docker workflow. The
# chart wraps no binary of its own, so it may only reference an image whose
# platform verification already passed; it must not inherit an unverified
# package's conclusion. A run without that evidence fails closed here and
# never reaches the chart repository.
release-certification:
package:
runs-on: ubuntu-latest
timeout-minutes: 30
permissions:
contents: read
actions: read
outputs:
version: ${{ steps.version.outputs.raw_tag }}
timeout-minutes: 15
steps:
- name: Checkout repository
uses: actions/checkout@f548e57e544e1ff5a4c46bf1e1b8685f8e4a348a # v7
- uses: actions/checkout@f548e57e544e1ff5a4c46bf1e1b8685f8e4a348a # v7
with:
ref: ${{ fromJson(inputs.candidate_manifest).source_sha }}
persist-credentials: false
- name: Normalize release version
id: version
env:
RAW_INPUT: ${{ github.event.inputs.version }}
RAW_BRANCH: ${{ github.event.workflow_run.head_branch }}
run: |
set -euo pipefail
if [ "${{ github.event_name }}" = "workflow_dispatch" ]; then
RAW="$RAW_INPUT"
else
RAW="$RAW_BRANCH"
fi
case "$RAW" in
refs/tags/*) RAW_TAG="${RAW#refs/tags/}" ;;
*) RAW_TAG="$RAW" ;;
esac
echo "raw_tag=$RAW_TAG" >> "$GITHUB_OUTPUT"
- name: Download and verify release certification and image record
env:
GH_TOKEN: ${{ secrets.RELEASE_VALIDATION_TOKEN || github.token }}
TRIGGERING_RUN_ID: ${{ github.event.workflow_run.id }}
RAW_TAG: ${{ steps.version.outputs.raw_tag }}
shell: bash
run: |
set -euo pipefail
mkdir -p certification image-record
build_run="$TRIGGERING_RUN_ID"
if [[ -z "$build_run" ]]; then
build_run="$(gh api --method GET \
"repos/${GITHUB_REPOSITORY}/actions/workflows/build.yml/runs" \
-f branch="$RAW_TAG" -f status=success -F per_page=1 \
--jq '.workflow_runs[0].id // empty' || true)"
fi
if [[ -z "$build_run" ]]; then
echo "No certified Build and Release run found for ${RAW_TAG}; failing closed" >&2
exit 1
fi
if ! gh run download "$build_run" -n release-certification -D certification; then
echo "Build run ${build_run} published no release-certification artifact; failing closed" >&2
exit 1
fi
python3 scripts/release/release_gate.py verify-promotion \
--plan .config/release-validation-plan.json \
--candidate certification/candidate.json \
--summary certification/release-gate-output \
--artifact-sha256 "$(jq -r '.deliverables[0].sha256' certification/candidate.json)" \
--artifact-name "$(jq -r '.deliverables[0].name' certification/candidate.json)"
docker_run="$(gh api --method GET \
"repos/${GITHUB_REPOSITORY}/actions/workflows/docker.yml/runs" \
-f branch="$RAW_TAG" -f status=success -F per_page=1 \
--jq '.workflow_runs[0].id // empty' || true)"
if [[ -z "$docker_run" ]] || ! gh run download "$docker_run" -n image-promotion-record -D image-record; then
echo "No image promotion record for ${RAW_TAG}; a chart may not wrap an unverified image" >&2
exit 1
fi
candidate_sha="$(python3 -c "import json,hashlib,sys; sys.path.insert(0,'scripts/release'); import release_gate; print(release_gate.canonical_sha256(json.load(open('certification/candidate.json'))))")"
record_sha="$(jq -r '.candidate_sha256 // empty' image-record/image-promotion-record.json)"
if [[ "$record_sha" != "$candidate_sha" ]]; then
echo "Image promotion record does not belong to this certification" >&2
exit 1
fi
overlap="$(jq -r --slurpfile candidate certification/candidate.json \
'[.binary_sha256s[]] - ($candidate[0].deliverables | map(.binary_sha256)) | length' \
image-record/image-promotion-record.json)"
if [[ "$overlap" -ne 0 ]]; then
echo "Image promotion record carries binary digests outside the certification" >&2
exit 1
fi
build-helm-package:
runs-on: ubuntu-latest
timeout-minutes: 30
needs: [ release-certification ]
if: |
(github.event_name == 'workflow_dispatch' && !contains(github.event.inputs.version, '-preview')) ||
(
github.event.workflow_run.conclusion == 'success' &&
github.event.workflow_run.event == 'push' &&
contains(github.event.workflow_run.head_branch, '.') &&
!contains(github.event.workflow_run.head_branch, '-preview')
)
outputs:
raw_tag: ${{ steps.version.outputs.raw_tag }}
app_version: ${{ steps.version.outputs.app_version }}
chart_version: ${{ steps.version.outputs.chart_version }}
steps:
- name: Checkout release source
uses: actions/checkout@f548e57e544e1ff5a4c46bf1e1b8685f8e4a348a # v7
with:
ref: ${{ github.event.workflow_run.head_sha || (startsWith(inputs.version, 'refs/tags/') && inputs.version || format('refs/tags/{0}', inputs.version)) }}
persist-credentials: false
# Both inputs reach the shell through env rather than `${{ }}`
# interpolation. A git ref name may contain `$(...)` — anything without a
# space is a legal tag — and interpolation pastes it into the script
# verbatim, where bash would run it. Reading "$RAW_INPUT" instead makes it
# data.
- name: Normalize release version
id: version
env:
RAW_INPUT: ${{ github.event.inputs.version }}
RAW_BRANCH: ${{ github.event.workflow_run.head_branch }}
run: |
set -eux
if [ "${{ github.event_name }}" = "workflow_dispatch" ]; then
RAW="$RAW_INPUT"
else
RAW="$RAW_BRANCH"
fi
case "$RAW" in
refs/tags/*)
RAW_TAG="${RAW#refs/tags/}"
;;
*)
RAW_TAG="$RAW"
;;
esac
./scripts/helm_chart_version.sh "$RAW_TAG"
# Docker and Helm share the binary-build trigger. Do not publish a chart
# until the image selected by its default appVersion is pullable.
- name: Wait for release image
timeout-minutes: 20
env:
APP_VERSION: ${{ steps.version.outputs.app_version }}
run: |
set -euo pipefail
if [[ ! "$APP_VERSION" =~ ^[0-9]+\.[0-9]+\.[0-9]+(-[0-9A-Za-z.-]+)?$ ]] ||
[[ "$APP_VERSION" == *-preview* ]]; then
echo "Invalid deliverable image version: $APP_VERSION" >&2
exit 1
fi
for attempt in {1..60}; do
if token=$(curl --fail --silent --show-error --connect-timeout 10 --max-time 20 \
'https://auth.docker.io/token?service=registry.docker.io&scope=repository:rustfs/rustfs:pull' \
| jq -er '.token // empty') &&
curl --fail --silent --show-error --head --connect-timeout 10 --max-time 20 \
--header "Authorization: Bearer $token" \
--header 'Accept: application/vnd.oci.image.index.v1+json, application/vnd.docker.distribution.manifest.list.v2+json' \
"https://registry-1.docker.io/v2/rustfs/rustfs/manifests/$APP_VERSION" > /dev/null; then
echo "Release image is available: rustfs/rustfs:$APP_VERSION"
exit 0
fi
echo "Waiting for rustfs/rustfs:$APP_VERSION ($attempt/60)"
if (( attempt < 60 )); then
sleep 15
fi
done
echo "Release image is unavailable; refusing to publish Helm chart" >&2
exit 1
- name: Replace chart version and app version
env:
CHART_VERSION: ${{ steps.version.outputs.chart_version }}
APP_VERSION: ${{ steps.version.outputs.app_version }}
run: |
set -eux
sed -i -E "s/^version:.*/version: \"${CHART_VERSION}\"/" helm/rustfs/Chart.yaml
sed -i -E "s/^appVersion:.*/appVersion: \"${APP_VERSION}\"/" helm/rustfs/Chart.yaml
- name: Set up Helm
uses: azure/setup-helm@b9e51907a09c216f16ebe8536097933489208112 # v4.3.0
- name: Install yq
- uses: azure/setup-helm@b9e51907a09c216f16ebe8536097933489208112 # v4.3.0
- name: Install pinned yq
run: |
set -euo pipefail
yq_version="4.54.1"
@@ -238,66 +35,43 @@ jobs:
echo "$tools_dir" >> "$GITHUB_PATH"
"$tools_dir/yq" --version
- name: Test Helm Chart Templates
run: ./scripts/test_helm_templates.sh
- name: Package Helm Chart
run: |
set -eux
cp helm/README.md helm/rustfs/
helm package ./helm/rustfs \
--destination helm/rustfs/ \
--version "${{ steps.version.outputs.chart_version }}"
- name: Upload helm package as artifact
uses: actions/upload-artifact@043fb46d1a93c77aae656e7c1c64a875d1fc6a0a # v7.0.1
with:
name: helm-package
path: |
helm/rustfs/*.tgz
helm/rustfs/README.md
retention-days: 1
publish-helm-package:
runs-on: ubuntu-latest
timeout-minutes: 30
needs: [ build-helm-package ]
if: needs.build-helm-package.result == 'success'
steps:
- name: Checkout helm package repo
uses: actions/checkout@f548e57e544e1ff5a4c46bf1e1b8685f8e4a348a # v7
with:
# persist-credentials-exempt: this checkout's token IS the push credential —
# the job git-pushes to rustfs/helm below. Clearing it breaks chart publishing.
repository: rustfs/helm
token: ${{ secrets.RUSTFS_HELM_PACKAGE }}
- name: Download helm package
uses: actions/download-artifact@3e5f45b2cfb9172054b4087a40e8e0b5a5461e7c # v8.0.1
with:
name: helm-package
path: ./
- name: Set up helm
uses: azure/setup-helm@b9e51907a09c216f16ebe8536097933489208112 # v4.3.0
- name: Generate index
run: helm repo index . --url https://charts.rustfs.com
# app_version is derived from the triggering tag name, and this job holds
# the cross-repository push token with rustfs/helm already checked out —
# the worst place in the repo to paste an attacker-influenced string into
# a shell line. Passed through env so bash treats it as data.
- name: Push helm package and index file
- name: Test and package the exact application version
id: chart
env:
GIT_USERNAME: ${{ secrets.USERNAME }}
GIT_EMAIL: ${{ secrets.EMAIL_ADDRESS }}
APP_VERSION: ${{ needs.build-helm-package.outputs.app_version }}
GH_TOKEN: ${{ github.token }}
CANDIDATE_MANIFEST: ${{ inputs.candidate_manifest }}
run: |
set -eux
git config --global user.name "${GIT_USERNAME}"
git config --global user.email "${GIT_EMAIL}"
git add .
git commit -m "Update rustfs helm package with ${APP_VERSION}." || echo "No changes to commit"
git push origin main
python3 - <<'PYCHART'
import sys, subprocess, shutil
from pathlib import Path
sys.path.insert(0, 'scripts/release')
import candidate as c, pipeline
inputs = pipeline.manifest_env()
current = pipeline.current_run('release-complete.yml')
if any(a['name'] == 'helm-package' for a in c.artifact_list(current['id'])):
c.download_artifact(current, 'helm-package', 'chart')
record = c.read_json('chart/chart.json')
c.require(record['inputs_sha256'] == c.digest(inputs), 'chart belongs to another candidate')
c.emit_output('reused', 'true')
else:
chart = Path('helm/rustfs/Chart.yaml')
import re
value = chart.read_text()
for key in ('version', 'appVersion'):
value, count = re.subn(rf'(?m)^{key}:.*$', f'{key}: "{inputs["version"]}"', value)
c.require(count == 1, 'ambiguous chart metadata')
chart.write_text(value)
subprocess.run(['bash', 'scripts/test_helm_templates.sh'], check=True)
shutil.copyfile('helm/README.md', 'helm/rustfs/README.md')
subprocess.run(['helm', 'package', 'helm/rustfs', '--destination', 'chart'], check=True)
path = Path('chart') / f"rustfs-{inputs['version']}.tgz"
c.write_json('chart/chart.json', {'version': inputs['version'], 'inputs_sha256': c.digest(inputs),
'name': path.name, 'sha256': c.file_digest(path), 'size': path.stat().st_size})
PYCHART
- uses: actions/upload-artifact@043fb46d1a93c77aae656e7c1c64a875d1fc6a0a # v7.0.1
if: steps.chart.outputs.reused != 'true'
with:
name: helm-package
path: chart/*
if-no-files-found: error
retention-days: 30
+415
View File
@@ -0,0 +1,415 @@
# Copyright 2026 RustFS Team
# SPDX-License-Identifier: Apache-2.0
# Manual packaging of the current origin/main tip. Workflow artifacts only.
# Dispatch uses this file from the selected branch. The build fetches
# refs/heads/main and checks out that commit, not the dispatch ref or a tag.
name: Manual main packages
on:
workflow_dispatch:
inputs:
architecture:
description: "Linux targets: x86 (x86_64 musl+gnu), arm (aarch64 musl+gnu), or all"
required: true
type: choice
options:
- x86
- arm
- all
default: all
version:
description: "Binary and package version: MAJOR.MINOR.PATCH, or MAJOR.MINOR.PATCH-(alpha|beta|rc).N. No leading v."
required: true
type: string
permissions:
contents: read
env:
CARGO_TERM_COLOR: always
RUST_BACKTRACE: 1
CARGO_INCREMENTAL: 0
jobs:
prepare:
name: Resolve main tip and version
if: github.repository == 'rustfs/rustfs'
runs-on: ubuntu-latest
timeout-minutes: 15
outputs:
source_sha: ${{ steps.pin.outputs.source_sha }}
version: ${{ steps.version.outputs.version }}
toolchain: ${{ steps.inputs.outputs.toolchain }}
console: ${{ steps.inputs.outputs.console }}
matrix: ${{ steps.matrix.outputs.matrix }}
steps:
- name: Checkout main
uses: actions/checkout@f548e57e544e1ff5a4c46bf1e1b8685f8e4a348a # v7
with:
ref: main
fetch-depth: 0
persist-credentials: false
- name: Pin the tip of origin/main
id: pin
shell: bash
run: |
set -euo pipefail
GIT_TERMINAL_PROMPT=0 git fetch --no-tags origin refs/heads/main
sha=$(git rev-parse FETCH_HEAD)
git checkout --detach "$sha"
test "$(git rev-parse HEAD)" = "$sha"
printf 'source_sha=%s\n' "$sha" >> "$GITHUB_OUTPUT"
printf 'Pinned origin/main %s\n' "$sha"
- name: Validate package version
id: version
shell: bash
env:
PACKAGE_VERSION: ${{ inputs.version }}
run: |
set -euo pipefail
bash scripts/release/package_manual_artifacts.sh validate \
--version "$PACKAGE_VERSION" \
--arch x86_64 \
--libc gnu | tee "$RUNNER_TEMP/package-version.txt"
printf 'version=%s\n' "$PACKAGE_VERSION" >> "$GITHUB_OUTPUT"
- name: Install stable Rust
uses: dtolnay/rust-toolchain@29eef336d9b2848a0b548edc03f92a220660cdb8 # stable
with:
toolchain: stable
- name: Resolve console assets and toolchain
id: inputs
shell: bash
env:
GH_TOKEN: ${{ github.token }}
run: |
set -euo pipefail
toolchain=$(rustc --version | awk '{print $2}')
[[ "$toolchain" =~ ^[0-9]+\.[0-9]+\.[0-9]+$ ]]
gh api repos/rustfs/console/releases/latest > console-release.json
TOOLCHAIN="$toolchain" python3 - <<'PY'
import json, os, sys
sys.path.insert(0, "scripts/release")
import candidate
console = candidate.console_asset(candidate.read_json("console-release.json"))
candidate.emit_output("console", json.dumps(console, separators=(",", ":")))
candidate.emit_output("toolchain", os.environ["TOOLCHAIN"])
print(f"toolchain={os.environ['TOOLCHAIN']}")
print(f"console={console['tag']} {console['name']}")
PY
- name: Select Linux targets
id: matrix
shell: bash
env:
ARCHITECTURE: ${{ inputs.architecture }}
run: |
set -euo pipefail
matrix=$(bash scripts/release/package_manual_artifacts.sh matrix "$ARCHITECTURE")
count=$(python3 -c 'import json,sys; print(len(json.loads(sys.argv[1])["include"]))' "$matrix")
[[ "$count" -gt 0 ]]
printf 'matrix=%s\n' "$matrix" >> "$GITHUB_OUTPUT"
printf 'Selected %s target(s) for %s\n' "$count" "$ARCHITECTURE"
package:
name: Package RustFS (${{ matrix.target_id }})
needs: prepare
if: github.repository == 'rustfs/rustfs'
# x86_64 uses sm-standard-4, the official release compile pool: the workspace
# release build needs that disk (placement rule: disk) and reads build.yml's
# cache without writing it. aarch64 uses ubicloud-standard-16-arm so the
# packaged binary and console smoke execute natively (placement rule: hardware).
runs-on: ${{ matrix.os }}
timeout-minutes: 210
env:
FORCE_JAVASCRIPT_ACTIONS_TO_NODE24: "true"
RUSTUP_TOOLCHAIN: ${{ needs.prepare.outputs.toolchain }}
RUSTFLAGS: "${{ matrix.rustflags }}"
strategy:
fail-fast: false
matrix: ${{ fromJson(needs.prepare.outputs.matrix) }}
steps:
- name: Checkout the pinned origin/main commit
uses: actions/checkout@f548e57e544e1ff5a4c46bf1e1b8685f8e4a348a # v7
with:
ref: ${{ needs.prepare.outputs.source_sha }}
fetch-depth: 0
persist-credentials: false
- name: Confirm the checkout is that main commit
shell: bash
env:
SOURCE_SHA: ${{ needs.prepare.outputs.source_sha }}
run: |
set -euo pipefail
test "$(git rev-parse HEAD)" = "$SOURCE_SHA"
GIT_TERMINAL_PROMPT=0 git fetch --no-tags origin refs/heads/main
git merge-base --is-ancestor "$SOURCE_SHA" FETCH_HEAD
- name: Setup Rust environment
uses: ./.github/actions/setup
with:
rust-version: ${{ needs.prepare.outputs.toolchain }}
target: ${{ matrix.target }}
cache-shared-key: build-${{ matrix.target }}${{ matrix.glibc && format('-glibc-{0}', matrix.glibc) || '' }}${{ matrix.os == 'ubicloud-standard-16-arm' && '-ubicloud-arm' || '' }}
# build.yml owns this key. This job only restores it.
cache-save-if: 'false'
install-cross-tools: ${{ matrix.cross }}
install-build-packaging-tools: 'true'
install-test-tools: 'false'
- name: Download static console assets
shell: bash
env:
GH_TOKEN: ${{ github.token }}
CONSOLE_IDENTITY: ${{ needs.prepare.outputs.console }}
run: |
set -euo pipefail
python3 - <<'PYCONSOLE'
import json, os, pathlib, subprocess, zipfile
value = json.loads(os.environ["CONSOLE_IDENTITY"])
subprocess.run(["python3", ".github/actions/setup/download_release_asset.py",
"--repository", "rustfs/console", "--tag", value["tag"], "--asset", value["name"],
"--sha256", value["sha256"], "--output", "console.zip"], check=True)
destination = pathlib.Path("rustfs/static")
destination.mkdir(parents=True, exist_ok=True)
with zipfile.ZipFile("console.zip") as archive:
for member in archive.infolist():
path = pathlib.PurePosixPath(member.filename)
if path.is_absolute() or ".." in path.parts or "\\" in member.filename:
raise SystemExit("unsafe Console archive member")
archive.extractall(destination)
if not (destination / "index.html").is_file():
raise SystemExit("Console archive has no index.html")
PYCONSOLE
rm -f console.zip
- name: Build RustFS
shell: bash
env:
RUSTFS_BUILD_VERSION: ${{ needs.prepare.outputs.version }}
BUILD_TARGET: ${{ matrix.target }}
BUILD_OS: ${{ matrix.build_os }}
BUILD_FEATURES: ${{ matrix.features }}
BUILD_GLIBC: ${{ matrix.glibc }}
BUILD_CROSS: ${{ matrix.cross }}
run: |
set -euo pipefail
touch rustfs/build.rs
feature_args=()
if [[ -n "$BUILD_FEATURES" ]]; then
feature_args=(--features "$BUILD_FEATURES")
fi
# gnu targets pin glibc 2.31 through cargo-zigbuild (<target>.<glibc>).
target_arg="$BUILD_TARGET"
if [[ -n "$BUILD_GLIBC" ]]; then
target_arg="${target_arg}.${BUILD_GLIBC}"
fi
profile_args=()
if [[ "$BUILD_TARGET" == *-linux-gnu && "$BUILD_FEATURES" == pyroscope ]]; then
profile_args=(--config 'profile.release.strip="none"')
fi
case "$BUILD_CROSS" in
true)
cargo zigbuild --release --locked --target "$target_arg" "${feature_args[@]}" "${profile_args[@]}" -p rustfs --bin rustfs
;;
false)
cargo build --release --locked --target "$target_arg" "${feature_args[@]}" "${profile_args[@]}" -p rustfs --bin rustfs
;;
*)
printf 'unexpected cross flag: %q\n' "$BUILD_CROSS" >&2
exit 1
;;
esac
bin="target/${BUILD_TARGET}/release/rustfs"
version_output=$("$bin" --version)
first=${version_output%%$'\n'*}
if [[ "$first" != "rustfs ${RUSTFS_BUILD_VERSION}" ]]; then
printf 'binary version line is %q, expected %q\n' "$first" "rustfs ${RUSTFS_BUILD_VERSION}" >&2
printf '%s\n' "$version_output" >&2
exit 1
fi
if ! grep -Fxq "build target : ${BUILD_TARGET}" <<< "$version_output" || \
! grep -Fxq "build os : ${BUILD_OS}" <<< "$version_output" || \
! grep -Fxq "rust channel : stable-${BUILD_TARGET}" <<< "$version_output"; then
printf 'release binary reports an unexpected compile target\n' >&2
printf '%s\n' "$version_output" >&2
exit 1
fi
if [[ -n "$BUILD_GLIBC" ]]; then
python3 scripts/release/check_glibc.py "$BUILD_GLIBC" "$bin"
fi
- name: Generate the CPU symbol catalog
if: endsWith(matrix.target, '-linux-gnu') && matrix.features == 'pyroscope'
shell: bash
env:
BUILD_TARGET: ${{ matrix.target }}
run: |
set -euo pipefail
cargo metadata --locked --format-version 1 --filter-platform "$BUILD_TARGET" \
--features rustfs/pyroscope > "$RUNNER_TEMP/cpu-catalog-metadata.json"
features=$(jq -ce '
(.packages[] | select(.name == "rustfs") | .id) as $id |
.resolve.nodes[] | select(.id == $id) | .features | map(select(. != "default")) | sort
' "$RUNNER_TEMP/cpu-catalog-metadata.json")
release_dir="target/${BUILD_TARGET}/release"
rm -f "$release_dir/rustfs.cpu-symbol-catalog.json" "$release_dir/rustfs.cpu-symbol-catalog.sha256"
cargo run --locked -p rustfs-connect-symbol-catalog -- \
"$release_dir/rustfs" "$(git rev-parse HEAD)" "$BUILD_TARGET" "$features" \
"$release_dir/rustfs.cpu-symbol-catalog.json" > "$release_dir/rustfs.cpu-symbol-catalog.sha256"
- name: Install DEB and RPM tools
shell: bash
run: |
set -euo pipefail
sudo apt-get update
sudo apt-get install -y ruby ruby-dev build-essential rpm fakeroot cpio
sudo gem install fpm --version 1.17.0 --no-document
- name: Package zip, deb, and rpm
shell: bash
env:
PACKAGE_VERSION: ${{ needs.prepare.outputs.version }}
ARCH: ${{ matrix.arch }}
LIBC: ${{ matrix.libc }}
BUILD_TARGET: ${{ matrix.target }}
BUILD_FEATURES: ${{ matrix.features }}
SOURCE_SHA: ${{ needs.prepare.outputs.source_sha }}
run: |
set -euo pipefail
binary="target/${BUILD_TARGET}/release/rustfs"
extra=()
if [[ "$BUILD_TARGET" == *-linux-gnu && "$BUILD_FEATURES" == pyroscope ]]; then
extra=(
--zip-extra "target/${BUILD_TARGET}/release/rustfs.cpu-symbol-catalog.json"
--zip-extra "target/${BUILD_TARGET}/release/rustfs.cpu-symbol-catalog.sha256"
)
fi
bash scripts/release/package_manual_artifacts.sh package \
--version "$PACKAGE_VERSION" \
--arch "$ARCH" \
--libc "$LIBC" \
--binary "$binary" \
--output dist \
"${extra[@]}"
printf 'Packaged %s from origin/main %s\n' "$PACKAGE_VERSION" "$SOURCE_SHA"
- name: Verify the packaged binary
shell: bash
env:
PACKAGE_VERSION: ${{ needs.prepare.outputs.version }}
ARCH: ${{ matrix.arch }}
LIBC: ${{ matrix.libc }}
BUILD_TARGET: ${{ matrix.target }}
BUILD_OS: ${{ matrix.build_os }}
BUILD_FEATURES: ${{ matrix.features }}
run: |
set -euo pipefail
stem="rustfs-linux-${ARCH}-${LIBC}-v${PACKAGE_VERSION}"
package_file="dist/${stem}.zip"
[[ -f "$package_file" && -f "dist/${stem}.deb" && -f "dist/${stem}.rpm" ]]
[[ "$(find dist -maxdepth 1 -type f | wc -l)" -eq 3 ]]
archive_members=$(unzip -Z1 "$package_file" | sort)
if [[ "$BUILD_TARGET" == *-linux-gnu && "$BUILD_FEATURES" == pyroscope ]]; then
expected_members=$(printf '%s\n' rustfs rustfs.cpu-symbol-catalog.json rustfs.cpu-symbol-catalog.sha256 | sort)
else
expected_members=$(printf '%s\n' rustfs)
fi
if [[ "$archive_members" != "$expected_members" ]]; then
printf 'Linux package does not contain the expected files\n' >&2
printf '%s\n' "$archive_members" >&2
exit 1
fi
binary_dir=$(mktemp -d)
trap 'rm -rf "$binary_dir"' EXIT
unzip -q "$package_file" rustfs -d "$binary_dir"
version_output=$("$binary_dir/rustfs" --version)
first=${version_output%%$'\n'*}
if [[ "$first" != "rustfs ${PACKAGE_VERSION}" ]]; then
printf 'packaged binary version line is %q, expected %q\n' "$first" "rustfs ${PACKAGE_VERSION}" >&2
exit 1
fi
if ! grep -Fxq "build target : ${BUILD_TARGET}" <<< "$version_output" || \
! grep -Fxq "build os : ${BUILD_OS}" <<< "$version_output"; then
printf 'packaged binary reports an unexpected compile target\n' >&2
exit 1
fi
- name: Verify packaged console
if: matrix.target == 'x86_64-unknown-linux-gnu' || matrix.target == 'aarch64-unknown-linux-gnu'
shell: bash
env:
PACKAGE_VERSION: ${{ needs.prepare.outputs.version }}
ARCH: ${{ matrix.arch }}
LIBC: ${{ matrix.libc }}
run: |
set -euo pipefail
package_file="dist/rustfs-linux-${ARCH}-${LIBC}-v${PACKAGE_VERSION}.zip"
find_free_port() {
python3 - <<'PY'
import socket
with socket.socket() as sock:
sock.bind(("127.0.0.1", 0))
print(sock.getsockname()[1])
PY
}
package_dir=$(mktemp -d)
data_dir=$(mktemp -d)
log_file="${RUNNER_TEMP}/rustfs-console-smoke.log"
server_pid=""
trap '
if [[ -n "${server_pid:-}" ]]; then
kill "$server_pid" 2>/dev/null || true
wait "$server_pid" 2>/dev/null || true
fi
rm -rf "$package_dir" "$data_dir"
' EXIT
unzip -q "$package_file" -d "$package_dir"
api_port=$(find_free_port)
console_port=$(find_free_port)
console_url="http://127.0.0.1:${console_port}/rustfs/console/"
"$package_dir/rustfs" server \
--address "127.0.0.1:${api_port}" \
--console-enable \
--console-address "127.0.0.1:${console_port}" \
--access-key console-smoke \
--secret-key console-smoke-secret \
"$data_dir" >"$log_file" 2>&1 &
server_pid=$!
for _ in {1..40}; do
response=$(curl --silent --output /dev/null --write-out '%{http_code} %{content_type}' "$console_url" || true)
if [[ "$response" == 200\ text/html* ]]; then
exit 0
fi
sleep 0.25
done
printf 'Console endpoint did not return 200 text/html: %s\n' "$response" >&2
cat "$log_file"
exit 1
- name: Upload packages
uses: actions/upload-artifact@043fb46d1a93c77aae656e7c1c64a875d1fc6a0a # v7.0.1
with:
name: packages-${{ matrix.target_id }}-v${{ needs.prepare.outputs.version }}
path: dist/*
if-no-files-found: error
compression-level: 0
retention-days: 30
+3 -1
View File
@@ -112,9 +112,11 @@ jobs:
# GitHub-hosted: reliably provides a running Docker daemon + buildx +
# python3. See the header note (ci-2) for why the self-hosted sm-standard-4
# label was abandoned (no docker.sock on the pod its only run landed on).
# sm-standard-2 has no docker.sock either (weekly runs failed at "Enable
# buildx" from 2026-09-27 to 2026-11-08), so keep this job on ubuntu-latest.
# TODO(ci-8): scheduled-failure alerting (auto-open issue on a red weekly
# run) is added by the ci-8 composite action; do not implement it here.
runs-on: sm-standard-2
runs-on: ubuntu-latest
timeout-minutes: 120
steps:
- uses: actions/checkout@f548e57e544e1ff5a4c46bf1e1b8685f8e4a348a # v7
+2 -2
View File
@@ -47,8 +47,8 @@ permissions:
contents: read
concurrency:
group: oidc-keycloak-live-${{ github.ref }}
cancel-in-progress: true
group: oidc-keycloak-live-${{ github.event_name }}-${{ github.ref }}
cancel-in-progress: ${{ github.event_name != 'schedule' }}
jobs:
oidc-keycloak-live:
+38 -825
View File
@@ -1,840 +1,53 @@
# Copyright 2024 RustFS Team
#
# Licensed under the Apache License, Version 2.0 (the "License");
# you may not use this file except in compliance with the License.
# You may obtain a copy of the License at
#
# http://www.apache.org/licenses/LICENSE-2.0
#
# Unless required by applicable law or agreed to in writing, software
# distributed under the License is distributed on an "AS IS" BASIS,
# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
# See the License for the specific language governing permissions and
# limitations under the License.
# Package Workflow - Build DEB/RPM packages
#
# This workflow builds DEB and RPM packages from pre-built Linux binaries
# and uploads them to Cloudflare R2 and the GitHub release.
#
# Trigger:
# - workflow_run: automatically package after "Build and Release" completes
# for a release tag (the mac/windows/linux binaries are already uploaded
# to the GitHub release before packaging starts)
# - workflow_dispatch: manual fallback with a release tag and/or exact build run ID
#
# Flow:
# 1. Resolve and validate the selected Build workflow run and source identity
# 2. Download Linux binaries (x86_64-gnu, aarch64-gnu, x86_64-musl,
# aarch64-musl) from build artifacts
# 3. Build DEB packages for amd64 and arm64, gnu and musl variants
# 4. Build RPM packages for x86_64 and aarch64, gnu and musl variants
# 5. Upload all packages to Cloudflare R2 and the GitHub release
name: Package DEB/RPM
permissions:
# contents: write is required to upload packages to the GitHub release
contents: write
actions: read
# Copyright 2026 RustFS Team
# SPDX-License-Identifier: Apache-2.0
name: Build candidate packages
on:
# Follows the same pattern as docker.yml: run after the release build
# workflow completes, so packaging is triggered only by release tags
# (e.g. 1.0.0-rc.2, 1.0.0-rc.3), never by development builds.
workflow_run:
workflows: [ "Build and Release" ]
types: [ completed ]
workflow_dispatch:
workflow_call:
inputs:
tag:
description: "Release tag to package (e.g. 1.0.0-beta.12). Leave empty for latest main build."
required: false
candidate_manifest:
type: string
build_run_id:
description: "Build workflow run ID (when combined with tag, both must identify the same release commit)"
required: false
type: string
concurrency:
group: ${{ github.workflow }}-${{ github.event.workflow_run.head_branch || github.event.inputs.tag || github.run_id }}
cancel-in-progress: true
env:
HEAD_BRANCH: ${{ github.event.workflow_run.head_branch }}
WORKFLOW_RUN_ID: ${{ github.event.workflow_run.id }}
required: true
permissions:
contents: read
actions: read
jobs:
# Resolve which build run to use and extract version info
resolve:
name: Resolve Build
# Auto-trigger only from successful tag builds of "Build and Release".
# Tag pushes arrive as event == push with head_branch != main (a
# non-main push head_branch is the release tag name). Manual dispatch
# stays available as a fallback for backfills and re-runs.
if: >-
github.event_name == 'workflow_dispatch' ||
(github.event.workflow_run.conclusion == 'success' &&
github.event.workflow_run.event == 'push' &&
github.event.workflow_run.head_branch != 'main')
runs-on: ubuntu-latest # gh CLI required; the sm-standard-2 fleet does not ship it
timeout-minutes: 10
outputs:
version: ${{ steps.resolve.outputs.version }}
build_type: ${{ steps.resolve.outputs.build_type }}
build_run_id: ${{ steps.resolve.outputs.build_run_id }}
build_run_number: ${{ steps.resolve.outputs.build_run_number }}
head_sha: ${{ steps.resolve.outputs.head_sha }}
dev_sequence: ${{ steps.resolve.outputs.dev_sequence }}
tag: ${{ steps.resolve.outputs.tag }}
steps:
- name: Resolve build run
id: resolve
shell: bash
env:
GH_TOKEN: ${{ github.token }}
EVENT_NAME: ${{ github.event_name }}
REPOSITORY: ${{ github.repository }}
INPUT_TAG: ${{ github.event.inputs.tag }}
INPUT_RUN_ID: ${{ github.event.inputs.build_run_id }}
run: |
set -euo pipefail
fail() {
echo "❌ $1" >&2
exit 1
}
TAG=""
BUILD_RUN_ID=""
case "$EVENT_NAME" in
workflow_run)
TAG="$HEAD_BRANCH"
BUILD_RUN_ID="$WORKFLOW_RUN_ID"
;;
workflow_dispatch)
TAG="$INPUT_TAG"
BUILD_RUN_ID="$INPUT_RUN_ID"
;;
*) fail "unsupported event: $EVENT_NAME" ;;
esac
# Validate and classify tags before using them in API paths or logs.
semver_core='(0|[1-9][0-9]*)\.(0|[1-9][0-9]*)\.(0|[1-9][0-9]*)'
prerelease_id='(alpha|beta|rc)\.(0|[1-9][0-9]*)'
if [[ -n "$TAG" ]]; then
if [[ "$TAG" =~ ^${semver_core}(-${prerelease_id})?-preview\.(0|[1-9][0-9]*)$ ]]; then
BUILD_TYPE=preview
elif [[ "$TAG" =~ ^${semver_core}-${prerelease_id}$ ]]; then
BUILD_TYPE=prerelease
elif [[ "$TAG" =~ ^${semver_core}$ ]]; then
BUILD_TYPE=release
else
fail "tag is not a supported strict package version"
fi
else
BUILD_TYPE=development
fi
if [[ -n "$BUILD_RUN_ID" ]]; then
[[ "$BUILD_RUN_ID" =~ ^[1-9][0-9]*$ ]] || fail "build run ID must be a positive decimal integer"
echo "Using selected build run: $BUILD_RUN_ID"
elif [[ -n "$TAG" ]]; then
echo "Looking for build run for tag: $TAG"
BUILD_RUN_ID=$(gh api --method GET \
"repos/${REPOSITORY}/actions/workflows/build.yml/runs" \
-f branch="$TAG" -f status=success -F per_page=1 \
--jq '.workflow_runs[0].id // empty' 2>/dev/null || true)
if [[ -z "$BUILD_RUN_ID" ]]; then
BUILD_RUN_ID=$(gh api --method GET \
"repos/${REPOSITORY}/actions/workflows/build.yml/runs" \
-f event=push -f status=success -F per_page=100 2>/dev/null |
jq -r --arg tag "$TAG" \
'[.workflow_runs[] | select(.head_branch == $tag)][0].id // empty' || true)
fi
[[ "$BUILD_RUN_ID" =~ ^[1-9][0-9]*$ ]] || fail "no successful build run found for tag"
echo "Found build run: $BUILD_RUN_ID"
else
echo "No tag specified, looking for latest main build"
BUILD_RUN_ID=$(gh api --method GET \
"repos/${REPOSITORY}/actions/workflows/build.yml/runs" \
-f branch=main -f status=success -F per_page=1 \
--jq '.workflow_runs[0].id // empty' 2>/dev/null || true)
[[ "$BUILD_RUN_ID" =~ ^[1-9][0-9]*$ ]] || fail "no successful main build found"
echo "Latest main build: $BUILD_RUN_ID"
fi
# Fetch once and use the same immutable run metadata for identity,
# ordering, workflow provenance, and release-channel validation.
RUN_JSON=$(gh api "repos/${REPOSITORY}/actions/runs/${BUILD_RUN_ID}") ||
fail "cannot read selected build run"
RUN_ID=$(jq -r '.id // empty' <<<"$RUN_JSON")
RUN_NUMBER=$(jq -r '.run_number // empty' <<<"$RUN_JSON")
RUN_STATUS=$(jq -r '.status // empty' <<<"$RUN_JSON")
RUN_CONCLUSION=$(jq -r '.conclusion // empty' <<<"$RUN_JSON")
RUN_PATH=$(jq -r '.path // empty' <<<"$RUN_JSON")
HEAD_SHA=$(jq -r '.head_sha // empty' <<<"$RUN_JSON")
RUN_HEAD_BRANCH=$(jq -r '.head_branch // empty' <<<"$RUN_JSON")
[[ "$RUN_ID" == "$BUILD_RUN_ID" ]] || fail "run metadata ID mismatch"
[[ "$RUN_NUMBER" =~ ^[1-9][0-9]*$ ]] || fail "build run number must be a positive decimal integer"
[[ "$RUN_STATUS" == completed && "$RUN_CONCLUSION" == success ]] || fail "selected build run is not successful"
[[ "$RUN_PATH" == .github/workflows/build.yml ]] || fail "selected run is not Build and Release"
[[ "$HEAD_SHA" =~ ^[0-9a-f]{40}$ ]] || fail "selected build run has an invalid head SHA"
[[ "$RUN_HEAD_BRANCH" != *$'\n'* && -n "$RUN_HEAD_BRANCH" ]] || fail "selected build run has an invalid head branch"
if [[ -n "$TAG" ]]; then
[[ "$RUN_HEAD_BRANCH" == "$TAG" ]] || fail "tag and build run head branch do not match"
TAG_REF_JSON=$(gh api "repos/${REPOSITORY}/git/ref/tags/${TAG}") ||
fail "cannot resolve release tag ref"
TAG_OBJECT_TYPE=$(jq -r '.object.type // empty' <<<"$TAG_REF_JSON")
TAG_OBJECT_SHA=$(jq -r '.object.sha // empty' <<<"$TAG_REF_JSON")
depth=0
while [[ "$TAG_OBJECT_TYPE" == tag && $depth -lt 5 ]]; do
TAG_OBJECT_JSON=$(gh api "repos/${REPOSITORY}/git/tags/${TAG_OBJECT_SHA}") ||
fail "cannot peel annotated release tag"
TAG_OBJECT_TYPE=$(jq -r '.object.type // empty' <<<"$TAG_OBJECT_JSON")
TAG_OBJECT_SHA=$(jq -r '.object.sha // empty' <<<"$TAG_OBJECT_JSON")
depth=$((depth + 1))
done
[[ "$TAG_OBJECT_TYPE" == commit && "$TAG_OBJECT_SHA" =~ ^[0-9a-f]{40}$ ]] ||
fail "release tag does not resolve to a commit"
[[ "$TAG_OBJECT_SHA" == "$HEAD_SHA" ]] || fail "release tag commit and build run head SHA do not match"
VERSION="$TAG"
DEV_SEQUENCE=""
else
VERSION="dev-${HEAD_SHA}"
DEV_SEQUENCE="$RUN_NUMBER"
fi
{
echo "version=$VERSION"
echo "build_type=$BUILD_TYPE"
echo "build_run_id=$BUILD_RUN_ID"
echo "build_run_number=$RUN_NUMBER"
echo "head_sha=$HEAD_SHA"
echo "dev_sequence=$DEV_SEQUENCE"
echo "tag=${TAG}"
} >> "$GITHUB_OUTPUT"
echo "📊 Resolved:"
echo " Version: $VERSION"
echo " Build type: $BUILD_TYPE"
echo " Build run ID: $BUILD_RUN_ID"
echo " Build run number: $RUN_NUMBER"
# Consume the trusted release certification of the resolved build run and
# verify the certified input digests before any package is wrapped. A run
# without that certification fails closed here; packages built later must
# complete their own verification and cannot inherit an unverified
# conclusion.
release-certification:
name: Release Certification
needs: resolve
if: needs.resolve.result == 'success'
runs-on: ubuntu-latest # gh CLI required; the sm-standard-2 fleet does not ship it
timeout-minutes: 30
permissions:
contents: read
actions: read
steps:
- name: Checkout repository
uses: actions/checkout@f548e57e544e1ff5a4c46bf1e1b8685f8e4a348a # v7
with:
persist-credentials: false
- name: Download and verify release certification
env:
GH_TOKEN: ${{ secrets.RELEASE_VALIDATION_TOKEN || github.token }}
BUILD_RUN_ID: ${{ needs.resolve.outputs.build_run_id }}
BUILD_TYPE: ${{ needs.resolve.outputs.build_type }}
shell: bash
run: |
set -euo pipefail
mkdir -p certification
if [[ "$BUILD_TYPE" == "development" ]]; then
# Development artifacts are explicitly uncertified and never touch
# official or latest channels; only the staging/dev paths remain.
jq -n '{schema_version: 1, certified: false, reason: "development artifacts are explicitly uncertified"}' \
> certification/development-uncertified.json
exit 0
fi
if ! gh run download "$BUILD_RUN_ID" -n release-certification -D certification; then
echo "Build run ${BUILD_RUN_ID} published no release-certification artifact; failing closed" >&2
exit 1
fi
python3 scripts/release/release_gate.py verify-promotion \
--plan .config/release-validation-plan.json \
--candidate certification/candidate.json \
--summary certification/release-gate-output \
--artifact-sha256 "$(jq -r '.deliverables[0].sha256' certification/candidate.json)" \
--artifact-name "$(jq -r '.deliverables[0].name' certification/candidate.json)"
# Build DEB and RPM packages for each architecture and libc variant
package:
name: Package (${{ matrix.arch }}-${{ matrix.libc }})
needs: [ resolve, release-certification ]
runs-on: ubuntu-latest # gh CLI required; the sm-standard-2 fleet does not ship it
timeout-minutes: 30
runs-on: ubuntu-latest
timeout-minutes: 45
strategy:
fail-fast: false
max-parallel: 1
matrix:
include:
- arch: x86_64
libc: gnu
deb_arch: amd64
rpm_arch: x86_64
artifact_name: "rustfs-linux-x86_64-gnu"
# gnu binaries dynamically link glibc; musl binaries are static
# and run without a libc dependency.
deb_depends: "libc6 (>= 2.31)"
rpm_depends: "glibc >= 2.31"
- arch: aarch64
libc: gnu
deb_arch: arm64
rpm_arch: aarch64
artifact_name: "rustfs-linux-aarch64-gnu"
deb_depends: "libc6 (>= 2.31)"
rpm_depends: "glibc >= 2.31"
- arch: x86_64
libc: musl
deb_arch: amd64
rpm_arch: x86_64
artifact_name: "rustfs-linux-x86_64-musl"
deb_depends: ""
rpm_depends: ""
- arch: aarch64
libc: musl
deb_arch: arm64
rpm_arch: aarch64
artifact_name: "rustfs-linux-aarch64-musl"
deb_depends: ""
rpm_depends: ""
target: [linux-x86_64-musl, linux-aarch64-musl, linux-x86_64-gnu, linux-aarch64-gnu]
steps:
- name: Checkout repository
uses: actions/checkout@f548e57e544e1ff5a4c46bf1e1b8685f8e4a348a # v7
- uses: actions/checkout@f548e57e544e1ff5a4c46bf1e1b8685f8e4a348a # v7
with:
ref: ${{ fromJson(inputs.candidate_manifest).source_sha }}
persist-credentials: false
- name: Normalize package metadata
id: versions
shell: bash
env:
BUILD_TYPE: ${{ needs.resolve.outputs.build_type }}
SOURCE_VERSION: ${{ needs.resolve.outputs.version }}
DEV_SEQUENCE: ${{ needs.resolve.outputs.dev_sequence }}
ARCH: ${{ matrix.rpm_arch }}
LIBC: ${{ matrix.libc }}
run: |
set -euo pipefail
normalized=$(./scripts/release/package_versions.sh \
"$BUILD_TYPE" "$SOURCE_VERSION" "$DEV_SEQUENCE" "$ARCH" "$LIBC")
printf '%s\n' "$normalized" >> "$GITHUB_OUTPUT"
- name: Download binary artifact from build run
uses: actions/download-artifact@3e5f45b2cfb9172054b4087a40e8e0b5a5461e7c # v8.0.1
with:
pattern: ${{ matrix.artifact_name }}*
path: ./binary-artifact
run-id: ${{ needs.resolve.outputs.build_run_id }}
github-token: ${{ github.token }}
merge-multiple: true
- name: Extract binary
id: binary
shell: bash
run: |
set -euo pipefail
ZIP_FILE=$(find ./binary-artifact -name "*.zip" -type f | head -1)
if [[ -z "$ZIP_FILE" ]]; then
echo "❌ No binary artifact found"
find ./binary-artifact -mindepth 1 -maxdepth 1 -print 2>/dev/null || true
exit 1
fi
echo "Found artifact: $ZIP_FILE"
mkdir -p ./bin
unzip -o "$ZIP_FILE" -d ./bin
if [[ ! -f ./bin/rustfs ]]; then
echo "❌ rustfs binary not found in archive"
exit 1
fi
chmod +x ./bin/rustfs
stat --printf='%n %s bytes\n' ./bin/rustfs
echo "✅ Binary extracted"
- name: Build DEB package
id: deb
shell: bash
env:
DEB_VERSION: ${{ steps.versions.outputs.deb_version }}
DEB_ARCH: ${{ matrix.deb_arch }}
DEB_FILE: ${{ steps.versions.outputs.deb_file }}
DEB_DEPENDS: ${{ matrix.deb_depends }}
LIBC: ${{ matrix.libc }}
run: |
set -euo pipefail
PKG_DIR="${DEB_FILE%.deb}"
echo "Building DEB: ${DEB_FILE}"
mkdir -p "${PKG_DIR}/DEBIAN"
mkdir -p "${PKG_DIR}/usr/bin"
mkdir -p "${PKG_DIR}/etc/default"
mkdir -p "${PKG_DIR}/lib/systemd/system"
mkdir -p "${PKG_DIR}/usr/share/doc/rustfs"
cp ./bin/rustfs "${PKG_DIR}/usr/bin/"
chmod 755 "${PKG_DIR}/usr/bin/rustfs"
cp deploy/build/rustfs.service "${PKG_DIR}/lib/systemd/system/"
cat > "${PKG_DIR}/etc/default/rustfs" << 'ENVEOF'
# RustFS Environment Configuration
# See https://docs.rustfs.com for more information
# RUSTFS_ACCESS_KEY=<changeme>
# RUSTFS_SECRET_KEY=<changeme>
# RUSTFS_VOLUMES=""
# RUSTFS_ADDRESS=":9000"
# RUSTFS_CONSOLE_ADDRESS=":9001"
# RUSTFS_CONSOLE_ENABLE=true
# RUSTFS_OBS_LOGGER_LEVEL=error
# RUSTFS_OBS_LOG_DIRECTORY="/var/log/rustfs/"
ENVEOF
cat > "${PKG_DIR}/DEBIAN/control" << EOF
Package: rustfs
Version: ${DEB_VERSION}
Section: utils
Priority: optional
Architecture: ${DEB_ARCH}
EOF
# gnu binaries dynamically link glibc and must declare it; musl
# binaries are statically linked and need no libc dependency.
if [[ -n "$DEB_DEPENDS" ]]; then
echo "Depends: ${DEB_DEPENDS}" >> "${PKG_DIR}/DEBIAN/control"
fi
cat >> "${PKG_DIR}/DEBIAN/control" << EOF
Maintainer: RustFS Team <support@rustfs.com>
Description: High-performance distributed object storage (${LIBC} build)
RustFS is a high-performance distributed object storage software
built using Rust. It is compatible with MinIO and S3 API.
Homepage: https://rustfs.com
EOF
# Declare /etc/default/rustfs as a conffile so dpkg preserves user
# modifications on upgrade instead of silently overwriting them.
cat > "${PKG_DIR}/DEBIAN/conffiles" << 'CONFFILES'
/etc/default/rustfs
CONFFILES
./scripts/release/package_service_scripts.sh after-install > "${PKG_DIR}/DEBIAN/postinst"
chmod 755 "${PKG_DIR}/DEBIAN/postinst"
./scripts/release/package_service_scripts.sh before-remove > "${PKG_DIR}/DEBIAN/prerm"
chmod 755 "${PKG_DIR}/DEBIAN/prerm"
./scripts/release/package_service_scripts.sh after-remove > "${PKG_DIR}/DEBIAN/postrm"
chmod 755 "${PKG_DIR}/DEBIAN/postrm"
cp LICENSE "${PKG_DIR}/usr/share/doc/rustfs/"
cp README.md "${PKG_DIR}/usr/share/doc/rustfs/"
fakeroot dpkg-deb --build "${PKG_DIR}" "$DEB_FILE"
[[ $(dpkg-deb -f "$DEB_FILE" Package) == rustfs ]]
[[ $(dpkg-deb -f "$DEB_FILE" Version) == "$DEB_VERSION" ]]
[[ $(dpkg-deb -f "$DEB_FILE" Architecture) == "$DEB_ARCH" ]]
dpkg-deb --fsys-tarfile "$DEB_FILE" | tar -tf - | grep -Fx './usr/bin/rustfs' >/dev/null
stat --printf='%n %s bytes\n' "$DEB_FILE"
echo "deb_file=$DEB_FILE" >> "$GITHUB_OUTPUT"
echo "✅ DEB built: $DEB_FILE"
- name: Build RPM package
id: rpm
shell: bash
env:
RPM_VERSION: ${{ steps.versions.outputs.rpm_version }}
RPM_RELEASE: ${{ steps.versions.outputs.rpm_release }}
RPM_ARCH: ${{ matrix.rpm_arch }}
RPM_FILE: ${{ steps.versions.outputs.rpm_file }}
RPM_DEPENDS: ${{ matrix.rpm_depends }}
LIBC: ${{ matrix.libc }}
run: |
set -euo pipefail
echo "Building RPM for ${RPM_ARCH} (${LIBC})"
sudo apt-get update && sudo apt-get install -y ruby ruby-dev build-essential rpm
sudo gem install fpm
./scripts/test_package_versions.sh --require-package-managers
# gnu binaries dynamically link glibc and must declare it; musl
# binaries are statically linked and need no libc dependency.
depends_args=()
if [[ -n "$RPM_DEPENDS" ]]; then
depends_args+=(--depends "$RPM_DEPENDS")
fi
# Create config file for fpm (DEB build creates it in its package dir structure,
# but fpm needs the file to exist before packaging)
mkdir -p ./tmp-pkg/etc/default
cat > ./tmp-pkg/etc/default/rustfs << 'ENVEOF'
# RustFS Environment Configuration
# See https://docs.rustfs.com for more information
# RUSTFS_ACCESS_KEY=<changeme>
# RUSTFS_SECRET_KEY=<changeme>
# RUSTFS_VOLUMES=""
# RUSTFS_ADDRESS=":9000"
# RUSTFS_CONSOLE_ADDRESS=":9001"
# RUSTFS_CONSOLE_ENABLE=true
# RUSTFS_OBS_LOGGER_LEVEL=error
# RUSTFS_OBS_LOG_DIRECTORY="/var/log/rustfs/"
ENVEOF
fpm -s dir -t rpm \
--name rustfs \
--version "$RPM_VERSION" \
--iteration "$RPM_RELEASE" \
--architecture "$RPM_ARCH" \
--package "$RPM_FILE" \
"${depends_args[@]}" \
--description "High-performance distributed object storage (${LIBC} build)" \
--maintainer "RustFS Team <support@rustfs.com>" \
--url "https://rustfs.com" \
--license "Apache-2.0" \
--before-install <(./scripts/release/package_service_scripts.sh rpm-before-install) \
--before-upgrade <(./scripts/release/package_service_scripts.sh rpm-before-upgrade) \
--after-install <(./scripts/release/package_service_scripts.sh after-install) \
--after-upgrade <(./scripts/release/package_service_scripts.sh after-install) \
--before-remove <(./scripts/release/package_service_scripts.sh before-remove) \
--after-remove <(./scripts/release/package_service_scripts.sh after-remove) \
--rpm-posttrans <(./scripts/release/package_service_scripts.sh rpm-posttrans) \
--config-files /etc/default/rustfs \
./bin/rustfs=/usr/bin/rustfs \
./tmp-pkg/etc/default/rustfs=/etc/default/rustfs \
deploy/build/rustfs.service=/lib/systemd/system/rustfs.service \
LICENSE=/usr/share/doc/rustfs/LICENSE \
README.md=/usr/share/doc/rustfs/README.md
if [[ ! -f "$RPM_FILE" ]]; then
echo "❌ RPM build failed"
exit 1
fi
RPM_METADATA=$(rpm -qp --qf '%{NAME}\n%{VERSION}\n%{RELEASE}\n%{ARCH}\n' "$RPM_FILE")
EXPECTED_METADATA=$(printf 'rustfs\n%s\n%s\n%s' "$RPM_VERSION" "$RPM_RELEASE" "$RPM_ARCH")
[[ "$RPM_METADATA" == "$EXPECTED_METADATA" ]]
rpm -qpl "$RPM_FILE" | grep -Fx '/usr/bin/rustfs' >/dev/null
stat --printf='%n %s bytes\n' "$RPM_FILE"
echo "rpm_file=$RPM_FILE" >> "$GITHUB_OUTPUT"
echo "✅ RPM built: $RPM_FILE"
- name: Download certification evidence
if: needs.resolve.outputs.build_type != 'development'
uses: actions/download-artifact@3e5f45b2cfb9172054b4087a40e8e0b5a5461e7c # v8.0.1
with:
name: release-certification
path: ./certification
# Later-built wrapper verification: the binary inside each package must
# match the certified binary digest for this platform before anything is
# uploaded, and the package itself stays on staging until promotion.
- name: Verify wrapped binary against certification
if: needs.resolve.outputs.build_type != 'development'
shell: bash
env:
DEB_FILE: ${{ steps.deb.outputs.deb_file }}
RPM_FILE: ${{ steps.rpm.outputs.rpm_file }}
ARTIFACT_NAME: ${{ matrix.artifact_name }}
run: |
set -euo pipefail
expected="$(jq -r --arg needle "$ARTIFACT_NAME" \
'[.deliverables[] | select(.name | contains($needle)) | .binary_sha256][0] // empty' \
certification/candidate.json)"
if [[ -z "$expected" ]]; then
echo "No certified binary digest recorded for ${ARTIFACT_NAME}" >&2
exit 1
fi
if ! command -v rpm2cpio >/dev/null 2>&1; then
sudo apt-get update && sudo apt-get install -y rpm2cpio
fi
if ! command -v cpio >/dev/null 2>&1; then
sudo apt-get update && sudo apt-get install -y cpio
fi
for package_file in "$DEB_FILE" "$RPM_FILE"; do
case "$package_file" in
*.deb)
dpkg-deb --fsys-tarfile "$package_file" | tar -xOf - ./usr/bin/rustfs > "$RUNNER_TEMP/wrapped-rustfs"
;;
*.rpm)
rpm2cpio "$package_file" | cpio -i --to-stdout ./usr/bin/rustfs > "$RUNNER_TEMP/wrapped-rustfs"
;;
esac
python3 scripts/release/release_gate.py verify-promotion \
--plan .config/release-validation-plan.json \
--candidate certification/candidate.json \
--summary certification/release-gate-output \
--artifact "$RUNNER_TEMP/wrapped-rustfs" \
--artifact-role binary
done
- name: Upload packages to artifacts
uses: actions/upload-artifact@043fb46d1a93c77aae656e7c1c64a875d1fc6a0a # v7.0.1
with:
name: packages-${{ matrix.arch }}-${{ matrix.libc }}
path: |
*.deb
*.rpm
retention-days: 30
- name: Upload packages to Cloudflare R2
if: needs.resolve.outputs.build_type != 'preview' && env.R2_ACCESS_KEY_ID != ''
env:
R2_ACCESS_KEY_ID: ${{ secrets.R2_ACCESS_KEY_ID }}
R2_SECRET_ACCESS_KEY: ${{ secrets.R2_SECRET_ACCESS_KEY }}
R2_ENDPOINT: ${{ secrets.R2_ENDPOINT }}
R2_BUCKET: ${{ secrets.R2_BUCKET }}
AWS_EC2_METADATA_DISABLED: true
BUILD_TYPE: ${{ needs.resolve.outputs.build_type }}
DEB_FILE: ${{ steps.deb.outputs.deb_file }}
RPM_FILE: ${{ steps.rpm.outputs.rpm_file }}
shell: bash
run: |
set -euo pipefail
if [[ -z "$R2_ACCESS_KEY_ID" || -z "$R2_SECRET_ACCESS_KEY" || -z "$R2_ENDPOINT" || -z "$R2_BUCKET" ]]; then
echo "⚠️ R2 credentials missing, skipping upload"
exit 0
fi
if ! command -v aws >/dev/null 2>&1; then
sudo apt-get update && sudo apt-get install -y awscli
fi
export AWS_ACCESS_KEY_ID="$R2_ACCESS_KEY_ID"
export AWS_SECRET_ACCESS_KEY="$R2_SECRET_ACCESS_KEY"
export AWS_DEFAULT_REGION="auto"
# Build-phase writes stay on staging for release types: the official
# packages/release and packages/latest paths are written only by the
# certification-gated promote-packages job, from these same staged
# objects. Development packages keep the uncertified dev channel.
if [[ "$BUILD_TYPE" == "development" ]]; then
R2_PREFIX="artifacts/rustfs/packages/dev"
else
R2_PREFIX="artifacts/rustfs/packages/staging/${GITHUB_RUN_ID}/${GITHUB_RUN_ATTEMPT}"
fi
R2_PATH="s3://${R2_BUCKET}/${R2_PREFIX}/"
echo "📤 Uploading to $R2_PATH"
for f in "$DEB_FILE" "$RPM_FILE"; do
if [[ -n "$f" && -f "$f" ]]; then
echo "Uploading: $f"
aws s3 cp "$f" "$R2_PATH" --endpoint-url "$R2_ENDPOINT" --only-show-errors
fi
done
echo "✅ Upload complete"
- name: Upload packages to GitHub Release
if: needs.resolve.outputs.tag != ''
- name: Reuse completed package set
id: reuse
env:
GH_TOKEN: ${{ github.token }}
TAG: ${{ needs.resolve.outputs.tag }}
DEB_FILE: ${{ steps.deb.outputs.deb_file }}
RPM_FILE: ${{ steps.rpm.outputs.rpm_file }}
shell: bash
CANDIDATE_MANIFEST: ${{ inputs.candidate_manifest }}
TARGET: ${{ matrix.target }}
run: python3 scripts/release/deliver.py reuse --kind packages --target "$TARGET"
- name: Install packaging tools
if: steps.reuse.outputs.reused != 'true'
run: |
set -euo pipefail
# Upload the packages, then refresh the release checksums so the new
# assets are covered, matching the binary release flow.
for f in "$DEB_FILE" "$RPM_FILE"; do
if [[ -n "$f" && -f "$f" ]]; then
echo "📤 Uploading $(basename "$f") to GitHub release ${TAG}..."
gh release upload "$TAG" "$f" --clobber
fi
done
CHECKSUM_DIR="$(mktemp -d)"
gh release download "$TAG" -p 'SHA256SUMS' -p 'SHA512SUMS' \
-D "$CHECKSUM_DIR" --clobber
for spec in "SHA256SUMS:sha256sum" "SHA512SUMS:sha512sum"; do
asset="${spec%%:*}"
checksum_cmd="${spec##*:}"
checksum_file="${CHECKSUM_DIR}/${asset}"
for f in "$DEB_FILE" "$RPM_FILE"; do
if [[ -n "$f" && -f "$f" ]]; then
base="$(basename "$f")"
# GitHub stores release asset names with '~' normalized to '.'.
# Current package file names embed the raw tag and contain no
# '~', but older releases may still ship such assets, so the
# checksum entries must reference the name as stored on the
# release.
github_base="${base//\~/.}"
# Remove any stale entry (both naming variants), then append
grep -Fv -- "$base" "$checksum_file" > "${checksum_file}.tmp" || true
grep -Fv -- "$github_base" "${checksum_file}.tmp" > "${checksum_file}.tmp2" || true
mv "${checksum_file}.tmp2" "$checksum_file"
digest=$("$checksum_cmd" -- "$f" | awk '{print $1}')
printf '%s %s\n' "$digest" "$github_base" >> "$checksum_file"
fi
done
echo "📤 Updating ${asset} for release ${TAG}..."
gh release upload "$TAG" "$checksum_file" --clobber
done
echo "✅ GitHub release assets updated"
# The only writer of the official packages/release and packages/latest R2
# paths. Promotion re-checks each package's wrapped binary against the
# passed certification list and copies the staged objects byte-for-byte:
# no rebuild-after-verify, no mutable latest resolution downstream.
promote-packages:
name: Promote Packages
needs: [ resolve, package, release-certification ]
# Target-channel lock for the official package paths.
concurrency:
group: release-channel-r2-packages
cancel-in-progress: false
if: needs.resolve.outputs.tag != '' && (needs.resolve.outputs.build_type == 'release' || needs.resolve.outputs.build_type == 'prerelease')
runs-on: ubuntu-latest # gh CLI required; the sm-standard-2 fleet does not ship it
timeout-minutes: 30
permissions:
contents: read
actions: read
env:
R2_ACCESS_KEY_ID: ${{ secrets.R2_ACCESS_KEY_ID }}
R2_SECRET_ACCESS_KEY: ${{ secrets.R2_SECRET_ACCESS_KEY }}
R2_ENDPOINT: ${{ secrets.R2_ENDPOINT }}
R2_BUCKET: ${{ secrets.R2_BUCKET }}
AWS_EC2_METADATA_DISABLED: true
steps:
- name: Checkout repository
uses: actions/checkout@f548e57e544e1ff5a4c46bf1e1b8685f8e4a348a # v7
with:
persist-credentials: false
- name: Download packages and certification
uses: actions/download-artifact@3e5f45b2cfb9172054b4087a40e8e0b5a5461e7c # v8.0.1
with:
path: ./packages
pattern: packages-*
merge-multiple: true
- name: Download certification evidence
uses: actions/download-artifact@3e5f45b2cfb9172054b4087a40e8e0b5a5461e7c # v8.0.1
with:
name: release-certification
path: ./certification
- name: Verify and promote staged packages
shell: bash
run: |
set -euo pipefail
if [[ -z "$R2_ACCESS_KEY_ID" || -z "$R2_SECRET_ACCESS_KEY" || -z "$R2_ENDPOINT" || -z "$R2_BUCKET" ]]; then
echo "❌ R2 credentials missing; refusing to promote to the official channel" >&2
exit 1
fi
STAGING_PREFIX="s3://${R2_BUCKET}/artifacts/rustfs/packages/staging/${GITHUB_RUN_ID}/${GITHUB_RUN_ATTEMPT}/"
RELEASE_PATH="s3://${R2_BUCKET}/artifacts/rustfs/packages/release/"
LATEST_PATH="s3://${R2_BUCKET}/artifacts/rustfs/packages/latest/"
if ! command -v aws >/dev/null 2>&1; then
sudo apt-get update && sudo apt-get install -y awscli
fi
export AWS_ACCESS_KEY_ID="$R2_ACCESS_KEY_ID"
export AWS_SECRET_ACCESS_KEY="$R2_SECRET_ACCESS_KEY"
export AWS_DEFAULT_REGION="auto"
if ! command -v rpm2cpio >/dev/null 2>&1; then
sudo apt-get install -y rpm2cpio
fi
if ! command -v cpio >/dev/null 2>&1; then
sudo apt-get install -y cpio
fi
promoted=0
for package_file in packages/*.deb packages/*.rpm; do
[[ -f "$package_file" ]] || continue
name="$(basename "$package_file")"
case "$package_file" in
*.deb)
dpkg-deb --fsys-tarfile "$package_file" | tar -xOf - ./usr/bin/rustfs > "$RUNNER_TEMP/wrapped-rustfs"
;;
*.rpm)
rpm2cpio "$package_file" | cpio -i --to-stdout ./usr/bin/rustfs > "$RUNNER_TEMP/wrapped-rustfs"
;;
esac
python3 scripts/release/release_gate.py verify-promotion \
--plan .config/release-validation-plan.json \
--candidate certification/candidate.json \
--summary certification/release-gate-output \
--artifact "$RUNNER_TEMP/wrapped-rustfs" \
--artifact-role binary
staged="$(mktemp)"
aws s3 cp "${STAGING_PREFIX}${name}" "$staged" --endpoint-url "$R2_ENDPOINT" --only-show-errors
staged_sha="$(sha256sum "$staged" | awk '{print $1}')"
built_sha="$(sha256sum "$package_file" | awk '{print $1}')"
if [[ "$staged_sha" != "$built_sha" ]]; then
echo "❌ Staged package ${name} does not match the verified bytes" >&2
exit 1
fi
aws s3 cp "$staged" "${RELEASE_PATH}${name}" --endpoint-url "$R2_ENDPOINT" --only-show-errors
aws s3 cp "$staged" "${LATEST_PATH}${name}" --endpoint-url "$R2_ENDPOINT" --only-show-errors
rm -f "$staged"
promoted=$((promoted + 1))
echo "✅ Promoted ${name} (${built_sha})"
done
if [[ "$promoted" -eq 0 ]]; then
echo "No staged packages found to promote" >&2
exit 1
fi
# Summary
summary:
name: Summary
needs: [ resolve, package ]
# Every completed Build and Release run triggers this workflow; when
# resolve declined it (not a release tag), there is nothing to summarise.
if: always() && needs.resolve.result != 'skipped'
runs-on: ubuntu-latest
timeout-minutes: 5
steps:
- name: Print summary
shell: bash
sudo apt-get update
sudo apt-get install -y ruby ruby-dev build-essential rpm fakeroot cpio
sudo gem install fpm --version 1.17.0 --no-document
- name: Package accepted executable and verify contents
if: steps.reuse.outputs.reused != 'true'
env:
SUMMARY_VERSION: ${{ needs.resolve.outputs.version }}
SUMMARY_BUILD_TYPE: ${{ needs.resolve.outputs.build_type }}
SUMMARY_BUILD_RUN_ID: ${{ needs.resolve.outputs.build_run_id }}
SUMMARY_PACKAGE_STATUS: ${{ needs.package.result }}
run: |
{
echo "## 📦 Package Summary"
echo ""
echo "| Item | Value |"
echo "|------|-------|"
echo "| Version | \`${SUMMARY_VERSION}\` |"
echo "| Build Type | ${SUMMARY_BUILD_TYPE} |"
echo "| Build Run | #${SUMMARY_BUILD_RUN_ID} |"
echo "| Package Status | ${SUMMARY_PACKAGE_STATUS} |"
} >> "$GITHUB_STEP_SUMMARY"
GH_TOKEN: ${{ github.token }}
CANDIDATE_MANIFEST: ${{ inputs.candidate_manifest }}
TARGET: ${{ matrix.target }}
run: python3 scripts/release/deliver.py package --target "$TARGET" --output delivery
- uses: actions/upload-artifact@043fb46d1a93c77aae656e7c1c64a875d1fc6a0a # v7.0.1
if: steps.reuse.outputs.reused != 'true'
with:
name: packages-${{ matrix.target }}
path: delivery/*
if-no-files-found: error
compression-level: 0
retention-days: 30
+308
View File
@@ -0,0 +1,308 @@
# Copyright 2026 RustFS Team
#
# Licensed under the Apache License, Version 2.0 (the "License");
# you may not use this file except in compliance with the License.
# You may obtain a copy of the License at
#
# http://www.apache.org/licenses/LICENSE-2.0
#
# Unless required by applicable law or agreed to in writing, software
# distributed under the License is distributed on an "AS IS" BASIS,
# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
# See the License for the specific language governing permissions and
# limitations under the License.
# s3-tests reference parity lane (rustfs/backlog#2773).
#
# Runs the ceph/s3-tests implemented list (scripts/s3-tests/implemented_tests.txt)
# against every target in .config/reference-parity-targets.toml and fails on
# any per-node-id difference between a candidate and the reference:
#
# - reference: the release pinned in .config/reference-binary.toml, downloaded
# and sha256-verified by .github/actions/reference-binary;
# - candidate targets: the debug binary built from this checkout exactly as
# ci.yml's build-rustfs-debug-binary job builds it for S3 Implemented Tests.
#
# Every target runs the same harness (scripts/s3-tests/run.sh from this
# checkout), test list and server configuration on its own hosted runner. A
# target's failures get one rerun against a fresh server; a case that passes it
# is flaky, not a difference. scripts/s3-tests/reference_parity.py judges the
# result and honours .config/reference-parity-tolerated.txt, whose entries must
# each match a real difference.
#
# Report-only: not part of the Test and Lint aggregate. Scheduled failures are
# owned by the scheduled-validation watchdog (.github/scheduled-validations.json).
name: Reference Parity
on:
pull_request:
paths:
- ".github/workflows/reference-parity.yml"
- ".github/actions/reference-binary/**"
- ".config/reference-binary.toml"
- ".config/reference-parity-targets.toml"
- ".config/reference-parity-tolerated.txt"
- "scripts/s3-tests/reference_parity.py"
- "scripts/junit_diff.py"
schedule:
- cron: "37 1 * * *"
workflow_dispatch:
# Matches ci.yml: these variables are hashed into the rust-cache key, so the
# candidate build reads the same ci-dev cache as build-rustfs-debug-binary.
env:
CARGO_TERM_COLOR: always
RUST_BACKTRACE: 1
concurrency:
group: ${{ github.workflow }}-${{ github.ref }}
cancel-in-progress: true
permissions:
contents: read
defaults:
run:
shell: bash
jobs:
plan:
name: Plan parity targets
runs-on: ubuntu-latest
timeout-minutes: 5
outputs:
targets: ${{ steps.targets.outputs.targets }}
steps:
- name: Checkout repository
uses: actions/checkout@f548e57e544e1ff5a4c46bf1e1b8685f8e4a348a # v7
with:
persist-credentials: false
- name: Validate target list
id: targets
run: |
set -euo pipefail
targets="$(python3 scripts/s3-tests/reference_parity.py targets --config .config/reference-parity-targets.toml)"
echo "targets=${targets}" >> "$GITHUB_OUTPUT"
echo "Parity targets: ${targets}"
build-candidate:
name: Build candidate binary
# Critical-path CPU and cache ownership: the same debug build as ci.yml's
# build-rustfs-debug-binary, reading the ci-dev cache written on sm-standard-4.
runs-on: sm-standard-4
timeout-minutes: 60
env:
FORCE_JAVASCRIPT_ACTIONS_TO_NODE24: "true"
steps:
- name: Checkout repository
uses: actions/checkout@f548e57e544e1ff5a4c46bf1e1b8685f8e4a348a # v7
with:
persist-credentials: false
- name: Setup Rust environment
uses: ./.github/actions/setup
with:
rust-version: stable
cache-shared-key: ci-dev
cache-save-if: 'false'
install-build-packaging-tools: 'false'
- name: Build debug binary
run: python3 scripts/e2e_binary.py build --bins --features e2e-test-hooks
- name: Upload candidate binary
uses: actions/upload-artifact@043fb46d1a93c77aae656e7c1c64a875d1fc6a0a # v7.0.1
with:
name: reference-parity-candidate-binary
overwrite: true
path: |
target/debug/rustfs
target/debug/rustfs.e2e.json
if-no-files-found: error
retention-days: 1
s3tests:
name: s3-tests (${{ matrix.target.name }})
needs: [ plan, build-candidate ]
runs-on: ubuntu-latest
timeout-minutes: 75
strategy:
fail-fast: false
matrix:
target: ${{ fromJSON(needs.plan.outputs.targets) }}
env:
TARGET_NAME: ${{ matrix.target.name }}
TARGET_KIND: ${{ matrix.target.binary }}
# Above this many first-run failures the target is systemically broken;
# a rerun would only repeat it, so the first run is judged as is.
RERUN_CAP: "50"
steps:
- name: Checkout repository
uses: actions/checkout@f548e57e544e1ff5a4c46bf1e1b8685f8e4a348a # v7
with:
persist-credentials: false
- name: Fetch pinned reference binary
id: reference
if: matrix.target.binary == 'reference'
uses: ./.github/actions/reference-binary
- name: Download candidate binary
if: matrix.target.binary == 'candidate'
uses: actions/download-artifact@3e5f45b2cfb9172054b4087a40e8e0b5a5461e7c # v8.0.1
with:
name: reference-parity-candidate-binary
path: target/debug
- name: Select target binary
env:
REFERENCE_PATH: ${{ steps.reference.outputs.path }}
run: |
set -euo pipefail
case "$TARGET_KIND" in
reference) binary="$REFERENCE_PATH" ;;
candidate) chmod +x target/debug/rustfs; binary="$PWD/target/debug/rustfs" ;;
*) echo "unknown target binary kind: $TARGET_KIND" >&2; exit 1 ;;
esac
test -x "$binary"
echo "TARGET_BINARY=$binary" >> "$GITHUB_ENV"
mkdir -p "parity/$TARGET_NAME"
{
echo "target=$TARGET_NAME"
echo "binary=$TARGET_KIND"
echo "sha256=$(sha256sum "$binary" | cut -d' ' -f1)"
echo "version=$("$binary" --version | tr '\n' ' ')"
} | tee "parity/$TARGET_NAME/target.txt"
- name: Apply target environment
env:
TARGET_JSON: ${{ toJSON(matrix.target) }}
run: |
set -euo pipefail
python3 scripts/s3-tests/reference_parity.py export-env --target-json "$TARGET_JSON" \
| tee -a "$GITHUB_ENV" "parity/$TARGET_NAME/target-env.txt"
- name: Run implemented s3-tests
run: |
set -uo pipefail
report="parity/$TARGET_NAME"
harness="artifacts/s3tests-single"
# Same invocation as ci.yml's S3 Implemented Tests, with a fresh data
# root and port per run. $1 optionally replaces the test list.
run_suite() {
local list="$1" label="$2" root port
root="${RUNNER_TEMP}/rustfs-s3tests-${label}"
rm -rf "$harness" "$root"
mkdir -p "$root"
port="$(python3 -c 'import socket; s=socket.socket(); s.bind(("127.0.0.1", 0)); print(s.getsockname()[1]); s.close()')"
env DEPLOY_MODE=binary \
RUSTFS_BINARY="$TARGET_BINARY" \
TEST_MODE=single \
MAXFAIL=0 \
XDIST=4 \
S3_PORT="$port" \
DATA_ROOT="$root" \
S3TESTS_CONF="$harness/s3tests.conf" \
${list:+IMPLEMENTED_TESTS_FILE="$list"} \
./scripts/s3-tests/run.sh
}
# A run is usable only if it produced a JUnit report; test failures
# are data for the comparison, a missing report is a lane failure.
keep() {
local prefix="$1"
for file in pytest.log compat-report.md; do
if [[ -f "$harness/$file" ]]; then cp "$harness/$file" "$report/${prefix}${file}"; fi
done
if [[ -f "$harness/rustfs-single/rustfs.log" ]]; then cp "$harness/rustfs-single/rustfs.log" "$report/${prefix}rustfs.log"; fi
test -s "$harness/junit.xml" || { echo "::error::$TARGET_NAME produced no JUnit report (${prefix:-first} run)"; return 1; }
cp "$harness/junit.xml" "$report/${prefix}junit.xml"
}
run_suite "" first
echo "first_run_exit=$?" >> "$report/target.txt"
keep "" || exit 1
cp "$harness/selected-nodeids.txt" "$report/selected-nodeids.txt" || exit 1
failures="$(python3 scripts/s3-tests/reference_parity.py failed --report "$report")" || exit 1
count="$(printf '%s' "$failures" | grep -c . || true)"
echo "first_run_failures=$count" >> "$report/target.txt"
if (( count == 0 )); then
exit 0
fi
if (( count > RERUN_CAP )); then
echo "::warning::$TARGET_NAME has $count failures (cap $RERUN_CAP); judging the first run without a rerun"
echo "rerun=skipped-over-cap" >> "$report/target.txt"
exit 0
fi
printf '%s\n' "$failures" > "$RUNNER_TEMP/rerun-tests.txt"
run_suite "$RUNNER_TEMP/rerun-tests.txt" rerun
echo "rerun_exit=$?" >> "$report/target.txt"
keep rerun- || exit 1
- name: Upload target report
if: always()
uses: actions/upload-artifact@043fb46d1a93c77aae656e7c1c64a875d1fc6a0a # v7.0.1
with:
name: reference-parity-report-${{ matrix.target.name }}
overwrite: true
path: parity/${{ matrix.target.name }}/
if-no-files-found: error
retention-days: 14
compare:
name: Compare targets
needs: [ plan, s3tests ]
if: always() && needs.plan.result == 'success'
runs-on: ubuntu-latest
timeout-minutes: 10
steps:
- name: Checkout repository
uses: actions/checkout@f548e57e544e1ff5a4c46bf1e1b8685f8e4a348a # v7
with:
persist-credentials: false
- name: Download target reports
uses: actions/download-artifact@3e5f45b2cfb9172054b4087a40e8e0b5a5461e7c # v8.0.1
with:
pattern: reference-parity-report-*
path: parity-download
- name: Compare per node id
env:
TARGETS: ${{ needs.plan.outputs.targets }}
S3TESTS_RESULT: ${{ needs.s3tests.result }}
run: |
set -uo pipefail
mkdir -p parity
for dir in parity-download/reference-parity-report-*; do
if [[ -d "$dir" ]]; then mv "$dir" "parity/${dir#parity-download/reference-parity-report-}"; fi
done
python3 scripts/s3-tests/reference_parity.py compare \
--targets-json "$TARGETS" \
--reports parity \
--tolerated .config/reference-parity-tolerated.txt \
--markdown "$GITHUB_STEP_SUMMARY" \
--json parity/diff.json > parity/diff.md
status=$?
cat parity/diff.md
if [[ "$S3TESTS_RESULT" != "success" ]]; then
echo "::error::s3-tests target jobs finished with '$S3TESTS_RESULT'"
status=1
fi
exit "$status"
- name: Upload parity diff
if: always()
uses: actions/upload-artifact@043fb46d1a93c77aae656e7c1c64a875d1fc6a0a # v7.0.1
with:
name: reference-parity-diff
overwrite: true
path: |
parity/diff.md
parity/diff.json
if-no-files-found: warn
retention-days: 14
+133
View File
@@ -0,0 +1,133 @@
# Copyright 2026 RustFS Team
# SPDX-License-Identifier: Apache-2.0
name: Release candidate (Linux)
on:
workflow_dispatch:
inputs:
version:
description: Final product version (no preview suffix)
type: string
required: true
source_sha:
description: Full reviewed source commit
type: string
required: true
permissions:
contents: read
actions: read
concurrency:
group: release-candidate-${{ inputs.version }}
cancel-in-progress: false
jobs:
freeze:
if: github.repository == 'rustfs/rustfs' && github.ref == 'refs/heads/main'
runs-on: ubuntu-latest
timeout-minutes: 15
outputs:
manifest: ${{ steps.freeze.outputs.manifest }}
steps:
- uses: actions/checkout@f548e57e544e1ff5a4c46bf1e1b8685f8e4a348a # v7
with:
fetch-depth: 0
persist-credentials: false
- name: Freeze or recover original inputs
id: freeze
env:
GH_TOKEN: ${{ github.token }}
VERSION: ${{ inputs.version }}
SOURCE_SHA: ${{ inputs.source_sha }}
run: python3 scripts/release/pipeline.py prepare --version "$VERSION" --source "$SOURCE_SHA" --directory frozen
- uses: actions/upload-artifact@043fb46d1a93c77aae656e7c1c64a875d1fc6a0a # v7.0.1
if: steps.freeze.outputs.reused != 'true'
with:
name: release-inputs
path: frozen/inputs.json
if-no-files-found: error
retention-days: 30
linux:
needs: freeze
uses: ./.github/workflows/build.yml
with:
candidate_manifest: ${{ needs.freeze.outputs.manifest }}
platforms: linux
upgrade:
needs: [freeze, linux]
runs-on: ubuntu-latest
timeout-minutes: 150
env:
RUSTUP_TOOLCHAIN: ${{ fromJson(needs.freeze.outputs.manifest).toolchain }}
steps:
- uses: actions/checkout@f548e57e544e1ff5a4c46bf1e1b8685f8e4a348a # v7
with:
ref: ${{ fromJson(needs.freeze.outputs.manifest).workflow_sha }}
persist-credentials: false
- uses: ./.github/actions/setup
with:
rust-version: ${{ fromJson(needs.freeze.outputs.manifest).toolchain }}
cache-shared-key: release-upgrade-harness-hosted
cache-save-if: ${{ github.ref == 'refs/heads/main' }}
install-build-packaging-tools: "false"
install-test-tools: "false"
- name: Verify archived server against previous releases
env:
GH_TOKEN: ${{ github.token }}
CANDIDATE_MANIFEST: ${{ needs.freeze.outputs.manifest }}
run: |
python3 - <<'PYVERIFY'
import sys
from pathlib import Path
sys.path.insert(0, 'scripts/release')
import candidate as c, pipeline, verify
inputs = pipeline.manifest_env()
run = pipeline.current_run('release-candidate.yml')
if any(a['name'] == 'release-upgrade' for a in c.artifact_list(run['id'])):
c.download_artifact(run, 'release-upgrade', 'receipts')
pipeline.verify_evidence(inputs, Path('receipts'), {run['id']: run})
c.emit_output('reused', 'true')
else:
target = 'linux-x86_64-gnu'
c.download_artifact(run, f"candidate-{inputs['candidate_id']}-{target}", 'binary')
verify.upgrade(inputs, Path('binary') / f"rustfs-{target}-v{inputs['version']}.zip", Path('receipts'))
PYVERIFY
id: verify
- uses: actions/upload-artifact@043fb46d1a93c77aae656e7c1c64a875d1fc6a0a # v7.0.1
if: steps.verify.outputs.reused != 'true'
with:
name: release-upgrade
path: receipts/*
if-no-files-found: error
retention-days: 30
- uses: actions/upload-artifact@043fb46d1a93c77aae656e7c1c64a875d1fc6a0a # v7.0.1
if: failure()
with:
name: release-upgrade-failure-${{ github.run_attempt }}
path: receipts/*
retention-days: 7
seal-linux:
needs: [freeze, linux, upgrade]
runs-on: ubuntu-latest
timeout-minutes: 30
steps:
- uses: actions/checkout@f548e57e544e1ff5a4c46bf1e1b8685f8e4a348a # v7
with:
persist-credentials: false
- name: Collect trusted receipts and seal Linux bytes
id: seal
env:
GH_TOKEN: ${{ github.token }}
run: |
python3 scripts/release/pipeline.py linux-seal --directory sealed
{
echo '## Linux acceptance'
echo 'Use this run ID and manifest SHA-256 when starting Release candidate (complete).'
sha256sum sealed/linux-manifest.json
echo 'Record cluster, performance, integrity and upgrade acceptance in backlog before continuing.'
} >> "$GITHUB_STEP_SUMMARY"
- uses: actions/upload-artifact@043fb46d1a93c77aae656e7c1c64a875d1fc6a0a # v7.0.1
if: steps.seal.outputs.reused != 'true'
with:
name: release-linux
path: sealed/*
if-no-files-found: error
compression-level: 0
retention-days: 30
+97
View File
@@ -0,0 +1,97 @@
# Copyright 2026 RustFS Team
# SPDX-License-Identifier: Apache-2.0
name: Release candidate (complete)
on:
workflow_dispatch:
inputs:
candidate_run_id:
description: Successful Linux candidate run ID
type: string
required: true
linux_manifest_sha256:
description: Exact accepted Linux manifest SHA-256
type: string
required: true
acceptance_report:
description: Backlog report accepting cluster, performance, integrity and upgrade results
type: string
required: true
permissions:
contents: read
actions: read
concurrency:
group: release-complete-${{ inputs.candidate_run_id }}
cancel-in-progress: false
jobs:
acceptance:
if: github.repository == 'rustfs/rustfs' && github.ref == 'refs/heads/main'
runs-on: ubuntu-latest
timeout-minutes: 30
environment: release-linux-acceptance
outputs:
manifest: ${{ steps.accept.outputs.manifest }}
steps:
- uses: actions/checkout@f548e57e544e1ff5a4c46bf1e1b8685f8e4a348a # v7
with:
fetch-depth: 0
persist-credentials: false
- name: Bind acceptance to the exact Linux manifest
id: accept
env:
GH_TOKEN: ${{ github.token }}
CANDIDATE_RUN: ${{ inputs.candidate_run_id }}
MANIFEST_SHA: ${{ inputs.linux_manifest_sha256 }}
REPORT: ${{ inputs.acceptance_report }}
run: python3 scripts/release/pipeline.py complete-prepare --run-id "$CANDIDATE_RUN" --digest "$MANIFEST_SHA" --report "$REPORT" --directory linux
desktop:
needs: acceptance
uses: ./.github/workflows/build.yml
with:
candidate_manifest: ${{ needs.acceptance.outputs.manifest }}
platforms: desktop
packages:
needs: acceptance
uses: ./.github/workflows/package.yml
with:
candidate_manifest: ${{ needs.acceptance.outputs.manifest }}
images:
needs: acceptance
uses: ./.github/workflows/docker.yml
with:
candidate_manifest: ${{ needs.acceptance.outputs.manifest }}
chart:
needs: acceptance
uses: ./.github/workflows/helm-package.yml
with:
candidate_manifest: ${{ needs.acceptance.outputs.manifest }}
seal:
needs: [acceptance, desktop, packages, images, chart]
runs-on: ubuntu-latest
timeout-minutes: 45
steps:
- uses: actions/checkout@f548e57e544e1ff5a4c46bf1e1b8685f8e4a348a # v7
with:
fetch-depth: 0
persist-credentials: false
- name: Verify every deliverable and seal the candidate
id: seal
env:
GH_TOKEN: ${{ github.token }}
CANDIDATE_RUN: ${{ inputs.candidate_run_id }}
MANIFEST_SHA: ${{ inputs.linux_manifest_sha256 }}
REPORT: ${{ inputs.acceptance_report }}
run: |
python3 scripts/release/pipeline.py complete-seal --run-id "$CANDIDATE_RUN" --digest "$MANIFEST_SHA" --report "$REPORT" --directory sealed
{
echo '## Final publication approval'
echo 'Review this manifest, then explicitly dispatch Publish accepted release with its SHA-256.'
sha256sum sealed/release-manifest.json
} >> "$GITHUB_STEP_SUMMARY"
- uses: actions/upload-artifact@043fb46d1a93c77aae656e7c1c64a875d1fc6a0a # v7.0.1
if: steps.seal.outputs.reused != 'true'
with:
name: release-seal
path: sealed/*
if-no-files-found: error
compression-level: 0
retention-days: 30
+193
View File
@@ -0,0 +1,193 @@
# Copyright 2026 RustFS Team
# SPDX-License-Identifier: Apache-2.0
name: Publish accepted release
on:
workflow_dispatch:
inputs:
complete_run_id:
description: Successful completed candidate run ID
type: string
required: true
manifest_sha256:
description: Exact approved final manifest SHA-256 (authorizes publication)
type: string
required: true
permissions:
contents: read
actions: read
concurrency:
group: release-publish-${{ inputs.complete_run_id }}
cancel-in-progress: false
jobs:
verify:
if: github.repository == 'rustfs/rustfs' && github.ref == 'refs/heads/main'
runs-on: ubuntu-latest
timeout-minutes: 45
outputs:
version: ${{ steps.verify.outputs.version }}
steps:
- uses: actions/checkout@f548e57e544e1ff5a4c46bf1e1b8685f8e4a348a # v7
with:
fetch-depth: 0
persist-credentials: false
- name: Recheck approved bytes and trusted certification
id: verify
env:
GH_TOKEN: ${{ github.token }}
COMPLETE_RUN: ${{ inputs.complete_run_id }}
MANIFEST_SHA: ${{ inputs.manifest_sha256 }}
run: python3 scripts/release/publish.py verify --run-id "$COMPLETE_RUN" --digest "$MANIFEST_SHA" --directory sealed
version:
needs: verify
runs-on: ubuntu-latest
timeout-minutes: 90
concurrency:
group: release-version-${{ needs.verify.outputs.version }}
cancel-in-progress: false
environment: release-publication
permissions:
contents: write
actions: read
packages: write
security-events: write
env:
GH_TOKEN: ${{ github.token }}
COMPLETE_RUN: ${{ inputs.complete_run_id }}
MANIFEST_SHA: ${{ inputs.manifest_sha256 }}
AWS_ACCESS_KEY_ID: ${{ secrets.R2_ACCESS_KEY_ID }}
AWS_SECRET_ACCESS_KEY: ${{ secrets.R2_SECRET_ACCESS_KEY }}
AWS_DEFAULT_REGION: auto
AWS_REQUEST_CHECKSUM_CALCULATION: when_required
AWS_RESPONSE_CHECKSUM_VALIDATION: when_required
AWS_EC2_METADATA_DISABLED: "true"
R2_ENDPOINT: ${{ secrets.R2_ENDPOINT }}
R2_BUCKET: ${{ secrets.R2_BUCKET }}
steps:
- uses: actions/checkout@f548e57e544e1ff5a4c46bf1e1b8685f8e4a348a # v7
with:
fetch-depth: 0
persist-credentials: false
- name: Install promotion tools
run: |
sudo apt-get update
sudo apt-get install -y skopeo
aws --version
- uses: docker/login-action@c94ce9fb468520275223c153574b00df6fe4bcc9 # v3
with:
username: rustfs
password: ${{ secrets.DOCKERHUB_TOKEN }}
- uses: docker/login-action@c94ce9fb468520275223c153574b00df6fe4bcc9 # v3
with:
registry: ghcr.io
username: ${{ secrets.GHCR_USERNAME }}
password: ${{ secrets.GHCR_PASSWORD }}
- uses: docker/login-action@c94ce9fb468520275223c153574b00df6fe4bcc9 # v3
with:
registry: quay.io
username: ${{ secrets.QUAY_USERNAME }}
password: ${{ secrets.QUAY_PASSWORD }}
- name: Share registry credentials with the OCI copier
run: |
install -m 600 "$HOME/.docker/config.json" "$RUNNER_TEMP/registry-auth.json"
echo "REGISTRY_AUTH_FILE=$RUNNER_TEMP/registry-auth.json" >> "$GITHUB_ENV"
- name: Promote verified version
run: python3 scripts/release/publish.py version --run-id "$COMPLETE_RUN" --digest "$MANIFEST_SHA" --directory sealed
- name: Publish accepted musl amd64 scan
uses: github/codeql-action/upload-sarif@99df26d4f13ea111d4ec1a7dddef6063f76b97e9 # v4
with:
sarif_file: sealed/scan-musl-amd64.sarif
category: container-image-musl-amd64
- name: Publish accepted musl arm64 scan
uses: github/codeql-action/upload-sarif@99df26d4f13ea111d4ec1a7dddef6063f76b97e9 # v4
with:
sarif_file: sealed/scan-musl-arm64.sarif
category: container-image-musl-arm64
- name: Publish accepted GNU amd64 scan
uses: github/codeql-action/upload-sarif@99df26d4f13ea111d4ec1a7dddef6063f76b97e9 # v4
with:
sarif_file: sealed/scan-gnu-amd64.sarif
category: container-image-gnu-amd64
- name: Publish accepted GNU arm64 scan
uses: github/codeql-action/upload-sarif@99df26d4f13ea111d4ec1a7dddef6063f76b97e9 # v4
with:
sarif_file: sealed/scan-gnu-arm64.sarif
category: container-image-gnu-arm64
- uses: actions/upload-artifact@043fb46d1a93c77aae656e7c1c64a875d1fc6a0a # v7.0.1
with:
name: release-version-receipt-${{ github.run_attempt }}
path: publication.json
if-no-files-found: error
retention-days: 90
channels:
needs: [verify, version]
runs-on: ubuntu-latest
timeout-minutes: 90
concurrency:
group: rustfs-release-channels
cancel-in-progress: false
permissions:
contents: write
actions: read
packages: write
env:
GH_TOKEN: ${{ github.token }}
COMPLETE_RUN: ${{ inputs.complete_run_id }}
MANIFEST_SHA: ${{ inputs.manifest_sha256 }}
AWS_ACCESS_KEY_ID: ${{ secrets.R2_ACCESS_KEY_ID }}
AWS_SECRET_ACCESS_KEY: ${{ secrets.R2_SECRET_ACCESS_KEY }}
AWS_DEFAULT_REGION: auto
AWS_REQUEST_CHECKSUM_CALCULATION: when_required
AWS_RESPONSE_CHECKSUM_VALIDATION: when_required
AWS_EC2_METADATA_DISABLED: "true"
R2_ENDPOINT: ${{ secrets.R2_ENDPOINT }}
R2_BUCKET: ${{ secrets.R2_BUCKET }}
steps:
- uses: actions/checkout@f548e57e544e1ff5a4c46bf1e1b8685f8e4a348a # v7
with:
fetch-depth: 0
persist-credentials: false
- name: Install promotion tools
run: |
sudo apt-get update
sudo apt-get install -y skopeo
aws --version
- uses: docker/login-action@c94ce9fb468520275223c153574b00df6fe4bcc9 # v3
with:
username: rustfs
password: ${{ secrets.DOCKERHUB_TOKEN }}
- uses: docker/login-action@c94ce9fb468520275223c153574b00df6fe4bcc9 # v3
with:
registry: ghcr.io
username: ${{ secrets.GHCR_USERNAME }}
password: ${{ secrets.GHCR_PASSWORD }}
- uses: docker/login-action@c94ce9fb468520275223c153574b00df6fe4bcc9 # v3
with:
registry: quay.io
username: ${{ secrets.QUAY_USERNAME }}
password: ${{ secrets.QUAY_PASSWORD }}
- name: Share registry credentials with the OCI copier
run: |
install -m 600 "$HOME/.docker/config.json" "$RUNNER_TEMP/registry-auth.json"
echo "REGISTRY_AUTH_FILE=$RUNNER_TEMP/registry-auth.json" >> "$GITHUB_ENV"
- uses: azure/setup-helm@b9e51907a09c216f16ebe8536097933489208112 # v4.3.0
- name: Install pinned OSS client
run: |
curl --fail --location --retry 3 --output "$RUNNER_TEMP/ossutil.zip" https://gosspublic.alicdn.com/ossutil/v2/2.1.1/ossutil-2.1.1-linux-amd64.zip
printf '%s %s\n' 60f3a808cbbdfd4b5269b8f967c6a66f564c9dacff52d93a5d21a588976ab5a2 "$RUNNER_TEMP/ossutil.zip" | sha256sum --check -
unzip -q "$RUNNER_TEMP/ossutil.zip" -d "$RUNNER_TEMP/ossutil-extract"
install -m 755 "$RUNNER_TEMP/ossutil-extract/ossutil-2.1.1-linux-amd64/ossutil" "$RUNNER_TEMP/ossutil"
echo "OSSUTIL_BIN=$RUNNER_TEMP/ossutil" >> "$GITHUB_ENV"
- name: Promote verified channels
env:
HELM_TOKEN: ${{ secrets.RUSTFS_HELM_PACKAGE }}
OSS_ACCESS_KEY_ID: ${{ secrets.ALICLOUDOSS_KEY_ID }}
OSS_ACCESS_KEY_SECRET: ${{ secrets.ALICLOUDOSS_KEY_SECRET }}
OSS_REGION: cn-beijing
OSS_ENDPOINT: https://oss-cn-beijing.aliyuncs.com
run: python3 scripts/release/publish.py channels --run-id "$COMPLETE_RUN" --digest "$MANIFEST_SHA" --directory sealed
- uses: actions/upload-artifact@043fb46d1a93c77aae656e7c1c64a875d1fc6a0a # v7.0.1
with:
name: release-channels-receipt-${{ github.run_attempt }}
path: publication.json
if-no-files-found: error
retention-days: 90
@@ -26,7 +26,13 @@ permissions:
jobs:
alert-on-incomplete-run:
name: Alert on incomplete scheduled run
if: github.repository == 'rustfs/rustfs'
# The registry still owns responsibility and cadence selection. Successful
# and PR runs can never produce an incomplete scheduled-run alert.
if: >-
github.repository == 'rustfs/rustfs' &&
github.event.workflow_run.conclusion != 'success' &&
(github.event.workflow_run.event == 'schedule' ||
github.event.workflow_run.event == 'workflow_run')
# Stay on GitHub-hosted runners: this monitor must keep running when the
# self-hosted sm-standard-* pool is down, which is exactly when the
# scheduled validations it watches stall.
Generated
+232 -103
View File
@@ -322,9 +322,9 @@ dependencies = [
[[package]]
name = "archive-trait"
version = "0.0.15"
version = "0.0.16"
source = "registry+https://github.com/rust-lang/crates.io-index"
checksum = "999bf88cc483664cab77700ca2fdfb0415a73412abe284ef5bc2319d00938e00"
checksum = "0b5040ca935c09888e7dfb04383b4d62fa3571e25bc66b6f345b591e97958524"
dependencies = [
"cap-std",
"thiserror 2.0.21",
@@ -811,9 +811,9 @@ dependencies = [
[[package]]
name = "async-nats"
version = "0.51.0"
version = "0.52.0"
source = "registry+https://github.com/rust-lang/crates.io-index"
checksum = "a4f92759f418774b78c36ef07885849150ce950c67d9bcdeff8a9c253f57d9a1"
checksum = "015f0a7a3b0b29701b78c76a4d4a122624c97c71fa30363381664eb729d619c7"
dependencies = [
"aws-lc-rs",
"base64 0.22.1",
@@ -3906,9 +3906,9 @@ dependencies = [
[[package]]
name = "dial9"
version = "0.5.7"
version = "0.5.8"
source = "registry+https://github.com/rust-lang/crates.io-index"
checksum = "2f3298a54fd82af243c9f122595762f5b0e48bd3d1ea8fe3f3eb5e95c130e881"
checksum = "c83757d61dc75b3e9d3579a3004c7fe0c42aff2f779ca9eeec7f0c8e1ae3fc10"
dependencies = [
"bon",
"dial9-core",
@@ -3990,9 +3990,9 @@ dependencies = [
[[package]]
name = "dial9-tokio-telemetry"
version = "0.5.7"
version = "0.5.8"
source = "registry+https://github.com/rust-lang/crates.io-index"
checksum = "0428cba60fbcf37ab82666f3a7ef3b6d0671e90dfb6a352376813b471319f061"
checksum = "6f0eb05627c12c57e40510e63f9a3d5f628ab4b32fec9ccdb0ae98544c13f15e"
dependencies = [
"bon",
"bytes",
@@ -4136,7 +4136,7 @@ checksum = "d0881ea181b1df73ff77ffaaf9c7544ecc11e82fba9b5f27b262a3c73a332555"
[[package]]
name = "e2e_test"
version = "1.0.2"
version = "1.0.5"
dependencies = [
"anyhow",
"async-compression",
@@ -4525,9 +4525,9 @@ dependencies = [
[[package]]
name = "flate2"
version = "1.1.10"
version = "1.1.11"
source = "registry+https://github.com/rust-lang/crates.io-index"
checksum = "6e634e2e0ebac1ee034020da1ca582e17ffe4e0f5e985823721e168928136dcb"
checksum = "626d4928326b646e70f65ed36fc7cf9f206bdb68a0019ee7c6b4709da4e084c4"
dependencies = [
"crc32fast",
"miniz_oxide",
@@ -4594,7 +4594,7 @@ checksum = "94e7099f6313ecacbe1256e8ff9d617b75d1bcb16a6fddef94866d225a01a14a"
dependencies = [
"io-lifetimes 2.0.4",
"rustix",
"windows-sys 0.59.0",
"windows-sys 0.52.0",
]
[[package]]
@@ -4698,6 +4698,12 @@ version = "0.3.34"
source = "registry+https://github.com/rust-lang/crates.io-index"
checksum = "cd417de3d1d015fc3bfd2b1ea46dfc7bab72ef86f1cc7cc9c78e728b34a6d1fd"
[[package]]
name = "futures-timer"
version = "3.0.4"
source = "registry+https://github.com/rust-lang/crates.io-index"
checksum = "af43fadb8a98512d547e37b4e92e0ced13e205c061b87b4623eff01d918d6968"
[[package]]
name = "futures-util"
version = "0.3.34"
@@ -5385,9 +5391,9 @@ dependencies = [
[[package]]
name = "hotpath"
version = "0.30.3"
version = "0.31.1"
source = "registry+https://github.com/rust-lang/crates.io-index"
checksum = "d9632c1fd213e63c1b88d8f3e59543da4999b17f859cc1e1d67cf8f2207e8935"
checksum = "4039758c8bca06a34c475d17aaafc17ac1073ce2cecd8dba8c82e69edb300bbe"
dependencies = [
"arc-swap",
"async-channel",
@@ -5420,15 +5426,15 @@ dependencies = [
[[package]]
name = "hotpath-drain"
version = "0.30.3"
version = "0.31.1"
source = "registry+https://github.com/rust-lang/crates.io-index"
checksum = "738324f75aec847dea984f6806402f4f1bdfab437b4700fc8c3ebcad2dff6424"
checksum = "bbba0a3badfb9ebf95f22dc6cdb4afcd0ff07b91a85fa3196fd1060576ea3065"
[[package]]
name = "hotpath-macros"
version = "0.30.3"
version = "0.31.1"
source = "registry+https://github.com/rust-lang/crates.io-index"
checksum = "5d92fdfec803932e5764ea93c201cf7c6c2d8cff1871da44f64f5094c9c3ef8a"
checksum = "7d980014875338771b00ac44f1c9e2ea1a10650aa68ce54226233bc767714361"
dependencies = [
"proc-macro2",
"quote",
@@ -6412,9 +6418,9 @@ dependencies = [
[[package]]
name = "lzma-rust2"
version = "0.24.0"
version = "0.25.0"
source = "registry+https://github.com/rust-lang/crates.io-index"
checksum = "490750560fa7851fbbf5d3ab1a8750f09350ed361ed8aed85b1e1e40587812ee"
checksum = "ef1d73c59a56771343712b22ea5615092c612c16ba291c0979e3105dfeec4890"
dependencies = [
"sha2 0.11.1",
]
@@ -7914,10 +7920,34 @@ version = "0.14.0"
source = "registry+https://github.com/rust-lang/crates.io-index"
checksum = "010378780309880b08997fae13be7834dba947d36393bd372f2b1556deb2a2f6"
dependencies = [
"phf_macros",
"phf_shared 0.14.0",
"serde",
]
[[package]]
name = "phf_generator"
version = "0.14.0"
source = "registry+https://github.com/rust-lang/crates.io-index"
checksum = "aeb62e0959d5a1bebc965f4d15d9e2b7cea002b6b0f5ba8cde6cc26738467100"
dependencies = [
"fastrand",
"phf_shared 0.14.0",
]
[[package]]
name = "phf_macros"
version = "0.14.0"
source = "registry+https://github.com/rust-lang/crates.io-index"
checksum = "5fa8d0ca26d424d27630da600c6624696e7dec8bf7b3b492b383c5dc49e5e085"
dependencies = [
"phf_generator",
"phf_shared 0.14.0",
"proc-macro2",
"quote",
"syn 2.0.119",
]
[[package]]
name = "phf_shared"
version = "0.12.1"
@@ -9529,7 +9559,7 @@ dependencies = [
[[package]]
name = "rustfs"
version = "1.0.2"
version = "1.0.5"
dependencies = [
"aes-gcm",
"anyhow",
@@ -9690,7 +9720,7 @@ dependencies = [
[[package]]
name = "rustfs-audit"
version = "1.0.2"
version = "1.0.5"
dependencies = [
"const-str",
"futures",
@@ -9712,7 +9742,7 @@ dependencies = [
[[package]]
name = "rustfs-checksums"
version = "1.0.2"
version = "1.0.5"
dependencies = [
"base64-simd",
"bytes",
@@ -9728,7 +9758,7 @@ dependencies = [
[[package]]
name = "rustfs-common"
version = "1.0.2"
version = "1.0.5"
dependencies = [
"arc-swap",
"hotpath",
@@ -9743,7 +9773,7 @@ dependencies = [
[[package]]
name = "rustfs-concurrency"
version = "1.0.2"
version = "1.0.5"
dependencies = [
"hotpath",
"insta",
@@ -9756,7 +9786,7 @@ dependencies = [
[[package]]
name = "rustfs-config"
version = "1.0.2"
version = "1.0.5"
dependencies = [
"const-str",
"hotpath",
@@ -9766,7 +9796,7 @@ dependencies = [
[[package]]
name = "rustfs-connect-symbol-catalog"
version = "1.0.2"
version = "1.0.5"
dependencies = [
"hex-simd",
"object 0.40.0",
@@ -9778,7 +9808,7 @@ dependencies = [
[[package]]
name = "rustfs-credentials"
version = "1.0.2"
version = "1.0.5"
dependencies = [
"base64-simd",
"hmac 0.13.0",
@@ -9792,7 +9822,7 @@ dependencies = [
[[package]]
name = "rustfs-crypto"
version = "1.0.2"
version = "1.0.5"
dependencies = [
"aes-gcm",
"argon2",
@@ -9813,7 +9843,7 @@ dependencies = [
[[package]]
name = "rustfs-data-usage"
version = "1.0.2"
version = "1.0.5"
dependencies = [
"hotpath",
"rmp-serde",
@@ -9823,7 +9853,7 @@ dependencies = [
[[package]]
name = "rustfs-ecstore"
version = "1.0.2"
version = "1.0.5"
dependencies = [
"arc-swap",
"async-channel",
@@ -9962,7 +9992,7 @@ dependencies = [
[[package]]
name = "rustfs-extension-schema"
version = "1.0.2"
version = "1.0.5"
dependencies = [
"hotpath",
"serde",
@@ -9972,7 +10002,7 @@ dependencies = [
[[package]]
name = "rustfs-filemeta"
version = "1.0.2"
version = "1.0.5"
dependencies = [
"arc-swap",
"base64-simd",
@@ -9999,6 +10029,114 @@ dependencies = [
"xxhash-rust",
]
[[package]]
name = "rustfs-gateway"
version = "0.57.0"
source = "git+https://github.com/rustfs/gateway.git?rev=21283e874c2381bab611e9347acbc5b92e9e2b3c#21283e874c2381bab611e9347acbc5b92e9e2b3c"
dependencies = [
"arc-swap",
"bytes",
"form_urlencoded",
"futures-timer",
"futures-util",
"http 1.5.0",
"http-body",
"http-body-util",
"httparse",
"hyper",
"pin-project-lite",
"rustfs-gateway-core",
"rustfs-gateway-http",
"rustfs-gateway-macros",
"rustfs-gateway-sig",
"rustfs-gateway-stream",
"rustfs-gateway-types",
"rustfs-gateway-xml",
"sha2 0.11.1",
"tokio",
"tower",
"tracing",
"url",
]
[[package]]
name = "rustfs-gateway-adapter"
version = "1.0.5"
dependencies = [
"http 1.5.0",
"rustfs-credentials",
"rustfs-gateway",
"rustfs-gateway-http",
"rustfs-iam",
"rustfs-policy",
"serde_json",
"time",
"tokio",
]
[[package]]
name = "rustfs-gateway-core"
version = "0.54.0"
source = "git+https://github.com/rustfs/gateway.git?rev=21283e874c2381bab611e9347acbc5b92e9e2b3c#21283e874c2381bab611e9347acbc5b92e9e2b3c"
dependencies = [
"bytes",
"http 1.5.0",
"md-5 0.11.0",
"percent-encoding",
"rustfs-gateway-http",
"rustfs-gateway-sig",
"rustfs-gateway-stream",
"rustfs-gateway-types",
"rustfs-gateway-xml",
"serde",
"serde_json",
"subtle",
"zeroize",
]
[[package]]
name = "rustfs-gateway-http"
version = "0.1.4"
source = "git+https://github.com/rustfs/gateway.git?rev=21283e874c2381bab611e9347acbc5b92e9e2b3c#21283e874c2381bab611e9347acbc5b92e9e2b3c"
dependencies = [
"hmac 0.13.0",
"http 1.5.0",
"httparse",
"memchr",
"phf 0.14.0",
"rustfs-gateway-stream",
"rustfs-gateway-types",
"sha2 0.11.1",
"smallvec",
"subtle",
"zeroize",
]
[[package]]
name = "rustfs-gateway-macros"
version = "0.1.4"
source = "git+https://github.com/rustfs/gateway.git?rev=21283e874c2381bab611e9347acbc5b92e9e2b3c#21283e874c2381bab611e9347acbc5b92e9e2b3c"
dependencies = [
"proc-macro2",
"quote",
"syn 3.0.9",
]
[[package]]
name = "rustfs-gateway-sig"
version = "0.13.0"
source = "git+https://github.com/rustfs/gateway.git?rev=21283e874c2381bab611e9347acbc5b92e9e2b3c#21283e874c2381bab611e9347acbc5b92e9e2b3c"
dependencies = [
"hmac 0.13.0",
"http 1.5.0",
"rustfs-gateway-http",
"sha1 0.11.0",
"sha2 0.11.1",
"smallvec",
"subtle",
"zeroize",
]
[[package]]
name = "rustfs-gateway-stream"
version = "0.2.0"
@@ -10043,7 +10181,7 @@ dependencies = [
[[package]]
name = "rustfs-heal"
version = "1.0.2"
version = "1.0.5"
dependencies = [
"async-trait",
"base64-simd",
@@ -10084,7 +10222,7 @@ dependencies = [
[[package]]
name = "rustfs-heal-contracts"
version = "1.0.2"
version = "1.0.5"
dependencies = [
"serde",
"serde_json",
@@ -10094,7 +10232,7 @@ dependencies = [
[[package]]
name = "rustfs-iam"
version = "1.0.2"
version = "1.0.5"
dependencies = [
"arc-swap",
"async-trait",
@@ -10144,7 +10282,7 @@ dependencies = [
[[package]]
name = "rustfs-io-core"
version = "1.0.2"
version = "1.0.5"
dependencies = [
"bytes",
"hotpath",
@@ -10157,7 +10295,7 @@ dependencies = [
[[package]]
name = "rustfs-io-metrics"
version = "1.0.2"
version = "1.0.5"
dependencies = [
"criterion",
"hotpath",
@@ -10221,7 +10359,7 @@ dependencies = [
[[package]]
name = "rustfs-keystone"
version = "1.0.2"
version = "1.0.5"
dependencies = [
"bytes",
"futures",
@@ -10247,7 +10385,7 @@ dependencies = [
[[package]]
name = "rustfs-kms"
version = "1.0.2"
version = "1.0.5"
dependencies = [
"aes-gcm",
"anyhow",
@@ -10297,14 +10435,14 @@ dependencies = [
[[package]]
name = "rustfs-license"
version = "1.0.2"
version = "1.0.5"
dependencies = [
"thiserror 2.0.21",
]
[[package]]
name = "rustfs-lifecycle"
version = "1.0.2"
version = "1.0.5"
dependencies = [
"async-trait",
"hotpath",
@@ -10326,7 +10464,7 @@ dependencies = [
[[package]]
name = "rustfs-lock"
version = "1.0.2"
version = "1.0.5"
dependencies = [
"async-trait",
"compact_str",
@@ -10350,7 +10488,7 @@ dependencies = [
[[package]]
name = "rustfs-log-analyzer"
version = "1.0.2"
version = "1.0.5"
dependencies = [
"chrono",
"flate2",
@@ -10369,7 +10507,7 @@ dependencies = [
[[package]]
name = "rustfs-madmin"
version = "1.0.2"
version = "1.0.5"
dependencies = [
"hotpath",
"http 1.5.0",
@@ -10388,25 +10526,25 @@ dependencies = [
[[package]]
name = "rustfs-mimalloc"
version = "0.7.3"
version = "0.7.4"
source = "registry+https://github.com/rust-lang/crates.io-index"
checksum = "216ecfc9341b5d6e2506d47ce57c9232ae1b8020bb10264ea05286fb234286e7"
checksum = "fc7914232a854336c62058c24a8362c553ac8e7adba2e2395ce9097db24d1510"
dependencies = [
"rustfs-mimalloc-sys",
]
[[package]]
name = "rustfs-mimalloc-sys"
version = "0.7.3"
version = "0.7.4"
source = "registry+https://github.com/rust-lang/crates.io-index"
checksum = "fb16ad68f96883407a75cdb88252b43dc2ea592428f0fc54ff7885756de4b814"
checksum = "8073624242781448512682f1931624ee67108b6b7873b22fb29c1a1df689ac0c"
dependencies = [
"cc",
]
[[package]]
name = "rustfs-notify"
version = "1.0.2"
version = "1.0.5"
dependencies = [
"arc-swap",
"async-trait",
@@ -10441,7 +10579,7 @@ dependencies = [
[[package]]
name = "rustfs-object-capacity"
version = "1.0.2"
version = "1.0.5"
dependencies = [
"criterion",
"futures",
@@ -10460,7 +10598,7 @@ dependencies = [
[[package]]
name = "rustfs-object-data-cache"
version = "1.0.2"
version = "1.0.5"
dependencies = [
"bytes",
"criterion",
@@ -10477,7 +10615,7 @@ dependencies = [
[[package]]
name = "rustfs-obs"
version = "1.0.2"
version = "1.0.5"
dependencies = [
"chrono",
"crossbeam-channel",
@@ -10540,7 +10678,7 @@ dependencies = [
[[package]]
name = "rustfs-policy"
version = "1.0.2"
version = "1.0.5"
dependencies = [
"async-trait",
"base64-simd",
@@ -10571,7 +10709,7 @@ dependencies = [
[[package]]
name = "rustfs-protocols"
version = "1.0.2"
version = "1.0.5"
dependencies = [
"async-compression",
"async-trait",
@@ -10637,7 +10775,7 @@ dependencies = [
[[package]]
name = "rustfs-protos"
version = "1.0.2"
version = "1.0.5"
dependencies = [
"flatbuffers",
"hotpath",
@@ -10662,7 +10800,7 @@ dependencies = [
[[package]]
name = "rustfs-replication"
version = "1.0.2"
version = "1.0.5"
dependencies = [
"byteorder",
"bytes",
@@ -10680,7 +10818,7 @@ dependencies = [
[[package]]
name = "rustfs-rio"
version = "1.0.2"
version = "1.0.5"
dependencies = [
"aes-gcm",
"arc-swap",
@@ -10726,7 +10864,7 @@ dependencies = [
[[package]]
name = "rustfs-rio-v2"
version = "1.0.2"
version = "1.0.5"
dependencies = [
"aes-gcm",
"bytes",
@@ -10749,7 +10887,7 @@ dependencies = [
[[package]]
name = "rustfs-s3-client"
version = "1.0.2"
version = "1.0.5"
dependencies = [
"base64-simd",
"bytes",
@@ -10795,7 +10933,7 @@ dependencies = [
[[package]]
name = "rustfs-s3-ops"
version = "1.0.2"
version = "1.0.5"
dependencies = [
"hotpath",
"rustfs-s3-types",
@@ -10803,7 +10941,7 @@ dependencies = [
[[package]]
name = "rustfs-s3-types"
version = "1.0.2"
version = "1.0.5"
dependencies = [
"bytes",
"futures-core",
@@ -10822,7 +10960,7 @@ dependencies = [
[[package]]
name = "rustfs-s3select-api"
version = "1.0.2"
version = "1.0.5"
dependencies = [
"arc-swap",
"async-compression",
@@ -10864,7 +11002,7 @@ dependencies = [
[[package]]
name = "rustfs-s3select-query"
version = "1.0.2"
version = "1.0.5"
dependencies = [
"async-trait",
"bytes",
@@ -10885,7 +11023,7 @@ dependencies = [
[[package]]
name = "rustfs-scanner"
version = "1.0.2"
version = "1.0.5"
dependencies = [
"async-trait",
"bytes",
@@ -10937,7 +11075,7 @@ dependencies = [
[[package]]
name = "rustfs-scanner-metrics"
version = "1.0.2"
version = "1.0.5"
dependencies = [
"chrono",
"jiff",
@@ -10952,7 +11090,7 @@ dependencies = [
[[package]]
name = "rustfs-security-governance"
version = "1.0.2"
version = "1.0.5"
dependencies = [
"hotpath",
"thiserror 2.0.21",
@@ -10960,7 +11098,7 @@ dependencies = [
[[package]]
name = "rustfs-signer"
version = "1.0.2"
version = "1.0.5"
dependencies = [
"base64-simd",
"bytes",
@@ -10980,7 +11118,7 @@ dependencies = [
[[package]]
name = "rustfs-storage-api"
version = "1.0.2"
version = "1.0.5"
dependencies = [
"async-trait",
"hotpath",
@@ -10995,7 +11133,7 @@ dependencies = [
[[package]]
name = "rustfs-targets"
version = "1.0.2"
version = "1.0.5"
dependencies = [
"arc-swap",
"async-nats",
@@ -11048,7 +11186,7 @@ dependencies = [
[[package]]
name = "rustfs-test-utils"
version = "1.0.2"
version = "1.0.5"
dependencies = [
"hotpath",
"rustfs-data-usage",
@@ -11064,7 +11202,7 @@ dependencies = [
[[package]]
name = "rustfs-tls-runtime"
version = "1.0.2"
version = "1.0.5"
dependencies = [
"arc-swap",
"hotpath",
@@ -11103,7 +11241,7 @@ dependencies = [
[[package]]
name = "rustfs-trusted-proxies"
version = "1.0.2"
version = "1.0.5"
dependencies = [
"async-trait",
"axum",
@@ -11140,7 +11278,7 @@ dependencies = [
[[package]]
name = "rustfs-utils"
version = "1.0.2"
version = "1.0.5"
dependencies = [
"aws-lc-rs",
"base64-simd",
@@ -11183,7 +11321,7 @@ dependencies = [
[[package]]
name = "rustfs-zip"
version = "1.0.2"
version = "1.0.5"
dependencies = [
"async-compression",
"futures",
@@ -11773,9 +11911,9 @@ dependencies = [
[[package]]
name = "serde_spanned"
version = "1.1.2"
version = "1.1.3"
source = "registry+https://github.com/rust-lang/crates.io-index"
checksum = "4e7523beb55eece201a2356bee0bbca0d1ab466c14c07703b2e0ee6d42cb0c2c"
checksum = "9c187e0a369471c550af028cba56d4f4fa2a489ad28151ae95a4d340a2a8b782"
dependencies = [
"serde_core",
]
@@ -12105,9 +12243,9 @@ dependencies = [
[[package]]
name = "smallvec"
version = "1.16.4"
version = "1.16.5"
source = "registry+https://github.com/rust-lang/crates.io-index"
checksum = "d0d5d4b1981b8bbbb7aed204cae5726d683d5d6ac7e74ec54122ea28f315e7c8"
checksum = "149309498a5f7334b12ddae0cad8aba81ce785fe91755068867e017cf98fee46"
dependencies = [
"serde",
]
@@ -12302,9 +12440,9 @@ dependencies = [
[[package]]
name = "starshard"
version = "2.4.0"
version = "2.4.1"
source = "registry+https://github.com/rust-lang/crates.io-index"
checksum = "fc45de5060fab0de00d7a42230877543ae00c37caf6f49975e69851089b023ac"
checksum = "b0883700b50fd0b62120ebc4c07230e02f0cf1ead474c3b3b44eaa13e0a999ea"
dependencies = [
"async-trait",
"hashbrown 0.17.1",
@@ -12553,9 +12691,9 @@ dependencies = [
[[package]]
name = "tar-codec"
version = "0.0.15"
version = "0.0.16"
source = "registry+https://github.com/rust-lang/crates.io-index"
checksum = "b9dad615e50d181afea4bafea23dacb8e33ed52dde06503813570634b932d467"
checksum = "6875ba507bbec6022500f46477159a03f07ebdffdb4ea043254974531df1dac2"
dependencies = [
"archive-trait",
"tar-framing",
@@ -12565,9 +12703,9 @@ dependencies = [
[[package]]
name = "tar-framing"
version = "0.0.15"
version = "0.0.16"
source = "registry+https://github.com/rust-lang/crates.io-index"
checksum = "563e8181b47c9cdcce569eae3ad431eec71543d6269dd60c7bd2350f70bcd233"
checksum = "49727b1a300514960d425d0e741a7da3e693170f3db39de3180fdbbc07e542c5"
dependencies = [
"thiserror 2.0.21",
"tokio",
@@ -12603,10 +12741,10 @@ source = "registry+https://github.com/rust-lang/crates.io-index"
checksum = "32497e9a4c7b38532efcdebeef879707aa9f794296a4f0244f6f69e9bc8574bd"
dependencies = [
"fastrand",
"getrandom 0.4.3",
"getrandom 0.3.4",
"once_cell",
"rustix",
"windows-sys 0.61.2",
"windows-sys 0.52.0",
]
[[package]]
@@ -12941,9 +13079,9 @@ dependencies = [
[[package]]
name = "toml"
version = "1.2.0+spec-1.1.0"
version = "1.2.1+spec-1.1.0"
source = "registry+https://github.com/rust-lang/crates.io-index"
checksum = "c5ffcdf8f50436dbbfd675c1fe1b8e0854f42b7defea8584bdebe1aee900e561"
checksum = "d5755c51863cdac711f1956baf8eb9588e060755f7d3c4c979e70ea1e1f32cd7"
dependencies = [
"indexmap 2.14.2",
"serde_core",
@@ -12956,9 +13094,9 @@ dependencies = [
[[package]]
name = "toml_datetime"
version = "1.1.2+spec-1.1.0"
version = "1.1.3+spec-1.1.0"
source = "registry+https://github.com/rust-lang/crates.io-index"
checksum = "2b86d767906c6c42421dcba507eb9d203e779497710a47782a224bb871653053"
checksum = "173f25c883cd2176a2d8212155638c0a60d484a9d84546d270866f40f56b6e49"
dependencies = [
"serde_core",
]
@@ -13851,15 +13989,6 @@ dependencies = [
"windows-targets 0.52.6",
]
[[package]]
name = "windows-sys"
version = "0.59.0"
source = "registry+https://github.com/rust-lang/crates.io-index"
checksum = "1e38bc4d79ed67fd075bcc251a1c39b32a1776bbe92e5bef1f0bf1f8c531853b"
dependencies = [
"windows-targets 0.52.6",
]
[[package]]
name = "windows-sys"
version = "0.60.2"
@@ -14032,7 +14161,7 @@ source = "registry+https://github.com/rust-lang/crates.io-index"
checksum = "3f3fd376f71958b862e7afb20cfe5a22830e1963462f3a17f49d82a6c1d1f42d"
dependencies = [
"bitflags 2.13.2",
"windows-sys 0.59.0",
"windows-sys 0.52.0",
]
[[package]]
@@ -14282,9 +14411,9 @@ dependencies = [
[[package]]
name = "zip"
version = "9.3.0"
version = "9.5.0"
source = "registry+https://github.com/rust-lang/crates.io-index"
checksum = "eaeb7f16d2605b49aef586996ded29e65642fab7259a3077e537351df6d80426"
checksum = "0700e6c427aff64acdbf02e85fbfd946c2ee12319d1af013092f3a2178914ff8"
dependencies = [
"aes 0.9.4",
"bzip2",
+63 -55
View File
@@ -26,6 +26,7 @@ members = [
"crates/ecstore", # Erasure coding storage implementation
"crates/e2e_test", # End-to-end test suite
"crates/filemeta", # File metadata management
"crates/gateway-adapter", # RustFS implementations of the gateway extension traits
"crates/heal", # Erasure set and object healing
"crates/heal-contracts", # Heal request/response channel contracts
"crates/iam", # Identity and Access Management
@@ -75,7 +76,7 @@ edition = "2024"
license = "Apache-2.0"
repository = "https://github.com/rustfs/rustfs"
rust-version = "1.99.0"
version = "1.0.2"
version = "1.0.5"
homepage = "https://rustfs.com"
description = "RustFS is a high-performance distributed object storage software built using Rust, one of the most popular languages worldwide. "
keywords = ["RustFS", "Minio", "object-storage", "filesystem", "s3"]
@@ -92,56 +93,57 @@ redundant_clone = "warn"
[workspace.dependencies]
# RustFS Internal Crates
rustfs = { path = "./rustfs", version = "1.0.2" }
rustfs-heal = { path = "crates/heal", version = "1.0.2" }
rustfs-heal-contracts = { path = "crates/heal-contracts", version = "1.0.2" }
rustfs-scanner-metrics = { path = "crates/scanner-metrics", version = "1.0.2" }
rustfs-audit = { path = "crates/audit", version = "1.0.2" }
rustfs-checksums = { path = "crates/checksums", version = "1.0.2" }
rustfs-common = { path = "crates/common", version = "1.0.2" }
rustfs-data-usage = { path = "crates/data-usage", version = "1.0.2" }
rustfs-config = { path = "./crates/config", version = "1.0.2" }
rustfs-concurrency = { path = "./crates/concurrency", version = "1.0.2" }
rustfs-credentials = { path = "crates/credentials", version = "1.0.2" }
rustfs-crypto = { path = "crates/crypto", version = "1.0.2" }
rustfs-ecstore = { path = "crates/ecstore", version = "1.0.2" }
rustfs-filemeta = { path = "crates/filemeta", version = "1.0.2" }
rustfs-iam = { path = "crates/iam", version = "1.0.2" }
rustfs-keystone = { path = "crates/keystone", version = "1.0.2" }
rustfs-license = { path = "crates/license", version = "1.0.2" }
rustfs-lifecycle = { path = "crates/lifecycle", version = "1.0.2" }
rustfs-kms = { path = "crates/kms", version = "1.0.2" }
rustfs-lock = { path = "crates/lock", version = "1.0.2" }
rustfs-madmin = { path = "crates/madmin", version = "1.0.2" }
rustfs-notify = { path = "crates/notify", version = "1.0.2" }
rustfs-io-metrics = { path = "crates/io-metrics", version = "1.0.2" }
rustfs-io-core = { path = "crates/io-core", version = "1.0.2" }
rustfs-object-capacity = { path = "crates/object-capacity", version = "1.0.2" }
rustfs-object-data-cache = { path = "crates/object-data-cache", version = "1.0.2", default-features = false }
rustfs-log-analyzer = { path = "crates/log-analyzer", version = "1.0.2" }
rustfs-obs = { path = "crates/obs", version = "1.0.2" }
rustfs-policy = { path = "crates/policy", version = "1.0.2" }
rustfs-protos = { path = "crates/protos", version = "1.0.2" }
rustfs-protocols = { path = "crates/protocols", version = "1.0.2" }
rustfs-replication = { path = "crates/replication", version = "1.0.2" }
rustfs-rio = { path = "crates/rio", version = "1.0.2" }
rustfs-rio-v2 = { path = "crates/rio-v2", version = "1.0.2" }
rustfs-s3-client = { path = "crates/s3-client", version = "1.0.2" }
rustfs-s3-types = { path = "crates/s3-types", version = "1.0.2" }
rustfs-s3-ops = { path = "crates/s3-ops", version = "1.0.2" }
rustfs-s3select-api = { path = "crates/s3select-api", version = "1.0.2" }
rustfs-s3select-query = { path = "crates/s3select-query", version = "1.0.2" }
rustfs-scanner = { path = "crates/scanner", version = "1.0.2" }
rustfs-security-governance = { path = "crates/security-governance", version = "1.0.2" }
rustfs-extension-schema = { path = "crates/extension-schema", version = "1.0.2" }
rustfs-signer = { path = "crates/signer", version = "1.0.2" }
rustfs-storage-api = { path = "crates/storage-api", version = "1.0.2" }
rustfs-trusted-proxies = { path = "crates/trusted-proxies", version = "1.0.2" }
rustfs-targets = { path = "crates/targets", version = "1.0.2" }
rustfs-test-utils = { path = "crates/test-utils", version = "1.0.2" }
rustfs-tls-runtime = { path = "crates/tls-runtime", version = "1.0.2" }
rustfs-utils = { path = "crates/utils", version = "1.0.2" }
rustfs-zip = { path = "./crates/zip", version = "1.0.2" }
rustfs = { path = "./rustfs", version = "1.0.5" }
rustfs-gateway-adapter = { path = "crates/gateway-adapter", version = "1.0.5" }
rustfs-heal = { path = "crates/heal", version = "1.0.5" }
rustfs-heal-contracts = { path = "crates/heal-contracts", version = "1.0.5" }
rustfs-scanner-metrics = { path = "crates/scanner-metrics", version = "1.0.5" }
rustfs-audit = { path = "crates/audit", version = "1.0.5" }
rustfs-checksums = { path = "crates/checksums", version = "1.0.5" }
rustfs-common = { path = "crates/common", version = "1.0.5" }
rustfs-data-usage = { path = "crates/data-usage", version = "1.0.5" }
rustfs-config = { path = "./crates/config", version = "1.0.5" }
rustfs-concurrency = { path = "./crates/concurrency", version = "1.0.5" }
rustfs-credentials = { path = "crates/credentials", version = "1.0.5" }
rustfs-crypto = { path = "crates/crypto", version = "1.0.5" }
rustfs-ecstore = { path = "crates/ecstore", version = "1.0.5" }
rustfs-filemeta = { path = "crates/filemeta", version = "1.0.5" }
rustfs-iam = { path = "crates/iam", version = "1.0.5" }
rustfs-keystone = { path = "crates/keystone", version = "1.0.5" }
rustfs-license = { path = "crates/license", version = "1.0.5" }
rustfs-lifecycle = { path = "crates/lifecycle", version = "1.0.5" }
rustfs-kms = { path = "crates/kms", version = "1.0.5" }
rustfs-lock = { path = "crates/lock", version = "1.0.5" }
rustfs-madmin = { path = "crates/madmin", version = "1.0.5" }
rustfs-notify = { path = "crates/notify", version = "1.0.5" }
rustfs-io-metrics = { path = "crates/io-metrics", version = "1.0.5" }
rustfs-io-core = { path = "crates/io-core", version = "1.0.5" }
rustfs-object-capacity = { path = "crates/object-capacity", version = "1.0.5" }
rustfs-object-data-cache = { path = "crates/object-data-cache", version = "1.0.5", default-features = false }
rustfs-log-analyzer = { path = "crates/log-analyzer", version = "1.0.5" }
rustfs-obs = { path = "crates/obs", version = "1.0.5" }
rustfs-policy = { path = "crates/policy", version = "1.0.5" }
rustfs-protos = { path = "crates/protos", version = "1.0.5" }
rustfs-protocols = { path = "crates/protocols", version = "1.0.5" }
rustfs-replication = { path = "crates/replication", version = "1.0.5" }
rustfs-rio = { path = "crates/rio", version = "1.0.5" }
rustfs-rio-v2 = { path = "crates/rio-v2", version = "1.0.5" }
rustfs-s3-client = { path = "crates/s3-client", version = "1.0.5" }
rustfs-s3-types = { path = "crates/s3-types", version = "1.0.5" }
rustfs-s3-ops = { path = "crates/s3-ops", version = "1.0.5" }
rustfs-s3select-api = { path = "crates/s3select-api", version = "1.0.5" }
rustfs-s3select-query = { path = "crates/s3select-query", version = "1.0.5" }
rustfs-scanner = { path = "crates/scanner", version = "1.0.5" }
rustfs-security-governance = { path = "crates/security-governance", version = "1.0.5" }
rustfs-extension-schema = { path = "crates/extension-schema", version = "1.0.5" }
rustfs-signer = { path = "crates/signer", version = "1.0.5" }
rustfs-storage-api = { path = "crates/storage-api", version = "1.0.5" }
rustfs-trusted-proxies = { path = "crates/trusted-proxies", version = "1.0.5" }
rustfs-targets = { path = "crates/targets", version = "1.0.5" }
rustfs-test-utils = { path = "crates/test-utils", version = "1.0.5" }
rustfs-tls-runtime = { path = "crates/tls-runtime", version = "1.0.5" }
rustfs-utils = { path = "crates/utils", version = "1.0.5" }
rustfs-zip = { path = "./crates/zip", version = "1.0.5" }
# The gateway's S3 types: the DTOs and persisted bucket-configuration codecs the
# engine crates move onto from s3s (rustfs/backlog#2745). The one pin of the
@@ -149,6 +151,12 @@ rustfs-zip = { path = "./crates/zip", version = "1.0.2" }
# the s3s-facing edge of the rustfs binary.
rustfs-gateway-types = { git = "https://github.com/rustfs/gateway.git", rev = "21283e874c2381bab611e9347acbc5b92e9e2b3c", version = "0.32.0", default-features = false }
rustfs-gateway-stream = { git = "https://github.com/rustfs/gateway.git", rev = "21283e874c2381bab611e9347acbc5b92e9e2b3c", version = "0.2.0", default-features = false }
# The gateway framework itself, consumed for its extension traits by the ring-2
# adapter. `default-features = false` keeps the `server` feature, and with it the
# server crate, out of the build.
rustfs-gateway = { git = "https://github.com/rustfs/gateway.git", rev = "21283e874c2381bab611e9347acbc5b92e9e2b3c", version = "0.57.0", default-features = false }
# Test-only: builds the `EffectiveHost` a host-resolution test queries with.
rustfs-gateway-http = { git = "https://github.com/rustfs/gateway.git", rev = "21283e874c2381bab611e9347acbc5b92e9e2b3c", default-features = false }
# Async Runtime and Networking
async-channel = "2.5.0"
@@ -156,7 +164,7 @@ async_zip = { default-features = false, version = "0.0.19" }
mysql_async = { default-features = false, version = "0.37.1" }
async-compression = { version = "0.4.52" }
async-trait = "0.1.92"
async-nats = { version = "0.51.0", default-features = false }
async-nats = { version = "0.52.0", default-features = false }
axum = "0.8.9"
futures = "0.3.34"
futures-core = "0.3.34"
@@ -256,8 +264,8 @@ anyhow = "1.0.104"
arc-swap = "1.9.3"
rustfs-tokio-tar = { version = "0.7.1" }
# Candidate Snowball parser versions exercised by rustfs-zip compatibility fixtures.
tar-codec = "0.0.15"
tar-framing = "0.0.15"
tar-codec = "0.0.16"
tar-framing = "0.0.16"
atoi = "3.1.1"
atomic_enum = "0.3.0"
aws-config = { version = "1.13.0" }
@@ -401,7 +409,7 @@ dav-server = "0.11.0"
# Performance Analysis and Memory Profiling
rustfs-mimalloc = { version = "0.7.3" }
# Preserve Unicode focus filters until rustfs/backlog#2302 is resolved.
hotpath = { version = "0.30.3", default-features = false }
hotpath = { version = "0.31.1", default-features = false }
# Snapshot testing for output format regression detection
insta = { version = "1.49" }
+11 -2
View File
@@ -12,7 +12,9 @@
# See the License for the specific language governing permissions and
# limitations under the License.
FROM alpine:3.24.2 AS build
ARG BINARY_SOURCE=download
FROM alpine:3.24.2 AS download
ARG TARGETARCH
ARG RELEASE=latest
@@ -74,6 +76,14 @@ COPY scripts/download_rc.sh /usr/local/bin/download_rc.sh
RUN sh /usr/local/bin/download_rc.sh "$TARGETARCH" musl "$RC_RELEASE" /build/rc
# Candidate packaging supplies the accepted executables from verified archives.
# The default download stage remains available for local image builds.
FROM scratch AS supplied
ARG TARGETARCH
COPY .release-inputs/${TARGETARCH}/ /build/
FROM ${BINARY_SOURCE} AS build
FROM alpine:3.24.2
ARG RELEASE=latest
@@ -103,7 +113,6 @@ RUN apk upgrade --no-cache && \
tzdata \
&& test "$(TZ=Asia/Kolkata date +%z)" = "+0530"
COPY --from=build /etc/ssl/certs/ca-certificates.crt /etc/ssl/certs/
COPY --from=build /build/rustfs /usr/bin/rustfs
COPY --from=build /build/rc /usr/bin/rc
COPY entrypoint.sh /etc/rustfs/entrypoint.sh
+11 -1
View File
@@ -12,7 +12,9 @@
# See the License for the specific language governing permissions and
# limitations under the License.
FROM ubuntu:26.04 AS build
ARG BINARY_SOURCE=download
FROM ubuntu:26.04 AS download
ARG TARGETARCH
ARG RELEASE=latest
@@ -80,6 +82,14 @@ RUN set -eux; \
COPY scripts/download_rc.sh /usr/local/bin/download_rc.sh
RUN sh /usr/local/bin/download_rc.sh "$TARGETARCH" gnu "$RC_RELEASE" /build/rc
# Candidate packaging supplies the accepted executables from verified archives.
# The default download stage remains available for local image builds.
FROM scratch AS supplied
ARG TARGETARCH
COPY .release-inputs/${TARGETARCH}/ /build/
FROM ${BINARY_SOURCE} AS build
FROM ubuntu:26.04
ARG RELEASE=latest
+22 -4
View File
@@ -24,9 +24,10 @@
#
# Supports cross-compilation for amd64 and arm64 via TARGETPLATFORM.
ARG TARGETPLATFORM
ARG BUILDPLATFORM
ARG TARGETARCH
# Do not declare TARGETPLATFORM, BUILDPLATFORM or TARGETARCH here: a global
# declaration without a value shadows BuildKit's automatic values, so the
# stage-level re-declarations below see empty strings. They are only
# declared inside the stage that uses them.
ARG RUSTFS_BUILD_FEATURES=""
ARG RC_RELEASE=latest
@@ -169,7 +170,24 @@ RUN --mount=type=cache,target=/usr/local/cargo/registry \
;; \
esac
RUN sh /usr/src/rustfs/scripts/download_rc.sh "$TARGETARCH" musl "$RC_RELEASE" /usr/local/bin/rc
# TARGETARCH is only provided by BuildKit. Fall back to TARGETPLATFORM (for
# example set by docker-compose) and then to the build host, like the cargo
# step above, so every path that builds this file passes a real architecture.
RUN set -eu; \
rc_arch="${TARGETARCH:-}"; \
if [ -z "${rc_arch}" ]; then \
case "${TARGETPLATFORM:-}" in \
linux/amd64) rc_arch=amd64 ;; \
linux/arm64) rc_arch=arm64 ;; \
*) \
case "$(uname -m)" in \
x86_64) rc_arch=amd64 ;; \
aarch64|arm64) rc_arch=arm64 ;; \
*) echo "TARGETARCH is empty and TARGETPLATFORM='${TARGETPLATFORM:-}' is not linux/amd64 or linux/arm64; cannot choose an rc architecture on $(uname -m)" >&2; exit 1 ;; \
esac ;; \
esac; \
fi; \
sh /usr/src/rustfs/scripts/download_rc.sh "${rc_arch}" musl "${RC_RELEASE}" /usr/local/bin/rc
# -----------------------------
# Development stage (keeps toolchain)
+3 -3
View File
@@ -145,7 +145,7 @@ chown -R 10001:10001 data logs
docker run -d -p 9000:9000 -p 9001:9001 -v $(pwd)/data:/data -v $(pwd)/logs:/logs rustfs/rustfs:latest
# Using specific version
docker run -d -p 9000:9000 -p 9001:9001 -v $(pwd)/data:/data -v $(pwd)/logs:/logs rustfs/rustfs:1.0.4
docker run -d -p 9000:9000 -p 9001:9001 -v $(pwd)/data:/data -v $(pwd)/logs:/logs rustfs/rustfs:1.0.5
```
If you use [podman](https://github.com/containers/podman) instead of docker, you can install the RustFS with the below command
@@ -224,7 +224,7 @@ For developers who want to build RustFS Docker images from source with multi-arc
./docker-buildx.sh --push
# Build specific version
./docker-buildx.sh --release 1.0.4 --push
./docker-buildx.sh --release 1.0.5 --push
# Build for custom registry
./docker-buildx.sh --registry your-registry.com --namespace yourname --push
@@ -242,7 +242,7 @@ You can also use Make targets for convenience:
```bash
make docker-buildx # Build locally
make docker-buildx-push # Build and push
make docker-buildx-version VERSION=1.0.4 # Build specific version
make docker-buildx-version VERSION=1.0.5 # Build specific version
make help-docker # Show all Docker-related commands
```
+3 -3
View File
@@ -138,7 +138,7 @@ chown -R 10001:10001 data logs
docker run -d -p 9000:9000 -p 9001:9001 -v $(pwd)/data:/data -v $(pwd)/logs:/logs rustfs/rustfs:latest
# 使用指定版本运行
docker run -d -p 9000:9000 -p 9001:9001 -v $(pwd)/data:/data -v $(pwd)/logs:/logs rustfs/rustfs:1.0.4
docker run -d -p 9000:9000 -p 9001:9001 -v $(pwd)/data:/data -v $(pwd)/logs:/logs rustfs/rustfs:1.0.5
```
如果您通过绑定挂载启用 TLS 证书目录,也请用同样方式准备该目录:
@@ -176,7 +176,7 @@ docker compose -f docker-compose-simple.yml up -d
./docker-buildx.sh --push
# 构建指定版本
./docker-buildx.sh --release 1.0.4 --push
./docker-buildx.sh --release 1.0.5 --push
# 构建并推送到自定义仓库
./docker-buildx.sh --registry your-registry.com --namespace yourname --push
@@ -193,7 +193,7 @@ docker compose -f docker-compose-simple.yml up -d
```bash
make docker-buildx # 本地构建
make docker-buildx-push # 构建并推送
make docker-buildx-version VERSION=1.0.4 # 构建指定版本
make docker-buildx-version VERSION=1.0.5 # 构建指定版本
make help-docker # 显示所有 Docker 相关命令
```
+100
View File
@@ -902,4 +902,104 @@ mod tests {
assert_eq!(raw.len(), 8);
assert_eq!(&raw[..], reference.digest().to_be_bytes().as_slice());
}
#[test]
fn checksums_streaming_chunks_and_header_encoding_match_oneshot_across_unaligned_boundaries() {
use crate::http::HttpChecksum;
let all = [
ChecksumAlgorithm::Crc32,
ChecksumAlgorithm::Crc32c,
ChecksumAlgorithm::Crc64Nvme,
ChecksumAlgorithm::Sha1,
ChecksumAlgorithm::Sha256,
ChecksumAlgorithm::Sha512,
ChecksumAlgorithm::Xxhash3,
ChecksumAlgorithm::Xxhash64,
ChecksumAlgorithm::Xxhash128,
];
for &payload_len in &[0usize, 1, 31, 33, 1024, 4096 + 37, 64 * 1024 + 19] {
let payload: Vec<u8> = (0..payload_len).map(|i| ((i * 19 + 5) % 253) as u8).collect();
for algorithm in all {
let mut one_shot = algorithm.into_impl();
one_shot.update(&payload);
let expected_raw = one_shot.finalize();
assert_eq!(expected_raw.len(), algorithm.raw_len());
let mut one_shot_hdr = algorithm.into_impl();
one_shot_hdr.update(&payload);
let expected_headers = one_shot_hdr.headers();
for &chunk_size in &[1usize, 31, 1021, 4096] {
if payload_len > 4096 && chunk_size == 1 {
continue;
}
let mut streaming: Box<dyn HttpChecksum> = algorithm.into_impl();
for chunk in payload.chunks(chunk_size) {
streaming.update(chunk);
}
let actual_headers = streaming.headers();
assert_eq!(
actual_headers.get(algorithm.http_header_name()),
expected_headers.get(algorithm.http_header_name()),
"{algorithm:?} streaming mismatch at payload_len={payload_len}, chunk_size={chunk_size}"
);
}
}
}
}
#[test]
#[ignore = "utility throughput microbenchmark; run via cargo test -p rustfs-checksums --lib --release -- --ignored --nocapture"]
fn microbench_checksums_streaming_chunks_and_header_encoding() {
use crate::http::HttpChecksum;
use std::hint::black_box;
use std::time::Instant;
let all = [
ChecksumAlgorithm::Crc32,
ChecksumAlgorithm::Crc32c,
ChecksumAlgorithm::Crc64Nvme,
ChecksumAlgorithm::Sha1,
ChecksumAlgorithm::Sha256,
ChecksumAlgorithm::Sha512,
ChecksumAlgorithm::Xxhash3,
ChecksumAlgorithm::Xxhash64,
ChecksumAlgorithm::Xxhash128,
];
let payload: Vec<u8> = (0..(256 * 1024)).map(|i| ((i * 19 + 5) % 253) as u8).collect();
for &chunk_size in &[4 * 1024usize, 64 * 1024, 256 * 1024] {
for algorithm in all {
let mut one_shot = algorithm.into_impl();
one_shot.update(&payload);
let expected_headers = one_shot.headers();
let iters = 8usize;
let t0 = Instant::now();
let mut last_headers = http::HeaderMap::new();
for _ in 0..iters {
let mut streaming: Box<dyn HttpChecksum> = algorithm.into_impl();
for chunk in payload.chunks(chunk_size) {
streaming.update(black_box(chunk));
}
last_headers = streaming.headers();
black_box(&last_headers);
}
let elapsed = t0.elapsed();
assert_eq!(
last_headers.get(algorithm.http_header_name()),
expected_headers.get(algorithm.http_header_name()),
"{algorithm:?} streaming chunk_size={chunk_size} header mismatch"
);
let total_mib = (payload.len() * iters) as f64 / (1024.0 * 1024.0);
let mibs = total_mib / elapsed.as_secs_f64().max(1e-9);
println!(
"[microbench][checksums] algo={algorithm:?} chunk_size={chunk_size}B throughput={mibs:.1} MiB/s (size={}B)",
algorithm.raw_len()
);
}
}
}
}
+1
View File
@@ -29,6 +29,7 @@ Registered in [`src/lib.rs`](src/lib.rs). Grouped by concern:
| **distributed 4×4** | [`src/distributed/`](src/distributed) | Nightly `e2e-distributed` lane: S3, object lock/WORM, versioning, bucket/site replication, quota, expand/decommission/rebalance, concurrency, chaos, 4-node upgrade of historical data and IAM AK/SK. Map: [`docs/testing/distributed-e2e.md`](../../docs/testing/distributed-e2e.md) |
| **chaos / reliability** | [`src/chaos.rs`](src/chaos.rs), `reliability_disk_fault_test`, `heal_erasure_disk_rebuild_test`, `server_startup_failfast_test` | Disk offline/replace/corrupt, EC rebuild, heal, fail-fast startup |
| **upgrade compatibility** | `upgrade_compatibility_test` | Pinned previous-release writes followed by current-build reads on the same data directory |
| **dual-stack differential** | `dual_stack_differential` | Two single-node servers answer one SDK script for all 73 S3 operations and every differing `(op, field, a, b)` is reported. Side A is the binary under test on `RUSTFS_S3_STACK=legacy`; side B runs `RUSTFS_E2E_STACK_B`. Set `RUSTFS_E2E_REFERENCE_BINARY` to a released `rustfs` binary to make side A that binary instead (started without `RUSTFS_S3_STACK`, which older releases do not know). A missing, non-file or non-executable path fails the run; it never skips or falls back to the same-binary mode. The report in `<target>/dual-stack/<run>.json` records `reference_side` and each side's `--version` output |
The external-tool `storage_metric_ownership_test` validates the OTLP/Collector/Prometheus path, including a rolling upgrade and node failures. See the [storage metrics guide](../../docs/operations/storage-metrics.md) for its required binaries and focused command.
+12
View File
@@ -774,6 +774,18 @@ impl RustFSTestEnvironment {
.await
}
/// Like [`Self::start_rustfs_server_from_binary`], but leaves other running RustFS processes
/// alone, so that two servers of different binaries can be alive at once.
pub async fn start_rustfs_server_from_binary_without_cleanup(
&mut self,
binary_path: &Path,
extra_args: Vec<&str>,
extra_env: &[(&str, &str)],
) -> Result<(), Box<dyn std::error::Error + Send + Sync>> {
self.start_rustfs_server_inner_with_binary(binary_path, extra_args, extra_env, false)
.await
}
/// Start RustFS server with basic configuration
pub async fn start_rustfs_server(&mut self, extra_args: Vec<&str>) -> Result<(), Box<dyn std::error::Error + Send + Sync>> {
self.start_rustfs_server_inner(extra_args, &[], true).await
@@ -1285,7 +1285,9 @@ async fn list_parts_disk_loss_after_metadata_read(
let error = result.expect_err("one missing directory and three unavailable participants cannot prove an empty upload");
let response = error.raw_response().expect("ListParts must return an S3 service response");
assert_eq!(response.status().as_u16(), 503);
assert_eq!(error.as_service_error().and_then(|err| err.meta().code()), Some("SlowDownRead"));
// Since #9149, SlowDownRead is reserved for GET/HEAD object reads; a
// ListParts read-quorum failure is reported as ServiceUnavailable.
assert_eq!(error.as_service_error().and_then(|err| err.meta().code()), Some("ServiceUnavailable"));
assert!(response.headers().get("x-rustfs-error-cause").is_none());
assert!(response.headers().get("retry-after").is_none());
assert!(
@@ -493,7 +493,12 @@ async fn compare_degraded_legacy_list_parts(dist: &mut DistCluster, seed: &Upgra
assert!(legacy.parts().is_empty());
let current = current.expect_err("current coordinator must not infer empty from missing directories and offline disks");
assert_eq!(current.raw_response().expect("current S3 response").status().as_u16(), 503);
assert_eq!(current.as_service_error().and_then(|error| error.meta().code()), Some("SlowDownRead"));
// Since #9149, SlowDownRead is reserved for GET/HEAD object reads; a
// ListParts read-quorum failure is reported as ServiceUnavailable.
assert_eq!(
current.as_service_error().and_then(|error| error.meta().code()),
Some("ServiceUnavailable")
);
dist.cluster.start_node_from_binary(3, previous).await?;
wait_for_upgrade_write_readiness(&clients, &seed.history_bucket, "mixed-list-restored", CREDENTIAL_TIMEOUT).await?;
for (path, before) in original_files {
@@ -232,6 +232,12 @@ pub(crate) struct Report {
pub run: String,
pub stack_a: String,
pub stack_b: String,
/// Which side is the pinned release binary: `"a"`, or `None` when both sides are the binary
/// under test.
pub reference_side: Option<&'static str>,
/// `--version` output of the binary each side actually ran.
pub version_a: String,
pub version_b: String,
pub region_a: Option<String>,
pub region_b: Option<String>,
pub compared_ops: usize,
@@ -480,10 +486,13 @@ mod tests {
fn the_report_lists_each_difference_as_op_field_a_b() {
let directory = std::env::temp_dir().join(format!("dual-stack-report-{}", uuid::Uuid::new_v4()));
let report = Report {
schema: 1,
schema: 2,
run: "unit".into(),
stack_a: "legacy".into(),
stack_b: "legacy".into(),
reference_side: Some("a"),
version_a: "rustfs 1.0.4".into(),
version_b: "rustfs 1.1.0".into(),
region_a: None,
region_b: Some("eu-west-1".into()),
compared_ops: 1,
@@ -498,6 +507,9 @@ mod tests {
let _ = std::fs::remove_dir_all(&directory);
assert_eq!(path.file_name().and_then(|name| name.to_str()), Some("unit.json"));
assert_eq!(json["compared_ops"], 1);
assert_eq!(json["reference_side"], "a");
assert_eq!(json["version_a"], "rustfs 1.0.4");
assert_eq!(json["version_b"], "rustfs 1.1.0");
assert_eq!(
json["differences"],
serde_json::json!([{
@@ -14,7 +14,10 @@
//! Dual-stack differential: one `rustfs` binary, two processes, one SDK script (rustfs/backlog#2740).
//!
//! Stack A always runs `RUSTFS_S3_STACK=legacy`. Stack B runs the value of `RUSTFS_E2E_STACK_B`
//! Stack A runs `RUSTFS_S3_STACK=legacy` of the binary under test, unless `RUSTFS_E2E_REFERENCE_BINARY`
//! names a released binary: then A is that binary, started without `RUSTFS_S3_STACK` (older
//! releases do not know it), and a bad path fails the run instead of falling back (see
//! `reference`). The report names which side is the reference and the `--version` each side ran. Stack B runs the value of `RUSTFS_E2E_STACK_B`
//! (default `legacy` until the gateway stack is assembled), and `RUSTFS_E2E_STACK_B_REGION`, when
//! set, starts B in another region so that a difference can be injected by hand. Both servers
//! answer the script in `scenarios`, which covers every operation of `impl S3 for FS`; `compare`
@@ -23,16 +26,20 @@
//! is process-global, so two stacks in one process would read each other's buckets.
mod compare;
mod reference;
mod scenarios;
use crate::common::{RustFSTestEnvironment, init_logging, workspace_root};
use crate::common::{RustFSTestEnvironment, init_logging, rustfs_binary_path, workspace_root};
use compare::{Difference, Observation, Report, compare, compared_cases, compared_ops, render};
use std::path::PathBuf;
use reference::{SideA, binary_version};
use std::path::{Path, PathBuf};
use std::time::{SystemTime, UNIX_EPOCH};
type BoxError = Box<dyn std::error::Error + Send + Sync>;
const STACK_A: &str = "legacy";
/// Label of side A in the report when it is the released reference binary.
const STACK_A_REFERENCE: &str = "reference";
const ENV_STACK_B: &str = "RUSTFS_E2E_STACK_B";
const ENV_STACK_B_REGION: &str = "RUSTFS_E2E_STACK_B_REGION";
/// Becomes `gateway` once rustfs/backlog#2734 T2.5 assembles the gateway stack.
@@ -52,22 +59,31 @@ const CONTROL_REGION_B: &str = "eu-west-1";
const MIN_NORMAL_SUCCESSES: usize = 68;
struct PairConfig {
side_a: SideA,
stack_b: String,
region_a: Option<String>,
region_b: Option<String>,
}
impl PairConfig {
fn from_env() -> Self {
Self {
fn from_env() -> Result<Self, BoxError> {
Ok(Self {
side_a: SideA::from_env()?,
stack_b: std::env::var(ENV_STACK_B).unwrap_or_else(|_| DEFAULT_STACK_B.to_owned()),
region_a: None,
region_b: std::env::var(ENV_STACK_B_REGION).ok(),
})
}
fn stack_a_label(&self) -> &'static str {
match self.side_a {
SideA::SameBinary => STACK_A,
SideA::Reference(_) => STACK_A_REFERENCE,
}
}
}
/// Starts one server on `stack`, overriding its region only when asked.
/// Starts one server of the binary under test on `stack`, overriding its region only when asked.
async fn start_stack(stack: &str, region: Option<&str>) -> Result<RustFSTestEnvironment, BoxError> {
let mut env = RustFSTestEnvironment::new().await?;
let mut vars = vec![("RUSTFS_S3_STACK", stack)];
@@ -78,12 +94,28 @@ async fn start_stack(stack: &str, region: Option<&str>) -> Result<RustFSTestEnvi
Ok(env)
}
/// Starts one server of a released binary. `RUSTFS_S3_STACK` is not passed: older releases do not
/// know it and have exactly one stack.
async fn start_reference(binary: &Path, region: Option<&str>) -> Result<RustFSTestEnvironment, BoxError> {
let mut env = RustFSTestEnvironment::new().await?;
let mut vars = Vec::new();
if let Some(region) = region {
vars.push(("RUSTFS_REGION", region));
}
env.start_rustfs_server_from_binary_without_cleanup(binary, vec![], &vars)
.await?;
Ok(env)
}
/// Starts stack A and stack B side by side, each with its own port and data directory.
async fn spawn_pair(config: &PairConfig) -> Result<(RustFSTestEnvironment, RustFSTestEnvironment), BoxError> {
tokio::try_join!(
start_stack(STACK_A, config.region_a.as_deref()),
start_stack(&config.stack_b, config.region_b.as_deref())
)
let side_a = async {
match &config.side_a {
SideA::SameBinary => start_stack(STACK_A, config.region_a.as_deref()).await,
SideA::Reference(binary) => start_reference(binary, config.region_a.as_deref()).await,
}
};
tokio::try_join!(side_a, start_stack(&config.stack_b, config.region_b.as_deref()))
}
fn report_directory() -> PathBuf {
@@ -104,16 +136,34 @@ struct Run {
/// Runs the script against both stacks concurrently and writes the report.
async fn run_differential(label: &str, config: &PairConfig) -> Run {
let binary_under_test = rustfs_binary_path();
let (version_a, version_b) = match &config.side_a {
SideA::SameBinary => {
let version = binary_version(&binary_under_test)
.await
.expect("capture the binary under test version");
(version.clone(), version)
}
SideA::Reference(binary) => (
binary_version(binary).await.expect("capture the reference binary version"),
binary_version(&binary_under_test)
.await
.expect("capture the binary under test version"),
),
};
let (server_a, server_b) = spawn_pair(config).await.expect("start stack A and stack B");
let (a, b) = tokio::join!(Box::pin(scenarios::run(&server_a)), Box::pin(scenarios::run(&server_b)));
let millis = SystemTime::now()
.duration_since(UNIX_EPOCH)
.map_or(0, |elapsed| elapsed.as_millis());
let report = Report {
schema: 1,
schema: 2,
run: format!("{label}-{millis}-{}", std::process::id()),
stack_a: STACK_A.to_owned(),
stack_a: config.stack_a_label().to_owned(),
stack_b: config.stack_b.clone(),
reference_side: matches!(config.side_a, SideA::Reference(_)).then_some("a"),
version_a,
version_b,
region_a: config.region_a.clone(),
region_b: config.region_b.clone(),
compared_ops: compared_ops(&a, &b).len(),
@@ -131,7 +181,7 @@ mod tests {
#[tokio::test]
async fn both_stacks_answer_every_operation_identically() {
init_logging();
let config = PairConfig::from_env();
let config = PairConfig::from_env().expect("RUSTFS_E2E_REFERENCE_BINARY must be valid when set");
let run = run_differential("self-comparison", &config).await;
let report = &run.report;
@@ -146,9 +196,11 @@ mod tests {
assert_eq!(report.compared_ops, scenarios::OPERATIONS.len(), "report: {}", run.path.display());
assert!(
report.differences.is_empty(),
"stack {} and stack {} differ on {} of {} cases (report: {}):\n{}",
"stack {} ({}) and stack {} ({}) differ on {} of {} cases (report: {}):\n{}",
report.stack_a,
report.version_a,
report.stack_b,
report.version_b,
report.differences.len(),
report.compared_cases,
run.path.display(),
@@ -160,7 +212,11 @@ mod tests {
.iter()
.filter(|observation| observation.case == "negative" && !(400..500).contains(&observation.status))
.collect();
assert!(not_4xx.is_empty(), "negative cases must fail with a 4xx on {STACK_A}: {not_4xx:#?}");
assert!(
not_4xx.is_empty(),
"negative cases must fail with a 4xx on side A ({}): {not_4xx:#?}",
report.version_a
);
let successes = run
.a
.iter()
@@ -168,7 +224,8 @@ mod tests {
.count();
assert!(
successes >= MIN_NORMAL_SUCCESSES,
"only {successes} normal cases succeeded on {STACK_A}: {:#?}",
"only {successes} normal cases succeeded on side A ({}): {:#?}",
report.version_a,
run.a
);
}
@@ -177,6 +234,7 @@ mod tests {
async fn differential_detects_an_injected_difference() {
init_logging();
let config = PairConfig {
side_a: SideA::SameBinary,
stack_b: STACK_A.to_owned(),
region_a: Some(CONTROL_REGION_A.to_owned()),
region_b: Some(CONTROL_REGION_B.to_owned()),
@@ -0,0 +1,163 @@
// Copyright 2024 RustFS Team
//
// Licensed under the Apache License, Version 2.0 (the "License");
// you may not use this file except in compliance with the License.
// You may obtain a copy of the License at
//
// http://www.apache.org/licenses/LICENSE-2.0
//
// Unless required by applicable law or agreed to in writing, software
// distributed under the License is distributed on an "AS IS" BASIS,
// WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
// See the License for the specific language governing permissions and
// limitations under the License.
//! Which binary side A of the differential runs, and what it reports about itself.
//!
//! `RUSTFS_E2E_REFERENCE_BINARY` names a released `rustfs` binary to use as the reference. Unset,
//! side A is the binary under test on `RUSTFS_S3_STACK=legacy`. Set, it must resolve to an
//! executable file or the run fails: a reference that was asked for and silently replaced by the
//! binary under test would report an agreement with itself as agreement with the release.
use std::ffi::OsString;
use std::path::{Path, PathBuf};
type BoxError = Box<dyn std::error::Error + Send + Sync>;
pub(super) const ENV_REFERENCE_BINARY: &str = "RUSTFS_E2E_REFERENCE_BINARY";
#[derive(Clone, Debug, PartialEq, Eq)]
pub(super) enum SideA {
/// Both sides are the binary under test; A runs `RUSTFS_S3_STACK=legacy`.
SameBinary,
/// A is this released binary, started without `RUSTFS_S3_STACK`.
Reference(PathBuf),
}
impl SideA {
pub(super) fn from_env() -> Result<Self, BoxError> {
Self::resolve(std::env::var_os(ENV_REFERENCE_BINARY))
}
/// `None` is the only value that selects same-binary mode; every other value must validate.
pub(super) fn resolve(value: Option<OsString>) -> Result<Self, BoxError> {
let Some(value) = value else {
return Ok(Self::SameBinary);
};
let path = PathBuf::from(value);
validate_executable(&path)?;
Ok(Self::Reference(path))
}
}
fn validate_executable(path: &Path) -> Result<(), BoxError> {
if path.as_os_str().is_empty() {
return Err(format!("{ENV_REFERENCE_BINARY} is set but empty").into());
}
let metadata = std::fs::metadata(path)
.map_err(|error| format!("{ENV_REFERENCE_BINARY} points to {}, which cannot be read: {error}", path.display()))?;
if !metadata.is_file() {
return Err(format!("{ENV_REFERENCE_BINARY} points to {}, which is not a file", path.display()).into());
}
if !is_executable(&metadata) {
return Err(format!("{ENV_REFERENCE_BINARY} points to {}, which is not executable", path.display()).into());
}
Ok(())
}
#[cfg(unix)]
fn is_executable(metadata: &std::fs::Metadata) -> bool {
use std::os::unix::fs::PermissionsExt;
metadata.permissions().mode() & 0o111 != 0
}
#[cfg(not(unix))]
fn is_executable(_metadata: &std::fs::Metadata) -> bool {
true
}
/// What `<binary> --version` printed, trimmed. Fails when the binary does not run, exits
/// non-zero, or prints nothing, because an empty version would report nothing as observed.
pub(super) async fn binary_version(binary: &Path) -> Result<String, BoxError> {
let output = tokio::process::Command::new(binary).arg("--version").output().await?;
if !output.status.success() {
return Err(format!("{} --version failed with {}", binary.display(), output.status).into());
}
let version = String::from_utf8_lossy(&output.stdout).trim().to_owned();
if version.is_empty() {
return Err(format!("{} --version printed nothing", binary.display()).into());
}
Ok(version)
}
#[cfg(all(test, unix))]
mod tests {
use super::*;
use std::os::unix::fs::PermissionsExt;
fn script(dir: &Path, name: &str, body: &str, mode: u32) -> PathBuf {
let path = dir.join(name);
std::fs::write(&path, body).expect("write the stand-in");
std::fs::set_permissions(&path, std::fs::Permissions::from_mode(mode)).expect("chmod");
path
}
#[test]
fn missing_path_is_an_error_not_a_fallback() {
let dir = tempfile::tempdir().expect("tempdir");
let error = SideA::resolve(Some(dir.path().join("absent").into_os_string())).expect_err("must fail closed");
assert!(error.to_string().contains("cannot be read"), "{error}");
}
#[test]
fn non_executable_file_is_an_error() {
let dir = tempfile::tempdir().expect("tempdir");
let path = script(dir.path(), "rustfs", "#!/bin/sh\necho v\n", 0o644);
let error = SideA::resolve(Some(path.into_os_string())).expect_err("must fail closed");
assert!(error.to_string().contains("not executable"), "{error}");
}
#[test]
fn directory_is_an_error() {
let dir = tempfile::tempdir().expect("tempdir");
let error = SideA::resolve(Some(dir.path().as_os_str().to_owned())).expect_err("must fail closed");
assert!(error.to_string().contains("not a file"), "{error}");
}
#[test]
fn empty_value_is_an_error() {
let error = SideA::resolve(Some(OsString::new())).expect_err("must fail closed");
assert!(error.to_string().contains("empty"), "{error}");
}
#[test]
fn unset_selects_same_binary_mode() {
assert_eq!(SideA::resolve(None).expect("unset is valid"), SideA::SameBinary);
}
#[test]
fn executable_file_selects_reference_mode() {
let dir = tempfile::tempdir().expect("tempdir");
let path = script(dir.path(), "rustfs", "#!/bin/sh\necho v\n", 0o755);
assert_eq!(
SideA::resolve(Some(path.clone().into_os_string())).expect("valid"),
SideA::Reference(path)
);
}
#[tokio::test]
async fn version_output_is_captured_from_the_binary() {
let dir = tempfile::tempdir().expect("tempdir");
let path = script(dir.path(), "rustfs", "#!/bin/sh\necho 'rustfs 1.0.4 (abc123)'\n", 0o755);
assert_eq!(binary_version(&path).await.expect("version"), "rustfs 1.0.4 (abc123)");
}
#[tokio::test]
async fn failing_or_silent_version_is_an_error() {
let dir = tempfile::tempdir().expect("tempdir");
let failing = script(dir.path(), "failing", "#!/bin/sh\nexit 3\n", 0o755);
assert!(binary_version(&failing).await.is_err());
let silent = script(dir.path(), "silent", "#!/bin/sh\nexit 0\n", 0o755);
assert!(binary_version(&silent).await.is_err());
}
}
+255 -2
View File
@@ -21,12 +21,15 @@
#[cfg(test)]
mod tests {
use crate::common::{RustFSTestEnvironment, init_logging};
use crate::common::{RustFSTestEnvironment, admin_create_user, init_logging};
use aws_sdk_s3::Client;
use aws_sdk_s3::primitives::ByteStream;
use aws_sdk_s3::types::{BucketVersioningStatus, Delete, ObjectIdentifier, VersioningConfiguration};
use std::error::Error;
use tracing::info;
type TestResult = Result<(), Box<dyn Error + Send + Sync>>;
async fn create_bucket(client: &Client, bucket: &str) -> Result<(), Box<dyn Error + Send + Sync>> {
client.create_bucket().bucket(bucket).send().await?;
Ok(())
@@ -90,7 +93,7 @@ mod tests {
Ok(())
}
/// Duplicate and repeated slashes after a leading slash collapse MinIO-style.
/// Duplicate and repeated slashes after a leading slash collapse MinIO-style across single and bulk operations.
#[tokio::test]
async fn test_put_object_with_duplicate_slashes_normalized() -> Result<(), Box<dyn Error + Send + Sync>> {
init_logging();
@@ -144,8 +147,258 @@ mod tests {
"GET after DELETE with raw key must return HTTP 404, got {error:?}"
);
assert_bulk_delete_normalizes_keys(&client).await?;
assert_bulk_delete_authorizes_normalized_keys(&env, &client).await?;
assert_bulk_delete_preserves_version_identity(&client).await?;
env.stop_server();
info!("Test completed successfully");
Ok(())
}
async fn assert_bulk_delete_normalizes_keys(client: &Client) -> TestResult {
let bucket = "test-bulk-leading-slash";
create_bucket(client, bucket).await?;
let cases = [
("/foo/bar", "foo/bar"),
("//keyname", "keyname"),
("/dir///sub//file", "dir/sub/file"),
("/folder///", "folder/"),
("/literal%2F+名", "literal%2F+名"),
("plain", "plain"),
];
let invalid = ["///", "/dir/../keep", "dir//keep"];
client
.put_object()
.bucket(bucket)
.key("keep")
.body(ByteStream::from_static(b"keep"))
.send()
.await?;
client
.put_object()
.bucket(bucket)
.key("dir/keep")
.body(ByteStream::from_static(b"keep"))
.send()
.await?;
for quiet in [false, true] {
for (raw, _) in cases {
client
.put_object()
.bucket(bucket)
.key(raw)
.body(ByteStream::from_static(b""))
.send()
.await?;
}
let listing = client.list_objects_v2().bucket(bucket).send().await?;
let stored: Vec<_> = listing.contents().iter().filter_map(|object| object.key()).collect();
let mut expected: Vec<_> = cases
.iter()
.map(|(_, normalized)| *normalized)
.chain(["dir/keep", "keep"])
.collect();
expected.sort_unstable();
assert_eq!(stored, expected, "PUT must store the exact normalized targets before bulk deletion");
let objects = cases
.iter()
.map(|(raw, _)| *raw)
.chain(invalid)
.map(|key| ObjectIdentifier::builder().key(key).build())
.collect::<Result<Vec<_>, _>>()?;
let delete = Delete::builder().set_objects(Some(objects)).quiet(quiet).build()?;
// A second identical batch also pins idempotent deletion of absent normalized keys.
for _ in 0..2 {
let output = client.delete_objects().bucket(bucket).delete(delete.clone()).send().await?;
let errors: Vec<_> = output.errors().iter().map(|error| (error.key(), error.code())).collect();
assert_eq!(
errors,
invalid.map(|key| (Some(key), Some("InvalidArgument"))),
"unexpected per-key errors: {output:?}"
);
let deleted: Vec<_> = output.deleted().iter().filter_map(|object| object.key()).collect();
let expected = if quiet {
vec![]
} else {
cases.iter().map(|(raw, _)| *raw).collect()
};
assert_eq!(deleted, expected, "responses must retain request keys, including directory slashes");
}
let listing = client.list_objects_v2().bucket(bucket).send().await?;
let remaining: Vec<_> = listing.contents().iter().filter_map(|object| object.key()).collect();
assert_eq!(
remaining,
["dir/keep", "keep"],
"bulk delete must remove normalized objects without resolving dot segments"
);
}
Ok(())
}
async fn assert_bulk_delete_authorizes_normalized_keys(env: &RustFSTestEnvironment, admin: &Client) -> TestResult {
let bucket = "test-bulk-slash-policy";
let user = "bulk-slash-user";
admin_create_user(env, user, "bulkSlashTestSecret123!").await?;
create_bucket(admin, bucket).await?;
for key in ["allowed/item", "protected/item"] {
admin
.put_object()
.bucket(bucket)
.key(key)
.body(ByteStream::from_static(b"keep"))
.send()
.await?;
}
let policy = serde_json::json!({
"Version": "2012-10-17",
"Statement": [
{"Effect": "Allow", "Principal": {"AWS": [user]}, "Action": ["s3:DeleteObject"],
"Resource": [format!("arn:aws:s3:::{bucket}/*")]},
{"Effect": "Deny", "Principal": {"AWS": [user]}, "Action": ["s3:DeleteObject"],
"Resource": [format!("arn:aws:s3:::{bucket}/protected/*")]}
]
});
admin
.put_bucket_policy()
.bucket(bucket)
.policy(policy.to_string())
.send()
.await?;
let client = env.create_s3_client_with_credentials(user, "bulkSlashTestSecret123!");
let output = client
.delete_objects()
.bucket(bucket)
.delete(
Delete::builder()
.objects(ObjectIdentifier::builder().key("/allowed///item").build()?)
.objects(ObjectIdentifier::builder().key("/protected///item").build()?)
.build()?,
)
.send()
.await?;
assert_eq!(output.deleted().len(), 1, "allowed normalized key must be deleted: {output:?}");
assert_eq!(output.deleted()[0].key(), Some("/allowed///item"));
assert_eq!(output.errors().len(), 1, "normalized resource denial must apply: {output:?}");
assert_eq!(output.errors()[0].key(), Some("/protected///item"));
assert_eq!(output.errors()[0].code(), Some("AccessDenied"));
let listing = admin.list_objects_v2().bucket(bucket).send().await?;
let remaining: Vec<_> = listing.contents().iter().filter_map(|object| object.key()).collect();
assert_eq!(remaining, ["protected/item"], "only the authorized object may be removed");
let protected = admin.get_object().bucket(bucket).key("protected/item").send().await?;
assert_eq!(protected.body.collect().await?.into_bytes().as_ref(), b"keep");
Ok(())
}
async fn assert_bulk_delete_preserves_version_identity(client: &Client) -> TestResult {
let bucket = "test-bulk-slash-versions";
let raw = "/history///file";
let normalized = "history/file";
create_bucket(client, bucket).await?;
client
.put_bucket_versioning()
.bucket(bucket)
.versioning_configuration(
VersioningConfiguration::builder()
.status(BucketVersioningStatus::Enabled)
.build(),
)
.send()
.await?;
let first = client
.put_object()
.bucket(bucket)
.key(raw)
.body(ByteStream::from_static(b"old"))
.send()
.await?;
let first_id = first.version_id().ok_or("first PUT omitted version ID")?;
let latest = client
.put_object()
.bucket(bucket)
.key(raw)
.body(ByteStream::from_static(b"latest"))
.send()
.await?;
let latest_id = latest.version_id().ok_or("latest PUT omitted version ID")?;
let invalid = client
.delete_objects()
.bucket(bucket)
.delete(
Delete::builder()
.objects(ObjectIdentifier::builder().key("///").build()?)
.build()?,
)
.send()
.await?;
assert!(invalid.deleted().is_empty(), "slash-only keys must not delete a versioned object");
assert_eq!(
invalid.errors().len(),
1,
"slash-only keys must remain invalid in versioned buckets: {invalid:?}"
);
assert_eq!(invalid.errors()[0].key(), Some("///"));
assert_eq!(invalid.errors()[0].code(), Some("InvalidArgument"));
let deleted = client
.delete_objects()
.bucket(bucket)
.delete(
Delete::builder()
.objects(ObjectIdentifier::builder().key(raw).version_id(first_id).build()?)
.build()?,
)
.send()
.await?;
assert!(deleted.errors().is_empty(), "historical delete failed: {deleted:?}");
assert_eq!(deleted.deleted().len(), 1);
assert_eq!(deleted.deleted()[0].key(), Some(raw));
assert_eq!(deleted.deleted()[0].version_id(), Some(first_id));
let listing = client.list_object_versions().bucket(bucket).send().await?;
assert_eq!(listing.versions().len(), 1, "only the latest data version should remain: {listing:?}");
assert_eq!(listing.versions()[0].version_id(), Some(latest_id));
let deleted = client
.delete_objects()
.bucket(bucket)
.delete(
Delete::builder()
.objects(ObjectIdentifier::builder().key(raw).build()?)
.build()?,
)
.send()
.await?;
assert!(deleted.errors().is_empty(), "delete marker creation failed: {deleted:?}");
assert_eq!(deleted.deleted().len(), 1);
assert_eq!(deleted.deleted()[0].key(), Some(raw));
assert_eq!(deleted.deleted()[0].delete_marker(), Some(true));
let marker = deleted.deleted()[0]
.delete_marker_version_id()
.ok_or("missing delete marker ID")?;
let listing = client.list_object_versions().bucket(bucket).send().await?;
assert_eq!(listing.delete_markers().len(), 1);
assert_eq!(listing.delete_markers()[0].key(), Some(normalized));
assert_eq!(listing.delete_markers()[0].version_id(), Some(marker));
let deleted = client
.delete_objects()
.bucket(bucket)
.delete(
Delete::builder()
.objects(ObjectIdentifier::builder().key(raw).version_id(marker).build()?)
.build()?,
)
.send()
.await?;
assert!(deleted.errors().is_empty(), "explicit marker removal failed: {deleted:?}");
assert_eq!(deleted.deleted().len(), 1);
assert_eq!(deleted.deleted()[0].key(), Some(raw));
assert_eq!(deleted.deleted()[0].version_id(), Some(marker));
let listing = client.list_object_versions().bucket(bucket).send().await?;
assert!(listing.delete_markers().is_empty());
let restored = client.get_object().bucket(bucket).key(normalized).send().await?;
assert_eq!(restored.version_id(), Some(latest_id));
assert_eq!(restored.body.collect().await?.into_bytes().as_ref(), b"latest");
Ok(())
}
}
+10
View File
@@ -17701,6 +17701,16 @@ impl ECStore {
.map(|_| ())
}
#[cfg(feature = "test-util")]
pub async fn complete_scanner_pause_backlog_retirement_for_test(&self, source_pool_index: usize) -> Result<()> {
// Native backlog fixtures use bounded capacity inputs rather than the
// host filesystem's unrelated usage. Keep the real ILM completion gate.
let run_token = self.durable_ilm_receipt_run_token(source_pool_index).await?;
self.persist_decommission_durable_ilm_manifest_for_run(source_pool_index, &run_token)
.await?;
self.complete_decommission(source_pool_index).await
}
#[cfg(feature = "test-util")]
pub async fn retire_scanner_pause_backlog_for_test(
self: &Arc<Self>,
@@ -85,6 +85,7 @@ impl ScannerPauseBacklogRetirementPhase {
pub struct ScannerPauseBacklogRetirementReplica {
pub pool_index: usize,
pub set_index: usize,
pub set_id: Option<uuid::Uuid>,
/// False for retiring members retained only to verify the old commit proof.
pub writable: bool,
pub data: Option<Vec<u8>>,
@@ -140,6 +141,7 @@ async fn read_replica((set, writable): (Arc<SetDisks>, bool)) -> Result<ScannerP
let mut replica = ScannerPauseBacklogRetirementReplica {
pool_index: set.pool_index,
set_index: set.set_index,
set_id: Some(set.persistent_set_id()?),
writable,
data: None,
};
@@ -222,6 +224,7 @@ pub(crate) fn plan_scanner_pause_backlog_retirement(
.map(|read| ScannerPauseBacklogRetirementReplica {
pool_index: read.replica.pool_index,
set_index: read.replica.set_index,
set_id: read.replica.set_id,
writable: read.replica.writable,
data: read.replica.data.clone(),
})
+60 -30
View File
@@ -685,12 +685,8 @@ async fn clear_bucket_usage_memory(bucket: &str, guard: Option<&rustfs_lock::Nam
}
pub async fn remove_bucket_usage_from_backend(store: Arc<ECStore>, bucket: &str) -> Result<(), Error> {
remove_bucket_usage_for_namespace_change(store.as_ref(), bucket).await
}
pub(crate) async fn remove_bucket_usage_for_namespace_change(store: &ECStore, bucket: &str) -> Result<(), Error> {
prepare_bucket_usage_for_namespace_change(bucket, None).await?;
remove_bucket_usage_from_backend_with_guard_fenced(store, bucket, None).await
remove_bucket_usage_with_owned_namespace(store, bucket, None).await
}
pub(crate) async fn prepare_bucket_usage_for_namespace_change(
@@ -730,11 +726,37 @@ where
pub(crate) async fn remove_bucket_usage_from_backend_with_guard_fenced(
store: &ECStore,
bucket: &str,
guard: Option<&rustfs_lock::NamespaceLockGuard>,
guard: Option<&Arc<rustfs_lock::NamespaceLockGuard>>,
) -> Result<(), Error> {
let result = remove_bucket_usage_from_backend_with_store_and_guard_and_publication(store, bucket, guard, Some(store)).await;
invalidate_bucket_usage_snapshot_caches(guard, bucket).await?;
result
let owned_store = crate::bucket::metadata_sys::object_store_in(&store.ctx).await?;
if !std::ptr::eq(store, owned_store.as_ref()) {
return Err(Error::other("usage cleanup requires its own metadata store"));
}
remove_bucket_usage_with_owned_namespace(owned_store, bucket, guard.cloned()).await
}
async fn remove_bucket_usage_with_owned_namespace(
store: Arc<ECStore>,
bucket: &str,
guard: Option<Arc<rustfs_lock::NamespaceLockGuard>>,
) -> Result<(), Error> {
let bucket = bucket.to_owned();
let ctx = Arc::clone(&store.ctx);
// Retain the bucket generation and movement/object guards through every
// physical commit even when the namespace request stops waiting.
ctx.run_detached_mutation(async move {
let result = remove_bucket_usage_from_backend_with_store_and_guard_and_publication(
store.as_ref(),
&bucket,
guard.as_deref(),
Some(store.as_ref()),
)
.await;
invalidate_bucket_usage_snapshot_caches(guard.as_deref(), &bucket).await?;
result
})
.await
.map_err(Error::from)?
}
async fn invalidate_bucket_usage_snapshot_caches(
@@ -849,9 +871,9 @@ where
let primary_seed_epoch = match publication_store {
Some(publication_store) => Some(
publication_store
.scanner_data_usage_publication_epoch()
.data_usage_namespace_cleanup_epoch()
.await
.ok_or_else(|| Error::other("data usage publication is blocked by data movement"))?,
.ok_or_else(|| Error::other("data usage namespace cleanup epoch is unavailable"))?,
),
None => None,
};
@@ -871,9 +893,9 @@ where
let backup_seed_epoch = match publication_store {
Some(publication_store) => Some(
publication_store
.scanner_data_usage_publication_epoch()
.data_usage_namespace_cleanup_epoch()
.await
.ok_or_else(|| Error::other("data usage publication is blocked by data movement"))?,
.ok_or_else(|| Error::other("data usage namespace cleanup epoch is unavailable"))?,
),
None => None,
};
@@ -985,9 +1007,9 @@ where
let read_epoch = match publication {
Some((publication_store, expected_publication_epoch)) => {
let epoch = publication_store
.scanner_data_usage_publication_epoch()
.data_usage_namespace_cleanup_epoch()
.await
.ok_or_else(|| Error::other("data usage publication is blocked by data movement"))?;
.ok_or_else(|| Error::other("data usage namespace cleanup epoch is unavailable"))?;
if expected_publication_epoch.is_some_and(|expected| expected != epoch) {
return Err(Error::other("data usage publication epoch changed before snapshot read"));
}
@@ -1022,9 +1044,8 @@ where
ensure_bucket_namespace_guard(guard, bucket, "data usage snapshot commit")?;
let publication_guard = match publication {
Some((publication_store, expected_publication_epoch)) => {
let Some((guard, publication_epoch)) = publication_store.scanner_data_usage_publication_admission_guard().await
else {
return Err(Error::other("data usage publication is blocked by data movement"));
let Some((guard, publication_epoch)) = publication_store.data_usage_namespace_cleanup_guard().await else {
return Err(Error::other("data usage namespace cleanup epoch is unavailable"));
};
if expected_publication_epoch
.or(read_epoch)
@@ -1036,18 +1057,27 @@ where
}
None => None,
};
let save_result = store
.put_object(
RUSTFS_META_BUCKET,
object,
&mut put_data,
&ObjectOptions {
max_parity: true,
http_preconditions: Some(http_preconditions),
..Default::default()
},
)
.await;
let save_result = match publication {
Some((publication_store, _)) => {
publication_store
.put_usage_snapshot_for_namespace_cleanup(object, revision.as_deref(), &mut put_data, guard)
.await
}
None => {
store
.put_object(
RUSTFS_META_BUCKET,
object,
&mut put_data,
&ObjectOptions {
max_parity: true,
http_preconditions: Some(http_preconditions),
..Default::default()
},
)
.await
}
};
drop(publication_guard);
match save_result.map_err(|err| crate::config::com::map_system_metadata_write_error(err, object)) {
Ok(_) => return Ok(()),
+180 -73
View File
@@ -579,6 +579,7 @@ struct PreparedPartTxnReceipt {
dst_meta_path: PathBuf,
transaction_path: PathBuf,
meta: Bytes,
had_old_data: bool,
had_old_meta: bool,
}
@@ -586,6 +587,7 @@ struct PublishedPartTxnReceipt {
current_data_path: PathBuf,
current_meta_path: PathBuf,
transaction_path: PathBuf,
had_old_data: bool,
had_old_meta: bool,
}
@@ -4323,20 +4325,28 @@ fn take_presynced_tmp_file(file: &Path) -> bool {
PRESYNCED_TMP_FILES.remove(file).is_some()
}
fn open_lazy_local_file_std(file_path: &Path, volume_dir: &Path, _reclaim_on_shutdown: bool) -> std::io::Result<std::fs::File> {
if let Some(parent) = file_path.parent()
&& (volume_dir.as_os_str().is_empty() || !volume_dir.starts_with(parent))
&& let Err(err) = super::fs::create_dir_all_std(parent)
&& err.kind() != std::io::ErrorKind::AlreadyExists
{
return Err(to_file_error(err));
}
let file = std::fs::OpenOptions::new()
.write(true)
.create(true)
.truncate(true)
.open(file_path)
.map_err(to_file_error)?;
fn open_lazy_local_file_std(
file_path: &Path,
volume_dir: &Path,
_reclaim_on_shutdown: bool,
parent_exists: bool,
) -> std::io::Result<std::fs::File> {
let mut open_opts = std::fs::OpenOptions::new();
open_opts.write(true).create(true).truncate(true);
let file = match parent_exists.then(|| open_opts.open(file_path)) {
Some(Ok(file)) => file,
Some(Err(err)) if err.kind() != std::io::ErrorKind::NotFound => return Err(to_file_error(err)),
_ => {
if let Some(parent) = file_path.parent()
&& (volume_dir.as_os_str().is_empty() || !volume_dir.starts_with(parent))
&& let Err(err) = super::fs::create_dir_all_std(parent)
&& err.kind() != std::io::ErrorKind::AlreadyExists
{
return Err(to_file_error(err));
}
open_opts.open(file_path).map_err(to_file_error)?
}
};
#[cfg(target_os = "macos")]
if _reclaim_on_shutdown {
let _ = set_std_fd_nocache(&file);
@@ -4351,6 +4361,7 @@ struct LazyLocalFileWriter {
reclaim_on_shutdown: bool,
sync_on_close: bool,
mark_dir_presynced: bool,
parent_exists: bool,
bytes_written: usize,
state: LazyLocalFileWriterState,
}
@@ -4371,6 +4382,7 @@ impl LazyLocalFileWriter {
reclaim_on_shutdown: bool,
sync_on_close: bool,
mark_dir_presynced: bool,
parent_exists: bool,
) -> Self {
Self {
file_path,
@@ -4379,6 +4391,7 @@ impl LazyLocalFileWriter {
reclaim_on_shutdown,
sync_on_close,
mark_dir_presynced,
parent_exists,
bytes_written: 0,
state: LazyLocalFileWriterState::Unopened,
}
@@ -4391,12 +4404,13 @@ impl LazyLocalFileWriter {
let reclaim_on_shutdown = self.reclaim_on_shutdown;
let sync_on_close = self.sync_on_close;
let mark_dir_presynced = self.mark_dir_presynced;
let parent_exists = self.parent_exists;
match std::mem::replace(&mut self.state, LazyLocalFileWriterState::Closed) {
LazyLocalFileWriterState::Unopened => {
let file_path = self.file_path.clone();
let volume_dir = self.volume_dir.clone();
self.state = LazyLocalFileWriterState::Writing(tokio::task::spawn_blocking(move || {
let mut file = open_lazy_local_file_std(&file_path, &volume_dir, reclaim_on_shutdown)?;
let mut file = open_lazy_local_file_std(&file_path, &volume_dir, reclaim_on_shutdown, parent_exists)?;
std::io::Write::write_all(&mut file, &payload).map_err(to_file_error)?;
if close_after {
if sync_on_close {
@@ -4538,8 +4552,9 @@ impl AsyncWrite for LazyLocalFileWriter {
let file_path = self.file_path.clone();
let volume_dir = self.volume_dir.clone();
let reclaim_on_shutdown = self.reclaim_on_shutdown;
let parent_exists = self.parent_exists;
self.state = LazyLocalFileWriterState::Closing(tokio::task::spawn_blocking(move || {
open_lazy_local_file_std(&file_path, &volume_dir, reclaim_on_shutdown).map(drop)
open_lazy_local_file_std(&file_path, &volume_dir, reclaim_on_shutdown, parent_exists).map(drop)
}));
}
LazyLocalFileWriterState::Open(file) => {
@@ -5591,11 +5606,14 @@ impl LocalIoBackend for StdBackend {
let file_path = local_disk_object_path(self.io_root(), volume, path)?;
check_path_length(file_path.to_string_lossy().as_ref())?;
let mut mark_dir_presynced = false;
let mut parent_exists = false;
if let Some(parent) = file_path.parent() {
check_path_length(parent.to_string_lossy().as_ref())?;
parent_exists = parent == volume_dir;
if volume == RUSTFS_META_TMP_BUCKET {
unmark_presynced_tmp_file(&file_path);
let parent_was_presynced = unmark_presynced_tmp_data_dir(parent);
parent_exists |= parent_was_presynced;
let file_name = file_path.file_name().and_then(|n| n.to_str()).unwrap_or_default();
mark_dir_presynced = sync_on_close
&& (file_name == "part.1"
@@ -5649,6 +5667,7 @@ impl LocalIoBackend for StdBackend {
reclaim_on_shutdown,
sync_on_close,
mark_dir_presynced,
parent_exists,
)))
}
WriteMode::Append => {
@@ -11521,9 +11540,20 @@ impl LocalDisk {
admission: Option<multipart_staging::MultipartSourceAdmission>,
) -> Result<()> {
multipart_staging::reject_reserved_anchor(path)?;
if volume == RUSTFS_META_MULTIPART_BUCKET {
self.multipart_index.record_head(path, &buf).await?;
}
let fuse_with_admission = admission.is_some() && !crash_inject::is_armed(CrashPoint::MetaWriteAfterTmpBeforeRename, path);
// Held until the staging root is published, by the filesystem work
// itself, so a concurrent listing cannot prune the marker in between.
let mut index_pin = None;
let prepared_index = if volume == RUSTFS_META_MULTIPART_BUCKET {
if fuse_with_admission {
self.multipart_index.prepare_record_head(path, &buf).await
} else {
index_pin = self.multipart_index.record_head(path, &buf).await?;
None
}
} else {
None
};
let volume_dir = self.io_get_bucket_path(volume)?;
let file_path = self.io_get_object_path(volume, path)?;
check_path_length(file_path.to_string_lossy().as_ref())?;
@@ -11539,11 +11569,16 @@ impl LocalDisk {
{
multipart_source_state::invalidate_active_multipart_source_head(admission.source());
}
if admission.is_some() && !crash_inject::is_armed(CrashPoint::MetaWriteAfterTmpBeforeRename, path) {
if fuse_with_admission {
let precondition = admission.as_ref().map(|a| a.precondition());
let root = self.publication_root.clone();
let lease = os::acquire_namespace_mutation_lease_with_owner(&file_path, namespace_owner).await;
os::run_blocking_namespace_operation_with_sync(lease, do_sync, move || {
let _index_pin = prepared_index.map(|record| record.run_std()).transpose()?;
#[cfg(test)]
if let Some(pin) = &_index_pin {
pin.pause_for_test();
}
super::fs::create_dir_all_std(&tmp_volume_dir)?;
#[cfg(test)]
run_owned_file_before_open(&tmp_file_path);
@@ -11586,12 +11621,16 @@ impl LocalDisk {
return Err(DiskError::Unexpected);
}
let publication_owner: Option<Arc<dyn Send + Sync>> = match index_pin {
Some(pin) => Some(Arc::new((namespace_owner.clone(), pin))),
None => namespace_owner.clone(),
};
os::rename_all_with_precondition_and_parent_sync(
&tmp_file_path,
&file_path,
&volume_dir,
&self.publication_root,
namespace_owner.clone(),
publication_owner,
admission.as_ref().map(|admission| admission.precondition()),
admission.is_some() && sync && durability.syncs_commit_metadata(),
)
@@ -14450,6 +14489,7 @@ impl DiskAPI for LocalDisk {
if bucket == RUSTFS_META_MULTIPART_BUCKET && !paths.is_empty() {
let mut cached_parts = Vec::with_capacity(paths.len());
let mut all_cached = true;
let mut verified_source: Option<(&str, PathBuf)> = None;
for path_str in paths {
let Some(num) = Path::new(path_str)
.file_name()
@@ -14461,12 +14501,21 @@ impl DiskAPI for LocalDisk {
all_cached = false;
break;
};
let Ok(source) = self.io_get_object_open_path(bucket, multipart_staging::upload_scope_root(path_str, true))
else {
all_cached = false;
break;
let scope_root = multipart_staging::upload_scope_root(path_str, true);
let (source, head_already_verified) = match verified_source.as_ref() {
Some((prev_root, prev_source)) if *prev_root == scope_root => (prev_source, true),
_ => {
let Ok(source) = self.io_get_object_open_path(bucket, scope_root) else {
all_cached = false;
break;
};
verified_source = Some((scope_root, source));
(&verified_source.as_ref().expect("just initialized").1, false)
}
};
let Some(part) = multipart_source_state::get_active_multipart_source_part(&source, num) else {
let Some(part) =
multipart_source_state::get_active_multipart_source_part_with_head(source, num, head_already_verified)
else {
all_cached = false;
break;
};
@@ -14651,6 +14700,7 @@ impl DiskAPI for LocalDisk {
Ok(_) => return Ok(Err(DiskError::FileAccessDenied)),
Err(err) => return Ok(Err(to_file_error(err).into())),
}
let mut created_transaction_parent = false;
if let Some((integrity, source, destination, expected_len)) = integrity_prep {
let stat = match std::fs::symlink_metadata(&source) {
Ok(stat) => stat,
@@ -14659,6 +14709,7 @@ impl DiskAPI for LocalDisk {
if !stat.is_file() || stat.len() != expected_len {
return Ok(Err(DiskError::FileCorrupt));
}
let presynced_proof = take_presynced_tmp_file(&source);
let verify_and_sync = (|| -> std::io::Result<bool> {
let mut file = os::open_file_for_sync_std(&source)?;
let mut header = [0u8; rustfs_filemeta::shard_integrity::INDEX_HEADER_SIZE];
@@ -14666,7 +14717,7 @@ impl DiskAPI for LocalDisk {
if header != integrity.index_header() {
return Ok(false);
}
if durability.syncs_data_shards() && !take_presynced_tmp_file(&source) {
if durability.syncs_data_shards() && !presynced_proof {
file.sync_all()?;
}
Ok(true)
@@ -14676,7 +14727,13 @@ impl DiskAPI for LocalDisk {
Ok(false) => return Ok(Err(DiskError::FileCorrupt)),
Err(err) => return Ok(Err(to_file_error(err).into())),
}
let precondition = admission.as_ref().map(|check| check.precondition());
#[cfg(all(any(test, feature = "test-util"), not(windows)))]
let has_hooks = os::prepared_publication_test_hooks::has_any_hooks();
#[cfg(not(all(any(test, feature = "test-util"), not(windows))))]
let has_hooks = false;
let precondition = has_hooks
.then(|| admission.as_ref().map(|check| check.precondition()))
.flatten();
// Publish the immutable generation before preparing the part switch.
// Rollback retains the old generation; an unreferenced new index is
// reclaimed with the upload directory if preparation is interrupted.
@@ -14693,12 +14750,13 @@ impl DiskAPI for LocalDisk {
) {
return Ok(Err(to_file_error(err).into()));
}
created_transaction_parent = same_parent;
}
let Some(transaction_parent) = transaction_path.parent() else {
return Ok(Err(DiskError::InvalidPath));
};
if let Err(err) = std::fs::create_dir_all(transaction_parent) {
if !created_transaction_parent && let Err(err) = std::fs::create_dir_all(transaction_parent) {
return Ok(Err(to_file_error(err).into()));
}
let staging_path = transaction_parent.join(format!(".part-txn-{}", Uuid::new_v4()));
@@ -14706,17 +14764,35 @@ impl DiskAPI for LocalDisk {
return Ok(Err(to_file_error(err).into()));
}
let prepare_result = (|| -> std::io::Result<bool> {
let prepare_result = (|| -> std::io::Result<(bool, bool)> {
let old_data_absent = staging_path.join(PART_TRANSACTION_OLD_DATA_ABSENT);
let old_meta_absent = staging_path.join(PART_TRANSACTION_OLD_META_ABSENT);
let had_old_data = snapshot_part_transaction_file(
&dst_file_path,
&staging_path.join(PART_TRANSACTION_OLD_DATA),
&staging_path.join(PART_TRANSACTION_OLD_DATA_ABSENT),
)?;
let had_old_meta = snapshot_part_transaction_file(
&dst_meta_path,
&staging_path.join(PART_TRANSACTION_OLD_META),
&staging_path.join(PART_TRANSACTION_OLD_META_ABSENT),
&old_data_absent,
)?;
let had_old_meta = match std::fs::symlink_metadata(&dst_meta_path) {
Ok(metadata) if metadata.is_file() => {
std::fs::hard_link(&dst_meta_path, staging_path.join(PART_TRANSACTION_OLD_META))?;
true
}
Ok(metadata) if metadata.file_type().is_symlink() => {
return Err(std::io::Error::new(ErrorKind::InvalidInput, "multipart transaction source is a symlink"));
}
Ok(_) => {
return Err(std::io::Error::new(ErrorKind::InvalidData, "multipart transaction source is not a file"));
}
Err(err) if err.kind() == ErrorKind::NotFound => {
if !had_old_data && std::fs::hard_link(&old_data_absent, &old_meta_absent).is_ok() {
false
} else {
std::fs::write(&old_meta_absent, [])?;
false
}
}
Err(err) => return Err(err),
};
let new_meta_path = staging_path.join(PART_TRANSACTION_NEW_META);
let mut new_meta = std::fs::OpenOptions::new()
@@ -14739,11 +14815,11 @@ impl DiskAPI for LocalDisk {
if sync_overwrite_backup {
os::fsync_dir_std(transaction_parent)?;
}
Ok(had_old_meta)
Ok((had_old_data, had_old_meta))
})();
let had_old_meta = match prepare_result {
Ok(had_old_meta) => had_old_meta,
let (had_old_data, had_old_meta) = match prepare_result {
Ok(flags) => flags,
Err(err) => {
let _ = remove_dir_all_if_exists(&staging_path);
return Ok(Err(to_file_error(err).into()));
@@ -14759,6 +14835,7 @@ impl DiskAPI for LocalDisk {
dst_meta_path,
transaction_path: transaction_path.clone(),
meta,
had_old_data,
had_old_meta,
},
);
@@ -14785,23 +14862,25 @@ impl DiskAPI for LocalDisk {
.io_get_object_open_path(volume, &txn_rel_path)
.ok()
.and_then(|txn_open| take_published_part_txn_receipt(&txn_open));
let (current_data_path, current_meta_path, transaction_path, skip_obsolete_read_on_commit) = match cached_published {
Some(entry) => (
entry.current_data_path,
entry.current_meta_path,
entry.transaction_path,
!entry.had_old_meta,
),
None => {
let (_, current_data_path) = self.io_get_bucket_and_object_path(volume, path)?;
let current_meta_path = self.io_get_object_path(volume, &format!("{path}.meta"))?;
let transaction_path = self.io_get_object_path(volume, &txn_rel_path)?;
for candidate in [&current_data_path, &current_meta_path, &transaction_path] {
check_path_length(candidate.to_string_lossy().as_ref())?;
let (current_data_path, current_meta_path, transaction_path, skip_obsolete_read_on_commit, cached_old_flags) =
match cached_published {
Some(entry) => (
entry.current_data_path,
entry.current_meta_path,
entry.transaction_path,
!entry.had_old_meta,
Some((entry.had_old_data, entry.had_old_meta)),
),
None => {
let (_, current_data_path) = self.io_get_bucket_and_object_path(volume, path)?;
let current_meta_path = self.io_get_object_path(volume, &format!("{path}.meta"))?;
let transaction_path = self.io_get_object_path(volume, &txn_rel_path)?;
for candidate in [&current_data_path, &current_meta_path, &transaction_path] {
check_path_length(candidate.to_string_lossy().as_ref())?;
}
(current_data_path, current_meta_path, transaction_path, false, None)
}
(current_data_path, current_meta_path, transaction_path, false)
}
};
};
let durability = effective_durability(volume);
let sync = durability.syncs_commit_metadata();
let part_number = current_data_path
@@ -14901,7 +14980,26 @@ impl DiskAPI for LocalDisk {
{
return Ok(Err(to_file_error(err).into()));
}
if let Err(err) = remove_dir_all_if_exists(&cleanup_path) {
if let Some((had_old_data, had_old_meta)) = cached_old_flags.filter(|_| action == PartTransactionAction::Commit) {
let old_data_name = if had_old_data {
PART_TRANSACTION_OLD_DATA
} else {
PART_TRANSACTION_OLD_DATA_ABSENT
};
let old_meta_name = if had_old_meta {
PART_TRANSACTION_OLD_META
} else {
PART_TRANSACTION_OLD_META_ABSENT
};
for name in [PART_TRANSACTION_NEW_META, old_data_name, old_meta_name] {
let _ = std::fs::remove_file(cleanup_path.join(name));
}
if std::fs::remove_dir(&cleanup_path).is_err()
&& let Err(err) = remove_dir_all_if_exists(&cleanup_path)
{
return Ok(Err(to_file_error(err).into()));
}
} else if let Err(err) = remove_dir_all_if_exists(&cleanup_path) {
return Ok(Err(to_file_error(err).into()));
}
Ok(Ok(true))
@@ -14973,7 +15071,7 @@ impl DiskAPI for LocalDisk {
dst_file_path,
dst_meta_path,
transaction_path,
preverified_meta_and_old_flag,
preverified_meta_and_old_flags,
) = match cached_prepared {
Some(entry) => (
entry.src_volume_dir,
@@ -14982,7 +15080,7 @@ impl DiskAPI for LocalDisk {
entry.dst_file_path,
Some(entry.dst_meta_path),
Some(entry.transaction_path),
Some(entry.had_old_meta),
Some((entry.had_old_data, entry.had_old_meta)),
),
None => {
let (src_volume_dir, src_file_path) = self.io_get_bucket_and_object_path(src_volume, src_path)?;
@@ -15079,7 +15177,7 @@ impl DiskAPI for LocalDisk {
let transaction_meta_path = transaction_path.join(PART_TRANSACTION_NEW_META);
let publish_meta_path = transaction_path.join(PART_TRANSACTION_PUBLISH_META);
if !cfg!(test)
&& preverified_meta_and_old_flag.is_some()
&& preverified_meta_and_old_flags.is_some()
&& publish_meta_path.is_file()
&& std::fs::metadata(&transaction_meta_path).is_ok_and(|m| m.is_file() && m.len() == meta.len() as u64)
{
@@ -15123,7 +15221,13 @@ impl DiskAPI for LocalDisk {
os::sync_file(&src_file_path).map_err(to_file_error)?;
}
let precondition = admission.as_ref().map(|check| check.precondition());
#[cfg(all(any(test, feature = "test-util"), not(windows)))]
let has_hooks = os::prepared_publication_test_hooks::has_any_hooks();
#[cfg(not(all(any(test, feature = "test-util"), not(windows))))]
let has_hooks = false;
let precondition = has_hooks
.then(|| admission.as_ref().map(|check| check.precondition()))
.flatten();
os::rename_all_with_precondition_and_parent_sync_std(
&src_file_path,
@@ -15135,17 +15239,19 @@ impl DiskAPI for LocalDisk {
)
.map_err(to_file_error)?;
let dst_meta = lstat_std(&dst_file_path).map_err(to_file_error)?;
if src_is_dir != dst_meta.is_dir() {
warn!(
event = EVENT_DISK_LOCAL_RENAME_REJECTED,
component = LOG_COMPONENT_ECSTORE,
subsystem = LOG_SUBSYSTEM_DISK_LOCAL,
reason = "dst_type_changed_after_rename",
path = ?dst_file_path,
"Disk local rename rejected"
);
return Err(DiskError::FileAccessDenied.into());
if cfg!(test) || src_is_dir {
let dst_meta = lstat_std(&dst_file_path).map_err(to_file_error)?;
if src_is_dir != dst_meta.is_dir() {
warn!(
event = EVENT_DISK_LOCAL_RENAME_REJECTED,
component = LOG_COMPONENT_ECSTORE,
subsystem = LOG_SUBSYSTEM_DISK_LOCAL,
reason = "dst_type_changed_after_rename",
path = ?dst_file_path,
"Disk local rename rejected"
);
return Err(DiskError::FileAccessDenied.into());
}
}
if let Some(transaction_publish_meta) = transaction_publish_meta {
@@ -15176,8 +15282,8 @@ impl DiskAPI for LocalDisk {
}
}
}
if let (Some(had_old_meta), Some(txn_path), Some(meta_path)) =
(preverified_meta_and_old_flag, transaction_path.as_ref(), dst_meta_path.as_path().into())
if let (Some((had_old_data, had_old_meta)), Some(txn_path), Some(meta_path)) =
(preverified_meta_and_old_flags, transaction_path.as_ref(), dst_meta_path.as_path().into())
{
put_published_part_txn_receipt(
txn_path,
@@ -15185,6 +15291,7 @@ impl DiskAPI for LocalDisk {
current_data_path: dst_file_path,
current_meta_path: meta_path.to_path_buf(),
transaction_path: txn_path.clone(),
had_old_data,
had_old_meta,
},
);
+17 -4
View File
@@ -286,12 +286,15 @@ pub(super) fn fsync_inline_commit_std(
if destination_was_absent {
let mut ancestor = dst.parent().and_then(Path::parent);
let mut ancestor_step = 0usize;
while let Some(ancestor_dir) = ancestor {
if !ancestor_dir.starts_with(dst_volume_dir) {
break;
}
let fsync_started = rustfs_io_metrics::put_stage_timer();
if let Err(err) = os::fsync_dir_limited_std(ancestor_dir) {
if let Err(err) =
os::fsync_commit_ancestor_directory_limited_std(ancestor_dir, dst_volume_dir, ancestor_step, commit_guard)
{
rustfs_io_metrics::record_put_object_stage_duration_from(
rustfs_io_metrics::PUT_STAGE_SET_DISK_RENAME_ANCESTOR_DIR_FSYNC,
fsync_started,
@@ -310,6 +313,7 @@ pub(super) fn fsync_inline_commit_std(
break;
}
ancestor = ancestor_dir.parent();
ancestor_step += 1;
}
}
Ok(())
@@ -517,8 +521,12 @@ impl LocalDisk {
state.preflight_rejection = Some(LocalRenamePreflightRejection(()));
})?;
crate::hp_guard!("LocalDisk::rename_data");
if dst_volume == super::super::RUSTFS_META_MULTIPART_BUCKET {
self.multipart_index.record_info(dst_path, &fi).await?;
if dst_volume == super::super::RUSTFS_META_MULTIPART_BUCKET
&& let Some(pin) = self.multipart_index.record_info(dst_path, &fi).await?
{
// The owner chain reaches every lease this rename publishes under,
// so the index marker stays pinned until that work has returned.
state.namespace_owner = Some(Arc::new((state.namespace_owner.take(), pin)));
}
#[cfg(feature = "e2e-test-hooks")]
if fi.is_healing() {
@@ -1473,7 +1481,12 @@ impl LocalDisk {
}
}
}
match publication_owned.finish_on_thread(sync_commit_metadata, !cfg!(test), staging_cleanup) {
match publication_owned.finish_on_thread(
sync_commit_metadata,
!cfg!(test),
staging_cleanup,
Some(&directory_guard_owned),
) {
Ok(result) => Ok(MultipartFusedCommitOutcome::Committed(Box::new(RenameDataResp {
multipart_source: Some(Box::new(result)),
old_data_dir: has_old_data_dir,
+391 -131
View File
@@ -41,19 +41,53 @@ struct DiscoverySession {
touched: Instant,
}
/// Plain (non-transaction) children of an object-scoped discovery root cost one
/// budget entry per this many names: enough to bound a hostile prefix without
/// making a real folder prefix undeletable.
/// Names an object-scoped scan skips without a stat (plain children of the
/// key's directory, and entries of a UUID child that cannot be delete evidence)
/// cost one budget entry per this many names: enough to bound a hostile prefix
/// without making a real folder prefix undeletable.
const PLAIN_CHILDREN_PER_BUDGET_ENTRY: u32 = 64;
/// Entries one step of the resumable (background) discovery may visit. The
/// foreground bounded protocol has its own, larger per-call allowance.
const DISCOVERY_STEP_ENTRIES: u32 = 4096;
/// Wall-clock allowance of one bounded discovery call: a third of the drive
/// timeout that wraps the call (including its wait for a lane worker), capped at
/// ten seconds, so an incomplete walk returns before the call can time out.
fn bounded_discovery_time_limit() -> std::time::Duration {
(crate::disk::disk_store::get_max_timeout_duration() / 3).min(std::time::Duration::from_secs(10))
}
/// Whether an entry inside a UUID child of an object-scoped root can name one
/// of the key's delete transactions: a journal file of a rollback directory or
/// a data-directory marker. Every other name is inert to discovery.
fn may_be_delete_evidence(name: &str) -> bool {
[PREPARED, STORAGE_FORMAT_FILE_BACKUP, DELETE_QUORUM_COMMITTED_FILE].contains(&name)
|| [
DELETE_DATA_DIR_MARKER_PREFIX,
RESERVED_DELETE_DATA_DIR_MARKER_PREFIX,
RECOVERY_DELETE_DATA_DIR_MARKER_PREFIX,
]
.iter()
.any(|prefix| delete_transaction_marker_id(name, prefix).is_some())
}
impl DiscoverySession {
/// Consume one bounded step, returning ownership only when traversal remains.
fn advance(self, limit: u32) -> std::io::Result<(Vec<DeleteTransactionId>, Option<Self>)> {
self.advance_bounded(limit, super::super::DELETE_RECOVERY_MAX_DISCOVERY_ENTRIES)
self.advance_bounded(limit, DISCOVERY_STEP_ENTRIES, &|| false)
.map(|(transactions, next, _)| (transactions, next))
}
fn advance_bounded(mut self, limit: u32, max_entries: u32) -> std::io::Result<(Vec<DeleteTransactionId>, Option<Self>, u32)> {
/// Walk until `limit` transactions, `max_entries` charged entries, the end
/// of the scope, or `should_stop` (checked before every entry); any early
/// stop returns the remaining traversal as continuation state.
fn advance_bounded(
mut self,
limit: u32,
max_entries: u32,
should_stop: &dyn Fn() -> bool,
) -> std::io::Result<(Vec<DeleteTransactionId>, Option<Self>, u32)> {
if self.frames.is_empty() {
match std::fs::read_dir(&self.root) {
Ok(entries) => self.frames.push((String::new(), entries)),
@@ -66,10 +100,11 @@ impl DiscoverySession {
let mut transactions = BTreeSet::new();
let mut seen_candidates = HashSet::new();
let mut visited = 0u32;
let mut plain_children = 0u32;
let mut skipped_by_name = 0u32;
while !self.frames.is_empty()
&& transactions.len() < usize::try_from(limit).map_err(|_| std::io::Error::from(ErrorKind::InvalidInput))?
&& visited < max_entries
&& !should_stop()
{
let (parent, entries) = self
.frames
@@ -90,13 +125,22 @@ impl DiscoverySession {
return Err(std::io::Error::from(ErrorKind::InvalidData));
}
// An object-scoped scan only ever descends into transaction (UUID)
// directories at the root. Plain child keys of a prefix directory are
// skipped by name without a stat; they are charged at one entry per
// readdir batch so the work stays bounded while deleting a missing key
// that names a prefix with thousands of children can still complete.
if self.object.is_some() && parent.is_empty() && !canonical_uuid(&name).is_some_and(|id| !id.is_nil()) {
plain_children += 1;
if plain_children.is_multiple_of(PLAIN_CHILDREN_PER_BUDGET_ENTRY) {
// directories at the root, and inside them only journal files and
// data-directory markers can be evidence. Every other name (plain
// child keys of a prefix directory, and the `xl.meta`, data
// directories or children of a UUID-named child key) is skipped by
// name without a stat and charged at one entry per readdir batch, so
// the work stays bounded while deleting a missing key that names a
// folder prefix with many children, UUID-named or not, can complete.
if self.object.is_some()
&& if parent.is_empty() {
!canonical_uuid(&name).is_some_and(|id| !id.is_nil())
} else {
!may_be_delete_evidence(&name)
}
{
skipped_by_name += 1;
if skipped_by_name.is_multiple_of(PLAIN_CHILDREN_PER_BUDGET_ENTRY) {
visited += 1;
}
continue;
@@ -1128,6 +1172,8 @@ struct PreparedDeleteReceiptEntry {
journal_raw: Vec<u8>,
journal: DeleteRecoveryJournal,
sole_transaction: bool,
had_old_data_dir: bool,
rollback_dir_fd: Option<Arc<std::fs::File>>,
}
#[derive(Clone)]
@@ -1137,6 +1183,8 @@ struct PublishedDeleteReceiptEntry {
file_path: PathBuf,
staging_root: PathBuf,
journal_raw: Vec<u8>,
had_old_data_dir: bool,
rollback_dir_fd: Option<Arc<std::fs::File>>,
}
const PREPARED_DELETE_RECEIPTS_PER_SHARD: usize = 64;
@@ -1387,8 +1435,16 @@ impl LocalDisk {
self.validate_delete_discovery_owner(&session)?;
// The blocking owner drops every directory handle even if the RPC waiter disappears.
// A partial inventory is never evidence of absence and never enters DISCOVERY_SESSIONS.
os::run_blocking_namespace_operation(lease, move || {
let (transactions, next, visited) = session.advance_bounded(256, max_entries)?;
// A large folder prefix may take a while to walk, so the probe rides the
// durable-mutation lane like bounded inspection, never the one-worker
// foreground lane that every other mutation on the drive queues behind.
// It also stops, incomplete, when its waiter goes away or well before the
// drive timeout, so a slow walk answers SlowDown instead of marking the
// drive failed or holding a lane worker after the request is gone.
let deadline = Instant::now() + bounded_discovery_time_limit();
os::run_cancellable_blocking_namespace_operation_with_sync(lease, true, move |is_cancelled| {
let should_stop = || is_cancelled() || Instant::now() >= deadline;
let (transactions, next, visited) = session.advance_bounded(256, max_entries, &should_stop)?;
Ok(super::super::BoundedDeleteRecoveryDiscovery {
transactions: if next.is_none() { transactions } else { Vec::new() },
visited,
@@ -1829,7 +1885,9 @@ impl LocalDisk {
) -> Result<DeleteRecoveryInspection> {
self.reject_multipart_delete_recovery_mutation(volume, object)?;
if let Ok(root) = self.io_get_object_path(volume, object) {
let _ = take_prepared_delete_receipt(&root.join(transaction.to_string()));
let tx_dir = root.join(transaction.to_string());
let _ = take_prepared_delete_receipt(&tx_dir);
let _ = take_published_delete_receipt(&tx_dir);
}
let (mut observed, mut has_data_dir_work) = self
@@ -2493,8 +2551,9 @@ impl LocalDisk {
Err(err) if err.kind() == ErrorKind::NotFound => return Ok(Ok(None)),
Err(err) => return Err(err),
};
type PreparedDeleteTuple = (Option<Uuid>, Option<Vec<u8>>, Vec<u8>, DeleteRecoveryJournal, bool);
let (old_dir, target, journal_raw, journal, sole_transaction) =
type PreparedDeleteTuple =
(Option<Uuid>, Option<Vec<u8>>, Vec<u8>, DeleteRecoveryJournal, bool, bool, Option<bool>);
let (old_dir, target, journal_raw, journal, sole_transaction, destination_missing, old_data_dir_state) =
match (|| -> Result<Option<PreparedDeleteTuple>> {
let mut meta = FileMeta::load(&buf)?;
let mut deleted_headers: Vec<_> = meta
@@ -2511,13 +2570,12 @@ impl LocalDisk {
if old_dir.is_some_and(|dir| dir == transaction || dir.is_nil()) {
return Err(DiskError::FileCorrupt);
}
let mut old_path_to_verify = None;
if let Some(uuid) = old_dir {
meta.data.remove(vec![fi_clone.version_id.unwrap_or_default(), uuid])?;
let old_path = path_join(&[file_path.as_path(), Path::new(uuid.to_string().as_str())]);
check_path_length(old_path.to_string_lossy().as_ref())?;
if old_path.join(STORAGE_FORMAT_FILE).exists() {
return Ok(None);
}
old_path_to_verify = Some(old_path);
}
let surviving: HashSet<_> = meta
.versions
@@ -2529,7 +2587,11 @@ impl LocalDisk {
let target = (!meta.versions.is_empty()).then(|| meta.marshal_msg()).transpose()?;
let mut predecessors = BTreeSet::new();
let mut sole_transaction = true;
let mut destination_exists = false;
let mut old_data_dir_exists = false;
let mut scanned_file_path = false;
if let Ok(entries) = std::fs::read_dir(&file_path) {
scanned_file_path = true;
for (visited, entry) in
recovery_entries(entries, recovery_scope.as_ref().map(|scope| scope.budget.as_ref())).enumerate()
{
@@ -2542,13 +2604,19 @@ impl LocalDisk {
sole_transaction = false;
continue;
};
let Some(id) = Uuid::parse_str(name_str).ok().filter(|id| !id.is_nil() && *id != transaction)
else {
let Some(id) = Uuid::parse_str(name_str).ok().filter(|id| !id.is_nil()) else {
continue;
};
if id == transaction {
destination_exists = true;
continue;
}
if !entry.file_type().map_err(to_file_error)?.is_dir() {
return Err(DiskError::InvalidPath);
}
if old_dir == Some(id) && name_str == id.hyphenated().to_string() {
old_data_dir_exists = true;
}
let dir_path = entry.path();
let prepared = regular_bytes(&dir_path.join(PREPARED))?;
let receipt = regular_bytes(&dir_path.join(DELETE_QUORUM_COMMITTED_FILE))?;
@@ -2589,6 +2657,12 @@ impl LocalDisk {
}
}
}
if let Some(old_path) = old_path_to_verify
&& (!scanned_file_path || old_data_dir_exists)
&& old_path.join(STORAGE_FORMAT_FILE).exists()
{
return Ok(None);
}
sole_transaction &= predecessors.is_empty();
let mut sparse = fi_clone;
sparse.data = None;
@@ -2607,14 +2681,28 @@ impl LocalDisk {
predecessors: predecessors.into_iter().collect(),
};
let raw = journal_bytes(&journal)?;
Ok(Some((old_dir, target, raw, journal, sole_transaction)))
Ok(Some((
old_dir,
target,
raw,
journal,
sole_transaction,
scanned_file_path && !destination_exists,
scanned_file_path.then_some(old_data_dir_exists),
)))
})() {
Ok(Some(tuple)) => tuple,
Ok(None) => return Ok(Ok(None)),
Err(err) => return Ok(Err(err)),
};
std::fs::create_dir_all(&staging_root)?;
let staged = tempfile::Builder::new().prefix("delete-prep-").tempdir_in(&staging_root)?;
let staged = match tempfile::Builder::new().prefix("delete-prep-").tempdir_in(&staging_root) {
Ok(dir) => dir,
Err(err) if err.kind() == ErrorKind::NotFound => {
std::fs::create_dir_all(&staging_root)?;
tempfile::Builder::new().prefix("delete-prep-").tempdir_in(&staging_root)?
}
Err(err) => return Err(err),
};
let staged_dir = staged.path().to_path_buf();
let backup_path = staged_dir.join(STORAGE_FORMAT_FILE_BACKUP);
if std::fs::hard_link(&xl_path, &backup_path).is_err() {
@@ -2638,19 +2726,25 @@ impl LocalDisk {
for file in written_files {
file.sync_data()?;
}
#[cfg(unix)]
let rollback_dir_fd = sync.then(|| std::fs::File::open(&staged_dir).ok().map(Arc::new)).flatten();
#[cfg(not(unix))]
let rollback_dir_fd: Option<Arc<std::fs::File>> = None;
if sync {
os::fsync_dir_std(&staged_dir)?;
os::fsync_directory_handle_std(&staged_dir, rollback_dir_fd.as_deref())?;
}
match std::fs::symlink_metadata(&destination) {
Ok(metadata) if metadata.is_dir() => match std::fs::remove_dir(&destination) {
Ok(()) => {}
if !destination_missing {
match std::fs::symlink_metadata(&destination) {
Ok(metadata) if metadata.is_dir() => match std::fs::remove_dir(&destination) {
Ok(()) => {}
Err(err) if err.kind() == ErrorKind::NotFound => {}
Err(err) if is_dir_not_empty_error(&err) => return Ok(Err(DiskError::OutdatedXLMeta)),
Err(err) => return Err(err),
},
Ok(_) => return Ok(Err(DiskError::OutdatedXLMeta)),
Err(err) if err.kind() == ErrorKind::NotFound => {}
Err(err) if is_dir_not_empty_error(&err) => return Ok(Err(DiskError::OutdatedXLMeta)),
Err(err) => return Err(err),
},
Ok(_) => return Ok(Err(DiskError::OutdatedXLMeta)),
Err(err) if err.kind() == ErrorKind::NotFound => {}
Err(err) => return Err(err),
}
}
let staged_dir = staged.keep();
os::record_native_publication(&destination);
@@ -2658,8 +2752,14 @@ impl LocalDisk {
let _ = std::fs::remove_dir_all(&staged_dir);
return Err(err);
}
let old_data_path = old_dir.map(|uuid| file_path.join(uuid.to_string())).filter(|p| p.is_dir());
if sync && let Err(err) = os::fsync_dir_std(&file_path) {
let old_data_path = old_dir
.map(|uuid| file_path.join(uuid.to_string()))
.filter(|p| old_data_dir_state.unwrap_or_else(|| p.is_dir()));
#[cfg(unix)]
let file_path_dir = sync.then(|| std::fs::File::open(&file_path).ok()).flatten();
#[cfg(not(unix))]
let file_path_dir: Option<std::fs::File> = None;
if sync && let Err(err) = os::fsync_directory_handle_std(&file_path, file_path_dir.as_ref()) {
let _ = std::fs::remove_dir_all(&destination);
return Err(err);
}
@@ -2668,11 +2768,15 @@ impl LocalDisk {
let mut reserved_data_path = None;
if let Some(old_path) = old_data_path {
let reserved_marker = old_path.join(format!("{RESERVED_DELETE_DATA_DIR_MARKER_PREFIX}{transaction}"));
#[cfg(unix)]
let old_dir_file = sync.then(|| std::fs::File::open(&old_path).ok()).flatten();
#[cfg(not(unix))]
let old_dir_file: Option<std::fs::File> = None;
let reserve_res = (|| -> std::io::Result<()> {
let marker = std::fs::File::create(&reserved_marker)?;
if sync {
marker.sync_all()?;
os::fsync_dir_std(&old_path)?;
os::fsync_directory_handle_std(&old_path, old_dir_file.as_ref())?;
}
Ok(())
})();
@@ -2681,15 +2785,22 @@ impl LocalDisk {
let _ = std::fs::remove_dir_all(&destination);
return Err(err);
}
reserved_data_path = Some((old_path, reserved_marker));
reserved_data_path = Some((old_path, reserved_marker, old_dir_file));
}
if let Some(ref target_bytes) = target {
// A provisional head must stay outside the published recovery transaction.
let target_in_rollback = destination.join(TARGET);
let linked_xl = staging_root.join(format!("delete-xl-{transaction}"));
let _ = std::fs::remove_file(&linked_xl);
os::record_native_publication(&xl_path);
let write_res = if std::fs::hard_link(&target_in_rollback, &linked_xl).is_ok() {
let linked = match std::fs::hard_link(&target_in_rollback, &linked_xl) {
Ok(()) => true,
Err(err) if err.kind() == ErrorKind::AlreadyExists => {
let _ = std::fs::remove_file(&linked_xl);
std::fs::hard_link(&target_in_rollback, &linked_xl).is_ok()
}
Err(_) => false,
};
let write_res = if linked {
let rename_res = std::fs::rename(&linked_xl, &xl_path);
if rename_res.is_err() {
let _ = std::fs::remove_file(&linked_xl);
@@ -2706,7 +2817,7 @@ impl LocalDisk {
})()
};
if let Err(err) = write_res {
if let Some((_, reserved_marker)) = &reserved_data_path {
if let Some((_, reserved_marker, _)) = &reserved_data_path {
let _ = std::fs::remove_file(reserved_marker);
}
let _ = std::fs::remove_dir_all(&destination);
@@ -2715,19 +2826,20 @@ impl LocalDisk {
} else if let Err(err) = std::fs::remove_file(&xl_path)
&& err.kind() != ErrorKind::NotFound
{
if let Some((_, reserved_marker)) = &reserved_data_path {
if let Some((_, reserved_marker, _)) = &reserved_data_path {
let _ = std::fs::remove_file(reserved_marker);
}
let _ = std::fs::remove_dir_all(&destination);
return Err(err);
}
if sync {
os::fsync_dir_std(&file_path)?;
os::fsync_directory_handle_std(&file_path, file_path_dir.as_ref())?;
}
if let Some((old_path, reserved_marker)) = reserved_data_path {
let had_old_data_dir = reserved_data_path.is_some();
if let Some((old_path, reserved_marker, old_dir_file)) = reserved_data_path {
let recovery_marker = old_path.join(format!("{RECOVERY_DELETE_DATA_DIR_MARKER_PREFIX}{transaction}"));
match std::fs::rename(&reserved_marker, &recovery_marker) {
Ok(()) if sync => os::fsync_dir_std(&old_path)?,
Ok(()) if sync => os::fsync_directory_handle_std(&old_path, old_dir_file.as_ref())?,
Ok(()) => {}
Err(err) if err.kind() == ErrorKind::NotFound => {}
Err(err) => return Err(err),
@@ -2743,9 +2855,11 @@ impl LocalDisk {
journal_raw,
journal,
sole_transaction,
had_old_data_dir,
rollback_dir_fd,
},
);
Ok(Ok(Some(old_dir)))
Ok(Ok(Some(old_dir.filter(|_| had_old_data_dir))))
},
)
.await
@@ -2798,7 +2912,13 @@ impl LocalDisk {
Some(entry.bucket_path),
entry.file_path,
entry.staging_root,
Some((entry.journal_raw, entry.journal, entry.sole_transaction)),
Some((
entry.journal_raw,
entry.journal,
entry.sole_transaction,
entry.had_old_data_dir,
entry.rollback_dir_fd,
)),
),
None => {
self.reject_multipart_delete_recovery_mutation(volume, object)?;
@@ -2829,7 +2949,7 @@ impl LocalDisk {
lease,
sync,
move || -> std::io::Result<Result<Option<(DeleteRecoveryJournal, bool, bool)>>> {
if let Some((journal_raw, journal, sole_transaction)) = cached_receipt
if let Some((journal_raw, journal, sole_transaction, had_old_data_dir, rollback_dir_fd)) = cached_receipt
&& rollback.join(PREPARED).is_file()
{
let tmp_receipt = staging_root.join(format!("receipt-{transaction}.tmp"));
@@ -2843,7 +2963,7 @@ impl LocalDisk {
os::record_native_publication(&receipt_path);
std::fs::rename(&tmp_receipt, &receipt_path)?;
if sync {
os::fsync_dir_std(&rollback)?;
os::fsync_directory_handle_std(&rollback, rollback_dir_fd.as_deref())?;
}
Ok(())
})();
@@ -2860,6 +2980,8 @@ impl LocalDisk {
file_path: root,
staging_root,
journal_raw,
had_old_data_dir,
rollback_dir_fd,
},
);
}
@@ -2996,29 +3118,34 @@ impl LocalDisk {
None
};
let cached_published = rollback_open_path.as_deref().and_then(take_published_delete_receipt);
let (object_path, bucket_path, staging_root, root, cached_journal_raw) = match cached_published {
Some(entry) => (
entry.object_path,
entry.bucket_path,
entry.staging_root,
entry.file_path,
Some(entry.journal_raw),
),
None => {
self.reject_multipart_delete_recovery_mutation(volume, object)?;
if !safe_path(object, false) {
return Err(DiskError::InvalidPath);
let (object_path, bucket_path, staging_root, root, cached_journal_raw, cached_had_old_data_dir, mut rollback_dir_fd) =
match cached_published {
Some(entry) => (
entry.object_path,
entry.bucket_path,
entry.staging_root,
entry.file_path,
Some(entry.journal_raw),
Some(entry.had_old_data_dir),
entry.rollback_dir_fd,
),
None => {
self.reject_multipart_delete_recovery_mutation(volume, object)?;
if !safe_path(object, false) {
return Err(DiskError::InvalidPath);
}
let obj_path = self.get_object_path(volume, object)?;
(
obj_path,
self.io_get_bucket_path(volume)?,
self.io_get_bucket_path(RUSTFS_META_TMP_BUCKET)?,
self.io_get_object_path(volume, object)?,
None,
None,
None,
)
}
let obj_path = self.get_object_path(volume, object)?;
(
obj_path,
self.io_get_bucket_path(volume)?,
self.io_get_bucket_path(RUSTFS_META_TMP_BUCKET)?,
self.io_get_object_path(volume, object)?,
None,
)
}
};
};
let metadata_owner = os::acquire_metadata_mutation_lease(&object_path, namespace_owner).await;
let volume_guard = os::disk_volume_mutation_lock(&self.root, volume).read_owned().await;
let owner: Option<Arc<dyn Send + Sync>> = Some(Arc::new((metadata_owner, volume_guard)));
@@ -3047,14 +3174,19 @@ impl LocalDisk {
}
let data_dirs = journal.data_dirs.clone();
let has_original = journal.original_digest.is_some();
let has_target = journal.target_digest.is_some();
let journal_for_receipt = cached_journal_raw.is_none().then(|| journal.clone());
let lease = os::acquire_namespace_mutation_lease_with_owner(&receipt_path, owner).await;
let prune_paths: Vec<_> = rollback
.ancestors()
.take_while(|dir| *dir != bucket_path.as_path() && dir.starts_with(&bucket_path))
.map(Path::to_path_buf)
.collect();
let prune_paths: Vec<_> = if has_target {
vec![rollback.clone()]
} else {
rollback
.ancestors()
.take_while(|dir| *dir != bucket_path.as_path() && dir.starts_with(&bucket_path))
.map(Path::to_path_buf)
.collect()
};
let res = os::run_blocking_namespace_operation_with_sync(lease, sync, move || -> std::io::Result<()> {
let persist_receipt = |finished: bool| -> std::io::Result<()> {
@@ -3087,80 +3219,118 @@ impl LocalDisk {
let recovery_committed = format!("{RECOVERY_DELETE_DATA_DIR_MARKER_PREFIX}{transaction}");
let reserved = format!("{RESERVED_DELETE_DATA_DIR_MARKER_PREFIX}{transaction}");
let mut had_physical_data_dirs = false;
for data_dir in &data_dirs {
for base in [&root, &rollback] {
let dir_path = base.join(data_dir.to_string());
match std::fs::read_dir(&dir_path) {
Ok(entries) => {
had_physical_data_dirs = true;
let mut removable = Vec::new();
let mut preserve_dir = false;
for entry in recovery_entries(entries, work_budget.as_deref()) {
let entry = entry?;
let name = entry.file_name();
let Some(name) = name.to_str() else {
preserve_dir = true;
continue;
};
if entry.file_type()?.is_file()
&& (delete_recovery_part_file(name)
|| name == committed
|| name == recovery_committed
|| name == reserved
|| is_delete_transaction_marker(name, RESERVED_DELETE_DATA_DIR_MARKER_PREFIX))
{
removable.push(entry.path());
} else {
preserve_dir = true;
}
}
for path in removable {
ignore_not_found(std::fs::remove_file(path))?;
}
if sync {
os::fsync_dir_std(&dir_path)?;
}
if !preserve_dir {
match std::fs::remove_dir(&dir_path) {
Ok(()) => {}
Err(err) if err.kind() == ErrorKind::NotFound || is_dir_not_empty_error(&err) => {}
Err(err) => return Err(err),
let fast_cleanup = cached_journal_raw.is_some();
if cached_had_old_data_dir != Some(false) {
let bases: &[&Path] = if fast_cleanup { &[&root] } else { &[&root, &rollback] };
for data_dir in &data_dirs {
for base in bases {
let dir_path = base.join(data_dir.to_string());
match std::fs::read_dir(&dir_path) {
Ok(entries) => {
had_physical_data_dirs = true;
let mut removable = Vec::new();
let mut preserve_dir = false;
for entry in recovery_entries(entries, work_budget.as_deref()) {
let entry = entry?;
let name = entry.file_name();
let Some(name) = name.to_str() else {
preserve_dir = true;
continue;
};
if entry.file_type()?.is_file()
&& (delete_recovery_part_file(name)
|| name == committed
|| name == recovery_committed
|| name == reserved
|| is_delete_transaction_marker(name, RESERVED_DELETE_DATA_DIR_MARKER_PREFIX))
{
removable.push(entry.path());
} else {
preserve_dir = true;
}
}
for path in removable {
ignore_not_found(std::fs::remove_file(path))?;
}
let mut dir_removed = false;
if !preserve_dir {
match std::fs::remove_dir(&dir_path) {
Ok(()) => dir_removed = true,
Err(err) if err.kind() == ErrorKind::NotFound => dir_removed = true,
Err(err) if is_dir_not_empty_error(&err) => {}
Err(err) => return Err(err),
}
}
if !dir_removed && sync {
os::fsync_dir_std(&dir_path)?;
}
}
Err(err) if err.kind() == ErrorKind::NotFound => {}
Err(err) => return Err(err),
}
Err(err) if err.kind() == ErrorKind::NotFound => {}
Err(err) => return Err(err),
}
}
}
if had_physical_data_dirs && sync && root.is_dir() {
os::fsync_dir_std(&root)?;
}
for name in [STORAGE_FORMAT_FILE_BACKUP, DELETE_MARKER_ROLLBACK_FILE] {
if let Err(err) = ignore_not_found(std::fs::remove_file(rollback.join(name))) {
if fast_cleanup {
let backup_or_marker = if has_original {
Some(STORAGE_FORMAT_FILE_BACKUP)
} else if has_target {
Some(DELETE_MARKER_ROLLBACK_FILE)
} else {
None
};
if let Some(name) = backup_or_marker
&& let Err(err) = ignore_not_found(std::fs::remove_file(rollback.join(name)))
{
if has_target || had_physical_data_dirs {
let _ = persist_receipt(true);
}
return Err(err);
}
}
for name in [PREPARED, TARGET, FINISHED] {
let path = rollback.join(name);
#[cfg(all(any(test, feature = "test-util"), not(windows)))]
os::prepared_publication_test_hooks::run(os::prepared_publication_test_hooks::Stage::Remove, &path);
if let Err(err) = ignore_not_found(std::fs::remove_file(path)) {
if has_target || had_physical_data_dirs {
let _ = persist_receipt(true);
for name in [PREPARED, FINISHED].into_iter().chain(has_target.then_some(TARGET)) {
let path = rollback.join(name);
#[cfg(all(any(test, feature = "test-util"), not(windows)))]
os::prepared_publication_test_hooks::run(os::prepared_publication_test_hooks::Stage::Remove, &path);
if let Err(err) = ignore_not_found(std::fs::remove_file(path)) {
if has_target || had_physical_data_dirs {
let _ = persist_receipt(true);
}
return Err(err);
}
}
} else {
for name in [STORAGE_FORMAT_FILE_BACKUP, DELETE_MARKER_ROLLBACK_FILE] {
if let Err(err) = ignore_not_found(std::fs::remove_file(rollback.join(name))) {
if has_target || had_physical_data_dirs {
let _ = persist_receipt(true);
}
return Err(err);
}
}
for name in [PREPARED, TARGET, FINISHED] {
let path = rollback.join(name);
#[cfg(all(any(test, feature = "test-util"), not(windows)))]
os::prepared_publication_test_hooks::run(os::prepared_publication_test_hooks::Stage::Remove, &path);
if let Err(err) = ignore_not_found(std::fs::remove_file(path)) {
if has_target || had_physical_data_dirs {
let _ = persist_receipt(true);
}
return Err(err);
}
return Err(err);
}
}
if sync && let Err(err) = os::fsync_dir_std(&rollback) {
let rollback_sync = rollback_dir_fd.take();
if sync && let Err(err) = os::fsync_directory_handle_std(&rollback, rollback_sync.as_deref()) {
drop(rollback_sync);
if has_target || had_physical_data_dirs {
let _ = persist_receipt(true);
}
return Err(err);
}
drop(rollback_sync);
#[cfg(all(any(test, feature = "test-util"), not(windows)))]
os::prepared_publication_test_hooks::run(os::prepared_publication_test_hooks::Stage::Remove, &receipt_path);
if let Err(err) = ignore_not_found(std::fs::remove_file(&receipt_path)) {
@@ -6443,6 +6613,93 @@ mod tests {
}
}
#[tokio::test]
async fn delete_recovery_bounded_discovery_charges_uuid_named_children_once_and_keeps_evidence() {
let (_temp, disk, _, _) = fixture().await;
let path = disk.delete_discovery_path("recovery", Some("prefix")).expect("prefix path");
std::fs::create_dir_all(&path).expect("physical prefix");
// UUID-named child keys of the folder: each holds its own `xl.meta` and a
// data directory, neither of which can be evidence for the folder's key.
const CHILDREN: u32 = 2000;
for _ in 0..CHILDREN {
let child = path.join(Uuid::new_v4().to_string());
std::fs::create_dir_all(child.join(Uuid::new_v4().to_string())).expect("child key data dir");
std::fs::write(child.join(STORAGE_FORMAT_FILE), b"meta").expect("child key metadata");
}
// A UUID-named sub-prefix with children of its own.
const SUB_PREFIX_CHILDREN: u32 = 640;
let sub_prefix = path.join(Uuid::new_v4().to_string());
for index in 0..SUB_PREFIX_CHILDREN {
std::fs::create_dir_all(sub_prefix.join(format!("child-{index}"))).expect("sub-prefix child");
}
// Evidence of two interrupted transactions of the key itself: a journaled
// rollback directory, and a data-directory marker whose rollback is gone.
let journaled = Uuid::new_v4();
std::fs::create_dir_all(path.join(journaled.to_string())).expect("rollback directory");
std::fs::write(path.join(journaled.to_string()).join(PREPARED), b"{}").expect("journal");
let marker_only = Uuid::new_v4();
let data_dir = path.join(Uuid::new_v4().to_string());
std::fs::create_dir_all(&data_dir).expect("residual data directory");
std::fs::write(data_dir.join(format!("{RESERVED_DELETE_DATA_DIR_MARKER_PREFIX}{marker_only}")), [])
.expect("reservation marker");
let page = disk
.discover_delete_transactions_bounded(
"recovery",
"prefix",
super::super::super::DELETE_RECOVERY_MAX_DISCOVERY_ENTRIES,
)
.await
.expect("bounded probe");
assert!(page.complete, "a folder of UUID-named keys is a complete inventory");
// One entry per descent (children, sub-prefix, rollback and data
// directory) and per evidence file; inert names at 1/64.
let inert = 2 * CHILDREN + SUB_PREFIX_CHILDREN;
assert_eq!(page.visited, CHILDREN + 3 + 2 + inert / PLAIN_CHILDREN_PER_BUDGET_ENTRY);
let found = page.transactions.iter().map(|id| id.transaction).collect::<BTreeSet<_>>();
assert!(found.contains(&journaled), "a journaled rollback directory is evidence");
assert!(found.contains(&marker_only), "a marker-only residue is evidence");
assert_eq!(found.len(), 2, "no child key is mistaken for a transaction");
}
#[tokio::test]
async fn delete_recovery_bounded_discovery_stops_on_request_and_skips_only_inert_irregular_entries() {
let (_temp, disk, _, _) = fixture().await;
let path = disk.delete_discovery_path("recovery", Some("prefix")).expect("prefix path");
let child = path.join(Uuid::new_v4().to_string());
std::fs::create_dir_all(&child).expect("UUID-named child key");
// A stop request (cancellation or the wall-clock limit) ends the walk
// before any entry, leaving continuation state: never evidence of absence.
let session = disk
.new_delete_discovery_session("recovery", Some("prefix"), &path)
.expect("private session");
let (transactions, next, visited) = session
.advance_bounded(256, super::super::super::DELETE_RECOVERY_MAX_DISCOVERY_ENTRIES, &|| true)
.expect("stopped walk");
assert!(transactions.is_empty());
assert_eq!(visited, 0);
assert!(next.is_some(), "a stopped walk is incomplete");
drop(next);
#[cfg(unix)]
{
// An irregular entry whose name cannot be evidence is skipped unread.
std::os::unix::fs::symlink(&path, child.join("inert-link")).expect("inert symlink");
let page = disk
.discover_delete_transactions_bounded("recovery", "prefix", 4096)
.await
.expect("inert irregular entries are not evidence");
assert!(page.complete && page.transactions.is_empty());
// One with an evidence name still fails closed.
std::os::unix::fs::symlink(&path, child.join(PREPARED)).expect("evidence-named symlink");
assert!(
disk.discover_delete_transactions_bounded("recovery", "prefix", 4096)
.await
.is_err(),
"an evidence-named symlink is corruption, not a skipped name"
);
}
}
#[tokio::test]
async fn delete_recovery_bounded_discovery_refuses_partial_inventory_without_public_cursor() {
let (_temp, disk, _, _) = fixture().await;
@@ -6545,8 +6802,11 @@ mod tests {
let filler_id = Uuid::new_v4().to_string();
let filler = path.join(&filler_id);
fs::create_dir_all(&filler).await.expect("non-evidence directory");
for index in 0..5000 {
fs::write(filler.join(format!("part-{index:04}")), [])
// Inert names are skipped at 1/64, so fill the step with entries that
// pass the evidence-name filter but are directories: each costs a full
// budget entry and names no transaction.
for _ in 0..5000 {
fs::create_dir(filler.join(format!("{DELETE_DATA_DIR_MARKER_PREFIX}{}", Uuid::new_v4())))
.await
.expect("non-evidence entry");
}
+409 -84
View File
@@ -4,6 +4,11 @@
//! Bucket-to-staging-root candidates: one marker per staging root, recorded
//! before the root is published and pruned once the root is gone. Source
//! metadata stays authoritative.
//!
//! A marker without a staging root is either a leftover or an upload whose
//! root is still being published. The publisher proves the second case by
//! holding a [`PublicationPin`] from before it records the marker until its
//! filesystem work has returned; no elapsed time is taken as that proof.
use super::{LocalDisk, read_metadata_only_std};
use crate::disk::{
@@ -29,9 +34,6 @@ use uuid::Uuid;
const INDEX: &str = ".rustfs.sys/multipart-index-v1";
const REBUILD_DIRENTS_PER_JOB: usize = 64;
/// A marker is recorded before its staging root is published. A marker younger
/// than this without a root is an upload in flight, never a leftover to prune.
const MULTIPART_INDEX_PRUNE_GRACE: std::time::Duration = std::time::Duration::from_secs(300);
type DirectoryEntries = Box<dyn Iterator<Item = io::Result<String>> + Send>;
type PhysicalRoot = (u64, [u8; 16]);
static OWNERS: LazyLock<parking_lot::Mutex<HashMap<PhysicalRoot, Weak<IndexOwner>>>> =
@@ -56,9 +58,15 @@ pub(super) struct IndexOwner {
// drop so a cancelled scan can be resumed by the next caller.
driving: AtomicBool,
disabled: AtomicBool,
// Markers whose staging root some publisher is creating right now, with
// the number of publishers. Boot-local like the index itself: every writer
// of this physical drive in this process shares this owner.
publishing: parking_lot::Mutex<HashMap<PathBuf, usize>>,
#[cfg(test)]
rebuild_head_pause: parking_lot::Mutex<Option<RebuildHeadPause>>,
#[cfg(test)]
publication_pause: parking_lot::Mutex<Option<RebuildHeadPause>>,
#[cfg(test)]
observed_rebuild_queue: parking_lot::Mutex<Option<Arc<os::DiskBlockingQueue>>>,
#[cfg(test)]
rebuild_single_job: AtomicBool,
@@ -120,23 +128,6 @@ impl IndexDirectory {
std::fs::File::open(self.path.join(name))
}
}
fn modified(&self, name: &str) -> io::Result<std::time::SystemTime> {
#[cfg(unix)]
{
self.open_file(name)?.metadata()?.modified()
}
#[cfg(not(unix))]
{
std::fs::metadata(self.path.join(name))?.modified()
}
}
/// Whether the marker is still inside the publication grace period. A
/// marker dated in the future (clock stepped back) is never a leftover.
fn marker_is_young(&self, name: &str) -> bool {
self.modified(name)
.ok()
.is_some_and(|modified| modified.elapsed().map_or(true, |age| age < MULTIPART_INDEX_PRUNE_GRACE))
}
fn remove_file(&self, name: &str) -> io::Result<()> {
#[cfg(unix)]
{
@@ -280,9 +271,12 @@ impl IndexOwner {
initialized: OnceLock::new(),
driving: AtomicBool::new(false),
disabled: AtomicBool::new(false),
publishing: parking_lot::Mutex::new(HashMap::new()),
#[cfg(test)]
rebuild_head_pause: parking_lot::Mutex::new(None),
#[cfg(test)]
publication_pause: parking_lot::Mutex::new(None),
#[cfg(test)]
observed_rebuild_queue: parking_lot::Mutex::new(None),
#[cfg(test)]
rebuild_single_job: AtomicBool::new(false),
@@ -394,8 +388,8 @@ impl IndexOwner {
// The index is boot-local: every start rescans the staging area
// before serving it, so a marker lost to a crash is recreated before
// it can be consulted. Creation is enough; no fsync of the marker or
// its ancestors. An existing marker is touched so pruning sees the
// new upload's age, not the previous one's.
// its ancestors. An existing marker needs nothing: the caller's
// publication pin, not the marker's age, keeps it from being pruned.
match openat(
&parent,
source_root,
@@ -407,16 +401,7 @@ impl IndexOwner {
return Err(io::Error::other("index marker is not a file"));
}
}
Err(rustix::io::Errno::EXIST) => {
// Best effort: a filesystem that refuses to set times keeps
// the previous upload's age, which only shortens the prune
// grace for this key; index completeness does not depend on it.
if let Ok(marker) =
openat(&parent, source_root, OFlags::WRONLY | OFlags::NOFOLLOW | OFlags::CLOEXEC, Mode::empty())
{
let _ = std::fs::File::from(marker).set_modified(std::time::SystemTime::now());
}
}
Err(rustix::io::Errno::EXIST) => {}
Err(error) => return Err(error.into()),
}
}
@@ -429,7 +414,23 @@ impl IndexOwner {
os::replace_multipart_source_metadata(&marker, b"1", &self.base, &root)?;
os::sync_multipart_owned_record_parents(&self.base.join(".rustfs.sys"), &directory, true, &self.base, &root)?;
}
self.ensure_current_root()
Ok(())
}
/// Declare that the caller is about to publish the staging root of
/// `marker`. Must precede the marker lease the caller records under, so a
/// prune that found no pin has released that lease before the marker is
/// (re)created.
fn pin_publication(self: &Arc<Self>, marker: PathBuf) -> PublicationPin {
*self.publishing.lock().entry(marker.clone()).or_default() += 1;
PublicationPin {
owner: Arc::clone(self),
marker,
}
}
fn is_publishing(&self, marker: &Path) -> bool {
self.publishing.lock().contains_key(marker)
}
fn ensure_current_root(&self) -> io::Result<()> {
@@ -513,9 +514,12 @@ impl IndexOwner {
}
}
pub(super) async fn record_info(self: &Arc<Self>, path: &str, info: &FileInfo) -> Result<()> {
/// Record the marker of `path`'s staging root. The returned pin must stay
/// alive until the filesystem work that publishes the root has returned,
/// including work that outlives a cancelled caller.
pub(super) async fn record_info(self: &Arc<Self>, path: &str, info: &FileInfo) -> Result<Option<PublicationPin>> {
if canonical_upload(path).is_none() {
return Ok(());
return Ok(None);
}
let owner = self.clone();
let path = path.to_owned();
@@ -531,6 +535,7 @@ impl IndexOwner {
.join(path.split('/').next().unwrap_or_default())
})
.unwrap_or_else(|| owner.base.join(INDEX));
let pin = self.pin_publication(marker.clone());
let lease = os::acquire_namespace_mutation_lease_with_owner(&marker, None).await;
os::run_blocking_namespace_operation_with_sync(lease, true, move || {
match owner.record(&path, &info) {
@@ -544,16 +549,13 @@ impl IndexOwner {
}
})
.await?;
Ok(())
Ok(Some(pin))
}
pub(super) async fn record_head(self: &Arc<Self>, path: &str, bytes: &[u8]) -> Result<()> {
let Some(upload) = path
pub(super) async fn prepare_record_head(self: &Arc<Self>, path: &str, bytes: &[u8]) -> Option<PreparedHeadRecord> {
let upload = path
.strip_suffix("/xl.meta")
.filter(|upload| canonical_upload(upload).is_some())
else {
return Ok(());
};
.filter(|upload| canonical_upload(upload).is_some())?;
let info = match get_file_info(
bytes,
RUSTFS_META_MULTIPART_BUCKET,
@@ -568,10 +570,36 @@ impl IndexOwner {
Ok(info) => info,
Err(_) => {
self.disabled.store(true, Ordering::Release);
return Ok(());
return None;
}
};
self.record_info(upload, &info).await
let marker = info
.metadata
.get(crate::disk::multipart_index::RUSTFS_MULTIPART_BUCKET_KEY)
.map(|bucket| {
self.base
.join(INDEX)
.join(sha(bucket))
.join(upload.split('/').next().unwrap_or_default())
})
.unwrap_or_else(|| self.base.join(INDEX));
let pin = self.pin_publication(marker.clone());
let lease = os::acquire_namespace_mutation_lease_with_owner(&marker, None).await;
Some(PreparedHeadRecord {
upload: upload.to_owned(),
info,
pin,
_lease: lease,
})
}
/// Record the marker of the staging root that `path` belongs to. See
/// [`Self::record_info`] for how long the returned pin must live.
pub(super) async fn record_head(self: &Arc<Self>, path: &str, bytes: &[u8]) -> Result<Option<PublicationPin>> {
let Some(prepared) = self.prepare_record_head(path, bytes).await else {
return Ok(None);
};
prepared.run().await.map(Some)
}
async fn page(self: &Arc<Self>, bucket: &str, marker: Option<&str>, disk_id: Uuid) -> Result<MultipartIndexPage> {
@@ -622,17 +650,20 @@ impl IndexOwner {
// to every key ever multipart-initiated on this drive.
let mut live = Vec::with_capacity(roots.len());
let mut stale = Vec::new();
let markers = owner.base.join(INDEX).join(sha(&bucket));
for root in roots {
// The marker precedes its staging root: a pinned one is an
// upload in flight and stays a candidate. The pin is read
// before the root, so a publisher that finished in between has
// already made its root visible.
if owner.is_publishing(&markers.join(&root)) {
live.push(root);
continue;
}
match owner.directory(&Path::new(RUSTFS_META_MULTIPART_BUCKET).join(&root)) {
Ok(_) => live.push(root),
Err(error) if matches!(error.kind(), io::ErrorKind::NotFound | io::ErrorKind::NotADirectory) => {
// The marker precedes its staging root; a young one is an
// upload in flight and stays a candidate.
if directory.as_ref().is_some_and(|index| index.marker_is_young(&root)) {
live.push(root);
} else {
stale.push(root);
}
stale.push(root);
}
Err(error) => return Err(DiskError::from(error)),
}
@@ -657,9 +688,11 @@ impl IndexOwner {
}
/// Remove one marker whose staging root was gone when a page examined it.
/// The marker's own namespace lease serializes this with `record_info`; the
/// staging root and the marker's age are re-checked under that lease, so an
/// upload recorded in between keeps its marker.
/// The marker's own namespace lease serializes this with recording. Under
/// that lease the publication pin is read first and the staging root
/// second: a publisher pinned before the read keeps the marker, one that
/// already finished has made its root visible, and one that pins later
/// records its marker only after this lease is released.
async fn prune_marker(self: &Arc<Self>, bucket: &str, root: &str) -> Result<()> {
let owner = self.clone();
let bucket_directory = Path::new(INDEX).join(sha(bucket));
@@ -670,6 +703,9 @@ impl IndexOwner {
// not queue behind every other mutation on the one-worker foreground lane.
os::run_blocking_namespace_operation_with_sync(lease, true, move || {
owner.ensure_current_root()?;
if owner.is_publishing(&marker) {
return Ok(());
}
match owner.directory(&Path::new(RUSTFS_META_MULTIPART_BUCKET).join(&root)) {
Ok(_) => return Ok(()),
Err(error) if matches!(error.kind(), io::ErrorKind::NotFound | io::ErrorKind::NotADirectory) => {}
@@ -680,9 +716,6 @@ impl IndexOwner {
Err(error) if error.kind() == io::ErrorKind::NotFound => return Ok(()),
Err(error) => return Err(error),
};
if directory.marker_is_young(&root) {
return Ok(());
}
match directory.remove_file(&root) {
Err(error) if error.kind() != io::ErrorKind::NotFound => Err(error),
_ => Ok(()),
@@ -693,6 +726,72 @@ impl IndexOwner {
}
}
/// Proof that the staging root of one marker is being published. While any pin
/// for a marker is alive, pruning leaves that marker alone.
#[must_use = "dropping the pin lets a concurrent listing prune the marker before its staging root exists"]
pub(super) struct PublicationPin {
owner: Arc<IndexOwner>,
marker: PathBuf,
}
impl PublicationPin {
/// Test seam between recording the marker and publishing the staging root.
#[cfg(test)]
pub(super) fn pause_for_test(&self) {
let pause = self.owner.publication_pause.lock().take();
if let Some(pause) = pause {
let _pause = os::enter_blocking_test_hook();
let _ = pause.started.send(());
let _ = pause.release.recv_timeout(std::time::Duration::from_secs(10));
}
}
}
impl Drop for PublicationPin {
fn drop(&mut self) {
let mut publishing = self.owner.publishing.lock();
if let Some(count) = publishing.get_mut(&self.marker) {
*count -= 1;
if *count == 0 {
publishing.remove(&self.marker);
}
}
}
}
pub(super) struct PreparedHeadRecord {
upload: String,
info: FileInfo,
pin: PublicationPin,
_lease: Arc<os::NamespaceMutationLease>,
}
impl PreparedHeadRecord {
/// Record the marker and release its lease. The caller publishes the
/// staging root while it holds the returned pin.
pub(super) fn run_std(self) -> io::Result<PublicationPin> {
let Self {
upload,
info,
pin,
_lease,
} = self;
match pin.owner.record(&upload, &info) {
Err(error) if error.kind() == io::ErrorKind::InvalidData => {
pin.owner.disabled.store(true, Ordering::Release);
Ok(pin)
}
Err(error) => Err(error),
Ok(()) => Ok(pin),
}
}
async fn run(self) -> Result<PublicationPin> {
let lease = Arc::clone(&self._lease);
Ok(os::run_blocking_namespace_operation_with_sync(lease, true, move || self.run_std()).await?)
}
}
impl LocalDisk {
#[cfg(any(test, feature = "test-util"))]
pub(in crate::disk) async fn wait_for_multipart_index_bootstrap(&self) {
@@ -907,47 +1006,274 @@ mod tests {
let live = format!("{:064x}", 1);
let finished = format!("{:064x}", 2);
let in_flight = format!("{:064x}", 3);
let clock_stepped = format!("{:064x}", 4);
for name in [&live, &finished, &in_flight, &clock_stepped] {
for name in [&live, &finished, &in_flight] {
std::fs::write(directory.join(name), b"1").expect("marker");
}
// Completion and abort remove the staging root and leave the marker.
std::fs::create_dir_all(root.path().join(RUSTFS_META_MULTIPART_BUCKET).join(&live)).expect("live staging root");
set_marker_modified(&directory.join(&finished), std::time::SystemTime::now() - MULTIPART_INDEX_PRUNE_GRACE * 2);
// A marker dated after a backwards clock step is never a leftover.
set_marker_modified(
&directory.join(&clock_stepped),
std::time::SystemTime::now() + MULTIPART_INDEX_PRUNE_GRACE * 2,
);
let page = disk.multipart_index.page("bucket", None, Uuid::new_v4()).await.expect("page");
// The in-flight marker has no staging root yet but is young: a candidate, not a leftover.
assert_eq!(page.roots, vec![live.clone(), in_flight.clone(), clock_stepped.clone()]);
let owner = &disk.multipart_index;
let publishing = owner.pin_publication(owner.base.join(INDEX).join(sha("bucket")).join(&in_flight));
let second_publisher = owner.pin_publication(owner.base.join(INDEX).join(sha("bucket")).join(&in_flight));
let page = owner.page("bucket", None, Uuid::new_v4()).await.expect("page");
// The in-flight marker has no staging root yet but is pinned: a candidate, not a leftover.
assert_eq!(page.roots, vec![live.clone(), in_flight.clone()]);
assert!(page.next_marker.is_none());
assert!(directory.join(&live).exists(), "a live root keeps its marker");
assert!(directory.join(&in_flight).exists(), "a marker inside the grace period is kept");
assert!(directory.join(&clock_stepped).exists(), "a future-dated marker is kept");
assert!(!directory.join(&finished).exists(), "a finished root's marker is pruned");
// The next page no longer pays for the pruned marker.
let page = disk
.multipart_index
assert!(directory.join(&in_flight).exists(), "a pinned marker is kept");
assert!(!directory.join(&finished).exists(), "a finished root's marker is pruned at once");
// One of two publishers of the same root finishing does not unpin it.
drop(publishing);
let page = owner
.page("bucket", None, Uuid::new_v4())
.await
.expect("page after pruning");
assert_eq!(page.roots, vec![live, in_flight, clock_stepped]);
.expect("page with one publisher left");
assert_eq!(page.roots, vec![live.clone(), in_flight.clone()]);
assert!(directory.join(&in_flight).exists(), "the remaining publisher keeps the marker");
// A publication that ended without a staging root leaves a leftover.
drop(second_publisher);
let page = owner
.page("bucket", None, Uuid::new_v4())
.await
.expect("page after the failed publication");
assert_eq!(page.roots, vec![live]);
assert!(!directory.join(&in_flight).exists(), "an unpinned marker without a root is pruned");
assert!(owner.publishing.lock().is_empty(), "finished publishers leave no pin behind");
}
/// Move a marker's modification time outside the publication grace period.
fn age_marker(marker: &Path) {
set_marker_modified(marker, std::time::SystemTime::now() - MULTIPART_INDEX_PRUNE_GRACE * 2);
fn pause_next_publication(owner: &IndexOwner) -> (tokio::sync::oneshot::Receiver<()>, std::sync::mpsc::Sender<()>) {
let (started, waiting) = tokio::sync::oneshot::channel();
let (release, released) = std::sync::mpsc::channel();
*owner.publication_pause.lock() = Some(RebuildHeadPause {
started,
release: released,
});
(waiting, release)
}
fn set_marker_modified(marker: &Path, modified: std::time::SystemTime) {
/// The window the pin closes: the marker is recorded and its lease is
/// released, the staging root is not published yet, and a listing examines
/// the bucket. The marker is dated a day back so that no grace period could
/// be what keeps it.
#[tokio::test(flavor = "multi_thread", worker_threads = 2)]
async fn multipart_index_listing_during_source_publication_keeps_the_new_upload_listed() {
let root = tempfile::tempdir().expect("disk root");
let disk = Arc::new(open(root.path()).await);
let (path, info) = upload("bucket", "published-while-listed");
let source_root = path.split('/').next().expect("source sha").to_owned();
let marker = root.path().join(INDEX).join(sha("bucket")).join(&source_root);
let source_head = root
.path()
.join(RUSTFS_META_MULTIPART_BUCKET)
.join(&path)
.join(STORAGE_FORMAT_FILE);
let (paused, release) = pause_next_publication(&disk.multipart_index);
let writer = tokio::spawn({
let disk = Arc::clone(&disk);
let path = path.clone();
async move { disk.write_metadata("bucket", RUSTFS_META_MULTIPART_BUCKET, &path, info).await }
});
tokio::time::timeout(std::time::Duration::from_secs(5), paused)
.await
.expect("the publication reaches the pause")
.expect("pause sender");
assert!(marker.exists(), "the marker precedes the staging root");
assert!(!source_head.exists(), "the staging root is not published yet");
std::fs::File::options()
.write(true)
.open(marker)
.open(&marker)
.expect("marker")
.set_modified(modified)
.expect("set marker time");
.set_modified(std::time::SystemTime::now() - std::time::Duration::from_secs(86_400))
.expect("age the marker");
let identity = Uuid::new_v4();
let page = disk
.multipart_index
.page("bucket", None, identity)
.await
.expect("page during publication");
assert_eq!(page.roots, vec![source_root.clone()], "an upload in flight stays a candidate");
assert!(marker.exists(), "a listing must not prune the marker of an upload in flight");
release.send(()).expect("resume the publication");
writer.await.expect("writer task").expect("the upload metadata is published");
assert!(source_head.exists());
let page = disk
.multipart_index
.page("bucket", None, identity)
.await
.expect("page after publication");
assert_eq!(page.roots, vec![source_root], "the published upload is listed");
assert!(marker.exists());
assert!(disk.multipart_index.publishing.lock().is_empty());
}
/// A timed-out caller drops its future while the filesystem work goes on.
/// The pin belongs to that work, so the marker survives until the staging
/// root the detached work publishes is visible.
#[tokio::test(flavor = "multi_thread", worker_threads = 2)]
async fn multipart_index_cancelled_publisher_keeps_its_marker_until_the_detached_work_returns() {
let root = tempfile::tempdir().expect("disk root");
let disk = Arc::new(open(root.path()).await);
let (path, info) = upload("bucket", "caller-timed-out");
let source_root = path.split('/').next().expect("source sha").to_owned();
let marker = root.path().join(INDEX).join(sha("bucket")).join(&source_root);
let source_head = root
.path()
.join(RUSTFS_META_MULTIPART_BUCKET)
.join(&path)
.join(STORAGE_FORMAT_FILE);
let (paused, release) = pause_next_publication(&disk.multipart_index);
let writer = tokio::spawn({
let disk = Arc::clone(&disk);
let path = path.clone();
async move { disk.write_metadata("bucket", RUSTFS_META_MULTIPART_BUCKET, &path, info).await }
});
tokio::time::timeout(std::time::Duration::from_secs(5), paused)
.await
.expect("the publication reaches the pause")
.expect("pause sender");
writer.abort();
assert!(writer.await.expect_err("the caller is gone").is_cancelled());
let identity = Uuid::new_v4();
let page = disk
.multipart_index
.page("bucket", None, identity)
.await
.expect("page after the caller left");
assert_eq!(page.roots, vec![source_root.clone()]);
assert!(marker.exists(), "the detached publication still owns the marker");
release.send(()).expect("resume the detached publication");
tokio::time::timeout(std::time::Duration::from_secs(10), async {
while !source_head.exists() || !disk.multipart_index.publishing.lock().is_empty() {
tokio::time::sleep(std::time::Duration::from_millis(5)).await;
}
})
.await
.expect("the detached publication completes");
let page = disk
.multipart_index
.page("bucket", None, identity)
.await
.expect("page after the detached publication");
assert_eq!(page.roots, vec![source_root], "the upload the detached work published is listed");
assert!(marker.exists());
}
/// A page classifies a marker as stale before the publisher pins it; the
/// prune that follows must read the pin again under the marker lease.
#[tokio::test]
async fn multipart_index_prune_rechecks_the_pin_under_the_marker_lease() {
let root = tempfile::tempdir().expect("disk root");
let disk = open(root.path()).await;
let owner = &disk.multipart_index;
let directory = root.path().join(INDEX).join(sha("bucket"));
std::fs::create_dir_all(&directory).expect("marker directory");
let name = format!("{:064x}", 9);
std::fs::write(directory.join(&name), b"1").expect("marker");
let pin = owner.pin_publication(owner.base.join(INDEX).join(sha("bucket")).join(&name));
owner.prune_marker("bucket", &name).await.expect("prune of a pinned marker");
assert!(directory.join(&name).exists(), "a publisher that pinned after the page keeps its marker");
drop(pin);
owner.prune_marker("bucket", &name).await.expect("prune of a leftover");
assert!(!directory.join(&name).exists(), "the same marker is a leftover once nobody publishes it");
}
/// The publisher pins before it queues on the marker lease. A prune that
/// holds the lease and found no pin has therefore unlinked the marker
/// before the publisher records it, and the record recreates it.
#[tokio::test(flavor = "multi_thread", worker_threads = 2)]
async fn multipart_index_publisher_pins_before_it_waits_for_the_marker_lease() {
let root = tempfile::tempdir().expect("disk root");
let disk = Arc::new(open(root.path()).await);
let owner = Arc::clone(&disk.multipart_index);
let (path, info) = upload("bucket", "queued-behind-a-prune");
let source_root = path.split('/').next().expect("source sha").to_owned();
let marker = owner.base.join(INDEX).join(sha("bucket")).join(&source_root);
let visible_marker = root.path().join(INDEX).join(sha("bucket")).join(&source_root);
// Stand in for a prune that owns the marker lease and already read "no pin".
let prune_lease = os::acquire_namespace_mutation_lease_with_owner(&marker, None).await;
assert!(!owner.is_publishing(&marker));
let writer = tokio::spawn({
let disk = Arc::clone(&disk);
let path = path.clone();
async move { disk.write_metadata("bucket", RUSTFS_META_MULTIPART_BUCKET, &path, info).await }
});
tokio::time::timeout(std::time::Duration::from_secs(5), async {
while !owner.is_publishing(&marker) {
tokio::time::sleep(std::time::Duration::from_millis(2)).await;
}
})
.await
.expect("the publisher pins while the marker lease is still held by the prune");
assert!(!visible_marker.exists(), "the marker is recorded only under the lease");
drop(prune_lease);
writer.await.expect("writer task").expect("the upload metadata is published");
assert!(visible_marker.exists(), "the record after the prune recreates the marker");
let page = owner.page("bucket", None, Uuid::new_v4()).await.expect("page");
assert_eq!(page.roots, vec![source_root]);
}
/// Stale markers at the start of a page, at its cursor, and filling a whole
/// page: every page still satisfies the receiver's contract and advances.
#[tokio::test]
async fn multipart_index_pages_of_stale_markers_keep_valid_cursors() {
let root = tempfile::tempdir().expect("disk root");
let disk = open(root.path()).await;
let directory = root.path().join(INDEX).join(sha("bucket"));
std::fs::create_dir_all(&directory).expect("marker directory");
// Page one is entirely stale. Page two is live except its first name
// and the name its cursor lands on. Two live names follow.
let total = MULTIPART_INDEX_PAGE_SIZE * 2 + 2;
let stale = |i: usize| i <= MULTIPART_INDEX_PAGE_SIZE || i == MULTIPART_INDEX_PAGE_SIZE * 2 - 1;
for i in 0..total {
let name = format!("{i:064x}");
std::fs::write(directory.join(&name), b"1").expect("marker");
if !stale(i) {
std::fs::create_dir_all(root.path().join(RUSTFS_META_MULTIPART_BUCKET).join(&name)).expect("staging root");
}
}
let disk_id = Uuid::new_v4();
let first = disk.multipart_index.page("bucket", None, disk_id).await.expect("first page");
first
.validate("bucket", None, disk_id)
.expect("an empty page with a cursor validates");
assert!(first.roots.is_empty(), "a fully stale page has no roots");
let cursor = first.next_marker.clone().expect("more markers follow a fully stale page");
assert_eq!(cursor, format!("{:064x}", MULTIPART_INDEX_PAGE_SIZE - 1));
let second = disk
.multipart_index
.page("bucket", Some(&cursor), disk_id)
.await
.expect("second page");
second
.validate("bucket", Some(&cursor), disk_id)
.expect("stale first and cursor names validate");
assert_eq!(second.roots.len(), MULTIPART_INDEX_PAGE_SIZE - 2);
assert_eq!(second.roots.first(), Some(&format!("{:064x}", MULTIPART_INDEX_PAGE_SIZE + 1)));
let cursor = second
.next_marker
.clone()
.expect("the cursor is the stale last examined name");
assert_eq!(cursor, format!("{:064x}", MULTIPART_INDEX_PAGE_SIZE * 2 - 1));
assert!(second.roots.last() < Some(&cursor));
let third = disk
.multipart_index
.page("bucket", Some(&cursor), disk_id)
.await
.expect("third page");
third.validate("bucket", Some(&cursor), disk_id).expect("last page validates");
assert_eq!(third.roots.len(), 2);
assert!(third.next_marker.is_none());
let remaining = std::fs::read_dir(&directory).expect("marker directory").count();
assert_eq!(remaining, total - MULTIPART_INDEX_PAGE_SIZE - 2, "every stale marker is pruned");
}
#[tokio::test]
@@ -964,7 +1290,6 @@ mod tests {
std::fs::create_dir_all(root.path().join(RUSTFS_META_MULTIPART_BUCKET).join(&name)).expect("staging root");
}
}
age_marker(&directory.join(&stale));
let disk_id = Uuid::new_v4();
let first = disk.multipart_index.page("bucket", None, disk_id).await.expect("first page");
first
@@ -428,7 +428,6 @@ impl LocalDisk {
let root = store.publication_root.clone();
let lease = os::acquire_namespace_mutation_lease_with_owner(&destination, Some(store.lease.clone())).await;
os::run_blocking_namespace_operation_with_sync(lease, true, move || {
permit.validate()?;
os::rename_multipart_source_owned_std(&source, &destination, &base, &root, || permit.validate())
})
.await?;
@@ -295,10 +295,17 @@ pub(super) fn put_active_multipart_source_head(source: impl AsRef<Path>, raw: By
}
fn stamp_matches_active_disk(source: &Path, expected_head_stamp: FileStamp) -> bool {
FileStamp::from_path(&source.join(STORAGE_FORMAT_FILE)) == Some(expected_head_stamp)
&& !source.join(STATE).exists()
&& !source.join(crate::disk::MULTIPART_RECOVERY_DIR).exists()
&& !source.join(".mpu-completions").exists()
if FileStamp::from_path(&source.join(STORAGE_FORMAT_FILE)) != Some(expected_head_stamp) {
return false;
}
#[cfg(test)]
if source.join(STATE).exists()
|| source.join(crate::disk::MULTIPART_RECOVERY_DIR).exists()
|| source.join(".mpu-completions").exists()
{
return false;
}
true
}
pub(super) fn get_active_multipart_source_head(source: &Path) -> Option<(Bytes, FileInfo)> {
@@ -375,7 +382,11 @@ pub(super) fn finish_active_multipart_part_transaction(
ACTIVE_MULTIPART_SOURCE_HEADS.remove_if_mut(source, |entry| !entry.unsettled_parts.remove(&part_number));
}
pub(super) fn get_active_multipart_source_part(source: &Path, part_number: usize) -> Option<rustfs_filemeta::ObjectPartInfo> {
pub(super) fn get_active_multipart_source_part_with_head(
source: &Path,
part_number: usize,
head_already_verified: bool,
) -> Option<rustfs_filemeta::ObjectPartInfo> {
let (head_stamp, data_dir, expected_stamp, part) = ACTIVE_MULTIPART_SOURCE_HEADS
.with_get(source, |entry| {
if !entry.unsettled_parts.is_empty() {
@@ -386,7 +397,7 @@ pub(super) fn get_active_multipart_source_part(source: &Path, part_number: usize
Some((entry.head_stamp, data_dir, *expected_stamp, part.clone()))
})
.flatten()?;
if !stamp_matches_active_disk(source, head_stamp) {
if !head_already_verified && !stamp_matches_active_disk(source, head_stamp) {
ACTIVE_MULTIPART_SOURCE_HEADS.remove(source);
return None;
}
@@ -609,25 +620,45 @@ impl SourceStore {
raw: &[u8],
terminal: Option<&Path>,
sync_source_before: bool,
) -> std::io::Result<()> {
Self::persist_state_with_guard_std(source, base, root, raw, terminal, sync_source_before, None)
}
fn persist_state_with_guard_std(
source: &Path,
base: &Path,
root: &os::PublicationRoot,
raw: &[u8],
terminal: Option<&Path>,
sync_source_before: bool,
directory_guard: Option<&os::RenameCommitGuard>,
) -> std::io::Result<()> {
invalidate_active_multipart_source_head(source);
#[cfg(unix)]
let mut pinned = os::pin_multipart_source_directory(source, base, root)?;
#[cfg(not(unix))]
#[cfg(not(target_os = "linux"))]
let _pinned = os::pin_multipart_source_directory(source, base, root)?;
if sync_source_before {
#[cfg(unix)]
#[cfg(target_os = "linux")]
{
let mut pinned = os::pin_multipart_source_directory(source, base, root)?;
let opened = std::fs::File::from(pinned.pop().ok_or_else(|| IoError::other("empty source directory guard"))?);
os::fsync_directory_handle_std(source, Some(&opened))?;
}
#[cfg(all(unix, not(target_os = "linux")))]
{
let opened = std::fs::File::from(
_pinned
.last()
.ok_or_else(|| IoError::other("empty source directory guard"))?
.try_clone()?,
);
os::fsync_directory_handle_std(source, Some(&opened))?;
}
#[cfg(not(unix))]
os::fsync_dir_std(source)?;
}
if let Some(path) = terminal {
os::replace_multipart_source_metadata(path, raw, base, root)?;
}
os::replace_multipart_source_metadata(&source.join(STATE), raw, base, root)
os::replace_multipart_source_metadata_with_guard(&source.join(STATE), raw, base, root, directory_guard)
}
async fn write_owned_backup_and_persist(&self, owner: Uuid, target: bool, bytes: &[u8], state: &SourceState) -> Result<()> {
if owner.is_nil() {
@@ -724,17 +755,39 @@ impl SourceStore {
bytes: &[u8],
sync_source_root: bool,
) -> std::io::Result<()> {
Self::write_owned_backup_inner_std(source, base, root, owner, target, bytes, sync_source_root, false, true, false)
}
#[expect(clippy::too_many_arguments)]
fn write_owned_backup_inner_std(
source: &Path,
base: &Path,
root: &os::PublicationRoot,
owner: Uuid,
target: bool,
bytes: &[u8],
sync_source_root: bool,
fresh_owner: bool,
sync_parents: bool,
source_already_pinned: bool,
) -> std::io::Result<()> {
invalidate_active_multipart_source_head(source);
let path = if target {
target_backup_path(source, owner)
} else {
original_path(source, owner)
};
let _pin = os::pin_multipart_source_directory(source, base, root)?;
let _pin = if source_already_pinned {
None
} else {
Some(os::pin_multipart_source_directory(source, base, root)?)
};
let parent = path
.parent()
.ok_or_else(|| std::io::Error::other("source backup has no parent"))?;
let _parents = os::mkdir_all_below_existing_base_std(parent, base, root)?;
let existed = if target {
let existed = if fresh_owner {
false
} else if target {
match os::observe_multipart_source_metadata(&path, base, root, bytes.len() as u64) {
Ok((length, sha)) if length == bytes.len() as u64 && sha == raw_sha(bytes) => true,
Ok(_) => {
@@ -754,10 +807,14 @@ impl SourceStore {
if !existed {
os::replace_multipart_source_metadata(&path, bytes, base, root)?;
}
os::sync_multipart_owned_record_parents_options(source, parent, existed, sync_source_root, base, root)
if sync_parents {
os::sync_multipart_owned_record_parents_options(source, parent, existed, sync_source_root, base, root)?;
}
Ok(())
}
pub async fn replace_head(&self, bytes: &[u8]) -> Result<()> {
self.multipart_index
let index_pin = self
.multipart_index
.record_head(&format!("{}/xl.meta", self.path), bytes)
.await?;
invalidate_active_multipart_source_head(&self.source);
@@ -766,6 +823,7 @@ impl SourceStore {
let root = self.publication_root.clone();
let bytes = bytes.to_vec();
os::run_blocking_namespace_operation(self.lease.clone(), move || {
let _index_pin = index_pin;
os::replace_multipart_source_metadata(&path, &bytes, &base, &root)
})
.await
@@ -2024,11 +2082,41 @@ impl NativeMultipartPublication {
}
let setup_res: std::io::Result<()> = (|| {
if let Some((original_bytes, directory, anchor_bytes)) = deferred_initial {
SourceStore::write_owned_backup_std(source, base, root, backup_owner, false, &original_bytes, false)?;
SourceStore::write_owned_backup_inner_std(
source,
base,
root,
backup_owner,
false,
&original_bytes,
false,
true,
old_metadata.is_none(),
true,
)?;
if let Some(previous) = old_metadata {
SourceStore::write_owned_backup_std(source, base, root, backup_owner, true, previous, false)?;
SourceStore::write_owned_backup_inner_std(
source,
base,
root,
backup_owner,
true,
previous,
false,
true,
true,
true,
)?;
}
SourceStore::persist_state_std(source, base, root, &next_state_raw, terminal.as_deref(), false)?;
SourceStore::persist_state_with_guard_std(
source,
base,
root,
&next_state_raw,
terminal.as_deref(),
false,
Some(directory_guard),
)?;
let _parent = os::mkdir_all_below_existing_base_std(&directory, base, root)?;
os::replace_multipart_source_metadata(&directory.join(RETIREMENT_ANCHOR), &anchor_bytes, base, root)?;
#[cfg(target_os = "linux")]
@@ -2037,9 +2125,28 @@ impl NativeMultipartPublication {
}
} else {
if let Some(previous) = old_metadata {
SourceStore::write_owned_backup_std(source, base, root, backup_owner, true, previous, false)?;
SourceStore::write_owned_backup_inner_std(
source,
base,
root,
backup_owner,
true,
previous,
false,
false,
true,
true,
)?;
}
SourceStore::persist_state_std(source, base, root, &next_state_raw, terminal.as_deref(), false)?;
SourceStore::persist_state_with_guard_std(
source,
base,
root,
&next_state_raw,
terminal.as_deref(),
false,
Some(directory_guard),
)?;
}
Ok(())
})();
@@ -2269,6 +2376,7 @@ impl NativeMultipartPublication {
target_already_synced: bool,
fused_in_place: bool,
staging_cleanup: Option<(PathBuf, Arc<std::sync::atomic::AtomicBool>)>,
directory_guard: Option<&os::RenameCommitGuard>,
) -> Result<MultipartPublicationResult> {
let mut state = self.state.lock().clone();
let expected_state_raw = encode(&state)?;
@@ -2354,7 +2462,15 @@ impl NativeMultipartPublication {
})?;
}
}
SourceStore::persist_state_std(source, base, root, &next_state_raw, terminal.as_deref(), !fused_in_place)?;
SourceStore::persist_state_with_guard_std(
source,
base,
root,
&next_state_raw,
terminal.as_deref(),
!fused_in_place,
directory_guard,
)?;
if let Some((io_path, cleaned)) = staging_cleanup {
match std::fs::remove_dir(&io_path) {
Ok(()) => cleaned.store(true, std::sync::atomic::Ordering::Release),
@@ -2387,7 +2503,7 @@ impl NativeMultipartPublication {
) -> Result<MultipartPublicationResult> {
let this = Arc::clone(self);
os::run_blocking_namespace_operation_with_sync(target_lease, true, move || {
this.finish_on_thread(target_already_synced, false, staging_cleanup)
this.finish_on_thread(target_already_synced, false, staging_cleanup, None)
.map_err(IoError::other)
})
.await
@@ -455,8 +455,11 @@ impl LocalDisk {
}
let meta_name = format!("part.{}.meta", part.number);
let actual = if let Some(cached_part) =
super::multipart_source_state::get_active_multipart_source_part(&source_object, part.number)
{
super::multipart_source_state::get_active_multipart_source_part_with_head(
&source_object,
part.number,
from_active_cache,
) {
cached_part
} else {
if pinned_source_data.is_none() {
+7 -1
View File
@@ -2585,7 +2585,13 @@ pub struct BoundedDeleteRecoveryDiscovery {
pub complete: bool,
}
pub const DELETE_RECOVERY_MAX_DISCOVERY_ENTRIES: u32 = 4096;
/// Entries one bounded discovery call may charge on one drive. Object-scoped
/// discovery charges each transaction-shaped (UUID) child of the key's
/// directory once and every name that cannot be delete evidence at
/// 1/`PLAIN_CHILDREN_PER_BUDGET_ENTRY`, so this admits a missing-key delete of a
/// folder prefix with this many UUID-named children per drive. The request
/// budget (`DeleteRecoveryWorkBudget`) still bounds the total across keys.
pub const DELETE_RECOVERY_MAX_DISCOVERY_ENTRIES: u32 = 262_144;
pub const DELETE_RECOVERY_MAX_INSPECTION_ENTRIES: u32 = 65536;
#[derive(Debug, Clone, Serialize, Deserialize)]
+95 -45
View File
@@ -1288,6 +1288,28 @@ pub(crate) fn fsync_dir_limited_std(dir: &Path) -> io::Result<()> {
fsync_dir_std(dir)
}
pub(crate) fn fsync_commit_ancestor_directory_limited_std(
dir: &Path,
base_dir: &Path,
step_from_destination: usize,
guard: &RenameCommitGuard,
) -> io::Result<()> {
#[cfg(target_os = "linux")]
if base_dir == guard.directories.destination_base
&& guard.directories.destination_path.ancestors().nth(1 + step_from_destination) == Some(dir)
&& let Some(ancestors) = guard.directories.destination_ancestors.get()
&& let Some(idx) = ancestors.len().checked_sub(1 + step_from_destination)
&& let Some(file) = ancestors.get(idx)
{
#[cfg(test)]
fsync_dir_recorder::record_limited(dir);
return fsync_directory_handle_std(dir, Some(file));
}
#[cfg(not(target_os = "linux"))]
let _ = (base_dir, step_from_destination, guard);
fsync_dir_limited_std(dir)
}
/// Reuse the published directory without bypassing group commit or sync admission.
pub(crate) async fn fsync_commit_directory(
dir: &Path,
@@ -4042,7 +4064,7 @@ pub(crate) fn rename_all_with_precondition_and_parent_sync_std(
#[cfg(all(test, not(windows)))]
prepared_publication_test_hooks::run_rename_destination(source, destination);
#[cfg(all(any(test, feature = "test-util"), not(windows)))]
{
if prepared_publication_test_hooks::has_any_hooks() {
prepared_publication_test_hooks::run(prepared_publication_test_hooks::Stage::Rename, source);
prepared_publication_test_hooks::run(prepared_publication_test_hooks::Stage::Rename, destination);
if let Some(check) = precondition {
@@ -5992,6 +6014,7 @@ struct LinuxCommitDirectories {
destination_path: PathBuf,
source: LinuxCommitDirectory,
destination: std::sync::OnceLock<LinuxCommitDirectory>,
destination_ancestors: std::sync::OnceLock<Vec<std::fs::File>>,
destination_base: PathBuf,
}
@@ -6008,6 +6031,8 @@ impl LinuxCommitDirectories {
.ok_or_else(|| io::Error::other("missing destination directory guard"))?,
);
let directory = LinuxCommitDirectory::new(directory)?;
let ancestors = parents.into_iter().map(std::fs::File::from).collect();
let _ = self.destination_ancestors.set(ancestors);
Ok(self.destination.get_or_init(|| directory))
}
@@ -6017,7 +6042,7 @@ impl LinuxCommitDirectories {
.ok_or_else(|| io::Error::new(io::ErrorKind::InvalidInput, "file parent does not match its commit guard"))
}
fn validate(&self) -> io::Result<()> {
fn validate(&self, check_destination: bool) -> io::Result<()> {
use std::os::unix::fs::MetadataExt;
// A pinned directory may have been unlinked or replaced. Do not report a
// successful publication into a detached object tree.
@@ -6026,8 +6051,9 @@ impl LinuxCommitDirectories {
return Err(io::Error::new(io::ErrorKind::InvalidData, "commit directory identity changed"));
}
let destination = self.prepare_destination()?;
if (self.source.device, self.source.inode) != (destination.device, destination.inode)
|| self.source_path.as_os_str() != self.destination_path.as_os_str()
if check_destination
&& ((self.source.device, self.source.inode) != (destination.device, destination.inode)
|| self.source_path.as_os_str() != self.destination_path.as_os_str())
{
let current = self.root.open_directory(&self.destination_path)?.metadata()?;
if (current.dev(), current.ino()) != (destination.device, destination.inode) {
@@ -6041,8 +6067,9 @@ impl LinuxCommitDirectories {
record_native_publication(destination);
let source_name = self.file_name(source, true)?;
let destination_name = self.file_name(destination, false)?;
let destination_was_prepared = self.destination.get().is_some();
let destination_parent = self.prepare_destination()?;
self.validate()?;
self.validate(destination_was_prepared)?;
let mut attempt = 0;
loop {
if let Some(prepared) = prepared {
@@ -6264,7 +6291,7 @@ pub(crate) fn multipart_payload_commit_guard_std(
source_parent: &Path,
) -> io::Result<RenameCommitGuard> {
let original = &metadata_guard.directories;
original.validate()?;
original.validate(true)?;
let source = LinuxCommitDirectory::new(original.root.open_directory(source_parent)?)?;
let destination = original.prepare_destination()?;
// Clone the pinned file identity, never recapture the target pathname
@@ -6281,6 +6308,7 @@ pub(crate) fn multipart_payload_commit_guard_std(
destination_path: original.destination_path.clone(),
source,
destination: std::sync::OnceLock::from(destination),
destination_ancestors: std::sync::OnceLock::new(),
destination_base: original.destination_base.clone(),
}),
})
@@ -6362,6 +6390,7 @@ pub(crate) fn prepare_rename_commit_guard(
destination_path: destination_parent.to_path_buf(),
source,
destination,
destination_ancestors: std::sync::OnceLock::new(),
destination_base: destination_base.to_path_buf(),
}),
})
@@ -7447,6 +7476,40 @@ pub(crate) fn pin_multipart_source_directory(
}
}
fn same_parent_commit_guard(
parent: &Path,
base: &Path,
publication_root: &PublicationRoot,
directory_guard: Option<&RenameCommitGuard>,
) -> io::Result<RenameCommitGuard> {
#[cfg(target_os = "linux")]
if let Some(directory_guard) = directory_guard.filter(|g| g.directories.source_path == parent) {
let dir = &directory_guard.directories.source;
return Ok(RenameCommitGuard {
directories: Arc::new(LinuxCommitDirectories {
root: directory_guard.directories.root.clone(),
source_path: parent.to_path_buf(),
destination_path: parent.to_path_buf(),
source: LinuxCommitDirectory {
file: Arc::clone(&dir.file),
device: dir.device,
inode: dir.inode,
},
destination: std::sync::OnceLock::from(LinuxCommitDirectory {
file: Arc::clone(&dir.file),
device: dir.device,
inode: dir.inode,
}),
destination_ancestors: std::sync::OnceLock::new(),
destination_base: base.to_path_buf(),
}),
});
}
#[cfg(not(target_os = "linux"))]
let _ = directory_guard;
prepare_rename_commit_guard(parent, parent, base, publication_root)
}
/// A small control record always gets its own inode and an atomic replacement.
/// The caller owns the upload SourceRW and keeps its namespace lease alive.
pub(crate) fn replace_multipart_source_metadata(
@@ -7454,11 +7517,21 @@ pub(crate) fn replace_multipart_source_metadata(
bytes: &[u8],
base: &Path,
publication_root: &PublicationRoot,
) -> io::Result<()> {
replace_multipart_source_metadata_with_guard(path, bytes, base, publication_root, None)
}
pub(crate) fn replace_multipart_source_metadata_with_guard(
path: &Path,
bytes: &[u8],
base: &Path,
publication_root: &PublicationRoot,
directory_guard: Option<&RenameCommitGuard>,
) -> io::Result<()> {
let parent = path.parent().ok_or_else(|| io::Error::other("MPU record has no parent"))?;
#[cfg(not(target_os = "linux"))]
let _pinned = pin_multipart_source_directory(parent, base, publication_root)?;
let guard = prepare_rename_commit_guard(parent, parent, base, publication_root)?;
let guard = same_parent_commit_guard(parent, base, publication_root, directory_guard)?;
let staged = parent.join(format!(".mpu-record-{}", uuid::Uuid::new_v4()));
let mut prepared = create_prepared_rename_source_with_commit_guard(&staged, path, &guard)?;
prepared.write_all(bytes, true)?;
@@ -7707,37 +7780,9 @@ pub(crate) fn install_multipart_prepared_head(
if original_path.parent() != Some(parent) || original_path == canonical_head {
return Err(io::Error::new(io::ErrorKind::InvalidInput, "MPU proxy must be a private sibling"));
}
#[cfg(target_os = "linux")]
let guard = if let Some(directory_guard) = directory_guard.filter(|g| g.directories.source_path == parent) {
let dir = &directory_guard.directories.source;
RenameCommitGuard {
directories: Arc::new(LinuxCommitDirectories {
root: directory_guard.directories.root.clone(),
source_path: parent.to_path_buf(),
destination_path: parent.to_path_buf(),
source: LinuxCommitDirectory {
file: Arc::clone(&dir.file),
device: dir.device,
inode: dir.inode,
},
destination: std::sync::OnceLock::from(LinuxCommitDirectory {
file: Arc::clone(&dir.file),
device: dir.device,
inode: dir.inode,
}),
destination_base: base.to_path_buf(),
}),
}
} else {
prepare_rename_commit_guard(parent, parent, base, publication_root)?
};
#[cfg(not(target_os = "linux"))]
let _pinned = pin_multipart_source_directory(parent, base, publication_root)?;
#[cfg(not(target_os = "linux"))]
let guard = {
let _ = directory_guard;
prepare_rename_commit_guard(parent, parent, base, publication_root)?
};
let guard = same_parent_commit_guard(parent, base, publication_root, directory_guard)?;
rename_prepared_source_std(&prepared, &original_path, canonical_head, base, publication_root, &guard)?;
prepared.path = canonical_head.to_path_buf();
fsync_commit_directory_limited_std(parent, &guard)?;
@@ -8039,8 +8084,7 @@ pub(crate) fn hardlink_multipart_source_files(
}
#[cfg(unix)]
{
use rustix::fs::{AtFlags, Mode, OFlags, linkat, openat, unlinkat};
use std::os::unix::fs::MetadataExt;
use rustix::fs::{AtFlags, FileType, linkat, statat, unlinkat};
#[cfg(test)]
let source_path = source.to_path_buf();
#[cfg(test)]
@@ -8055,20 +8099,26 @@ pub(crate) fn hardlink_multipart_source_files(
.ok_or_else(|| io::Error::other("missing multipart attempt directory"))?;
#[cfg(test)]
prepared_publication_test_hooks::run_hardlink_destination(&source_path, &destination_path);
let flags = OFlags::RDONLY | OFlags::NOFOLLOW | OFlags::CLOEXEC;
for name in names {
if cancelled() {
return Err(io::Error::new(io::ErrorKind::Interrupted, "multipart staging cancelled"));
}
let original = std::fs::File::from(openat(source, name.as_str(), flags, Mode::empty())?);
let expected = original.metadata()?;
if !expected.is_file() {
let expected = statat(source, name.as_str(), AtFlags::SYMLINK_NOFOLLOW)?;
let expected_type = FileType::from_raw_mode(expected.st_mode);
if expected_type.is_symlink() {
return Err(io::Error::from_raw_os_error(libc::ELOOP));
}
if !expected_type.is_file() {
return Err(io::Error::new(io::ErrorKind::InvalidData, "multipart shard is not a regular file"));
}
linkat(source, name.as_str(), destination, name.as_str(), AtFlags::empty())?;
let linked = std::fs::File::from(openat(destination, name.as_str(), flags, Mode::empty())?);
let actual = linked.metadata()?;
if actual.dev() != expected.dev() || actual.ino() != expected.ino() {
let actual = statat(destination, name.as_str(), AtFlags::SYMLINK_NOFOLLOW)?;
if !FileType::from_raw_mode(actual.st_mode).is_file()
|| actual.st_dev != expected.st_dev
|| actual.st_ino != expected.st_ino
|| actual.st_size != expected.st_size
|| actual.st_mtime != expected.st_mtime
{
let _ = unlinkat(destination, name.as_str(), AtFlags::empty());
return Err(io::Error::new(io::ErrorKind::InvalidData, "multipart shard changed during staging"));
}
+5 -3
View File
@@ -975,9 +975,11 @@ pub struct DeleteRecoveryWorkBudget {
remaining: std::sync::atomic::AtomicU32,
}
/// Entries one disk may visit per recovery step (discovery plus inspection).
/// Every disk of every set a request fans out to is admitted up to this much
/// and charged on completion, so a budget must cover that many disks.
/// Per-drive unit a request budget is sized by (a typical discovery plus one
/// inspection). Admission charges what a call visited, not this unit, so it only
/// scales the request total with the drives a request can reach; it is kept
/// independent of the per-call caps, which may let one key's probe use more of
/// the same total on a large folder.
const DELETE_RECOVERY_RESERVATION_PER_DISK: u32 = 4096 + 65_536;
impl Default for DeleteRecoveryWorkBudget {
+263 -1
View File
@@ -94,6 +94,7 @@ use super::{
delete_tier_mutation_intent_record_if_current, list_tier_coordinator_mutation_intent_records,
list_tier_mutation_intent_records, load_tier_coordinator_mutation_intent_record_with_etag,
load_tier_mutation_intent_record_with_etag, save_tier_coordinator_mutation_intent_record_if_absent,
save_tier_mutation_intent_record_if_current,
},
warm_backend::WarmBackendImpl,
};
@@ -621,7 +622,11 @@ struct CommittedMutationRecoveryPlan {
impl TierMutationRecoverySnapshot {
fn requires_config_lock(&self) -> bool {
self.has_committed_mutation_blocks || self.intents.iter().any(|recovered| recovered.has_coordinator_record)
self.has_committed_mutation_blocks
|| self.intents.iter().any(|recovered| {
recovered.has_coordinator_record
|| (recovered.has_peer_record && recovered.intent.state == TierMutationIntentState::Prepared)
})
}
fn allowance(&self) -> MutationBlockAllowance {
@@ -5103,6 +5108,9 @@ impl TierConfigMgr {
Self::reconcile_terminal_dual_prefix_mutation_intents(api.clone(), &mut snapshot).await?;
Self::recover_prepared_coordinator_mutation_intents(api.clone(), &mut snapshot.coordinator_intents).await?;
if config_lock.is_some() {
Self::recover_expired_peer_only_prepares(api.clone(), &snapshot).await?;
}
snapshot = Self::load_and_reconcile_mutation_recovery_snapshot(handle, api.clone(), true).await?;
if snapshot.requires_config_lock() && config_lock.is_none() {
@@ -5503,6 +5511,58 @@ impl TierConfigMgr {
}
}
async fn recover_expired_peer_only_prepares<S>(api: Arc<S>, snapshot: &TierMutationRecoverySnapshot) -> io::Result<()>
where
S: EcstoreObjectIO,
{
let now = i64::try_from(OffsetDateTime::now_utc().unix_timestamp_nanos()).unwrap_or(i64::MAX);
let orphaned = snapshot
.intents
.iter()
.filter(|recovered| {
recovered.has_peer_record
&& !recovered.has_coordinator_record
&& recovered.intent.state == TierMutationIntentState::Prepared
&& Self::tier_mutation_tombstone_retention_elapsed(&recovered.intent, now)
})
.map(|recovered| recovered.intent.clone())
.collect::<Vec<_>>();
if orphaned.is_empty() {
return Ok(());
}
// The caller holds the tier-config namespace lock. Only the exact
// persisted candidate proves a commit; an old or different config
// leaves the peer fence intact.
let loaded = load_tier_config_for_recovery(api.clone()).await?;
let Some(config_etag) = loaded.etag else {
return Ok(());
};
for intent in orphaned {
if loaded.persisted_digest != Some(intent.candidate_digest) {
continue;
}
let _mutation_guard = acquire_tier_mutation_mutex(intent.mutation_id).await;
let (mut current, record_etag) =
match load_tier_mutation_intent_record_with_etag(api.clone(), intent.mutation_id).await {
Ok(record) => record,
Err(Error::ConfigNotFound) => continue,
Err(err) => return Err(tier_mutation_replay_error(err)),
};
if current != intent {
continue;
}
current
.advance(TierMutationIntentState::Committed, Some(config_etag.clone()))
.map_err(io::Error::other)?;
match save_tier_mutation_intent_record_if_current(api.clone(), &current, &record_etag).await {
Ok(()) | Err(Error::PreconditionFailed) => {}
Err(err) => return Err(tier_mutation_replay_error(err)),
}
}
Ok(())
}
async fn recover_prepared_coordinator_mutation_intents<S>(api: Arc<S>, intents: &mut [TierMutationIntent]) -> io::Result<()>
where
S: EcstoreObjectIO,
@@ -13415,6 +13475,208 @@ mod tests {
);
}
#[tokio::test]
async fn reload_recovers_expired_peer_only_prepare_for_persisted_add() {
use crate::services::tier::tier_mutation_intent::save_tier_mutation_intent_record;
let store = Arc::new(CasConfigStore::default());
let previous = empty_mgr();
let mut candidate = empty_mgr();
candidate.tiers.insert("COLD-A".to_string(), build_rustfs_tier("COLD-A"));
let targets = build_tier_mutation_affected_targets(
TierMutationIntentKind::Add,
HashSet::from(["COLD-A".to_string()]),
&previous,
&candidate,
)
.expect("added tier should have a target");
let mut intent = build_coordinator_tier_mutation_intent(TierMutationIntentKind::Add, None, &candidate, targets)
.expect("intent should build")
.expect("add should require an intent");
intent.expires_at_unix_nanos = 1;
candidate
.save_tiering_config_if_current(store.clone(), None)
.await
.expect("candidate should persist");
save_tier_mutation_intent_record(store.clone(), &intent)
.await
.expect("peer prepare should persist without a coordinator record");
let manager = TierConfigMgr::new();
TIER_MUTATION_TEST_PEERS
.scope(Vec::new(), async {
TIER_DRIVER_TEST_FACTORY
.scope(healthy_driver_factory(), async {
TierConfigMgr::reload_handle_with(&manager, store.clone())
.await
.expect("matching saved candidate should recover");
TierConfigMgr::acquire_operation_lease(&manager, "COLD-A")
.await
.expect("recovered tier should be readable");
})
.await;
})
.await;
assert!(
TierConfigMgr::load_tier_mutation_intents(store)
.await
.expect("peer scan should succeed")
.iter()
.all(|record| record.state != TierMutationIntentState::Prepared),
"the orphaned prepare must no longer block later reloads"
);
}
#[tokio::test]
async fn reload_keeps_peer_only_prepare_without_commit_proof() {
use crate::services::tier::tier_mutation_intent::{
save_tier_mutation_intent_record, tier_mutation_intent_record_object_name,
};
for case in [
"unexpired",
"recently_expired",
"different_candidate",
"missing_config",
"concurrent_change",
] {
let store = Arc::new(CasConfigStore::default());
let previous = empty_mgr();
let mut candidate = empty_mgr();
candidate.tiers.insert("COLD-A".to_string(), build_rustfs_tier("COLD-A"));
let targets = build_tier_mutation_affected_targets(
TierMutationIntentKind::Add,
HashSet::from(["COLD-A".to_string()]),
&previous,
&candidate,
)
.expect("added tier should have a target");
let mut intent = build_coordinator_tier_mutation_intent(TierMutationIntentKind::Add, None, &candidate, targets)
.expect("intent should build")
.expect("add should require an intent");
intent.expires_at_unix_nanos = match case {
"unexpired" => tier_mutation_intent_expiry_unix_nanos(),
"recently_expired" => {
i64::try_from(OffsetDateTime::now_utc().unix_timestamp_nanos()).expect("test clock should fit")
- 1_000_000_000
}
_ => 1,
};
if case == "different_candidate" {
intent.candidate_digest = [0x51; 32];
}
if case != "missing_config" {
candidate
.save_tiering_config_if_current(store.clone(), None)
.await
.expect("candidate should persist");
}
save_tier_mutation_intent_record(store.clone(), &intent)
.await
.expect("peer prepare should persist");
let expected = if case == "concurrent_change" {
let mut changed = intent.clone();
changed.candidate_digest = [0x52; 32];
let object = tier_mutation_intent_record_object_name(intent.mutation_id).expect("record path should build");
store
.rewrite_on_next_if_match(object, changed.encode().expect("changed intent should encode"))
.await;
changed
} else {
intent
};
let manager = TierConfigMgr::new();
TIER_MUTATION_TEST_PEERS
.scope(Vec::new(), async {
TierConfigMgr::reload_handle_with(&manager, store.clone())
.await
.expect("unproven prepare should remain fenced");
})
.await;
let error = match TierConfigMgr::acquire_operation_lease(&manager, "COLD-A").await {
Ok(_) => panic!("{case}: unproven prepare must block reads"),
Err(error) => error,
};
assert!(error.message.contains("being replaced"), "{case}: {error}");
let records = TierConfigMgr::load_tier_mutation_intents(store)
.await
.expect("peer scan should succeed");
assert_eq!(records, vec![expected], "{case}: prepared evidence must remain unchanged");
}
}
#[tokio::test]
async fn peer_only_prepare_recovery_retains_fence_until_peer_commit_succeeds() {
use crate::services::tier::tier_mutation_intent::save_tier_mutation_intent_record;
let store = Arc::new(CasConfigStore::default());
let previous = empty_mgr();
let mut candidate = empty_mgr();
candidate.tiers.insert("COLD-A".to_string(), build_rustfs_tier("COLD-A"));
let targets = build_tier_mutation_affected_targets(
TierMutationIntentKind::Add,
HashSet::from(["COLD-A".to_string()]),
&previous,
&candidate,
)
.expect("added tier should have a target");
let mut intent = build_coordinator_tier_mutation_intent(TierMutationIntentKind::Add, None, &candidate, targets)
.expect("intent should build")
.expect("add should require an intent");
intent.expires_at_unix_nanos = 1;
candidate
.save_tiering_config_if_current(store.clone(), None)
.await
.expect("candidate should persist");
save_tier_mutation_intent_record(store.clone(), &intent)
.await
.expect("peer prepare should persist");
let manager = TierConfigMgr::new();
let calls = Arc::new(Mutex::new(Vec::new()));
TIER_MUTATION_TEST_PEERS
.scope(vec![FakeTierMutationPeer::boxed("peer-a", calls.clone(), Err("offline"))], async {
TierConfigMgr::reload_handle_with(&manager, store.clone())
.await
.expect_err("failed peer commit must leave recovery incomplete");
})
.await;
let retained = TierConfigMgr::load_tier_mutation_intents(store.clone())
.await
.expect("peer scan should succeed");
assert_eq!(retained.len(), 1);
assert_eq!(retained[0].state, TierMutationIntentState::Committed);
let error = match TierConfigMgr::acquire_operation_lease(&manager, "COLD-A").await {
Ok(_) => panic!("failed peer commit must retain the fence"),
Err(error) => error,
};
assert!(error.message.contains("being replaced"), "{error}");
TIER_MUTATION_TEST_PEERS
.scope(
vec![FakeTierMutationPeer::boxed(
"peer-a",
calls.clone(),
Ok(PeerTierMutationState::Committed),
)],
async {
TIER_DRIVER_TEST_FACTORY
.scope(healthy_driver_factory(), async {
TierConfigMgr::reload_handle_with(&manager, store.clone())
.await
.expect("peer commit retry should recover");
TierConfigMgr::acquire_operation_lease(&manager, "COLD-A")
.await
.expect("tier should be readable after the retry");
})
.await;
},
)
.await;
assert_eq!(lock_unpoisoned(&calls).len(), 2, "recovery should retry the peer commit");
}
#[tokio::test]
async fn coordinator_mutation_recovery_scan_blocks_new_operation_lease() {
let store = Arc::new(CasConfigStore::default());
+27 -2
View File
@@ -4608,6 +4608,19 @@ fn marker_purge_receipt_path(
}
impl SetDisks {
/// The first disk slot UUID is assigned when this erasure set is formatted.
/// It survives pool reordering and replacement of a disk in that slot.
pub fn persistent_set_id(&self) -> Result<Uuid> {
self.format
.erasure
.sets
.get(self.set_index)
.and_then(|disks| disks.first())
.copied()
.filter(|id| !id.is_nil())
.ok_or_else(|| Error::other("erasure set has no persistent format identity"))
}
/// Persist a quorum DELETE receipt outside the object marker itself. The
/// receipt is written to the metadata bucket so it survives a node restart
/// while an inaccessible member still holds the stale marker.
@@ -11027,7 +11040,12 @@ mod tests {
let live = Uuid::new_v4();
let orphan = Uuid::new_v4();
let object_dir = dir.path().join("bucket/obj");
write_object_meta_with_data_dirs(&object_dir, "bucket", "obj", &[live]).await;
// Part files are inert to discovery and cost 1/64 of an entry, so the
// first step is exhausted by what still costs a full entry each: the
// data directories of many versions, all referenced by the metadata.
let versions = (0..4200).map(|_| Uuid::new_v4()).collect::<Vec<_>>();
let referenced = std::iter::once(live).chain(versions.iter().copied()).collect::<Vec<_>>();
write_object_meta_with_data_dirs(&object_dir, "bucket", "obj", &referenced).await;
let live_dir = object_dir.join(live.to_string());
let orphan_dir = object_dir.join(orphan.to_string());
fs::create_dir_all(&live_dir).await.expect("create live multipart directory");
@@ -11036,10 +11054,14 @@ mod tests {
.await
.expect("write orphan payload");
let parts_dir = live_dir.clone();
let versions_root = object_dir.clone();
tokio::task::spawn_blocking(move || {
for part in 1..=4200 {
std::fs::write(parts_dir.join(format!("part.{part}")), b"live").expect("write live multipart entry");
}
for version in versions {
std::fs::create_dir(versions_root.join(version.to_string())).expect("create referenced version directory");
}
})
.await
.expect("create multipart fixture");
@@ -11049,7 +11071,10 @@ mod tests {
.await
.expect("read first discovery page");
assert!(first.transactions.is_empty());
assert!(first.next.is_some(), "ordinary multipart entries must exhaust the first scan budget");
assert!(
first.next.is_some(),
"the data directories of many versions must exhaust the first scan step"
);
let last = disk
.discover_delete_transactions("bucket", Some("obj"), first.next, 1)
.await
@@ -1709,6 +1709,37 @@ mod tests {
);
}
#[tokio::test]
async fn delete_recovery_missing_key_naming_a_folder_of_uuid_named_keys_completes() {
let bucket = "recover-uuid-named-folder";
let (dirs, store, _) = fixture(bucket).await;
// More UUID-named child keys per drive than the former 4096-entry probe
// could cover once each child's `xl.meta` and data directory were charged
// (three entries per child).
const CHILDREN: usize = 1500;
for dir in &dirs {
let prefix = dir.path().join(bucket).join("uploads");
for _ in 0..CHILDREN {
let child = prefix.join(Uuid::new_v4().to_string());
std::fs::create_dir_all(child.join(Uuid::new_v4().to_string())).expect("child key data dir");
std::fs::write(child.join(crate::disk::STORAGE_FORMAT_FILE), b"meta").expect("child key metadata");
}
}
let error = store
.delete_object(bucket, "uploads", ObjectOptions::default())
.await
.expect_err("the folder's own key does not exist");
assert!(
crate::error::is_err_object_not_found(&error),
"a complete inventory keeps the missing-key no-op semantics: {error:?}"
);
assert!(dirs.iter().all(|dir| {
std::fs::read_dir(dir.path().join(bucket).join("uploads"))
.map(|entries| entries.count() == CHILDREN)
.unwrap_or(false)
}));
}
#[tokio::test]
async fn delete_recovery_batch_budget_survives_pool_and_set_fanout() {
use crate::layout::endpoint::Endpoint;
+22 -18
View File
@@ -720,6 +720,7 @@ impl ECStore {
},
other => StorageError::Lock(other),
})?;
let ns_guard = Arc::new(ns_guard);
let recover = Box::pin(async {
self.ensure_bucket_volume_absent_on_all_sets(bucket).await?;
@@ -1207,15 +1208,16 @@ impl ECStore {
Ok(())
}
async fn cleanup_bucket_usage(&self, bucket: &str, guard: Option<&rustfs_lock::NamespaceLockGuard>) -> Result<()> {
run_bucket_usage_cleanup(guard, bucket, async {
crate::data_usage::prepare_bucket_usage_for_namespace_change(bucket, guard).await?;
async fn cleanup_bucket_usage(&self, bucket: &str, guard: Option<&Arc<rustfs_lock::NamespaceLockGuard>>) -> Result<()> {
let namespace_guard = guard.map(Arc::as_ref);
run_bucket_usage_cleanup(namespace_guard, bucket, async {
crate::data_usage::prepare_bucket_usage_for_namespace_change(bucket, namespace_guard).await?;
crate::data_usage::remove_bucket_usage_from_backend_with_guard_fenced(self, bucket, guard).await
})
.await
}
async fn cleanup_bucket_usage_best_effort(&self, bucket: &str, guard: Option<&rustfs_lock::NamespaceLockGuard>) {
async fn cleanup_bucket_usage_best_effort(&self, bucket: &str, guard: Option<&Arc<rustfs_lock::NamespaceLockGuard>>) {
if let Err(err) = self.cleanup_bucket_usage(bucket, guard).await {
warn!(
bucket = %bucket,
@@ -1347,6 +1349,7 @@ impl ECStore {
} else {
None
};
let ns_guard = ns_guard.map(Arc::new);
let existing_bucket_info = match self.get_bucket_info_from_sets(bucket, &BucketOptions::default()).await {
Ok(info) => Some(info),
@@ -1494,7 +1497,7 @@ impl ECStore {
};
await_bucket_lifecycle_operation(
bucket_lifecycle_guard.as_ref(),
ns_guard.as_ref(),
ns_guard.as_deref(),
bucket,
"bucket metadata creation intent",
await_bucket_namespace_operation(
@@ -1520,7 +1523,7 @@ impl ECStore {
}
if bucket_lifecycle_guard.as_ref().is_some_and(|guard| guard.is_lock_lost())
|| metadata_transaction_guard.as_ref().is_some_and(|guard| guard.is_lock_lost())
|| ns_guard.as_ref().is_some_and(|guard| guard.is_lock_lost())
|| ns_guard.as_deref().is_some_and(|guard| guard.is_lock_lost())
{
if let Ok(sys) = metadata_sys::require_bucket_metadata_sys_in(&self.ctx) {
let sys = sys.read().await.clone();
@@ -1549,7 +1552,7 @@ impl ECStore {
let mut continuation_publication_started = false;
if let Err(err) = await_bucket_lifecycle_operation(
bucket_lifecycle_guard.as_ref(),
ns_guard.as_ref(),
ns_guard.as_deref(),
bucket,
"physical bucket creation",
await_bucket_namespace_operation(
@@ -1608,13 +1611,13 @@ impl ECStore {
{
warn!("best-effort bucket heal after BucketExists failed: {heal_err}");
}
if confirmed_missing && !is_err_bucket_exists(&err) && ns_guard.as_ref().is_none_or(|guard| !guard.is_lock_lost()) {
if confirmed_missing && !is_err_bucket_exists(&err) && ns_guard.as_deref().is_none_or(|guard| !guard.is_lock_lost()) {
error!("make bucket failed: {err}");
self.rollback_failed_bucket_creation(
bucket,
bucket_lifecycle_guard.as_ref(),
metadata_transaction_guard.as_ref(),
ns_guard.as_ref(),
ns_guard.as_deref(),
)
.await;
}
@@ -1636,14 +1639,14 @@ impl ECStore {
.await;
let metadata_lock_lost = bucket_lifecycle_guard.as_ref().is_some_and(|guard| guard.is_lock_lost())
|| metadata_transaction_guard.as_ref().is_some_and(|guard| guard.is_lock_lost())
|| ns_guard.as_ref().is_some_and(|guard| guard.is_lock_lost());
|| ns_guard.as_deref().is_some_and(|guard| guard.is_lock_lost());
if let Err(err) = metadata_result {
if !metadata_lock_lost {
self.rollback_failed_bucket_creation(
bucket,
bucket_lifecycle_guard.as_ref(),
metadata_transaction_guard.as_ref(),
ns_guard.as_ref(),
ns_guard.as_deref(),
)
.await;
}
@@ -2007,6 +2010,7 @@ impl ECStore {
} else {
None
};
let ns_guard = ns_guard.map(Arc::new);
let sr_mark_delete = opts.srdelete_op == SRBucketDeleteOp::MarkDelete;
let sr_purge = opts.srdelete_op == SRBucketDeleteOp::Purge;
@@ -2042,7 +2046,7 @@ impl ECStore {
crate::bucket::metadata_sys::get_object_lock_config_and_incarnation_from_disk_with_guard_in(
&self.ctx,
bucket,
Some(ns_guard.as_ref().unwrap_or(&guard)),
Some(ns_guard.as_deref().unwrap_or(&guard)),
)
.await?;
match state {
@@ -2068,7 +2072,7 @@ impl ECStore {
if sr_mark_delete {
await_bucket_lifecycle_operation(
bucket_lifecycle_guard.as_ref(),
ns_guard.as_ref(),
ns_guard.as_deref(),
bucket,
"bucket delete marker creation",
await_bucket_namespace_operation(
@@ -2107,7 +2111,7 @@ impl ECStore {
object_lock_metadata_guard.as_ref().map(|(guard, _)| guard),
bucket,
"Object Lock configuration fence during bucket deletion",
run_physical_bucket_deletion(ns_guard.as_ref(), bucket, async {
run_physical_bucket_deletion(ns_guard.as_deref(), bucket, async {
self.delete_bucket_on_sets(bucket, &delete_opts)
.await
.map_err(|err| to_object_err(err, vec![bucket]))
@@ -2136,12 +2140,12 @@ impl ECStore {
if let Some(guard) = bucket_lifecycle_guard.as_ref() {
record_opts.add_bucket_lifecycle_lock_guard(guard);
}
if let Some(guard) = ns_guard.as_ref() {
if let Some(guard) = ns_guard.as_deref() {
record_opts.add_namespace_lock_guard(guard);
}
await_bucket_lifecycle_operation(
bucket_lifecycle_guard.as_ref(),
ns_guard.as_ref(),
ns_guard.as_deref(),
bucket,
"bucket retirement publication",
crate::bucket::retirement::commit_retirement(store.clone(), bucket, incarnation, &record_opts),
@@ -2151,7 +2155,7 @@ impl ECStore {
// V1 ledger must not block quota admission after same-name recreation.
await_bucket_lifecycle_operation(
bucket_lifecycle_guard.as_ref(),
ns_guard.as_ref(),
ns_guard.as_deref(),
bucket,
"retired bucket quota ledger cleanup",
crate::bucket::quota::reservation::cleanup_retired_bucket_ledger(store, bucket, incarnation),
@@ -2162,7 +2166,7 @@ impl ECStore {
self.cleanup_bucket_usage_best_effort(bucket, ns_guard.as_ref()).await;
if let Err(err) = self
.cleanup_deleted_bucket_metadata(bucket, sr_purge, ns_guard.as_ref())
.cleanup_deleted_bucket_metadata(bucket, sr_purge, ns_guard.as_deref())
.await
{
warn!(
+14
View File
@@ -1122,6 +1122,20 @@ impl ECStore {
Some(epoch)
}
/// Namespace cleanup edits an existing usage baseline; it does not certify
/// a scan. Retain the movement fence without requiring scanning to resume.
pub(crate) async fn data_usage_namespace_cleanup_guard(&self) -> Option<(tokio::sync::OwnedRwLockReadGuard<()>, u64)> {
let operation_guard = self.ctx.data_movement_operation_gate().read_owned().await;
if self.ctx.data_movement_operation_epoch_exhausted() || self.ctx.data_movement_generation_exhausted() {
return None;
}
Some((operation_guard, self.ctx.data_movement_operation_epoch()))
}
pub(crate) async fn data_usage_namespace_cleanup_epoch(&self) -> Option<u64> {
self.data_usage_namespace_cleanup_guard().await.map(|(_, epoch)| epoch)
}
/// Acquire a storage-owned read admission for a coordinator's final
/// scanner publication. The guard remains in the context's lease table,
/// so a local movement writer cannot pass the peer while its authoritative
+372 -1
View File
@@ -4941,6 +4941,87 @@ impl ECStore {
Ok((idx, result))
}
/// Compare the logical revision across all pools before publishing a
/// namespace-only usage update to a writable pool. A suspended source is
/// retained for decommission to reconcile against the newer target.
pub(crate) async fn put_usage_snapshot_for_namespace_cleanup(
&self,
object: &str,
expected_revision: Option<&str>,
data: &mut PutObjReader,
namespace_guard: Option<&rustfs_lock::NamespaceLockGuard>,
) -> Result<ObjectInfo> {
// The caller owns the bucket namespace and movement guards. Acquire
// the fixed object domain, then every physical domain in pool order.
let guards = self
.acquire_all_physical_object_write_locks("usage_namespace_cleanup", RUSTFS_META_BUCKET, object)
.await?;
let mut opts = ObjectOptions {
no_lock: true,
max_parity: true,
write_completion: crate::object_api::WriteCompletion::TailDrained,
suppress_read_repair: true,
..Default::default()
};
for guard in &guards {
guard.add_namespace_lock_fence(&mut opts);
}
if let Some(guard) = namespace_guard {
opts.add_namespace_lock_guard(guard);
}
let source = match self
.get_latest_object_info_for_conditional_write(RUSTFS_META_BUCKET, object, &opts)
.await
.and_then(|(info, idx)| resolve_latest_object_access(RUSTFS_META_BUCKET, object, info, idx, &opts))
{
Ok((info, _)) => Some(info),
Err(err) if is_err_object_not_found(&err) || is_err_version_not_found(&err) => None,
Err(err) => return Err(err),
};
if source.as_ref().and_then(|info| info.etag.as_deref()) != expected_revision {
return Err(StorageError::PreconditionFailed);
}
if source.as_ref().is_some_and(|info| {
info.version_id.is_some_and(|version| !version.is_nil()) || info.etag.as_ref().is_none_or(String::is_empty)
}) {
return Err(StorageError::other("usage cleanup requires an unversioned snapshot with a revision"));
}
let target = self.get_pool_idx_no_lock(RUSTFS_META_BUCKET, object, data.size()).await?;
let target_info = match self.pools[target].get_object_info(RUSTFS_META_BUCKET, object, &opts).await {
Ok(info) => Some(info),
Err(err) if is_err_object_not_found(&err) || is_err_version_not_found(&err) => None,
Err(err) => return Err(err),
};
opts.http_preconditions = Some(match target_info.as_ref() {
Some(info) => crate::storage_api_contracts::object::HTTPPreconditions {
if_match: Some(
info.etag
.as_ref()
.filter(|etag| !etag.is_empty())
.cloned()
.ok_or_else(|| StorageError::other("usage cleanup target has no revision"))?,
),
..Default::default()
},
None => crate::storage_api_contracts::object::HTTPPreconditions {
if_none_match: Some("*".to_string()),
..Default::default()
},
});
let latest = source
.iter()
.chain(target_info.iter())
.filter_map(|info| info.mod_time)
.fold(OffsetDateTime::now_utc(), std::cmp::max);
opts.mod_time = Some(
latest
.checked_add(time::Duration::nanoseconds(1))
.ok_or_else(|| StorageError::other("usage cleanup snapshot timestamp is exhausted"))?,
);
opts.decommission_capacity_admission = crate::bucket::metadata_sys::object_store_if_initialized_in(&self.ctx).await;
self.pools[target].put_object(RUSTFS_META_BUCKET, object, data, &opts).await
}
#[instrument(level = "debug", skip(self, data))]
#[hotpath::measure(impl_type = "ECStore")]
pub(super) async fn handle_put_object(
@@ -7009,7 +7090,7 @@ mod tests {
};
use crate::set_disk::same_distributed_lock_domain;
use crate::set_disk::{SetDisks, disk_call_counters};
use crate::storage_api_contracts::bucket::MakeBucketOptions;
use crate::storage_api_contracts::bucket::{BucketOptions, MakeBucketOptions};
use crate::storage_api_contracts::lifecycle::TransitionedObject;
use crate::storage_api_contracts::list::ListOperations as _;
use bytes::Bytes;
@@ -10351,6 +10432,296 @@ mod tests {
multipool_delete_test_store(bucket, true).await
}
#[tokio::test]
async fn create_fresh_bucket_after_canceled_decommission() {
let (_dirs, store) = multipool_version_test_store("canceled-decommission-existing").await;
let mut canceled = prepared_pool_test_status(1, false);
canceled.decommission = Some(PoolDecommissionInfo {
canceled: true,
..Default::default()
});
*store.pool_meta.write().await = PoolMeta {
pools: vec![prepared_pool_test_status(0, false), canceled],
..Default::default()
};
store
.handle_make_bucket("canceled-decommission-fresh", &MakeBucketOptions::default())
.await
.expect("canceled decommission must not block creation of a new bucket");
assert!(
store.scanner_data_usage_publication_blocked().await,
"scanner publication must remain fenced"
);
}
#[tokio::test]
async fn create_bucket_after_failed_decommission() {
let (_dirs, store) = multipool_version_test_store("decommission-existing-bucket").await;
let bucket = "decommission-new-bucket";
let snapshot_time = std::time::SystemTime::now() - Duration::from_secs(60);
let mut snapshot = rustfs_data_usage::DataUsageInfo {
last_update: Some(snapshot_time),
scanner_cycle: Some(37),
scanner_epoch: Some(2),
buckets_count: 1,
usage_snapshot_complete: true,
..Default::default()
};
snapshot.buckets_usage.insert(
bucket.to_string(),
rustfs_data_usage::BucketUsageInfo {
size: 42,
objects_count: 1,
versions_count: 1,
..Default::default()
},
);
snapshot.bucket_sizes.insert(bucket.to_string(), 42);
snapshot.calculate_totals();
crate::data_usage::store_data_usage_in_backend(snapshot, store.clone())
.await
.expect("store a stale bucket-usage baseline before the failed decommission");
let (old_snapshot, usage_owner) = store
.get_latest_accessible_object_info_with_idx(
RUSTFS_META_BUCKET,
crate::data_usage::DATA_USAGE_OBJ_NAME_PATH.as_str(),
&ObjectOptions::default(),
)
.await
.expect("identify the pool holding the old usage baseline");
let mut pool_statuses = vec![prepared_pool_test_status(0, false), prepared_pool_test_status(1, false)];
pool_statuses[usage_owner].decommission = Some(PoolDecommissionInfo {
failed: true,
..Default::default()
});
*store.pool_meta.write().await = PoolMeta {
pools: pool_statuses,
..Default::default()
};
store
.handle_make_bucket(bucket, &MakeBucketOptions::default())
.await
.expect("a failed decommission must not block creation of an unrelated bucket");
store
.handle_get_bucket_info(bucket, &BucketOptions::default())
.await
.expect("the created bucket must be visible");
let usage = crate::data_usage::load_data_usage_from_backend(store.clone())
.await
.expect("load the cleaned usage baseline");
assert!(!usage.buckets_usage.contains_key(bucket), "the new bucket must not inherit old usage");
assert_eq!(usage.scanner_cycle, Some(37), "namespace cleanup must not certify a new scan");
assert_eq!(usage.scanner_epoch, Some(2), "namespace cleanup must preserve the scanner authority");
let object = crate::data_usage::DATA_USAGE_OBJ_NAME_PATH.as_str();
let (_, current_owner) = store
.get_latest_accessible_object_info_with_idx(RUSTFS_META_BUCKET, object, &ObjectOptions::default())
.await
.expect("read the new snapshot authority");
assert_ne!(current_owner, usage_owner, "cleanup must publish to a writable pool");
let retained_source = store.pools[usage_owner]
.get_object_info(RUSTFS_META_BUCKET, object, &ObjectOptions::default())
.await
.expect("decommission retains its original source for reconciliation");
assert_eq!(retained_source.etag, old_snapshot.etag);
let stale_update = store
.put_usage_snapshot_for_namespace_cleanup(
object,
old_snapshot.etag.as_deref(),
&mut PutObjReader::from_vec(b"stale snapshot".to_vec()),
None,
)
.await;
assert!(matches!(stale_update, Err(StorageError::PreconditionFailed)));
let usage = crate::data_usage::load_data_usage_from_backend(store.clone())
.await
.expect("a stale source CAS must not replace the writable target");
assert!(!usage.buckets_usage.contains_key(bucket));
assert!(
store.scanner_data_usage_publication_blocked().await,
"scanner publication must remain fenced"
);
}
#[tokio::test]
async fn canceled_create_retains_usage_cleanup_namespace_owner() {
use crate::set_disk::{PutObjectCommitBarrier, PutObjectCommitPause};
let (_dirs, store) = multipool_version_test_store("cleanup-cancellation-existing").await;
let bucket = "cleanup-cancellation-new";
let mut snapshot = rustfs_data_usage::DataUsageInfo {
last_update: Some(std::time::SystemTime::now()),
usage_snapshot_complete: true,
..Default::default()
};
snapshot.buckets_usage.insert(
bucket.to_owned(),
rustfs_data_usage::BucketUsageInfo {
size: 42,
objects_count: 1,
versions_count: 1,
..Default::default()
},
);
snapshot.bucket_sizes.insert(bucket.to_owned(), 42);
snapshot.calculate_totals();
crate::data_usage::store_data_usage_in_backend(snapshot, store.clone())
.await
.expect("seed the stale namespace usage");
let object = crate::data_usage::DATA_USAGE_OBJ_NAME_PATH.as_str();
let (_, source) = store
.get_latest_accessible_object_info_with_idx(RUSTFS_META_BUCKET, object, &ObjectOptions::default())
.await
.expect("find the source snapshot");
let mut pools = vec![prepared_pool_test_status(0, false), prepared_pool_test_status(1, false)];
pools[source].decommission = Some(PoolDecommissionInfo {
canceled: true,
..Default::default()
});
*store.pool_meta.write().await = PoolMeta {
pools,
..Default::default()
};
let barrier = PutObjectCommitBarrier::install(RUSTFS_META_BUCKET, object, PutObjectCommitPause::AfterNamespace);
let create_store = store.clone();
let create = tokio::spawn(async move { create_store.handle_make_bucket(bucket, &MakeBucketOptions::default()).await });
tokio::time::timeout(Duration::from_secs(10), barrier.wait_until_paused())
.await
.expect("cleanup reaches its physical commit with the namespace held");
create.abort();
assert!(create.await.expect_err("the request waiter was canceled").is_cancelled());
let namespace = store
.new_ns_lock(bucket, bucket)
.await
.expect("open the bucket namespace lock");
assert!(
tokio::time::timeout(Duration::from_millis(50), namespace.get_write_lock(Duration::from_secs(1)))
.await
.is_err(),
"cancellation must not release the namespace before its cleanup finishes"
);
barrier.release();
drop(barrier);
tokio::time::timeout(Duration::from_secs(10), store.handle_make_bucket(bucket, &MakeBucketOptions::default()))
.await
.expect("the resumed create must not remain blocked")
.expect("a new request resumes after the old cleanup releases its namespace");
let usage = crate::data_usage::load_data_usage_from_backend(store.clone())
.await
.expect("read the cleaned baseline");
assert!(!usage.buckets_usage.contains_key(bucket));
assert!(store.scanner_data_usage_publication_blocked().await);
}
#[tokio::test]
async fn usage_cleanup_rejects_an_unreadable_pool() {
let (_dirs, store) = multipool_version_test_store("cleanup-unreadable-pool").await;
let snapshot = rustfs_data_usage::DataUsageInfo {
last_update: Some(std::time::SystemTime::now()),
usage_snapshot_complete: true,
..Default::default()
};
crate::data_usage::store_data_usage_in_backend(snapshot, store.clone())
.await
.expect("seed the usage baseline");
let object = crate::data_usage::DATA_USAGE_OBJ_NAME_PATH.as_str();
let (before, source) = store
.get_latest_accessible_object_info_with_idx(RUSTFS_META_BUCKET, object, &ObjectOptions::default())
.await
.expect("find the baseline owner");
let unavailable = 1 - source;
for disk in store.pools[unavailable]
.get_disks_by_key(object)
.disks
.read()
.await
.iter()
.flatten()
{
disk.close().await.expect("stop disk health monitoring");
disk.force_offline_for_test();
}
let update = store
.put_usage_snapshot_for_namespace_cleanup(
object,
before.etag.as_deref(),
&mut PutObjReader::from_vec(b"unproven authority".to_vec()),
None,
)
.await;
assert!(update.is_err(), "an unreadable pool may hold a newer snapshot");
let after = store.pools[source]
.get_object_info(RUSTFS_META_BUCKET, object, &ObjectOptions::default())
.await
.expect("the readable baseline remains intact");
assert_eq!(after.etag, before.etag);
}
#[tokio::test]
async fn usage_cleanup_rejects_namespace_loss_before_physical_commit() {
use crate::set_disk::{PutObjectCommitBarrier, PutObjectCommitPause};
use rustfs_lock::{LocalClient, LockRequest, LockType, NamespaceLock, ObjectKey};
let (_dirs, store) = multipool_version_test_store("cleanup-fence-existing").await;
let snapshot = rustfs_data_usage::DataUsageInfo {
last_update: Some(std::time::SystemTime::now()),
usage_snapshot_complete: true,
..Default::default()
};
crate::data_usage::store_data_usage_in_backend(snapshot, store.clone())
.await
.expect("seed the protected baseline");
let object = crate::data_usage::DATA_USAGE_OBJ_NAME_PATH.as_str();
let (before, _) = store
.get_latest_accessible_object_info_with_idx(RUSTFS_META_BUCKET, object, &ObjectOptions::default())
.await
.expect("read the baseline revision");
let ttl = Duration::from_secs(60);
let namespace = NamespaceLock::new("usage-cleanup-commit-fence".to_owned(), Arc::new(LocalClient::new()));
let request = LockRequest::new(ObjectKey::new("cleanup-fence-existing", ""), LockType::Exclusive, "usage-cleanup")
.with_acquire_timeout(Duration::from_secs(1))
.with_ttl(ttl)
.with_refresh_interval(ttl);
let guard = Arc::new(
namespace
.acquire_guard(&request)
.await
.expect("acquire the namespace lease")
.expect("namespace lease is available"),
);
let barrier = PutObjectCommitBarrier::install(RUSTFS_META_BUCKET, object, PutObjectCommitPause::AfterNamespace);
let update_store = store.clone();
let update_guard = guard.clone();
let revision = before.etag.clone();
let update = tokio::spawn(async move {
let _movement = update_store
.data_usage_namespace_cleanup_guard()
.await
.expect("admit the namespace-only update");
update_store
.put_usage_snapshot_for_namespace_cleanup(
object,
revision.as_deref(),
&mut PutObjReader::from_vec(b"must not commit".to_vec()),
Some(update_guard.as_ref()),
)
.await
});
tokio::time::timeout(Duration::from_secs(10), barrier.wait_until_paused())
.await
.expect("the update reaches its commit barrier");
tokio::time::pause();
tokio::time::advance(ttl + Duration::from_millis(1)).await;
assert!(guard.is_lock_lost(), "the parent namespace lease has expired");
tokio::time::resume();
barrier.release();
assert!(update.await.expect("the cleanup task completes").is_err());
let (after, _) = store
.get_latest_accessible_object_info_with_idx(RUSTFS_META_BUCKET, object, &ObjectOptions::default())
.await
.expect("the original baseline remains readable");
assert_eq!(after.etag, before.etag, "a lost namespace lease must fence the physical write");
}
async fn multipool_delete_test_store(bucket: &str, versioned: bool) -> (Vec<tempfile::TempDir>, Arc<ECStore>) {
let ctx = Arc::new(crate::runtime::instance::InstanceContext::new());
let (mut dirs, first_set) = make_local_set_disks_with_ctx(4, 2, Arc::clone(&ctx)).await;
+48
View File
@@ -0,0 +1,48 @@
# Copyright 2024 RustFS Team
#
# Licensed under the Apache License, Version 2.0 (the "License");
# you may not use this file except in compliance with the License.
# You may obtain a copy of the License at
#
# http://www.apache.org/licenses/LICENSE-2.0
#
# Unless required by applicable law or agreed to in writing, software
# distributed under the License is distributed on an "AS IS" BASIS,
# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
# See the License for the specific language governing permissions and
# limitations under the License.
[package]
name = "rustfs-gateway-adapter"
version.workspace = true
edition.workspace = true
license.workspace = true
repository.workspace = true
rust-version.workspace = true
homepage.workspace = true
description = "RustFS implementations of the rustfs-gateway extension traits (credentials, policy facts, virtual hosts, bucket owner and request id)."
keywords = ["rustfs", "gateway", "s3", "iam"]
categories = ["web-programming", "authentication"]
authors.workspace = true
publish = false
[package.metadata.gateway]
# Documentary only: this crate is a ring-2 consumer of the gateway (it depends on
# RustFS business crates). The RustFS repository does not run the gateway's ring guard.
ring = 2
[lints]
workspace = true
[dependencies]
http = { workspace = true }
rustfs-gateway = { workspace = true }
rustfs-credentials = { workspace = true }
rustfs-iam = { workspace = true }
rustfs-policy = { workspace = true }
serde_json = { workspace = true }
[dev-dependencies]
rustfs-gateway-http = { workspace = true }
time = { workspace = true }
tokio = { workspace = true, features = ["macros", "rt"] }
+215
View File
@@ -0,0 +1,215 @@
// Copyright 2024 RustFS Team
//
// Licensed under the Apache License, Version 2.0 (the "License");
// you may not use this file except in compliance with the License.
// You may obtain a copy of the License at
//
// http://www.apache.org/licenses/LICENSE-2.0
//
// Unless required by applicable law or agreed to in writing, software
// distributed under the License is distributed on an "AS IS" BASIS,
// WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
// See the License for the specific language governing permissions and
// limitations under the License.
//! The gateway's [`BucketOwnerSource`]: the one owner RustFS reports, for a bucket that exists.
//!
//! Responsible for: confirming through a [`BucketLookup`] that a bucket exists, answering the
//! single owner id the deployment reports ([`RustfsBucketOwners::new`]) for it, and turning every
//! outcome that is not a usable owner into the gateway's one refusal, [`BucketOwnerError`].
//! NOT responsible for: choosing the owner id (the binary passes the constant it already reports
//! as the owner in listings), comparing it with `x-amz-expected-bucket-owner` (the gateway does,
//! byte for byte), or authorizing anything.
//! Upstream: the gateway's expected-owner check, which consults this source only when a request
//! carries the header, after authorization allowed it. Downstream: the lookup the binary supplies
//! over the ecstore bucket metadata system.
//!
//! # What an owner is in RustFS
//!
//! RustFS stores no per-bucket owner: bucket ACL writes are no-ops and nothing writes the ACL
//! document, so the only owner RustFS reports anywhere (ListBuckets, ACL and multipart listings)
//! is one constant. The source answers exactly that constant. A caller therefore passes the
//! gateway's check only by presenting the owner id RustFS lists, which separates no tenants; it is
//! a compatibility answer, and the constant's own documentation says it must not decide
//! authorization.
//!
//! # Difference from legacy (ledger entry R-071)
//!
//! Legacy RustFS ignores `x-amz-expected-bucket-owner` entirely. On the gateway stack a request
//! carrying it is served only when the value equals the reported owner id; otherwise it is `403
//! AccessDenied`. A missing bucket and a backend failure with the header present are also `403`
//! at the pinned gateway, where legacy answers `404` and `5xx`: [`BucketOwnerError`] has one value,
//! so it cannot express either. [`RustfsBucketOwners::resolve`] keeps the two apart at this
//! adapter's boundary so the gateway can surface them once it grows the classes.
use std::sync::Arc;
use rustfs_gateway::{BoxFuture, BucketName, BucketOwnerError, BucketOwnerSource};
/// Why a bucket yielded no owner.
#[derive(Clone, Copy, Debug, PartialEq, Eq)]
pub enum OwnerLookupError {
/// The bucket does not exist.
NoSuchBucket,
/// The metadata backend failed, could not be reached, or the owner is not usable (an empty
/// owner id is a misconfiguration), so the bucket's state is unknown.
Backend,
}
/// The narrow slice of RustFS bucket metadata an owner answer needs: whether a bucket exists.
///
/// Exists so the source can be tested against a fake. The binary implements it over the ecstore
/// bucket metadata system, mapping `BucketNotFound` to [`OwnerLookupError::NoSuchBucket`] and every
/// other error to [`OwnerLookupError::Backend`].
pub trait BucketLookup: Send + Sync + 'static {
/// `Ok(())` when `bucket` exists.
fn check<'a>(&'a self, bucket: &'a str) -> BoxFuture<'a, Result<(), OwnerLookupError>>;
}
/// A [`BucketOwnerSource`] answering one owner id for every bucket that exists.
pub struct RustfsBucketOwners<L> {
lookup: L,
owner: Arc<str>,
}
impl<L: BucketLookup> RustfsBucketOwners<L> {
/// A source answering `owner` for every bucket `lookup` finds.
///
/// An empty `owner` is no owner: every answer is then [`OwnerLookupError::Backend`].
#[must_use]
pub fn new(lookup: L, owner: impl Into<Arc<str>>) -> Self {
Self {
lookup,
owner: owner.into(),
}
}
/// The owner of `bucket`, or why there is none. Missing and failing stay distinct here.
///
/// # Errors
///
/// [`OwnerLookupError::NoSuchBucket`] when the bucket does not exist;
/// [`OwnerLookupError::Backend`] on any lookup failure or when the configured owner is empty.
pub async fn resolve(&self, bucket: &str) -> Result<Arc<str>, OwnerLookupError> {
self.lookup.check(bucket).await?;
if self.owner.is_empty() {
return Err(OwnerLookupError::Backend);
}
Ok(Arc::clone(&self.owner))
}
}
impl<L: BucketLookup> BucketOwnerSource for RustfsBucketOwners<L> {
fn owner<'a>(&'a self, bucket: &'a BucketName) -> BoxFuture<'a, Result<Arc<str>, BucketOwnerError>> {
Box::pin(async move {
// The pinned gateway collapses both classes (R-071), so both are the one refusal.
self.resolve(bucket.as_str())
.await
.map_err(|_: OwnerLookupError| BucketOwnerError::unavailable())
})
}
}
#[cfg(test)]
#[allow(clippy::expect_used, clippy::panic, clippy::unwrap_used)]
mod tests {
use super::*;
use std::sync::Mutex;
/// Stands for the owner id RustFS reports; the binary passes its own constant.
const OWNER: &str = "reported-owner-id";
struct Fake {
answer: Result<(), OwnerLookupError>,
asked: Mutex<Vec<String>>,
}
impl Fake {
fn answering(answer: Result<(), OwnerLookupError>) -> Self {
Self {
answer,
asked: Mutex::new(Vec::new()),
}
}
}
impl BucketLookup for Fake {
fn check<'a>(&'a self, bucket: &'a str) -> BoxFuture<'a, Result<(), OwnerLookupError>> {
self.asked.lock().unwrap().push(bucket.to_owned());
let answer = self.answer;
Box::pin(async move { answer })
}
}
fn source(answer: Result<(), OwnerLookupError>) -> RustfsBucketOwners<Fake> {
RustfsBucketOwners::new(Fake::answering(answer), OWNER)
}
async fn gateway_answer(answer: Result<(), OwnerLookupError>) -> Result<Arc<str>, BucketOwnerError> {
source(answer).owner(&BucketName::new("photos").expect("a bucket name")).await
}
/// Positive: an existing bucket's owner is the configured id, byte for byte.
#[tokio::test]
async fn an_existing_bucket_has_the_configured_owner() {
assert_eq!(gateway_answer(Ok(())).await.as_deref(), Ok(OWNER));
}
/// Positive: every bucket gets the same owner, since RustFS stores none per bucket.
#[tokio::test]
async fn every_bucket_has_the_same_owner() {
let source = source(Ok(()));
assert_eq!(source.resolve("photos").await.as_deref(), Ok(OWNER));
assert_eq!(source.resolve("backups").await.as_deref(), Ok(OWNER));
}
/// Positive: the lookup is asked about exactly the bucket the gateway named.
#[tokio::test]
async fn the_lookup_is_asked_about_the_named_bucket() {
let source = source(Ok(()));
source
.owner(&BucketName::new("photos").expect("a bucket name"))
.await
.expect("an owner");
assert_eq!(*source.lookup.asked.lock().unwrap(), vec!["photos".to_owned()]);
}
/// Negative: a missing bucket is reported as missing at the adapter boundary.
#[tokio::test]
async fn a_missing_bucket_is_reported_as_missing() {
assert_eq!(
source(Err(OwnerLookupError::NoSuchBucket)).resolve("photos").await,
Err(OwnerLookupError::NoSuchBucket)
);
}
/// Negative: a failing backend is reported as a failure, never as an owner and never as missing.
#[tokio::test]
async fn a_backend_failure_is_reported_as_a_failure() {
assert_eq!(
source(Err(OwnerLookupError::Backend)).resolve("photos").await,
Err(OwnerLookupError::Backend)
);
}
/// Negative: the pinned gateway sees one refusal for both, and never an owner.
#[tokio::test]
async fn the_gateway_is_refused_in_both_cases() {
assert_eq!(
gateway_answer(Err(OwnerLookupError::NoSuchBucket)).await,
Err(BucketOwnerError::unavailable())
);
assert_eq!(gateway_answer(Err(OwnerLookupError::Backend)).await, Err(BucketOwnerError::unavailable()));
}
/// Negative: an empty configured owner is no owner; it must not equal an empty assertion.
#[tokio::test]
async fn an_empty_owner_is_refused() {
let source = RustfsBucketOwners::new(Fake::answering(Ok(())), "");
assert_eq!(source.resolve("photos").await, Err(OwnerLookupError::Backend));
assert_eq!(
source.owner(&BucketName::new("photos").expect("a bucket name")).await,
Err(BucketOwnerError::unavailable())
);
}
}
+446
View File
@@ -0,0 +1,446 @@
// Copyright 2024 RustFS Team
//
// Licensed under the Apache License, Version 2.0 (the "License");
// you may not use this file except in compliance with the License.
// You may obtain a copy of the License at
//
// http://www.apache.org/licenses/LICENSE-2.0
//
// Unless required by applicable law or agreed to in writing, software
// distributed under the License is distributed on an "AS IS" BASIS,
// WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
// See the License for the specific language governing permissions and
// limitations under the License.
//! The gateway's [`CredentialProvider`], answered by RustFS IAM.
//!
//! Responsible for: resolving one access key id to the secret the gateway verifies a signature
//! with, and classifying every other outcome as either a business result
//! ([`CredentialLookup::NotFound`], eligible for the gateway's negative cache) or an operational
//! fault ([`ProviderError`], counted and never cached).
//! NOT responsible for: authorization of any kind, session-token claim validation, or deciding
//! what the client is answered with. The gateway answers every unusable credential identically.
//!
//! # Secret handling
//!
//! Secrets travel only inside the gateway's [`Credentials`], whose `Debug` redacts them. No type
//! in this module derives or implements `Debug`, and no error produced here carries text: the
//! IAM error is classified and dropped. A secret that fails the crate's effective-length rule
//! (`crate::MIN_SECRET_KEY_LEN` bytes once trailing NULs are removed) is never handed to the
//! gateway, whatever the lookup says about its validity.
use std::sync::Arc;
use rustfs_gateway::{BoxFuture, CredentialLookup, CredentialProvider, Credentials, ProviderError, SessionBinding};
use rustfs_iam::error::Error as IamError;
use rustfs_iam::store::Store;
use rustfs_iam::sys::IamSys;
use rustfs_policy::auth::{ACCOUNT_OFF, UserIdentity};
use crate::root::{RootSource, is_usable_secret};
/// The opaque issuer recorded in the [`rustfs_gateway::SessionBinding`] of every RustFS STS
/// credential.
pub const STS_SESSION_ISSUER: &str = "rustfs-sts";
/// Access keys with this prefix belong to Keystone token authentication, which never verifies an
/// AWS signature.
const KEYSTONE_ACCESS_KEY_PREFIX: &str = "keystone:";
/// The narrow slice of IAM that a credential lookup needs.
///
/// Exists so the adapter can be tested against a fake: IAM exports no in-memory backend.
/// Implemented for [`IamSys`], whose `check_key` loads a user from storage on a cache miss.
pub trait IamKeyLookup: Send + Sync + 'static {
/// Whether the IAM store has finished loading.
fn is_ready(&self) -> bool;
/// Resolves an access key, with [`IamSys::check_key`]'s contract: `Ok((None, _))` for an
/// unknown key, `Ok((Some(identity), valid))` otherwise, where `valid` is false for a
/// disabled, expired, or malformed credential.
fn check_key<'a>(&'a self, access_key: &'a str) -> BoxFuture<'a, Result<(Option<UserIdentity>, bool), IamError>>;
}
impl<T: Store> IamKeyLookup for IamSys<T> {
fn is_ready(&self) -> bool {
IamSys::is_ready(self)
}
fn check_key<'a>(&'a self, access_key: &'a str) -> BoxFuture<'a, Result<(Option<UserIdentity>, bool), IamError>> {
Box::pin(IamSys::check_key(self, access_key))
}
}
/// A [`CredentialProvider`] over RustFS IAM and the live root credentials.
///
/// Built only by [`crate::IamAdapters::new`], so it always reads the same [`RootSource`] as the
/// policy source it was assembled with.
pub struct IamCredentials {
iam: Arc<dyn IamKeyLookup>,
root: Arc<dyn RootSource>,
}
impl IamCredentials {
pub(crate) fn new(iam: Arc<dyn IamKeyLookup>, root: Arc<dyn RootSource>) -> Self {
Self { iam, root }
}
async fn resolve(&self, access_key: &str) -> Result<CredentialLookup, ProviderError> {
// Deliberate change from the legacy s3s path (`rustfs/src/auth.rs` `get_secret_key`),
// which answered a `keystone:` key with an empty secret. Keystone authenticates by token,
// so there is no AWS secret to verify a signature against: the key is unknown here.
if access_key.starts_with(KEYSTONE_ACCESS_KEY_PREFIX) {
return Ok(CredentialLookup::NotFound);
}
// Legacy answers every key with an internal error while the root credentials are not
// initialized; without them no key can be classified as root or not.
let Some(root) = self.root.current() else {
return Err(ProviderError::Unavailable);
};
// Byte-exact, as the trait requires: the access key id is signed material.
if root.access_key() == access_key {
return Ok(CredentialLookup::Found(root.credentials().clone_credentials()));
}
if !self.iam.is_ready() {
return Err(ProviderError::Unavailable);
}
match self.iam.check_key(access_key).await {
// IAM refuses such a secret through `valid`, but this provider does not rely on every
// lookup doing so: a short secret is never usable, so it is not a credential at all.
Ok((Some(identity), _)) if !is_usable_secret(identity.credentials.secret_key.as_bytes()) => {
Ok(CredentialLookup::NotFound)
}
Ok((Some(identity), valid)) => to_gateway_credentials(access_key, &identity, valid).map(CredentialLookup::Found),
Ok((None, _)) => Ok(CredentialLookup::NotFound),
Err(error) => classify_iam_error(&error),
}
}
}
/// Classifies an IAM failure without keeping any of its text.
fn classify_iam_error(error: &IamError) -> Result<CredentialLookup, ProviderError> {
match error {
IamError::NoSuchUser(_)
| IamError::NoSuchServiceAccount(_)
| IamError::NoSuchTempAccount(_)
| IamError::NoSuchAccount(_) => Ok(CredentialLookup::NotFound),
IamError::IamSysNotInitialized => Err(ProviderError::Unavailable),
_ => Err(ProviderError::Backend),
}
}
/// Builds the gateway credential for one IAM identity. No authorization happens here: a
/// disabled or invalid principal still resolves, marked disabled, and the gateway refuses it
/// exactly as it refuses an unknown key.
fn to_gateway_credentials(access_key: &str, identity: &UserIdentity, valid: bool) -> Result<Credentials, ProviderError> {
let cred = &identity.credentials;
let mut credentials = Credentials::new(access_key, cred.secret_key.as_bytes()).map_err(|_| ProviderError::Backend)?;
// A service account carries a claims token internally, but its clients never present it, so
// it authenticates as a long-term credential (as `check_session_token` treats it in legacy).
if !cred.session_token.is_empty() && !cred.is_service_account() {
// A temporary credential without an expiry cannot become a gateway session, which always
// ends. Refusing it as a fault is the fail-closed answer; a never-ending session is not.
let expiration = cred.expiration.ok_or(ProviderError::Backend)?;
let binding = SessionBinding::new(STS_SESSION_ISSUER, expiration.unix_timestamp()).map_err(|_| ProviderError::Backend)?;
credentials = credentials
.with_session(&cred.session_token, binding)
.map_err(|_| ProviderError::Backend)?;
}
if !valid || cred.status == ACCOUNT_OFF {
credentials = credentials.disable();
}
Ok(credentials)
}
impl CredentialProvider for IamCredentials {
fn lookup<'a>(&'a self, access_key_id: &'a str) -> BoxFuture<'a, Result<CredentialLookup, ProviderError>> {
Box::pin(self.resolve(access_key_id))
}
}
#[cfg(test)]
#[allow(clippy::expect_used, clippy::panic, clippy::unwrap_used)]
mod tests {
use super::*;
use crate::root::{RootCell, RootCredentials};
use rustfs_credentials::Credentials as IamCredential;
use rustfs_gateway::RequestNow;
use std::sync::atomic::{AtomicUsize, Ordering};
use time::OffsetDateTime;
const ACCESS_KEY: &str = "AKIDUSER0001";
const SECRET: &str = "user-secret-key-0001";
type Answer = Box<dyn Fn() -> Result<(Option<UserIdentity>, bool), IamError> + Send + Sync>;
struct FakeIam {
ready: bool,
answer: Answer,
calls: AtomicUsize,
}
impl IamKeyLookup for FakeIam {
fn is_ready(&self) -> bool {
self.ready
}
fn check_key<'a>(&'a self, _access_key: &'a str) -> BoxFuture<'a, Result<(Option<UserIdentity>, bool), IamError>> {
self.calls.fetch_add(1, Ordering::SeqCst);
let answer = (self.answer)();
Box::pin(async move { answer })
}
}
fn fake(
ready: bool,
answer: impl Fn() -> Result<(Option<UserIdentity>, bool), IamError> + Send + Sync + 'static,
) -> Arc<FakeIam> {
Arc::new(FakeIam {
ready,
answer: Box::new(answer),
calls: AtomicUsize::new(0),
})
}
fn user(status: &str) -> IamCredential {
IamCredential {
access_key: ACCESS_KEY.to_owned(),
secret_key: SECRET.to_owned(),
status: status.to_owned(),
..Default::default()
}
}
fn found(cred: IamCredential, valid: bool) -> Result<(Option<UserIdentity>, bool), IamError> {
Ok((Some(UserIdentity::new(cred)), valid))
}
fn root_cell() -> Arc<RootCell> {
let root = RootCredentials::new("rootadmin", "root-secret-key", "sts-signing-key").expect("valid root credentials");
Arc::new(RootCell::new(Some(root)))
}
async fn lookup(iam: Arc<FakeIam>, access_key: &str) -> Result<CredentialLookup, ProviderError> {
IamCredentials::new(iam, root_cell()).lookup(access_key).await
}
fn expect_found(result: Result<CredentialLookup, ProviderError>) -> Credentials {
match result {
Ok(CredentialLookup::Found(credentials)) => credentials,
Ok(CredentialLookup::NotFound) => panic!("expected Found, got NotFound"),
Err(error) => panic!("expected Found, got {error:?}"),
}
}
fn is_not_found(result: &Result<CredentialLookup, ProviderError>) -> bool {
matches!(result, Ok(CredentialLookup::NotFound))
}
/// Positive — an enabled user resolves to exactly its secret, as a long-term credential.
#[tokio::test]
async fn credentials_found_user_resolves_to_its_secret() {
let iam = fake(true, || found(user("on"), true));
let credentials = expect_found(lookup(iam, ACCESS_KEY).await);
assert_eq!(credentials.secret().expose(), SECRET.as_bytes());
assert_eq!(credentials.identity().access_key_id(), ACCESS_KEY);
assert!(!credentials.is_disabled());
assert!(credentials.session_binding().is_none());
}
/// Negative — a key IAM does not know is a business result, not a fault.
#[tokio::test]
async fn credentials_unknown_key_is_not_found() {
let iam = fake(true, || Ok((None, false)));
assert!(is_not_found(&lookup(iam, "AKIDUNKNOWN").await));
}
/// Negative — every "no such principal" error IAM can surface is `NotFound`. Answering these
/// as `Backend` would keep them out of the negative cache and count every guessed key as a
/// backend fault.
#[tokio::test]
async fn credentials_no_such_principal_errors_are_not_found() {
let errors: [fn() -> IamError; 4] = [
|| IamError::NoSuchUser(ACCESS_KEY.to_owned()),
|| IamError::NoSuchServiceAccount(ACCESS_KEY.to_owned()),
|| IamError::NoSuchTempAccount(ACCESS_KEY.to_owned()),
|| IamError::NoSuchAccount(ACCESS_KEY.to_owned()),
];
for error in errors {
let iam = fake(true, move || Err(error()));
let result = lookup(iam, ACCESS_KEY).await;
assert!(is_not_found(&result), "{:?} must be NotFound", error());
}
}
/// Negative — any other IAM failure is a backend fault, never `NotFound` and never `Found`.
#[tokio::test]
async fn credentials_backend_error_is_backend() {
let iam = fake(true, || Err(IamError::StringError("storage read failed".to_owned())));
assert!(matches!(lookup(iam, ACCESS_KEY).await, Err(ProviderError::Backend)));
}
/// Negative — a `keystone:` key never reaches IAM and never resolves to a secret.
///
/// Deliberate change from the legacy s3s path, which answered such a key with an empty
/// secret: Keystone authenticates by token, so there is no AWS secret to verify against.
#[tokio::test]
async fn credentials_keystone_prefix_is_not_found_without_iam() {
let iam = fake(true, || found(user("on"), true));
let result = lookup(iam.clone(), "keystone:project-user").await;
assert!(is_not_found(&result));
assert_eq!(iam.calls.load(Ordering::SeqCst), 0);
}
/// Negative — a switched-off or otherwise invalid user resolves, but disabled, and the same
/// user enabled does not.
#[tokio::test]
async fn credentials_disabled_user_is_disabled() {
let off = expect_found(lookup(fake(true, || found(user("off"), false)), ACCESS_KEY).await);
assert!(off.is_disabled());
assert!(off.admit(None, RequestNow::from_unix_seconds(0)).is_err());
let invalid = expect_found(lookup(fake(true, || found(user("on"), false)), ACCESS_KEY).await);
assert!(invalid.is_disabled());
let on = expect_found(lookup(fake(true, || found(user("on"), true)), ACCESS_KEY).await);
assert!(!on.is_disabled());
}
/// Positive — an STS credential carries its token and its lifetime as one session binding.
#[tokio::test]
async fn credentials_sts_carries_session_binding() {
let expires_at = 4_000_000_000_i64;
let iam = fake(true, move || {
let mut cred = user("on");
cred.session_token = "sts-session-token".to_owned();
cred.parent_user = "parent".to_owned();
cred.expiration = Some(OffsetDateTime::from_unix_timestamp(expires_at).expect("valid timestamp"));
found(cred, true)
});
let credentials = expect_found(lookup(iam, ACCESS_KEY).await);
let binding = credentials.session_binding().expect("an STS credential has a session");
assert_eq!(binding.expires_at_unix_seconds(), expires_at);
assert_eq!(binding.issuer(), STS_SESSION_ISSUER);
let now = RequestNow::from_unix_seconds(expires_at - 1);
assert!(credentials.admit(Some(b"sts-session-token"), now).is_ok());
assert!(credentials.admit(None, now).is_err());
}
/// Negative — an STS credential with no expiry cannot be expressed as a session the gateway
/// would ever expire, so it is a backend fault rather than a session that never ends.
#[tokio::test]
async fn credentials_sts_without_expiry_is_backend() {
let iam = fake(true, || {
let mut cred = user("on");
cred.session_token = "sts-session-token".to_owned();
found(cred, true)
});
assert!(matches!(lookup(iam, ACCESS_KEY).await, Err(ProviderError::Backend)));
}
/// Positive — a service account carries a token internally but authenticates as a long-term
/// credential: clients never present that token.
#[tokio::test]
async fn credentials_service_account_is_long_term() {
let iam = fake(true, || {
let mut cred = user("on");
cred.session_token = "service-account-claims-token".to_owned();
cred.parent_user = "parent".to_owned();
cred.claims = Some([(rustfs_credentials::IAM_POLICY_CLAIM_NAME_SA.to_owned(), "inherited-policy".into())].into());
found(cred, true)
});
let credentials = expect_found(lookup(iam, ACCESS_KEY).await);
assert!(credentials.session_binding().is_none());
assert!(credentials.admit(None, RequestNow::from_unix_seconds(0)).is_ok());
}
/// Negative — before IAM is ready a lookup is `Unavailable`, not `NotFound`: a key must not
/// be negatively cached because the store had not loaded yet.
#[tokio::test]
async fn credentials_iam_not_ready_is_unavailable() {
let iam = fake(false, || found(user("on"), true));
assert!(matches!(lookup(iam.clone(), ACCESS_KEY).await, Err(ProviderError::Unavailable)));
assert_eq!(iam.calls.load(Ordering::SeqCst), 0);
let uninitialized = fake(true, || Err(IamError::IamSysNotInitialized));
assert!(matches!(lookup(uninitialized, ACCESS_KEY).await, Err(ProviderError::Unavailable)));
}
/// Positive — the injected root credentials resolve without IAM, even before it is ready, and
/// nothing else is answered with the root secret.
#[tokio::test]
async fn credentials_root_resolves_without_iam() {
let iam = fake(false, || found(user("on"), true));
let provider = IamCredentials::new(iam.clone(), root_cell());
let root = expect_found(provider.lookup("rootadmin").await);
assert_eq!(root.secret().expose(), b"root-secret-key");
assert!(matches!(provider.lookup("ROOTADMIN").await, Err(ProviderError::Unavailable)));
assert_eq!(iam.calls.load(Ordering::SeqCst), 0);
}
/// Negative — a lookup that calls an identity valid but holds an empty or short secret is not
/// trusted: such a secret is not a credential, so the key is `NotFound`, never `Found`.
#[tokio::test]
async fn credentials_short_secret_is_never_found() {
for weak in ["", "1234567", "\0\0\0\0\0\0\0\0", "abc\0\0\0\0\0"] {
let iam = fake(true, move || {
let mut cred = user("on");
cred.secret_key = weak.to_owned();
found(cred, true)
});
let result = lookup(iam, ACCESS_KEY).await;
assert!(is_not_found(&result), "{weak:?} must be NotFound");
}
for usable in ["12345678", "abc\0defg"] {
let iam = fake(true, move || {
let mut cred = user("on");
cred.secret_key = usable.to_owned();
found(cred, true)
});
let credentials = expect_found(lookup(iam, ACCESS_KEY).await);
assert_eq!(credentials.secret().expose(), usable.as_bytes());
}
}
/// Negative — a switched-off account is disabled even when the lookup calls it valid.
#[tokio::test]
async fn credentials_off_status_alone_disables() {
let off = expect_found(lookup(fake(true, || found(user("off"), true)), ACCESS_KEY).await);
assert!(off.is_disabled());
assert!(off.admit(None, RequestNow::from_unix_seconds(0)).is_err());
}
/// Negative — an STS credential admits only its own session's token: the token of another
/// session, or none, is refused.
#[tokio::test]
async fn credentials_sts_refuses_another_sessions_token() {
let expires_at = 4_000_000_000_i64;
let iam = fake(true, move || {
let mut cred = user("on");
cred.session_token = "session-token-a".to_owned();
cred.parent_user = "parent".to_owned();
cred.expiration = Some(OffsetDateTime::from_unix_timestamp(expires_at).expect("valid timestamp"));
found(cred, true)
});
let credentials = expect_found(lookup(iam, ACCESS_KEY).await);
let now = RequestNow::from_unix_seconds(expires_at - 1);
assert!(credentials.admit(Some(b"session-token-b"), now).is_err());
assert!(credentials.admit(Some(b"session-token-a"), now).is_ok());
}
/// Negative — without root credentials no key can be classified, so every lookup is
/// `Unavailable` and IAM is not asked.
#[tokio::test]
async fn credentials_without_root_are_unavailable() {
let iam = fake(true, || found(user("on"), true));
let provider = IamCredentials::new(iam.clone(), Arc::new(RootCell::new(None)));
assert!(matches!(provider.lookup(ACCESS_KEY).await, Err(ProviderError::Unavailable)));
assert!(matches!(provider.lookup("rootadmin").await, Err(ProviderError::Unavailable)));
assert_eq!(iam.calls.load(Ordering::SeqCst), 0);
}
}
+306
View File
@@ -0,0 +1,306 @@
// Copyright 2024 RustFS Team
//
// Licensed under the Apache License, Version 2.0 (the "License");
// you may not use this file except in compliance with the License.
// You may obtain a copy of the License at
//
// http://www.apache.org/licenses/LICENSE-2.0
//
// Unless required by applicable law or agreed to in writing, software
// distributed under the License is distributed on an "AS IS" BASIS,
// WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
// See the License for the specific language governing permissions and
// limitations under the License.
//! The gateway's [`HostResolver`], built from RustFS's configured domain list.
//!
//! Responsible for: turning the `RUSTFS_SERVER_DOMAINS` list into the gateway's legacy RustFS
//! virtual-host resolver.
//! NOT responsible for: parsing a `Host` header, matching a domain, validating bucket names (all
//! done by [`LegacyRustfsVirtualHosts`]), reading the environment, or deciding that a console
//! deployment serves no domains (the caller passes an empty list then, as `s3_host_domains` in
//! `rustfs/src/server/http.rs` returns `None`). An empty list needs no branch here:
//! [`LegacyRustfsVirtualHosts`] built from no domain reads every host path-style, exactly as the
//! gateway's `PathStyleOnly` does.
//! Upstream: the RustFS configuration (`Config::server_domains`). Downstream: the gateway's
//! `ServiceBuilder`, which holds the resolver as `Arc<dyn HostResolver>`.
//!
//! # What "the same as `MultiDomain`" means
//!
//! RustFS today hands the same list to the s3s framework's multi-domain host, which ignores ports
//! on both sides when matching a domain, takes the whole prefix in front of a matched domain as
//! the bucket, compares a domain exactly (so a host is matched on the case it arrives in), and,
//! for a host no domain matches, falls back to CNAME-style: a host that is itself a bucket name
//! (lower-cased) is that bucket. [`LegacyRustfsVirtualHosts`] reproduces exactly that.
//!
//! What reads **path-style**: an empty domain list; a configured domain itself, with or without a
//! port; an IP address, or a socket address (`10.0.0.1`, `[::1]:9000`); a host that is a valid
//! domain but, lower-cased, is no bucket name (it carries a port, is over 63 bytes, or starts
//! with `xn--`). What reads as a **bucket**: anything under a configured domain, and any other
//! bucket-shaped host, which is a bucket of its own (`localhost`, `badexample.com`, `EXAMPLE.COM`).
//! What is **refused** before routing: a host that is no domain (`under_score.org`, a bare
//! `[::1]`, a trailing dot), and a bucket under a domain that the bucket rules refuse
//! (`PHOTOS.example.com`, `ab.example.com`). The tests below pin each of those readings.
use std::sync::Arc;
use rustfs_gateway::{HostResolver, LegacyDomainError, LegacyRustfsVirtualHosts};
/// Builds the host resolver for RustFS's configured domain list.
///
/// An empty list reads every host path-style. Otherwise the list is read as legacy RustFS reads
/// it: a later entry whose name without its port repeats an earlier one is dropped, and a list
/// that legacy RustFS refuses at start-up is refused here.
///
/// # Errors
///
/// [`LegacyDomainError`] for an entry that is not a domain, or for two entries where one equals or
/// is a subdomain of the other.
pub fn rustfs_host_resolver<I, S>(domains: I) -> Result<Arc<dyn HostResolver>, LegacyDomainError>
where
I: IntoIterator<Item = S>,
S: AsRef<str>,
{
Ok(Arc::new(LegacyRustfsVirtualHosts::new(domains)?))
}
#[cfg(test)]
#[allow(clippy::expect_used, clippy::panic, clippy::unwrap_used)]
mod tests {
use super::*;
use http::Method;
use rustfs_gateway::{Addressing, HostQuery, HostRefusal, TargetKind};
/// What the resolver reads `GET path` with `Host: host` as: the host-named bucket (`None` for
/// path style), the routed target, and the refusal, if any.
fn read(resolver: &dyn HostResolver, path: &str, host: &str) -> (Option<String>, TargetKind, Option<HostRefusal>) {
let method = Method::GET;
let request = http::Request::builder()
.method(method.clone())
.uri(path)
.header("host", host)
.body(())
.expect("a valid request");
let effective = rustfs_gateway_http::effective_host(&request).expect("an acceptable host");
let query = HostQuery {
host: &effective,
path,
method: &method,
};
let resolved = resolver.resolve(&query);
let bucket = match &resolved.addressing {
Addressing::VirtualHosted { bucket, .. } => Some(bucket.as_str().to_owned()),
Addressing::Path => None,
};
(bucket, resolved.target, resolver.refusal(&query))
}
fn with_domains(domains: &[&str]) -> Arc<dyn HostResolver> {
rustfs_host_resolver(domains).expect("a list RustFS accepts")
}
fn bucket(domains: &[&str], host: &str) -> Option<String> {
read(&*with_domains(domains), "/key", host).0
}
/// Positive: a host under a configured domain names the bucket in front of it.
#[test]
fn host_under_domain_names_the_bucket() {
assert_eq!(
read(&*with_domains(&["example.com"]), "/key", "photos.example.com"),
(Some("photos".to_owned()), TargetKind::Object, None)
);
assert_eq!(read(&*with_domains(&["example.com"]), "/", "photos.example.com").1, TargetKind::Bucket);
}
/// Positive: the port on the host, or on the configured domain, does not take part in matching.
#[test]
fn ports_are_ignored_on_both_sides() {
assert_eq!(bucket(&["example.com"], "photos.example.com:9000").as_deref(), Some("photos"));
assert_eq!(bucket(&["example.com:9000"], "photos.example.com").as_deref(), Some("photos"));
assert_eq!(bucket(&["example.com:9000"], "photos.example.com:1234").as_deref(), Some("photos"));
}
/// Positive: the whole prefix in front of the domain is the bucket, dots kept.
#[test]
fn whole_prefix_is_the_bucket() {
assert_eq!(bucket(&["example.com"], "my.dotted.bkt.example.com").as_deref(), Some("my.dotted.bkt"));
}
/// Positive: with several domains, each one resolves its own subdomains.
#[test]
fn every_configured_domain_resolves() {
let domains = ["example.com", "s3.local"];
assert_eq!(bucket(&domains, "abc.example.com").as_deref(), Some("abc"));
assert_eq!(bucket(&domains, "xyz.s3.local").as_deref(), Some("xyz"));
}
/// Positive: the domain itself is the endpoint, so the bucket comes from the path.
#[test]
fn the_domain_itself_is_path_style() {
assert_eq!(bucket(&["example.com"], "example.com"), None);
assert_eq!(bucket(&["example.com"], "example.com:9000"), None);
}
/// Negative: no configured domain means path style only, for every host shape.
#[test]
fn no_domain_is_path_style_only() {
for host in ["photos.example.com", "localhost", "localhost:9000", "10.0.0.1"] {
assert_eq!(bucket(&[], host), None, "{host}");
}
}
/// Negative: an address, and a host carrying a port that no domain matches, are path style.
#[test]
fn addresses_and_unmatched_ports_are_path_style() {
for host in ["10.0.0.1", "10.0.0.1:9000", "localhost:9000", "other.org:9000"] {
assert_eq!(bucket(&["example.com"], host), None, "{host}");
}
}
/// Negative: a host that only ends in the same letters as a domain is not under it. It is not
/// stripped to `bad`; the only reading it gets is the CNAME-style one, the whole host.
#[test]
fn suffix_lookalike_is_not_under_the_domain() {
assert_eq!(bucket(&["example.com"], "badexample.com").as_deref(), Some("badexample.com"));
assert_eq!(bucket(&["example.com"], "example.com.evil.org").as_deref(), Some("example.com.evil.org"));
}
/// Negative: matching a configured domain is exact on case, as the legacy multi-domain host does it, so a
/// mixed-case host is not stripped to its prefix; it is the lower-cased whole host.
#[test]
fn mixed_case_host_is_not_stripped_to_a_prefix() {
assert_eq!(bucket(&["example.com"], "Photos.Example.COM").as_deref(), Some("photos.example.com"));
}
/// Negative: a host that is no domain at all is refused, not routed.
#[test]
fn a_host_that_is_no_domain_is_refused() {
let resolver = with_domains(&["example.com"]);
assert_eq!(read(&*resolver, "/k", "under_score.org").2, Some(HostRefusal::UnusableHost));
}
/// Negative: a bucket the host names that bucket rules refuse is refused.
#[test]
fn a_host_named_bucket_the_rules_refuse_is_refused() {
let resolver = with_domains(&["example.com"]);
assert_eq!(read(&*resolver, "/k", "ab.example.com").2, Some(HostRefusal::RefusedBucket));
assert_eq!(read(&*resolver, "/k", "bad_host.example.com").2, Some(HostRefusal::RefusedBucket));
}
/// Negative: a list legacy RustFS refuses at start-up is refused.
#[test]
fn domain_lists_legacy_rustfs_refuses_are_refused() {
for list in [
vec!["example.com", "a.example.com"],
vec!["example.com", "com"],
vec!["bad_domain"],
vec![""],
vec!["example.com:notaport"],
] {
assert!(rustfs_host_resolver(&list).is_err(), "{list:?}");
}
}
/// Positive: a later repeat of a name, whatever its port, is dropped rather than refused, as
/// `s3_host_domains` drops it before the legacy multi-domain host sees the list.
#[test]
fn a_repeat_of_a_domain_is_dropped() {
assert_eq!(
bucket(&["example.com:9411", "example.com:9000"], "xyz.example.com").as_deref(),
Some("xyz")
);
}
/// The refusal alone, for hosts whose bucket reading is not the point.
fn refusal(domains: &[&str], host: &str) -> Option<HostRefusal> {
read(&*with_domains(domains), "/k", host).2
}
/// Positive: a bucket-shaped host no domain matches is a bucket of its own, `localhost` and
/// an upper-case spelling of the domain among them.
#[test]
fn bucket_shaped_hosts_outside_every_domain_are_their_own_bucket() {
assert_eq!(bucket(&["example.com"], "localhost").as_deref(), Some("localhost"));
assert_eq!(bucket(&["example.com"], "EXAMPLE.COM").as_deref(), Some("example.com"));
assert_eq!(bucket(&["example.com"], "cdn.other.org").as_deref(), Some("cdn.other.org"));
}
/// Negative: a bucket the rules refuse under a domain is refused, and reads no bucket.
#[test]
fn an_upper_case_bucket_under_the_domain_is_refused() {
assert_eq!(bucket(&["example.com"], "PHOTOS.example.com"), None);
assert_eq!(refusal(&["example.com"], "PHOTOS.example.com"), Some(HostRefusal::RefusedBucket));
}
/// Negative: an IPv6 literal. The bare form is no domain and is refused; with a port it is a
/// socket address and reads path-style.
#[test]
fn ipv6_literals_follow_the_address_rules() {
assert_eq!(refusal(&["example.com"], "[::1]"), Some(HostRefusal::UnusableHost));
assert_eq!(bucket(&["example.com"], "[::1]:9000"), None);
assert_eq!(refusal(&["example.com"], "[::1]:9000"), None);
}
/// Negative: a trailing root dot makes the host no domain, so it is refused, not stripped.
#[test]
fn a_trailing_dot_is_refused() {
assert_eq!(refusal(&["example.com"], "photos.example.com."), Some(HostRefusal::UnusableHost));
assert_eq!(bucket(&["example.com"], "photos.example.com."), None);
}
/// Negative: an empty label never reaches the resolver; the gateway's wire layer refuses the
/// host first (legacy answers 400 as well).
#[test]
fn an_empty_label_is_refused_before_resolution() {
for host in ["a..b.example.com", ".example.com"] {
let request = http::Request::builder()
.uri("/k")
.header("host", host)
.body(())
.expect("a valid request");
assert!(rustfs_gateway_http::effective_host(&request).is_err(), "{host}");
}
}
/// Negative: a punycode label is no bucket under a domain; outside every domain the host is
/// a valid domain that is no bucket name, so it reads path-style.
#[test]
fn a_punycode_label_is_never_a_bucket() {
assert_eq!(refusal(&["example.com"], "xn--bcher-kva.example.com"), Some(HostRefusal::RefusedBucket));
assert_eq!(bucket(&["example.com"], "xn--bcher-kva.de"), None);
assert_eq!(refusal(&["example.com"], "xn--bcher-kva.de"), None);
}
/// Negative: a label over 63 bytes is no bucket, under a domain or outside one.
#[test]
fn a_label_over_63_bytes_is_never_a_bucket() {
let long = "a".repeat(64);
let under = format!("{long}.example.com");
assert_eq!(refusal(&["example.com"], &under), Some(HostRefusal::RefusedBucket));
let outside = format!("{long}.org");
assert_eq!(bucket(&["example.com"], &outside), None);
assert_eq!(refusal(&["example.com"], &outside), None);
let at_limit = "a".repeat(63);
assert_eq!(bucket(&["example.com"], &format!("{at_limit}.example.com")), Some(at_limit));
}
/// Positive and negative: two configured domains where one merely ends in the other's letters
/// both work, and each domain itself is path-style.
#[test]
fn a_domain_and_its_suffix_lookalike_are_both_served() {
let domains = ["example.com", "badexample.com"];
assert_eq!(bucket(&domains, "photos.example.com").as_deref(), Some("photos"));
assert_eq!(bucket(&domains, "photos.badexample.com").as_deref(), Some("photos"));
assert_eq!(bucket(&domains, "badexample.com"), None);
assert_eq!(bucket(&domains, "bad.example.com").as_deref(), Some("bad"));
}
/// Negative: a configured domain is matched exactly, case included.
#[test]
fn a_configured_domain_is_matched_on_its_case() {
assert_eq!(bucket(&["Example.COM"], "photos.Example.COM").as_deref(), Some("photos"));
assert_eq!(bucket(&["Example.COM"], "photos.example.com").as_deref(), Some("photos.example.com"));
assert_eq!(bucket(&["Example.COM"], "Example.COM"), None);
}
}
+105
View File
@@ -0,0 +1,105 @@
// Copyright 2024 RustFS Team
//
// Licensed under the Apache License, Version 2.0 (the "License");
// you may not use this file except in compliance with the License.
// You may obtain a copy of the License at
//
// http://www.apache.org/licenses/LICENSE-2.0
//
// Unless required by applicable law or agreed to in writing, software
// distributed under the License is distributed on an "AS IS" BASIS,
// WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
// See the License for the specific language governing permissions and
// limitations under the License.
//! RustFS implementations of the `rustfs-gateway` extension traits.
//!
//! This is the gateway's ring 2: it may depend on RustFS business crates, and nothing in the
//! gateway depends on it. Each module adapts one extension point to the RustFS subsystem that
//! owns the answer.
pub mod bucket_owner;
pub mod credentials;
pub mod host;
pub mod policy;
pub mod root;
pub mod trace;
use std::sync::Arc;
use rustfs_gateway::{Authenticator, CredentialProvider, ServiceBuilder};
pub use bucket_owner::{BucketLookup, OwnerLookupError, RustfsBucketOwners};
pub use credentials::{IamCredentials, IamKeyLookup, STS_SESSION_ISSUER};
pub use host::rustfs_host_resolver;
pub use policy::{IamPolicySource, IamPrincipalLookup, PrincipalKind, RustfsPrincipal};
pub use root::{MIN_SECRET_KEY_LEN, RootCredentials, RootCredentialsError, RootSource};
pub use trace::{AttachOutcome, RequestIdCarrier, attach_host_request_id, host_request_id};
/// The IAM-backed credential provider and policy source, assembled together.
///
/// Both halves read the same IAM handle and the same live [`RootSource`], so a key cannot be root
/// to one and an ordinary user to the other. Neither half can be taken out on its own: the fields
/// are private, and [`IamAdapters::install`] consumes the pair and hands both to one builder.
///
/// Mixing the halves of two assemblies does not compile:
///
/// ```compile_fail
/// fn mix(a: rustfs_gateway_adapter::IamAdapters, b: rustfs_gateway_adapter::IamAdapters) {
/// let _mixed = (a.credentials, b.policy);
/// }
/// ```
///
/// Nor does building a half outside the pair:
///
/// ```compile_fail
/// use std::sync::Arc;
/// use rustfs_gateway_adapter::{IamPolicySource, IamPrincipalLookup, RootSource};
/// fn build(iam: Arc<dyn IamPrincipalLookup>, root: Arc<dyn RootSource>) {
/// let _alone = IamPolicySource::new(iam, root);
/// }
/// ```
///
/// Installing the pair is the one supported path:
///
/// ```no_run
/// use std::sync::Arc;
/// use rustfs_gateway::{RegionSet, ServiceBuilder, SigV4Authenticator};
/// use rustfs_gateway_adapter::{IamAdapters, IamPrincipalLookup, RootSource};
/// fn assemble(iam: Arc<dyn IamPrincipalLookup>, root: Arc<dyn RootSource>, regions: RegionSet) -> ServiceBuilder {
/// IamAdapters::new(iam, root).install(ServiceBuilder::new(), |provider| SigV4Authenticator::new(provider, regions))
/// }
/// ```
pub struct IamAdapters {
credentials: IamCredentials,
policy: IamPolicySource,
}
impl IamAdapters {
/// Assembles both adapters over one IAM handle and one root source. The binary passes a
/// source that reads its live root credentials on every call (see [`RootSource`]).
#[must_use]
pub fn new(iam: Arc<dyn IamPrincipalLookup>, root: Arc<dyn RootSource>) -> Self {
Self {
credentials: IamCredentials::new(iam.clone(), root.clone()),
policy: IamPolicySource::new(iam, root),
}
}
/// Installs both halves into `builder`: the credential provider through the authenticator
/// `authenticator` builds over it (for example `SigV4Authenticator::new(provider, regions)`
/// with the RustFS profile applied), and the policy source.
///
/// Consumes the pair and gives both halves to this one builder. The `authenticator` closure
/// receives the provider handle and may keep a clone of it; nothing here prevents that, so
/// the closure must not hand it to a second builder.
#[must_use]
pub fn install<A: Authenticator>(
self,
builder: ServiceBuilder,
authenticator: impl FnOnce(Arc<dyn CredentialProvider>) -> A,
) -> ServiceBuilder {
let authenticator = authenticator(Arc::new(self.credentials));
builder.authenticator(authenticator).policy_source(self.policy)
}
}
+779
View File
@@ -0,0 +1,779 @@
// Copyright 2024 RustFS Team
//
// Licensed under the Apache License, Version 2.0 (the "License");
// you may not use this file except in compliance with the License.
// You may obtain a copy of the License at
//
// http://www.apache.org/licenses/LICENSE-2.0
//
// Unless required by applicable law or agreed to in writing, software
// distributed under the License is distributed on an "AS IS" BASIS,
// WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
// See the License for the specific language governing permissions and
// limitations under the License.
//! The gateway's [`PolicySource`], answered by RustFS IAM.
//!
//! Responsible for: the one reading of the caller's IAM facts a request is judged against — which
//! principal kind it is, the user it acts for, its groups, its token claims, and whether it owns
//! the deployment — packed as a [`RustfsPrincipal`] into the gateway's [`PolicySnapshot`].
//! NOT responsible for: evaluating any policy, choosing which policy documents apply (that stays
//! in `IamSys::is_allowed`, which owns the STS, OIDC, session-policy and service-account rules), or
//! verifying the signature or the session token (the gateway did both before calling this).
//! Upstream: the gateway's verified [`Identity`]. Downstream: the RustFS authorizer, which reads
//! the payload back with `PolicySnapshot::get::<RustfsPrincipal>()`.
//!
//! # Failure classes
//!
//! Every refusal is a [`PolicyError`] with a constant label and nothing from IAM or the request:
//! IAM not ready, not initialized, or failing, or root credentials not initialized, is
//! [`PolicyError::unavailable`]; a principal IAM no longer knows is [`UNKNOWN_PRINCIPAL`]; a
//! disabled one, or one whose secret fails the effective-length rule, is
//! [`DISABLED_PRINCIPAL`]; an identity
//! whose credential kind disagrees with IAM's record is [`SESSION_MISMATCH`]; a token whose claims
//! cannot be read is [`PolicyError::malformed`]. None of them is ever an empty snapshot: an empty
//! snapshot is what an anonymous caller gets, and answering a fault with it would judge an
//! authenticated caller as anonymous.
//!
//! # Secret handling
//!
//! [`RustfsPrincipal`] holds no secret key and no session token, and implements no `Debug`, so a
//! snapshot formatted into a log cannot print its claims either. The root credentials and the STS
//! token-signing key are read per request from the shared [`RootSource`] and never stored here.
use std::collections::HashMap;
use std::sync::Arc;
use rustfs_credentials::Credentials as IamCredential;
use rustfs_gateway::{BoxFuture, Identity, PolicyError, PolicySnapshot, PolicySource};
use rustfs_iam::error::Error as IamError;
use rustfs_iam::store::Store;
use rustfs_iam::sys::{
IamSys, SESSION_POLICY_NAME, get_claims_from_token_with_secret, get_claims_from_token_with_secret_allow_missing_exp,
is_rustfs_oidc_claims,
};
use rustfs_policy::auth::ACCOUNT_OFF;
use serde_json::Value;
use crate::credentials::IamKeyLookup;
use crate::root::{RootCredentials, RootSource, is_usable_secret};
/// [`PolicyError`] label: IAM does not know the authenticated principal (any longer).
pub const UNKNOWN_PRINCIPAL: &str = "unknown-principal";
/// [`PolicyError`] label: IAM holds the principal disabled, expired, or otherwise invalid.
pub const DISABLED_PRINCIPAL: &str = "disabled-principal";
/// [`PolicyError`] label: the identity's credential kind (session or long-term) disagrees with
/// what IAM holds for its access key.
pub const SESSION_MISMATCH: &str = "session-mismatch";
/// The narrow slice of IAM a policy reading needs, on top of [`IamKeyLookup`].
///
/// Implemented for [`IamSys`]; exists so the source can be tested against a fake.
pub trait IamPrincipalLookup: IamKeyLookup {
/// The built-in IAM groups `user` belongs to now, as `IamSys::user_group_memberships`
/// answers. These may include groups granted after a credential was issued.
fn group_memberships(&self, user: &str) -> Vec<String>;
}
impl<T: Store> IamPrincipalLookup for IamSys<T> {
fn group_memberships(&self, user: &str) -> Vec<String> {
IamSys::user_group_memberships(self, user)
}
}
/// Which kind of credential the principal authenticated with.
#[derive(Clone, Copy, PartialEq, Eq)]
pub enum PrincipalKind {
/// The deployment's root credentials.
Root,
/// A long-term IAM user.
User,
/// A service account, acting for its parent user.
ServiceAccount,
/// An STS temporary credential, acting for its parent user.
Temporary,
}
/// The caller's IAM facts: the payload of every [`PolicySnapshot`] [`IamPolicySource`] produces.
///
/// Carries no secret key and no session token, and deliberately implements no `Debug`.
pub struct RustfsPrincipal {
kind: PrincipalKind,
account: String,
username: String,
parent_user: String,
credential_groups: Option<Vec<String>>,
group_memberships: Vec<String>,
claims: Option<HashMap<String, Value>>,
is_owner: bool,
}
impl RustfsPrincipal {
/// The kind of credential the caller authenticated with.
#[must_use]
pub fn kind(&self) -> PrincipalKind {
self.kind
}
/// The access key the caller signed with (IAM `Args::account`).
#[must_use]
pub fn account(&self) -> &str {
&self.account
}
/// The user the caller acts as: the parent user for an STS credential or a service account,
/// the access key otherwise.
#[must_use]
pub fn username(&self) -> &str {
&self.username
}
/// The parent user IAM records for the credential; empty for root and long-term users.
#[must_use]
pub fn parent_user(&self) -> &str {
&self.parent_user
}
/// The groups recorded on the credential itself (IAM `Args::groups`).
#[must_use]
pub fn credential_groups(&self) -> &Option<Vec<String>> {
&self.credential_groups
}
/// The groups IAM currently lists for [`Self::username`], read once with the snapshot.
#[must_use]
pub fn group_memberships(&self) -> &[String] {
&self.group_memberships
}
/// The claims the credential's token carries; `None` when it has none.
#[must_use]
pub fn claims(&self) -> Option<&HashMap<String, Value>> {
self.claims.as_ref()
}
/// Whether the caller owns the deployment (IAM `Args::is_owner`): root, or a credential of
/// root that no session policy or external OIDC identity narrows.
#[must_use]
pub fn is_owner(&self) -> bool {
self.is_owner
}
}
/// A [`PolicySource`] over RustFS IAM and the live root credentials.
///
/// Built only by [`crate::IamAdapters::new`], so it always reads the same [`RootSource`] as the
/// credential provider it was assembled with.
pub struct IamPolicySource {
iam: Arc<dyn IamPrincipalLookup>,
root: Arc<dyn RootSource>,
}
impl IamPolicySource {
pub(crate) fn new(iam: Arc<dyn IamPrincipalLookup>, root: Arc<dyn RootSource>) -> Self {
Self { iam, root }
}
async fn read(&self, identity: Option<&Identity>) -> Result<PolicySnapshot, PolicyError> {
let Some(identity) = identity else {
return Ok(PolicySnapshot::empty());
};
let principal = self.principal(identity).await?;
Ok(PolicySnapshot::of(Arc::new(principal)))
}
async fn principal(&self, identity: &Identity) -> Result<RustfsPrincipal, PolicyError> {
let access_key = identity.access_key_id();
let has_session = identity.session().is_some();
// Without root credentials neither ownership nor STS claims can be decided.
let Some(root) = self.root.current() else {
return Err(PolicyError::unavailable());
};
let root_access_key = root.access_key();
if access_key == root_access_key {
// Root has no session token, so a session identity for its key is not one IAM issued.
if has_session {
return Err(PolicyError::because(SESSION_MISMATCH));
}
return Ok(RustfsPrincipal {
kind: PrincipalKind::Root,
account: access_key.to_owned(),
username: access_key.to_owned(),
parent_user: String::new(),
credential_groups: None,
group_memberships: Vec::new(),
claims: None,
is_owner: true,
});
}
if !self.iam.is_ready() {
return Err(PolicyError::unavailable());
}
let cred = match self.iam.check_key(access_key).await {
Ok((Some(identity), true)) => identity.credentials,
Ok((Some(_), false)) => return Err(PolicyError::because(DISABLED_PRINCIPAL)),
Ok((None, _)) => return Err(PolicyError::because(UNKNOWN_PRINCIPAL)),
Err(error) => return Err(classify_iam_error(&error)),
};
if cred.status == ACCOUNT_OFF || !is_usable_secret(cred.secret_key.as_bytes()) {
return Err(PolicyError::because(DISABLED_PRINCIPAL));
}
let kind = if cred.is_service_account() {
PrincipalKind::ServiceAccount
} else if !cred.session_token.is_empty() {
PrincipalKind::Temporary
} else {
PrincipalKind::User
};
// The gateway admitted this identity against the credential IAM returned, so the kinds
// agree unless IAM's record changed in between; refuse rather than judge a stale reading.
if has_session != (kind == PrincipalKind::Temporary) {
return Err(PolicyError::because(SESSION_MISMATCH));
}
let claims = read_claims(kind, &cred, &root)?;
let username = match kind {
PrincipalKind::Temporary | PrincipalKind::ServiceAccount => cred.parent_user.clone(),
PrincipalKind::Root | PrincipalKind::User => cred.access_key.clone(),
};
let membership_subject = if username.is_empty() { access_key } else { username.as_str() };
let group_memberships = self.iam.group_memberships(membership_subject);
let is_owner = acts_for_root(&cred, claims.as_ref(), root_access_key);
Ok(RustfsPrincipal {
kind,
account: access_key.to_owned(),
username,
parent_user: cred.parent_user,
credential_groups: cred.groups,
group_memberships,
claims,
is_owner,
})
}
}
/// The claims of the credential's own token. The gateway admitted the presented token only when it
/// equals the stored one, so reading the stored token reads the presented one.
fn read_claims(
kind: PrincipalKind,
cred: &IamCredential,
root: &RootCredentials,
) -> Result<Option<HashMap<String, Value>>, PolicyError> {
// As in legacy `check_claims_from_token_with_context`: no token, no claims.
if cred.session_token.is_empty() {
return Ok(None);
}
let claims = match kind {
PrincipalKind::ServiceAccount => {
get_claims_from_token_with_secret_allow_missing_exp(&cred.session_token, &cred.secret_key)
.map_err(|_| PolicyError::malformed())?
}
PrincipalKind::Temporary => get_claims_from_token_with_secret(&cred.session_token, root.sts_signing_key())
.map_err(|_| PolicyError::malformed())?,
PrincipalKind::Root | PrincipalKind::User => return Ok(None),
};
Ok(if claims.is_empty() { None } else { Some(claims) })
}
/// Classifies an IAM failure without keeping any of its text.
fn classify_iam_error(error: &IamError) -> PolicyError {
match error {
IamError::NoSuchUser(_)
| IamError::NoSuchServiceAccount(_)
| IamError::NoSuchTempAccount(_)
| IamError::NoSuchAccount(_) => PolicyError::because(UNKNOWN_PRINCIPAL),
_ => PolicyError::unavailable(),
}
}
/// Root ownership as the legacy edge decides it (`rustfs/src/auth.rs` `has_root_access`): the
/// credential is root's or acts for root, and neither a session policy nor an external OIDC
/// identity narrows it.
fn acts_for_root(cred: &IamCredential, claims: Option<&HashMap<String, Value>>, root_access_key: &str) -> bool {
(cred.access_key == root_access_key || cred.parent_user == root_access_key)
&& !claims.is_some_and(|claims| claims.contains_key(SESSION_POLICY_NAME) || is_rustfs_oidc_claims(claims))
}
impl PolicySource for IamPolicySource {
fn snapshot<'a>(&'a self, identity: Option<&'a Identity>) -> BoxFuture<'a, Result<PolicySnapshot, PolicyError>> {
Box::pin(self.read(identity))
}
}
#[cfg(test)]
#[allow(clippy::expect_used, clippy::panic, clippy::unwrap_used)]
mod tests {
use super::*;
use crate::root::RootCell;
use rustfs_credentials::{Credentials as IamCredential, IAM_POLICY_CLAIM_NAME_SA};
use rustfs_gateway::SessionBinding;
use rustfs_iam::error::Error as IamError;
use rustfs_iam::sys::SESSION_POLICY_NAME;
use rustfs_iam::utils::generate_jwt;
use rustfs_policy::auth::UserIdentity;
use std::sync::atomic::{AtomicUsize, Ordering};
use time::OffsetDateTime;
const ROOT_ACCESS_KEY: &str = "rootadmin";
const ROOT_SECRET: &str = "root-secret-key-0001";
/// Deliberately not the root secret, so every STS test proves which key claims are read with.
const SIGNING_KEY: &str = "sts-signing-key-0001";
const ACCESS_KEY: &str = "AKIDUSER0001";
const SECRET: &str = "user-secret-key-0001";
const PARENT: &str = "alice";
const EXPIRES_AT: i64 = 4_000_000_000;
type KeyAnswer = Result<(Option<UserIdentity>, bool), IamError>;
type Answer = Box<dyn Fn() -> KeyAnswer + Send + Sync>;
struct FakeIam {
ready: bool,
answer: Answer,
calls: AtomicUsize,
}
impl IamKeyLookup for FakeIam {
fn is_ready(&self) -> bool {
self.ready
}
fn check_key<'a>(&'a self, _access_key: &'a str) -> BoxFuture<'a, KeyAnswer> {
self.calls.fetch_add(1, Ordering::SeqCst);
let answer = (self.answer)();
Box::pin(async move { answer })
}
}
impl IamPrincipalLookup for FakeIam {
/// Distinct groups per name, so a test can tell which name the source asked about.
fn group_memberships(&self, user: &str) -> Vec<String> {
match user {
PARENT => vec!["parent-group".to_owned()],
ACCESS_KEY => vec!["key-group".to_owned()],
ROOT_ACCESS_KEY => vec!["root-group".to_owned()],
_ => Vec::new(),
}
}
}
fn fake(ready: bool, answer: impl Fn() -> KeyAnswer + Send + Sync + 'static) -> Arc<FakeIam> {
Arc::new(FakeIam {
ready,
answer: Box::new(answer),
calls: AtomicUsize::new(0),
})
}
fn root() -> RootCredentials {
RootCredentials::new(ROOT_ACCESS_KEY, ROOT_SECRET, SIGNING_KEY).expect("valid root credentials")
}
fn source(iam: Arc<FakeIam>) -> IamPolicySource {
IamPolicySource::new(iam, Arc::new(RootCell::new(Some(root()))))
}
fn user() -> IamCredential {
IamCredential {
access_key: ACCESS_KEY.to_owned(),
secret_key: SECRET.to_owned(),
status: "on".to_owned(),
groups: Some(vec!["cred-group".to_owned()]),
..Default::default()
}
}
fn sts_token(claims: &[(&str, Value)], secret: &str) -> String {
let mut map: HashMap<String, Value> = claims.iter().map(|(k, v)| ((*k).to_owned(), v.clone())).collect();
map.insert("exp".to_owned(), Value::from(EXPIRES_AT));
generate_jwt(&map, secret).expect("a token")
}
fn sts(parent: &str, token: String) -> IamCredential {
IamCredential {
session_token: token,
parent_user: parent.to_owned(),
expiration: Some(OffsetDateTime::from_unix_timestamp(EXPIRES_AT).expect("valid timestamp")),
..user()
}
}
fn found(cred: IamCredential) -> KeyAnswer {
Ok((Some(UserIdentity::new(cred)), true))
}
fn long_term(access_key: &str) -> Identity {
Identity::new(access_key).expect("valid access key")
}
fn session(access_key: &str) -> Identity {
let binding = SessionBinding::new(crate::STS_SESSION_ISSUER, EXPIRES_AT).expect("valid binding");
long_term(access_key).with_session_binding(binding)
}
async fn principal_of(source: &IamPolicySource, identity: &Identity) -> PolicySnapshot {
source.snapshot(Some(identity)).await.expect("a snapshot")
}
fn payload(snapshot: &PolicySnapshot) -> &RustfsPrincipal {
snapshot.get::<RustfsPrincipal>().expect("the payload is a RustfsPrincipal")
}
fn refusal(result: Result<PolicySnapshot, PolicyError>) -> PolicyError {
match result {
Ok(snapshot) => panic!("expected a refusal, got a snapshot (empty: {})", snapshot.is_empty()),
Err(error) => error,
}
}
/// Compile-time: `RustfsPrincipal` has no `Debug`, so no formatting of a snapshot payload can
/// reach its claims. The two blanket impls make the path ambiguous, and so a compile error, as
/// soon as `RustfsPrincipal: Debug` holds.
#[test]
fn policy_principal_has_no_debug() {
trait AmbiguousIfDebug<A> {
fn some_item() {}
}
impl<T: ?Sized> AmbiguousIfDebug<()> for T {}
impl<T: ?Sized + core::fmt::Debug> AmbiguousIfDebug<u8> for T {}
let _ = <RustfsPrincipal as AmbiguousIfDebug<_>>::some_item;
}
/// Positive — a long-term user's facts: itself as username, both group sources, no claims,
/// not the owner.
#[tokio::test]
async fn policy_user_snapshot_carries_iam_facts() {
let iam = fake(true, || found(user()));
let snapshot = principal_of(&source(iam), &long_term(ACCESS_KEY)).await;
let principal = payload(&snapshot);
assert!(principal.kind() == PrincipalKind::User);
assert_eq!(principal.account(), ACCESS_KEY);
assert_eq!(principal.username(), ACCESS_KEY);
assert_eq!(principal.parent_user(), "");
assert_eq!(principal.credential_groups(), &Some(vec!["cred-group".to_owned()]));
assert_eq!(principal.group_memberships(), ["key-group".to_owned()]);
assert!(principal.claims().is_none());
assert!(!principal.is_owner());
}
/// Positive — an STS credential acts for its parent user, and its claims are the ones its
/// token carries, read with the root secret that signed it.
#[tokio::test]
async fn policy_sts_snapshot_carries_parent_user_and_claims() {
let token = sts_token(&[("sub", Value::from("alice-session"))], SIGNING_KEY);
let iam = fake(true, move || found(sts(PARENT, token.clone())));
let snapshot = principal_of(&source(iam), &session(ACCESS_KEY)).await;
let principal = payload(&snapshot);
assert!(principal.kind() == PrincipalKind::Temporary);
assert_eq!(principal.account(), ACCESS_KEY);
assert_eq!(principal.parent_user(), PARENT);
assert_eq!(principal.username(), PARENT);
let claims = principal.claims().expect("an STS principal has claims");
assert_eq!(claims.get("sub"), Some(&Value::from("alice-session")));
assert_eq!(principal.group_memberships(), ["parent-group".to_owned()]);
assert!(!principal.is_owner());
}
/// Positive — a service account's claims are read with its own secret, and it acts for its
/// parent.
#[tokio::test]
async fn policy_service_account_snapshot_reads_its_own_token() {
let mut token_claims: HashMap<String, Value> = HashMap::new();
token_claims.insert(IAM_POLICY_CLAIM_NAME_SA.to_owned(), Value::from("inherited-policy"));
token_claims.insert("parent".to_owned(), Value::from(PARENT));
let token = generate_jwt(&token_claims, SECRET).expect("a token");
let iam = fake(true, move || {
let mut cred = user();
cred.session_token = token.clone();
cred.parent_user = PARENT.to_owned();
cred.claims = Some([(IAM_POLICY_CLAIM_NAME_SA.to_owned(), Value::from("inherited-policy"))].into());
found(cred)
});
let snapshot = principal_of(&source(iam), &long_term(ACCESS_KEY)).await;
let principal = payload(&snapshot);
assert!(principal.kind() == PrincipalKind::ServiceAccount);
assert_eq!(principal.username(), PARENT);
assert_eq!(principal.claims().and_then(|c| c.get("parent")), Some(&Value::from(PARENT)));
assert_eq!(principal.group_memberships(), ["parent-group".to_owned()]);
assert!(!principal.is_owner());
}
/// Positive and negative — root is the owner without asking IAM, even before IAM is ready;
/// an STS credential of root is the owner too, but not once a session policy narrows it.
#[tokio::test]
async fn policy_owner_is_root_or_unrestricted_root_session() {
let not_ready = fake(false, || found(user()));
let snapshot = principal_of(&source(not_ready.clone()), &long_term(ROOT_ACCESS_KEY)).await;
let root = payload(&snapshot);
assert!(root.kind() == PrincipalKind::Root);
assert!(root.is_owner());
assert_eq!(not_ready.calls.load(Ordering::SeqCst), 0);
let plain = sts_token(&[], SIGNING_KEY);
let iam = fake(true, move || found(sts(ROOT_ACCESS_KEY, plain.clone())));
assert!(payload(&principal_of(&source(iam), &session(ACCESS_KEY)).await).is_owner());
let narrowed = sts_token(&[(SESSION_POLICY_NAME, Value::from("e30"))], SIGNING_KEY);
let iam = fake(true, move || found(sts(ROOT_ACCESS_KEY, narrowed.clone())));
assert!(!payload(&principal_of(&source(iam), &session(ACCESS_KEY)).await).is_owner());
}
/// Negative — an anonymous caller is an empty snapshot, and IAM is not asked.
#[tokio::test]
async fn policy_anonymous_is_empty_snapshot() {
let iam = fake(true, || found(user()));
let snapshot = source(iam.clone()).snapshot(None).await.expect("an anonymous snapshot");
assert!(snapshot.is_empty());
assert_eq!(iam.calls.load(Ordering::SeqCst), 0);
}
/// Negative — IAM that is not ready, not initialized, or failing is `unavailable`: never an
/// empty snapshot, which an authorizer would judge as an anonymous caller.
#[tokio::test]
async fn policy_iam_unavailable_is_unavailable() {
let not_ready = fake(false, || found(user()));
let error = refusal(source(not_ready.clone()).snapshot(Some(&long_term(ACCESS_KEY))).await);
assert_eq!(error, PolicyError::unavailable());
assert_eq!(not_ready.calls.load(Ordering::SeqCst), 0);
let failures: [fn() -> IamError; 2] = [
|| IamError::IamSysNotInitialized,
|| IamError::StringError("storage read failed".to_owned()),
];
for failure in failures {
let iam = fake(true, move || Err(failure()));
let error = refusal(source(iam).snapshot(Some(&long_term(ACCESS_KEY))).await);
assert_eq!(error, PolicyError::unavailable(), "{:?}", failure());
}
}
/// Negative — a principal IAM no longer knows is refused as unknown, not as unavailable and
/// not as an empty snapshot.
#[tokio::test]
async fn policy_unknown_principal_is_refused() {
let answers: [fn() -> KeyAnswer; 2] = [|| Ok((None, false)), || Err(IamError::NoSuchUser(ACCESS_KEY.to_owned()))];
for answer in answers {
let error = refusal(source(fake(true, answer)).snapshot(Some(&long_term(ACCESS_KEY))).await);
assert_eq!(error, PolicyError::because(UNKNOWN_PRINCIPAL));
}
}
/// Negative — a disabled or invalid principal is refused rather than given a snapshot.
#[tokio::test]
async fn policy_disabled_principal_is_refused() {
let invalid = fake(true, || Ok((Some(UserIdentity::new(user())), false)));
let error = refusal(source(invalid).snapshot(Some(&long_term(ACCESS_KEY))).await);
assert_eq!(error, PolicyError::because(DISABLED_PRINCIPAL));
let off = fake(true, || {
let mut cred = user();
cred.status = "off".to_owned();
found(cred)
});
let error = refusal(source(off).snapshot(Some(&long_term(ACCESS_KEY))).await);
assert_eq!(error, PolicyError::because(DISABLED_PRINCIPAL));
}
/// Negative — an STS token not signed by root yields no claims at all: it is `malformed`.
#[tokio::test]
async fn policy_sts_token_with_foreign_signature_is_malformed() {
let forged = sts_token(&[("sub", Value::from("alice-session"))], "not-the-root-secret");
let iam = fake(true, move || found(sts(PARENT, forged.clone())));
let error = refusal(source(iam).snapshot(Some(&session(ACCESS_KEY))).await);
assert_eq!(error, PolicyError::malformed());
}
/// Negative — the identity's credential kind must match what IAM holds: a session identity for
/// a long-term key, or a long-term identity for an STS key, is refused.
#[tokio::test]
async fn policy_session_kind_mismatch_is_refused() {
let token = sts_token(&[], SIGNING_KEY);
let iam = fake(true, move || found(sts(PARENT, token.clone())));
let error = refusal(source(iam).snapshot(Some(&long_term(ACCESS_KEY))).await);
assert_eq!(error, PolicyError::because(SESSION_MISMATCH));
let iam = fake(true, || found(user()));
let error = refusal(source(iam).snapshot(Some(&session(ACCESS_KEY))).await);
assert_eq!(error, PolicyError::because(SESSION_MISMATCH));
let iam = fake(false, || found(user()));
let error = refusal(source(iam).snapshot(Some(&session(ROOT_ACCESS_KEY))).await);
assert_eq!(error, PolicyError::because(SESSION_MISMATCH));
}
fn service_account(parent: &str, extra_claims: &[(&str, Value)]) -> KeyAnswer {
let mut token_claims: HashMap<String, Value> = extra_claims.iter().map(|(k, v)| ((*k).to_owned(), v.clone())).collect();
token_claims.insert(IAM_POLICY_CLAIM_NAME_SA.to_owned(), Value::from("inherited-policy"));
let mut cred = user();
cred.session_token = generate_jwt(&token_claims, SECRET).expect("a token");
cred.parent_user = parent.to_owned();
cred.claims = Some([(IAM_POLICY_CLAIM_NAME_SA.to_owned(), Value::from("inherited-policy"))].into());
found(cred)
}
/// Negative — a valid-flagged identity whose secret is empty or short gets no snapshot.
#[tokio::test]
async fn policy_short_secret_principal_is_refused() {
for weak in ["", "1234567", "\0\0\0\0\0\0\0\0", "abc\0\0\0\0\0"] {
let iam = fake(true, move || {
let mut cred = user();
cred.secret_key = weak.to_owned();
found(cred)
});
let error = refusal(source(iam).snapshot(Some(&long_term(ACCESS_KEY))).await);
assert_eq!(error, PolicyError::because(DISABLED_PRINCIPAL), "{weak:?}");
}
for usable in ["12345678", "abc\0defg"] {
let iam = fake(true, move || {
let mut cred = user();
cred.secret_key = usable.to_owned();
found(cred)
});
assert!(payload(&principal_of(&source(iam), &long_term(ACCESS_KEY)).await).kind() == PrincipalKind::User);
}
}
/// Negative — a root session issued through external OIDC is not the owner, though its parent
/// is root; the same session without the OIDC issuer is.
#[tokio::test]
async fn policy_oidc_root_session_is_not_owner() {
let oidc = sts_token(
&[
("iss", Value::from("rustfs-oidc")),
("oidc_provider", Value::from("corp-idp")),
("sub", Value::from("alice@corp")),
],
SIGNING_KEY,
);
let iam = fake(true, move || found(sts(ROOT_ACCESS_KEY, oidc.clone())));
assert!(!payload(&principal_of(&source(iam), &session(ACCESS_KEY)).await).is_owner());
let local = sts_token(
&[
("iss", Value::from("rustfs-sts")),
("oidc_provider", Value::from("corp-idp")),
("sub", Value::from("alice@corp")),
],
SIGNING_KEY,
);
let iam = fake(true, move || found(sts(ROOT_ACCESS_KEY, local.clone())));
assert!(payload(&principal_of(&source(iam), &session(ACCESS_KEY)).await).is_owner());
}
/// Positive and negative — a service account of root is the owner; one narrowed by a session
/// policy is not, and neither is a service account of an ordinary user.
#[tokio::test]
async fn policy_service_account_of_root_is_owner() {
let iam = fake(true, || service_account(ROOT_ACCESS_KEY, &[]));
let principal = principal_of(&source(iam), &long_term(ACCESS_KEY)).await;
assert!(payload(&principal).kind() == PrincipalKind::ServiceAccount);
assert!(payload(&principal).is_owner());
let iam = fake(true, || service_account(ROOT_ACCESS_KEY, &[(SESSION_POLICY_NAME, Value::from("e30"))]));
assert!(!payload(&principal_of(&source(iam), &long_term(ACCESS_KEY)).await).is_owner());
let iam = fake(true, || service_account(PARENT, &[]));
assert!(!payload(&principal_of(&source(iam), &long_term(ACCESS_KEY)).await).is_owner());
}
/// Negative — STS claims are read with the STS signing key, not with the root secret: a token
/// signed with the root secret while the signing key differs is `malformed`.
#[tokio::test]
async fn policy_sts_claims_use_the_signing_key() {
let root_signed = sts_token(&[("sub", Value::from("alice-session"))], ROOT_SECRET);
let iam = fake(true, move || found(sts(PARENT, root_signed.clone())));
let error = refusal(source(iam).snapshot(Some(&session(ACCESS_KEY))).await);
assert_eq!(error, PolicyError::malformed());
}
/// Negative — without root credentials no principal can be judged: `unavailable`, never an
/// empty snapshot, and IAM is not asked.
#[tokio::test]
async fn policy_without_root_is_unavailable() {
let iam = fake(true, || found(user()));
let source = IamPolicySource::new(iam.clone(), Arc::new(RootCell::new(None)));
for identity in [long_term(ACCESS_KEY), long_term(ROOT_ACCESS_KEY)] {
assert_eq!(refusal(source.snapshot(Some(&identity)).await), PolicyError::unavailable());
}
assert_eq!(iam.calls.load(Ordering::SeqCst), 0);
}
type LiveRoot = Arc<std::sync::Mutex<Option<(String, String)>>>;
/// The primary path: a source that reads a live accessor on every call, as the binary's does,
/// with nothing cached in between.
fn live_root_source(live: &LiveRoot) -> Arc<dyn RootSource> {
let accessor = live.clone();
Arc::new(move || {
let guard = accessor.lock().ok()?;
let (access_key, secret_key) = guard.as_ref()?;
RootCredentials::new(access_key, secret_key, SIGNING_KEY).ok().map(Arc::new)
})
}
fn set_live(live: &LiveRoot, value: Option<(&str, &str)>) {
*live.lock().expect("live root lock") = value.map(|(ak, sk)| (ak.to_owned(), sk.to_owned()));
}
/// Positive and negative — the two adapters read one live root accessor: a root change is seen
/// by both on the next request with no second step, the old root key stops being root to
/// both, and an uninitialized or unusable root makes both unavailable.
#[tokio::test]
async fn adapters_share_one_live_root() {
use rustfs_gateway::{CredentialLookup, CredentialProvider, ProviderError};
let live: LiveRoot = Arc::default();
set_live(&live, Some((ROOT_ACCESS_KEY, ROOT_SECRET)));
let iam = fake(true, || Ok((None, false)));
let adapters = crate::IamAdapters::new(iam, live_root_source(&live));
let found = adapters.credentials.lookup(ROOT_ACCESS_KEY).await;
assert!(matches!(found, Ok(CredentialLookup::Found(_))));
let snapshot = principal_of(&adapters.policy, &long_term(ROOT_ACCESS_KEY)).await;
assert!(payload(&snapshot).is_owner());
set_live(&live, Some(("newroot", "new-root-secret-01")));
match adapters.credentials.lookup("newroot").await {
Ok(CredentialLookup::Found(credentials)) => assert_eq!(credentials.secret().expose(), b"new-root-secret-01"),
_ => panic!("the new root key must resolve to the new secret"),
}
assert!(matches!(
adapters.credentials.lookup(ROOT_ACCESS_KEY).await,
Ok(CredentialLookup::NotFound)
));
assert!(payload(&principal_of(&adapters.policy, &long_term("newroot")).await).is_owner());
let old = refusal(adapters.policy.snapshot(Some(&long_term(ROOT_ACCESS_KEY))).await);
assert_eq!(old, PolicyError::because(UNKNOWN_PRINCIPAL));
for unusable in [None, Some(("newroot", "\0\0\0\0\0\0\0\0"))] {
set_live(&live, unusable);
assert!(matches!(adapters.credentials.lookup("newroot").await, Err(ProviderError::Unavailable)));
assert_eq!(
refusal(adapters.policy.snapshot(Some(&long_term("newroot"))).await),
PolicyError::unavailable()
);
}
}
/// Positive — `install` builds the authenticator exactly once, over a provider that reads the
/// pair's root source.
#[tokio::test]
async fn adapters_install_hands_over_the_paired_provider_once() {
use rustfs_gateway::{CredentialLookup, CredentialProvider, RegionSet, ServiceBuilder, SigV4Authenticator};
let live: LiveRoot = Arc::default();
set_live(&live, Some((ROOT_ACCESS_KEY, ROOT_SECRET)));
let adapters = crate::IamAdapters::new(fake(true, || Ok((None, false))), live_root_source(&live));
let mut handed = Vec::new();
let _builder = adapters.install(ServiceBuilder::new(), |provider| {
handed.push(provider.clone());
SigV4Authenticator::new(provider, RegionSet::new(["us-east-1"]).expect("a region"))
});
assert_eq!(handed.len(), 1);
let found = handed[0].lookup(ROOT_ACCESS_KEY).await;
assert!(matches!(found, Ok(CredentialLookup::Found(_))));
}
}
+216
View File
@@ -0,0 +1,216 @@
// Copyright 2024 RustFS Team
//
// Licensed under the Apache License, Version 2.0 (the "License");
// you may not use this file except in compliance with the License.
// You may obtain a copy of the License at
//
// http://www.apache.org/licenses/LICENSE-2.0
//
// Unless required by applicable law or agreed to in writing, software
// distributed under the License is distributed on an "AS IS" BASIS,
// WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
// See the License for the specific language governing permissions and
// limitations under the License.
//! The deployment's root credentials, as every adapter in this crate reads them.
//!
//! Responsible for: [`RootCredentials`], the validated root key pair plus the key STS session
//! tokens are signed with; [`RootSource`], the one live handle both the credential provider and
//! the policy source read on every request; and the effective-secret-length rule every secret in
//! this crate is held to.
//! NOT responsible for: where the root keys come from or when they change. The binary implements
//! [`RootSource`] directly over its live root accessor — a closure is enough — so there is no
//! second copy of root to keep in step.
//! Upstream: the binary's runtime credentials. Downstream: `crate::credentials`, `crate::policy`.
//!
//! # Why one source
//!
//! Legacy reads the current root credentials on every request. Two copies taken at assembly could
//! disagree, or go stale after a root change, and a key one adapter still treats as root would be
//! answered with the root secret or judged as the owner. Both adapters therefore hold the same
//! `Arc<dyn RootSource>`, handed to them together by [`crate::IamAdapters::new`].
use std::sync::Arc;
use rustfs_gateway::Credentials;
/// The shortest effective secret key, in bytes, that any credential in this crate may verify a
/// signature with. The same bound IAM applies to its own credentials; a signature over a shorter
/// (or empty) key is one anybody can compute.
pub const MIN_SECRET_KEY_LEN: usize = 8;
/// The number of bytes of `secret` a signature actually depends on: its length after trailing NUL
/// bytes are removed.
///
/// HMAC pads a key shorter than its block with zero bytes, and SigV4's first key is `"AWS4"`
/// followed by the secret, so trailing NULs add nothing: eight NUL bytes sign exactly like the
/// empty secret, and `"abc"` followed by five NULs signs like `"abc"`. A secret of NULs only has
/// an effective length of zero. An interior NUL is followed by real key bytes, which change the
/// padded key, so it counts like any other byte.
pub(crate) fn effective_secret_len(secret: &[u8]) -> usize {
secret.iter().rposition(|byte| *byte != 0).map_or(0, |last| last + 1)
}
/// Whether `secret` may verify a signature: its [`effective_secret_len`] is at least
/// [`MIN_SECRET_KEY_LEN`]. The one rule for root, IAM credentials and the STS signing key.
pub(crate) fn is_usable_secret(secret: &[u8]) -> bool {
effective_secret_len(secret) >= MIN_SECRET_KEY_LEN
}
/// Why [`RootCredentials::new`] refused its input. Carries no input text.
#[derive(Clone, Copy, Debug, PartialEq, Eq)]
pub enum RootCredentialsError {
/// The access key id is not one the gateway can hold.
InvalidAccessKey,
/// The root secret key fails the effective-length rule.
WeakSecretKey,
/// The STS token-signing key fails the effective-length rule.
WeakSigningKey,
}
impl core::fmt::Display for RootCredentialsError {
fn fmt(&self, f: &mut core::fmt::Formatter<'_>) -> core::fmt::Result {
f.write_str(match self {
Self::InvalidAccessKey => "root access key is not usable",
Self::WeakSecretKey => "root secret key is shorter than the minimum length",
Self::WeakSigningKey => "STS token-signing key is shorter than the minimum length",
})
}
}
impl std::error::Error for RootCredentialsError {}
/// The root key pair and the STS token-signing key, validated together.
///
/// Implements no `Debug`; the secrets are readable only inside this crate.
pub struct RootCredentials {
credentials: Credentials,
sts_signing_key: Box<str>,
}
impl RootCredentials {
/// Validates the root credentials and the key IAM signs STS session tokens with (today the
/// root secret, `rustfs_iam::manager::get_token_signing_key`; pass it separately so a future
/// key separation changes no adapter).
///
/// # Errors
///
/// [`RootCredentialsError`] when either secret fails the effective-length rule (fewer than
/// [`MIN_SECRET_KEY_LEN`] bytes once trailing NUL bytes are removed) or the access key id is not
/// one the gateway can hold.
pub fn new(access_key: &str, secret_key: &str, sts_signing_key: &str) -> Result<Self, RootCredentialsError> {
if !is_usable_secret(secret_key.as_bytes()) {
return Err(RootCredentialsError::WeakSecretKey);
}
if !is_usable_secret(sts_signing_key.as_bytes()) {
return Err(RootCredentialsError::WeakSigningKey);
}
let credentials =
Credentials::new(access_key, secret_key.as_bytes()).map_err(|_| RootCredentialsError::InvalidAccessKey)?;
Ok(Self {
credentials,
sts_signing_key: sts_signing_key.into(),
})
}
/// The root access key id, a public identifier.
#[must_use]
pub fn access_key(&self) -> &str {
self.credentials.identity().access_key_id()
}
pub(crate) fn credentials(&self) -> &Credentials {
&self.credentials
}
pub(crate) fn sts_signing_key(&self) -> &str {
&self.sts_signing_key
}
}
/// The live root credentials, read on every request by both adapters.
///
/// The binary implements it directly over RustFS's live root accessor, so a root change needs no
/// second step. Any `Fn() -> Option<Arc<RootCredentials>>` is a source; for example, over the
/// server context:
///
/// ```ignore
/// let root = move || {
/// let cred = ctx.action_credentials().get()?;
/// let signing_key = ctx.token_signing_key()?;
/// RootCredentials::new(&cred.access_key, &cred.secret_key, &signing_key).ok().map(Arc::new)
/// };
/// let adapters = IamAdapters::new(iam, Arc::new(root));
/// ```
///
/// Root credentials that fail validation read as `None`, which both adapters answer as
/// unavailable.
pub trait RootSource: Send + Sync + 'static {
/// The current root credentials; `None` while they are not initialized, which both adapters
/// answer as unavailable rather than guessing.
fn current(&self) -> Option<Arc<RootCredentials>>;
}
impl<F> RootSource for F
where
F: Fn() -> Option<Arc<RootCredentials>> + Send + Sync + 'static,
{
fn current(&self) -> Option<Arc<RootCredentials>> {
self()
}
}
/// Test helper: a [`RootSource`] holding fixed root credentials. Production code reads the live
/// accessor instead, so that there is no copy of root to keep in step.
#[cfg(test)]
pub(crate) struct RootCell(Option<Arc<RootCredentials>>);
#[cfg(test)]
impl RootCell {
pub(crate) fn new(root: Option<RootCredentials>) -> Self {
Self(root.map(Arc::new))
}
}
#[cfg(test)]
impl RootSource for RootCell {
fn current(&self) -> Option<Arc<RootCredentials>> {
self.0.clone()
}
}
#[cfg(test)]
#[allow(clippy::expect_used, clippy::panic, clippy::unwrap_used)]
mod tests {
use super::*;
/// Negative — an empty or short root secret is refused at construction, so no source can ever
/// hand out a root key anybody could sign with; eight bytes is accepted.
#[test]
fn root_short_secret_is_refused() {
for weak in ["", "1234567", "\0\0\0\0\0\0\0\0", "abc\0\0\0\0\0", "abcdefg\0"] {
let refused = RootCredentials::new("rootadmin", weak, "signing-key-0001");
assert!(matches!(refused, Err(RootCredentialsError::WeakSecretKey)), "{} bytes", weak.len());
}
assert!(RootCredentials::new("rootadmin", "12345678", "signing-key-0001").is_ok());
}
/// Positive — an interior NUL is followed by real key bytes and counts: `"abc\0defg"` has an
/// effective length of eight and is accepted, as is the same secret with trailing NULs.
#[test]
fn root_interior_nul_counts_toward_length() {
for secret in ["abc\0defg", "abc\0defg\0\0"] {
assert!(RootCredentials::new("rootadmin", secret, "signing-key-0001").is_ok(), "{secret:?}");
assert!(RootCredentials::new("rootadmin", "root-secret-key-0001", secret).is_ok(), "{secret:?}");
}
}
/// Negative — an empty or short STS signing key is refused at construction.
#[test]
fn root_short_signing_key_is_refused() {
for weak in ["", "1234567", "\0\0\0\0\0\0\0\0", "abc\0\0\0\0\0"] {
let refused = RootCredentials::new("rootadmin", "root-secret-key-0001", weak);
assert!(matches!(refused, Err(RootCredentialsError::WeakSigningKey)), "{} bytes", weak.len());
}
}
}
+230
View File
@@ -0,0 +1,230 @@
// Copyright 2024 RustFS Team
//
// Licensed under the Apache License, Version 2.0 (the "License");
// you may not use this file except in compliance with the License.
// You may obtain a copy of the License at
//
// http://www.apache.org/licenses/LICENSE-2.0
//
// Unless required by applicable law or agreed to in writing, software
// distributed under the License is distributed on an "AS IS" BASIS,
// WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
// See the License for the specific language governing permissions and
// limitations under the License.
//! The request id RustFS's outer layer minted, handed to the gateway as its own when that is safe.
//!
//! Responsible for: reading the id from the context the outer layer put in the request's
//! extensions ([`RequestIdCarrier`]) and, when it is inside the gateway's identifier alphabet and
//! length, inserting it as the gateway's [`HostRequestId`].
//! NOT responsible for: minting any id, refusing any request (nothing here can fail a request),
//! choosing the response headers (the gateway writes `x-amz-request-id` from the id it was
//! handed, and RustFS's outer layer overwrites that header on S3 paths), or the layer that mints
//! (`rustfs/src/server/layer.rs`). It never reads a header itself.
//! Upstream: RustFS's `RequestContext`, which the binary makes a [`RequestIdCarrier`]. Downstream:
//! the gateway service, which reads the [`HostRequestId`] extension once, at the top.
//!
//! # Whose id it is
//!
//! On S3 paths the outer layer mints a server-owned UUID. On admin, STS and console paths the
//! context copies the caller's `x-request-id` (or `x-amz-request-id`), as legacy does, so there
//! the id can be caller-chosen. Either way the id is attached only when it is inert, that is 1 to
//! 64 bytes of ASCII letters, digits, `-`, `.`, `_` and `~`, which is safe in a header, an XML
//! document and a log line. Anything else is not attached and the request is **forwarded
//! unchanged**: the gateway then mints its own id. No request is ever refused because of its id.
//!
//! # Known difference when nothing is attached
//!
//! The gateway answers with the id it minted (error body `<RequestId>`, its events and audit
//! records) while, on S3 paths, the outer layer overwrites the response header with RustFS's id.
//! So for such a request the header and the error body name different ids. That needs a caller-
//! chosen id outside the alphabet, or a request that never passed the outer layer.
//!
//! # Why this is not a `TraceSource`
//!
//! The gateway's `TraceSource::mint` takes no request, so a source cannot read the id RustFS
//! already minted; the gateway's documented path for a host that identifies its own requests is
//! the [`HostRequestId`] extension, and this module fills it.
//!
//! # Why a carrier trait
//!
//! `RequestContext` lives in the `rustfs` binary crate, which a library crate cannot depend on.
//! The binary implements [`RequestIdCarrier`] for it, and this module looks it up in the
//! extensions by that type.
use rustfs_gateway::HostRequestId;
/// A type in the request extensions that holds the id the outer layer minted or propagated.
pub trait RequestIdCarrier: Send + Sync + 'static {
/// The canonical request id: the value RustFS logs and the response should carry.
fn request_id(&self) -> &str;
}
/// What [`attach_host_request_id`] did. Ignoring it is harmless: every outcome forwards the request.
#[derive(Clone, Copy, Debug, PartialEq, Eq)]
pub enum AttachOutcome {
/// The carrier's id was inserted as the request's [`HostRequestId`].
Attached,
/// The request carries no context from the outer layer; nothing was inserted.
NoContext,
/// The carrier's id is empty, too long, or outside the gateway's alphabet; nothing was
/// inserted and the id was not repaired.
UnusableId,
}
/// The gateway's [`HostRequestId`] for the id `carrier` holds, or `None` when it is not inert.
#[must_use]
pub fn host_request_id<C: RequestIdCarrier + ?Sized>(carrier: &C) -> Option<HostRequestId> {
HostRequestId::new(carrier.request_id()).ok()
}
/// Reads the `C` the outer layer put in `request`'s extensions and, when its id is inert, inserts
/// it as the request's [`HostRequestId`].
///
/// Never refuses the request: the caller forwards it whatever the outcome.
pub fn attach_host_request_id<C: RequestIdCarrier, B>(request: &mut http::Request<B>) -> AttachOutcome {
let Some(carrier) = request.extensions().get::<C>() else {
return AttachOutcome::NoContext;
};
let Some(id) = host_request_id(carrier) else {
return AttachOutcome::UnusableId;
};
request.extensions_mut().insert(id);
AttachOutcome::Attached
}
#[cfg(test)]
#[allow(clippy::expect_used, clippy::panic, clippy::unwrap_used)]
mod tests {
use super::*;
#[derive(Clone)]
struct Ctx(String);
impl RequestIdCarrier for Ctx {
fn request_id(&self) -> &str {
&self.0
}
}
const UUID: &str = "7c9e6679-7425-40de-944b-e07fc1f90ae7";
fn request_with(id: &str) -> http::Request<()> {
let mut request = http::Request::new(());
request.extensions_mut().insert(Ctx(id.to_owned()));
request
}
fn handed_over(request: &http::Request<()>) -> Option<String> {
request
.extensions()
.get::<HostRequestId>()
.map(|id| id.request_id().as_str().to_owned())
}
fn attach(request: &mut http::Request<()>) -> AttachOutcome {
attach_host_request_id::<Ctx, _>(request)
}
/// Positive: the id RustFS minted is the id the gateway is handed, byte for byte.
#[test]
fn the_minted_id_is_handed_over_unchanged() {
let mut request = request_with(UUID);
assert_eq!(attach(&mut request), AttachOutcome::Attached);
assert_eq!(handed_over(&request).as_deref(), Some(UUID));
}
/// Positive: every shape RustFS mints is accepted, including its two fallbacks.
#[test]
fn every_shape_rustfs_mints_is_accepted() {
for id in [UUID, "trace-0123456789abcdef", "req-0123456789abcdef"] {
let mut request = request_with(id);
assert_eq!(attach(&mut request), AttachOutcome::Attached, "{id}");
assert_eq!(handed_over(&request).as_deref(), Some(id));
}
}
/// Positive: a caller-supplied id that is inert (admin and STS paths copy it) is attached, and
/// the longest allowed one is accepted.
#[test]
fn an_inert_caller_supplied_id_is_attached() {
let longest = "a".repeat(64);
for id in ["abc", "my.trace_id~1-2", longest.as_str()] {
let mut request = request_with(id);
assert_eq!(attach(&mut request), AttachOutcome::Attached, "{id}");
assert_eq!(handed_over(&request).as_deref(), Some(id));
}
}
/// Positive: two requests keep their own ids; nothing is shared or minted between them.
#[test]
fn each_request_keeps_its_own_id() {
let mut first = request_with("first-id");
let mut second = request_with("second-id");
attach(&mut first);
attach(&mut second);
assert_eq!(handed_over(&first).as_deref(), Some("first-id"));
assert_eq!(handed_over(&second).as_deref(), Some("second-id"));
}
/// Positive: with both a carrier and an `x-amz-request-id` header, the carrier's id wins.
#[test]
fn the_carrier_wins_over_a_request_id_header() {
let mut request = http::Request::builder()
.header("x-amz-request-id", "caller-chosen")
.header("x-request-id", "caller-chosen")
.body(())
.expect("a request");
request.extensions_mut().insert(Ctx(UUID.to_owned()));
assert_eq!(attach(&mut request), AttachOutcome::Attached);
assert_eq!(handed_over(&request).as_deref(), Some(UUID));
}
/// Negative: a request the outer layer never saw gets no id, and the request is not refused.
#[test]
fn a_missing_context_hands_over_nothing() {
let mut request = http::Request::new(());
assert_eq!(attach(&mut request), AttachOutcome::NoContext);
assert_eq!(handed_over(&request), None);
}
/// Negative: a header named like the id is not the id; only the context is read.
#[test]
fn a_request_id_header_alone_is_not_read() {
let mut request = http::Request::builder()
.header("x-amz-request-id", "caller-chosen")
.header("x-request-id", "caller-chosen")
.body(())
.expect("a request");
assert_eq!(attach(&mut request), AttachOutcome::NoContext);
assert_eq!(handed_over(&request), None);
}
/// Negative: an id outside the alphabet or the length is never attached or repaired, and the
/// outcome says so rather than failing the request.
#[test]
fn an_id_the_gateway_cannot_carry_is_not_attached() {
let too_long = "a".repeat(65);
for id in [
"",
"has space",
"line\r\nbreak",
"<RequestId>",
"quo\"te",
"Root=1-5759e988",
too_long.as_str(),
] {
let mut request = request_with(id);
assert_eq!(attach(&mut request), AttachOutcome::UnusableId, "{id:?}");
assert_eq!(handed_over(&request), None, "{id:?}");
}
}
/// Negative: the unusable id is not repaired into a shorter or cleaned-up one.
#[test]
fn host_request_id_does_not_repair() {
assert!(host_request_id(&Ctx("has space".to_owned())).is_none());
assert!(host_request_id(&Ctx(format!("{UUID} "))).is_none());
assert!(host_request_id(&Ctx(UUID.to_owned())).is_some());
}
}
+119
View File
@@ -1873,4 +1873,123 @@ mod tests {
assert_eq!(err.kind(), std::io::ErrorKind::InvalidData);
assert!(err.to_string().contains("invalid checksum value"), "unexpected error: {err}");
}
fn software_crc32_bitwise(reflected_poly: u32, data: &[u8]) -> [u8; 4] {
let mut crc = !0u32;
for &b in data {
crc ^= u32::from(b);
for _ in 0..8 {
let mask = (crc & 1).wrapping_neg();
crc = (crc >> 1) ^ (reflected_poly & mask);
}
}
(!crc).to_be_bytes()
}
fn software_crc64_nvme_bitwise(data: &[u8]) -> [u8; 8] {
let reflected_poly = super::CRC64_NVME_POLYNOMIAL.reverse_bits();
let mut crc = !0u64;
for &b in data {
crc ^= u64::from(b);
for _ in 0..8 {
let mask = (crc & 1).wrapping_neg();
crc = (crc >> 1) ^ (reflected_poly & mask);
}
}
(!crc).to_be_bytes()
}
#[test]
fn rio_crc_simd_matches_bitwise_scalar_across_tail_and_vector_boundaries() {
for &size in &[
0usize,
1,
7,
8,
15,
16,
17,
31,
32,
33,
63,
64,
65,
127,
128,
129,
255,
256,
257,
1024,
4096 + 17,
64 * 1024,
] {
let payload: Vec<u8> = (0..size).map(|i| ((i * 31 + 17) % 251) as u8).collect();
let crc32_simd = Checksum::new_from_data(ChecksumType::CRC32, &payload).expect("crc32");
let crc32c_simd = Checksum::new_from_data(ChecksumType::CRC32C, &payload).expect("crc32c");
let crc64_simd = Checksum::new_from_data(ChecksumType::CRC64_NVME, &payload).expect("crc64nvme");
let crc32_sw = software_crc32_bitwise(0xEDB88320, &payload);
let crc32c_sw = software_crc32_bitwise(0x82F63B78, &payload);
let crc64_sw = software_crc64_nvme_bitwise(&payload);
assert_eq!(crc32_simd.raw.as_slice(), &crc32_sw, "CRC32 SIMD vs scalar mismatch at {size}B");
assert_eq!(crc32c_simd.raw.as_slice(), &crc32c_sw, "CRC32C SIMD vs scalar mismatch at {size}B");
assert_eq!(crc64_simd.raw.as_slice(), &crc64_sw, "CRC64_NVME SIMD vs scalar mismatch at {size}B");
}
}
#[test]
#[ignore = "utility throughput microbenchmark; run via cargo test -p rustfs-rio --lib --release -- --ignored --nocapture"]
fn microbench_rio_checksum_simd_vs_scalar_and_all_base_algorithms() {
use std::hint::black_box;
use std::io::Write as _;
use std::time::Instant;
for &(size, iters) in &[(1024usize, 100usize), (64 * 1024, 20), (1024 * 1024, 4)] {
let payload: Vec<u8> = (0..size).map(|i| ((i * 31 + 17) % 251) as u8).collect();
let total_mib = (size * iters) as f64 / (1024.0 * 1024.0);
// Verify SIMD-accelerated CRC32 / CRC32C / CRC64_NVME against bitwise scalar reference at 1 KiB & 64 KiB.
if size <= 64 * 1024 {
let crc32_simd = Checksum::new_from_data(ChecksumType::CRC32, &payload).expect("crc32");
let crc32c_simd = Checksum::new_from_data(ChecksumType::CRC32C, &payload).expect("crc32c");
let crc64_simd = Checksum::new_from_data(ChecksumType::CRC64_NVME, &payload).expect("crc64nvme");
let t_sw = Instant::now();
let crc32_sw = software_crc32_bitwise(0xEDB88320, black_box(&payload));
let crc32c_sw = software_crc32_bitwise(0x82F63B78, black_box(&payload));
let crc64_sw = software_crc64_nvme_bitwise(black_box(&payload));
let sw_elapsed = t_sw.elapsed();
assert_eq!(crc32_simd.raw.as_slice(), &crc32_sw, "CRC32 SIMD vs scalar mismatch at {size}B");
assert_eq!(crc32c_simd.raw.as_slice(), &crc32c_sw, "CRC32C SIMD vs scalar mismatch at {size}B");
assert_eq!(crc64_simd.raw.as_slice(), &crc64_sw, "CRC64_NVME SIMD vs scalar mismatch at {size}B");
println!(
"[microbench][rio::checksum] size={size}B bitwise scalar reference check passed in {:?}",
sw_elapsed
);
}
for &checksum_type in super::BASE_CHECKSUM_TYPES {
let mut hasher = checksum_type.hasher().expect("base checksum type must have hasher");
let t0 = Instant::now();
let mut digest = Vec::new();
for _ in 0..iters {
hasher.reset();
hasher.write_all(black_box(&payload)).expect("hasher write_all");
digest = hasher.finalize();
black_box(&digest);
}
let elapsed = t0.elapsed();
assert_eq!(digest.len(), checksum_type.raw_byte_len());
let mibs = total_mib / elapsed.as_secs_f64().max(1e-9);
println!(
"[microbench][rio::checksum] size={size}B algo={} throughput={mibs:.1} MiB/s",
checksum_type
);
}
}
}
}
+396 -51
View File
@@ -690,6 +690,19 @@ pub(super) enum ScannerPauseBacklogAttemptDecision {
pub struct ScannerPauseBacklogReplicaId {
pub pool_index: usize,
pub set_index: usize,
/// Stable format identity; indices are only the current routing location.
#[serde(default, skip_serializing_if = "Option::is_none")]
pub set_id: Option<uuid::Uuid>,
}
impl ScannerPauseBacklogReplicaId {
fn from_set(set: &SetDisks) -> Result<Self, EcstoreError> {
Ok(Self {
pool_index: set.pool_index,
set_index: set.set_index,
set_id: Some(set.persistent_set_id()?),
})
}
}
#[derive(Clone, Debug, Deserialize, PartialEq, Eq, Serialize)]
@@ -706,9 +719,23 @@ impl ScannerPauseBacklogCommitRecord {
fn validate(&self) -> Result<(), String> {
self.ledger.validate()?;
if self.replicas.is_empty() || !self.replicas.windows(2).all(|pair| pair[0] < pair[1]) {
if self.replicas.is_empty()
|| !self
.replicas
.windows(2)
.all(|pair| (pair[0].pool_index, pair[0].set_index) < (pair[1].pool_index, pair[1].set_index))
{
return Err("scanner pause backlog commit has invalid replica membership".to_string());
}
let mut identities = BTreeSet::new();
if self
.replicas
.iter()
.filter_map(|id| id.set_id)
.any(|id| id.is_nil() || !identities.insert(id))
{
return Err("scanner pause backlog commit has invalid persistent set identities".to_string());
}
Ok(())
}
}
@@ -747,6 +774,10 @@ impl ScannerPauseBacklogReplicaRecord {
}
if let Some(committed) = &self.committed {
committed.validate()?;
if committed.replicas.iter().any(|id| id.set_id.is_some()) && committed.replicas.iter().any(|id| id.set_id.is_none())
{
return Err("scanner pause backlog membership mixes positional and persistent identities".to_string());
}
if let Some(stable) = &self.stable {
let stable_key = (stable.writer_epoch, stable.generation);
let committed_key = (committed.ledger.writer_epoch, committed.ledger.generation);
@@ -907,9 +938,19 @@ fn unix_now() -> u64 {
}
async fn read_scanner_pause_backlog_replica(store: Arc<SetDisks>) -> ScannerPauseBacklogReplica {
let id = ScannerPauseBacklogReplicaId {
pool_index: store.pool_index,
set_index: store.set_index,
let id = match ScannerPauseBacklogReplicaId::from_set(&store) {
Ok(id) => id,
Err(error) => {
return ScannerPauseBacklogReplica {
id: ScannerPauseBacklogReplicaId {
pool_index: store.pool_index,
set_index: store.set_index,
set_id: None,
},
revision: None,
state: ScannerPauseBacklogReplicaState::Unavailable(error.to_string()),
};
}
};
let reader = match store
.get_object_reader(
@@ -1333,8 +1374,48 @@ fn select_scanner_pause_backlog_roll_forward(replicas: &[ScannerPauseBacklogRepl
.then(|| newest.clone())
}
/// Resolve stored membership against immutable set identities before checking
/// commit proofs. A missing identity is never rebound to a new set at its old
/// index. Legacy records retain their positional interpretation until an
/// existing commit or rollback proof permits publishing the bound membership.
fn normalize_scanner_pause_backlog_membership(
replicas: &mut [ScannerPauseBacklogReplica],
) -> Result<(), ScannerPauseBacklogSelectionError> {
let mut by_location = HashMap::with_capacity(replicas.len());
let mut by_identity = HashMap::with_capacity(replicas.len());
for replica in replicas.iter() {
let id = replica.id;
if by_location.insert((id.pool_index, id.set_index), id).is_some()
|| id
.set_id
.is_some_and(|identity| identity.is_nil() || by_identity.insert(identity, id).is_some())
{
return Err(ScannerPauseBacklogSelectionError::AuthorityConflict(
"scanner pause backlog has duplicate or invalid storage identities".to_string(),
));
}
}
for replica in replicas {
if let ScannerPauseBacklogReplicaState::Valid(record) = &mut replica.state
&& let Some(commit) = record.committed.as_mut()
{
for member in &mut commit.replicas {
let current = match member.set_id {
Some(identity) => by_identity.get(&identity),
None => by_location.get(&(member.pool_index, member.set_index)),
};
if let Some(current) = current {
*member = *current;
}
}
commit.replicas.sort_unstable();
}
}
Ok(())
}
fn select_scanner_pause_backlog_replicas(
replicas: Vec<ScannerPauseBacklogReplica>,
mut replicas: Vec<ScannerPauseBacklogReplica>,
) -> Result<LoadedScannerPauseBacklog, ScannerPauseBacklogSelectionFailure> {
if replicas.is_empty() {
return Err(scanner_pause_backlog_selection_failure(
@@ -1354,6 +1435,9 @@ fn select_scanner_pause_backlog_replicas(
)),
));
}
if let Err(error) = normalize_scanner_pause_backlog_membership(&mut replicas) {
return Err(scanner_pause_backlog_selection_failure(replicas, error));
}
let replica_ids = scanner_pause_backlog_replica_ids(&replicas);
let authoritative_commit = match select_scanner_pause_backlog_commit(&replicas, &replica_ids) {
@@ -1512,12 +1596,13 @@ fn decode_scanner_pause_backlog_retirement_replicas(
) -> Result<(Vec<ScannerPauseBacklogReplica>, Vec<ScannerPauseBacklogReplica>), ScannerPauseBacklogRetirementError> {
let mut ids = BTreeSet::new();
let mut decoded = Vec::with_capacity(replicas.len());
let mut surviving = Vec::new();
let mut writable_ids = BTreeSet::new();
let mut has_source = false;
for replica in replicas {
let id = ScannerPauseBacklogReplicaId {
pool_index: replica.pool_index,
set_index: replica.set_index,
set_id: replica.set_id,
};
if !ids.insert(id) {
return Err(scanner_pause_backlog_retirement_invalid_record(
@@ -1548,7 +1633,7 @@ fn decode_scanner_pause_backlog_retirement_replicas(
state,
};
if replica.pool_index != source_pool_index && replica.writable {
surviving.push(decoded_replica.clone());
writable_ids.insert(id);
}
decoded.push(decoded_replica);
}
@@ -1557,6 +1642,12 @@ fn decode_scanner_pause_backlog_retirement_replicas(
"scanner pause backlog retirement has no native source record",
));
}
normalize_scanner_pause_backlog_membership(&mut decoded).map_err(scanner_pause_backlog_retirement_selection_error)?;
let surviving = decoded
.iter()
.filter(|replica| writable_ids.contains(&replica.id))
.cloned()
.collect();
Ok((decoded, surviving))
}
@@ -1874,11 +1965,9 @@ where
let loaded_ids = loaded.replicas.iter().map(|replica| replica.id).collect::<BTreeSet<_>>();
let writable_ids = writable
.iter()
.map(|set| ScannerPauseBacklogReplicaId {
pool_index: set.pool_index,
set_index: set.set_index,
})
.collect::<BTreeSet<_>>();
.map(|set| ScannerPauseBacklogReplicaId::from_set(set))
.collect::<Result<BTreeSet<_>, _>>()
.map_err(|error| error.to_string())?;
if writable_ids != loaded_ids {
return Err("scanner pause backlog replica topology changed during commit".to_string());
}
@@ -1901,10 +1990,7 @@ where
let results = tokio::spawn(async move {
let mut results = Vec::with_capacity(writable.len());
for set in writable {
let id = ScannerPauseBacklogReplicaId {
pool_index: set.pool_index,
set_index: set.set_index,
};
let id = ScannerPauseBacklogReplicaId::from_set(&set).map_err(|error| error.to_string())?;
if only.as_ref().is_some_and(|only| !only.contains(&id)) {
continue;
}
@@ -1928,10 +2014,10 @@ where
break;
}
}
results
Ok::<_, String>(results)
})
.await
.map_err(|err| format!("scanner pause backlog publication owner failed: {err}"))?;
.map_err(|err| format!("scanner pause backlog publication owner failed: {err}"))??;
let failures = results
.iter()
@@ -2444,6 +2530,142 @@ mod tests {
(root, store)
}
#[test]
#[serial_test::serial]
fn native_legacy_upgrade_and_pool_reindex_preserve_backlog_authority() {
assert_native_legacy_upgrade_and_pool_reindex_preserve_backlog_authority(false);
}
#[test]
#[serial_test::serial]
fn native_pool_reindex_with_failed_membership_publication_preserves_authority() {
assert_native_legacy_upgrade_and_pool_reindex_preserve_backlog_authority(true);
}
fn assert_native_legacy_upgrade_and_pool_reindex_preserve_backlog_authority(fail_publication: bool) {
run_native_retirement_test(async move || {
use crate::storage_api::owner::NativeScannerPauseBacklogWriteFault;
let (root, store) = native_pause_backlog_store(3, 1).await;
let now = unix_now();
let original = seed_native_expansion_ledger(&store, now).await;
// Model bytes written by the previous positional-membership writer.
for id in native_replica_ids(&store) {
let (bytes, _) = native_replica_bytes(&store.pools[id.pool_index].disk_set[0]).await;
let ScannerPauseBacklogReplicaState::Valid(mut record) = decode_scanner_pause_backlog_ledger(&bytes) else {
panic!("native seed is valid");
};
record.replica_schema_version = 1;
for member in &mut record.committed.as_mut().expect("seed commit").replicas {
member.set_id = None;
}
record.validate().expect("previous schema fixture");
replace_native_pause_backlog_replica(&store, id, *record).await;
}
let fault =
NativeScannerPauseBacklogWriteFault::fail_before_write(Arc::clone(&store.pools[1].disk_set[0]), "publish", 1);
assert!(
ScannerPauseBacklogController::claim(Arc::clone(&store), now + 60)
.await
.is_err()
);
drop(fault);
let interrupted = load_scanner_pause_backlog(Arc::clone(&store))
.await
.expect("mixed legacy and bound records retain the rollback proof");
assert_eq!(interrupted.ledger, original);
let upgraded = ScannerPauseBacklogController::claim(Arc::clone(&store), now + 61)
.await
.expect("retry finishes the format upgrade");
let expected = upgraded.loaded.ledger.clone();
drop(upgraded);
for pool in &store.pools {
let (bytes, _) = native_replica_bytes(&pool.disk_set[0]).await;
let ScannerPauseBacklogReplicaState::Valid(record) = decode_scanner_pause_backlog_ledger(&bytes) else {
panic!("upgraded native record is valid");
};
assert_eq!(record.replica_schema_version, 1);
assert!(
record
.committed
.expect("upgraded membership")
.replicas
.iter()
.all(|id| id.set_id.is_some())
);
}
let surviving_id = native_replica_id(&store, 2, 0).set_id;
let (source, _) = native_replica_bytes(&store.pools[1].disk_set[0]).await;
store
.prepare_scanner_pause_backlog_retirement_for_test(1, source.len() * 2)
.await
.expect("start retiring the middle pool");
store
.retire_scanner_pause_backlog_for_test(1, 0)
.await
.expect("hand off the native backlog");
store
.complete_scanner_pause_backlog_retirement_for_test(1)
.await
.expect("persist the completed middle pool through the ILM receipt checks");
shutdown_native_retirement_store(store).await;
let reordered = super::super::tests::setup_scanner_cycle_store_with_pool_order(
root.path(),
false,
&[0, 2, 3],
1,
NATIVE_RETIREMENT_DRIVES_PER_SET,
false,
)
.await;
assert_eq!(native_replica_id(&reordered, 1, 0).set_id, surviving_id);
assert_ne!(native_replica_id(&reordered, 2, 0).set_id, surviving_id);
let loaded = load_scanner_pause_backlog(Arc::clone(&reordered))
.await
.expect("removing a middle pool and appending a new one preserves native authority");
assert_eq!(loaded.ledger, expected);
assert_eq!(loaded.persistence_state, "membership_repair_pending");
if fail_publication {
// Match the reported partial publication: the two existing
// pools reject the new commit while the replacement accepts it.
let faults = [0, 1].map(|pool| {
NativeScannerPauseBacklogWriteFault::fail_before_write(
Arc::clone(&reordered.pools[pool].disk_set[0]),
"publish",
2,
)
});
let error = ScannerPauseBacklogController::claim(Arc::clone(&reordered), now + 120)
.await
.err()
.expect("the incomplete publication must fail");
assert!(error.contains("pool 0 set 0") && error.contains("pool 1 set 0"), "{error}");
drop(faults);
let recovered = load_scanner_pause_backlog(Arc::clone(&reordered))
.await
.expect("partial publication after reindexing retains proven authority");
assert_eq!(recovered.ledger, expected);
assert!(recovered.durable);
}
let (source, _) = native_replica_bytes(&reordered.pools[1].disk_set[0]).await;
reordered
.prepare_scanner_pause_backlog_retirement_for_test(1, source.len() * 2)
.await
.expect("retire the reindexed member before a scanner writer completes the membership change");
reordered
.retire_scanner_pause_backlog_for_test(1, 0)
.await
.expect("retirement uses the same bound authority");
let surviving = load_scanner_pause_backlog(Arc::clone(&reordered))
.await
.expect("remaining members retain the complete ledger");
assert_eq!(surviving.ledger, expected);
assert!(surviving.all_current_replicas_stable());
shutdown_native_retirement_store(reordered).await;
});
}
#[test]
#[serial_test::serial]
fn native_single_pool_partial_commit_converges_across_24_sets() {
@@ -2592,9 +2814,7 @@ mod tests {
);
let stable = durable_ledger(unix_now());
let members = (0..3)
.flat_map(|pool_index| (0..2).map(move |set_index| replica_id(pool_index, set_index)))
.collect::<Vec<_>>();
let members = native_replica_ids(&store);
let committed = ScannerPauseBacklogCommitRecord::new(stable.clone(), members);
let record = ScannerPauseBacklogReplicaRecord::new(Some(stable), Some(committed));
let bytes = encode_scanner_pause_backlog_record(&record).expect("stable replica should encode");
@@ -3021,10 +3241,7 @@ mod tests {
let now = unix_now().saturating_add(60);
let replacement = claim_scanner_pause_backlog_writer(&original, now).expect("next writer generation");
let new_commit = ScannerPauseBacklogCommitRecord::new(replacement.clone(), current_ids.clone());
let failed_id = ScannerPauseBacklogReplicaId {
pool_index: failed_pool,
set_index: 0,
};
let failed_id = native_replica_id(&store, failed_pool, 0);
let fault = NativeScannerPauseBacklogWriteFault::fail_before_write(
Arc::clone(&store.pools[failed_pool].disk_set[0]),
"publish",
@@ -3553,10 +3770,8 @@ mod tests {
.authoritative_commit
.expect("old cohort commit");
let now = unix_now().saturating_add(60);
let ids = (0..3)
.flat_map(|pool_index| (0..2).map(move |set_index| replica_id(pool_index, set_index)))
.collect::<Vec<_>>();
let newer_member = replica_id(2, 1);
let ids = native_replica_ids(&store);
let newer_member = native_replica_id(&store, 2, 1);
let newer = claim_scanner_pause_backlog_writer(&original, now).expect("newer generation");
let newer_commit = ScannerPauseBacklogCommitRecord::new(newer.clone(), ids.clone());
for id in &ids {
@@ -3617,14 +3832,12 @@ mod tests {
let (_root, store, original) = native_expanded_backlog_store(1, false).await;
let now = unix_now().saturating_add(60);
let wedge = native_interleaved_expansion_wedge(&store, &original, now).await;
let ids = (0..3)
.flat_map(|pool_index| (0..2).map(move |set_index| replica_id(pool_index, set_index)))
.collect::<Vec<_>>();
let ids = native_replica_ids(&store);
// The retiring source holds the newest stable ledger.
let newest = claim_scanner_pause_backlog_writer(&wedge.acknowledged, now + 1).expect("newest generation");
replace_native_pause_backlog_replica(
&store,
replica_id(0, 0),
native_replica_id(&store, 0, 0),
ScannerPauseBacklogReplicaRecord::new(
Some(newest.clone()),
Some(ScannerPauseBacklogCommitRecord::new(newest.clone(), ids)),
@@ -3675,10 +3888,8 @@ mod tests {
set_runtime_error(None);
let (_root, store, original) = native_expanded_backlog_store(1, false).await;
let now = unix_now().saturating_add(60);
let ids = (0..3)
.flat_map(|pool_index| (0..2).map(move |set_index| replica_id(pool_index, set_index)))
.collect::<Vec<_>>();
let newer_member = replica_id(2, 1);
let ids = native_replica_ids(&store);
let newer_member = native_replica_id(&store, 2, 1);
let others = ids.iter().copied().filter(|id| *id != newer_member).collect::<Vec<_>>();
let middle = claim_scanner_pause_backlog_writer(&original, now).expect("middle generation");
let newest = claim_scanner_pause_backlog_writer(&middle, now + 1).expect("newest generation");
@@ -3837,10 +4048,8 @@ mod tests {
let (_root, store, original) = native_expanded_backlog_store(1, false).await;
let now = unix_now().saturating_add(60);
let wedge = native_interleaved_expansion_wedge(&store, &original, now).await;
let ids = (0..3)
.flat_map(|pool_index| (0..2).map(move |set_index| replica_id(pool_index, set_index)))
.collect::<Vec<_>>();
assert_eq!(wedge.last, replica_id(2, 1));
let ids = native_replica_ids(&store);
assert_eq!(wedge.last, native_replica_id(&store, 2, 1));
// A different stable ledger at the newest key is a fork.
let fork = claim_scanner_pause_backlog_writer(&original, now + 7).expect("forked writer generation");
@@ -3851,7 +4060,7 @@ mod tests {
assert_ne!(fork, wedge.acknowledged);
replace_native_pause_backlog_replica(
&store,
replica_id(1, 0),
native_replica_id(&store, 1, 0),
ScannerPauseBacklogReplicaRecord::new(
Some(fork.clone()),
Some(ScannerPauseBacklogCommitRecord::new(fork, ids.clone())),
@@ -3868,7 +4077,7 @@ mod tests {
"{error}"
);
assert_eq!(native_replica_snapshot(&store).await, forked, "a refused claim keeps every record");
replace_native_pause_backlog_replica(&store, replica_id(1, 0), wedge.rollback.clone()).await;
replace_native_pause_backlog_replica(&store, native_replica_id(&store, 1, 0), wedge.rollback.clone()).await;
// A missing member that a stored commit names may have lost a
// newer stable ledger.
@@ -3903,7 +4112,7 @@ mod tests {
.expect_err("retirement must refuse an unproven authority");
assert!(refused.to_string().contains("unsafe authority conflict"), "{refused}");
assert_eq!(native_replica_bytes(&store.pools[0].disk_set[0]).await.0, source_bytes);
replace_native_pause_backlog_replica(&store, replica_id(1, 1), wedge.rollback.clone()).await;
replace_native_pause_backlog_replica(&store, native_replica_id(&store, 1, 1), wedge.rollback.clone()).await;
// The same records without the conflict converge and retire.
for set in 0..2 {
@@ -4020,7 +4229,7 @@ mod tests {
.authoritative_commit
.as_ref()
.expect("seed retains the old cohort proof");
assert_eq!(committed.replicas, vec![replica_id(0, 0), replica_id(0, 1)]);
assert_eq!(committed.replicas, vec![native_replica_id(&store, 0, 0), native_replica_id(&store, 0, 1)]);
assert_eq!(loaded.replica_count, 6);
if loaded.healthy_replicas == loaded.replica_count {
break loaded;
@@ -4037,13 +4246,13 @@ mod tests {
.authoritative_commit
.as_ref()
.expect("seed retains the old cohort proof");
assert_eq!(committed.replicas, vec![replica_id(0, 0), replica_id(0, 1)]);
assert_eq!(committed.replicas, vec![native_replica_id(&store, 0, 0), native_replica_id(&store, 0, 1)]);
assert_eq!(loaded.replica_count, 6);
assert_eq!(loaded.healthy_replicas, 6);
let seeded = loaded
.replicas
.iter()
.find(|replica| replica.id == replica_id(2, 0))
.find(|replica| replica.id == native_replica_id(&store, 2, 0))
.expect("the canceled caller's admitted seed replica");
assert!(matches!(
&seeded.state,
@@ -4055,7 +4264,7 @@ mod tests {
let remaining = loaded
.replicas
.iter()
.find(|replica| replica.id == replica_id(2, 1))
.find(|replica| replica.id == native_replica_id(&store, 2, 1))
.expect("the remaining seed replica");
assert!(matches!(
&remaining.state,
@@ -4268,7 +4477,12 @@ mod tests {
.prepare_scanner_pause_backlog_retirement_for_test(0, source_before.len() * 2)
.await
.expect("retire the old cohort's first pool");
let surviving = [replica_id(1, 0), replica_id(1, 1), replica_id(2, 0), replica_id(2, 1)];
let surviving = [
native_replica_id(&store, 1, 0),
native_replica_id(&store, 1, 1),
native_replica_id(&store, 2, 0),
native_replica_id(&store, 2, 1),
];
let newer = claim_scanner_pause_backlog_writer(&old.ledger, now + 2).expect("surviving writer");
replace_native_pause_backlog_replica(
&store,
@@ -4333,7 +4547,7 @@ mod tests {
assert_eq!(loaded.ledger, original);
assert!(loaded.durable && loaded.stable_matches_ledger);
let committed = loaded.authoritative_commit.expect("survivor commit proof");
assert_eq!(committed.replicas, vec![replica_id(2, 0), replica_id(2, 1)]);
assert_eq!(committed.replicas, vec![native_replica_id(&store, 2, 0), native_replica_id(&store, 2, 1)]);
}
shutdown_native_retirement_store(store).await;
});
@@ -4911,7 +5125,135 @@ mod tests {
}
fn replica_id(pool_index: usize, set_index: usize) -> ScannerPauseBacklogReplicaId {
ScannerPauseBacklogReplicaId { pool_index, set_index }
ScannerPauseBacklogReplicaId {
pool_index,
set_index,
set_id: None,
}
}
fn native_replica_id(store: &ECStore, pool_index: usize, set_index: usize) -> ScannerPauseBacklogReplicaId {
ScannerPauseBacklogReplicaId::from_set(&store.pools[pool_index].disk_set[set_index])
.expect("native replica has an immutable format identity")
}
fn native_replica_ids(store: &ECStore) -> Vec<ScannerPauseBacklogReplicaId> {
store
.pools
.iter()
.flat_map(|pool| pool.disk_set.iter())
.map(|set| ScannerPauseBacklogReplicaId::from_set(set).expect("native replica format identity"))
.collect()
}
fn bound_replica_id(pool_index: usize, identity: u128) -> ScannerPauseBacklogReplicaId {
ScannerPauseBacklogReplicaId {
pool_index,
set_index: 0,
set_id: Some(uuid::Uuid::from_u128(identity)),
}
}
fn missing_replica(id: ScannerPauseBacklogReplicaId) -> ScannerPauseBacklogReplica {
ScannerPauseBacklogReplica {
id,
revision: Some(DataUsageCacheRevision::Missing),
state: ScannerPauseBacklogReplicaState::Missing,
}
}
#[test]
fn bound_membership_survives_pool_reindexing() {
let ledger = durable_ledger(50);
let previous = bound_replica_id(2, 22);
let current = bound_replica_id(1, 22);
let added = bound_replica_id(2, 33);
let record = replica_record_for_members(&ledger, &ledger, &[previous]);
assert_eq!(record.replica_schema_version, 1);
let loaded = select_scanner_pause_backlog_replicas(vec![
missing_replica(bound_replica_id(0, 11)),
decoded_replica(current, &record),
missing_replica(added),
])
.expect("a surviving set keeps its commit proof after its routing index changes");
assert_eq!(loaded.ledger, ledger);
assert!(loaded.durable);
assert_eq!(loaded.authoritative_commit.expect("surviving native proof").replicas, vec![current]);
let mut replicas = vec![retirement_replica(current, &record)];
for id in [bound_replica_id(0, 11), added] {
replicas.push(ScannerPauseBacklogRetirementReplica {
pool_index: id.pool_index,
set_index: id.set_index,
set_id: id.set_id,
writable: true,
data: None,
});
}
let plan = plan_scanner_pause_backlog_retirement(1, &replicas)
.expect("reindexed native authority permits safe retirement")
.expect("new survivors need the existing ledger");
assert_eq!(planned_stable_ledger(&plan), ledger);
}
#[test]
fn bound_membership_does_not_rebind_a_missing_set_to_its_reused_index() {
let ledger = durable_ledger(50);
let missing = bound_replica_id(2, 22);
let replacement = bound_replica_id(2, 33);
let record = replica_record_for_members(&ledger, &ledger, &[missing]);
assert!(
select_scanner_pause_backlog_replicas(vec![
missing_replica(bound_replica_id(0, 11)),
decoded_replica(replacement, &record),
])
.is_err(),
"copied bytes on a new set cannot impersonate the missing commit member"
);
}
#[test]
fn legacy_membership_binds_only_after_its_existing_proof_is_verified() {
let ledger = durable_ledger(50);
let legacy = replica_record_for_members(&ledger, &ledger, &[replica_id(2, 0)]);
assert_eq!(legacy.replica_schema_version, 1);
let current = bound_replica_id(2, 22);
let loaded = select_scanner_pause_backlog_replicas(vec![decoded_replica(current, &legacy)])
.expect("unchanged legacy membership remains readable");
let upgraded = stable_scanner_pause_backlog_record(
&loaded.ledger,
loaded.authoritative_commit.as_ref(),
&scanner_pause_backlog_replica_ids(&loaded.replicas),
);
upgraded
.validate()
.expect("the replacement record binds every committed member");
assert_eq!(upgraded.replica_schema_version, 1);
assert_eq!(upgraded.committed.expect("bound membership").replicas, vec![current]);
assert!(
select_scanner_pause_backlog_replicas(vec![
decoded_replica(bound_replica_id(1, 22), &legacy),
missing_replica(bound_replica_id(2, 33)),
])
.is_err(),
"an already stranded legacy proof must not guess its old member's identity"
);
}
#[test]
fn bound_membership_rejects_nil_duplicate_and_mixed_identities() {
let ledger = durable_ledger(50);
for members in [
vec![bound_replica_id(0, 0)],
vec![bound_replica_id(0, 11), bound_replica_id(1, 11)],
vec![bound_replica_id(0, 11), bound_replica_id(0, 22)],
vec![bound_replica_id(0, 11), replica_id(1, 0)],
] {
let record = replica_record_for_members(&ledger, &ledger, &members);
assert_eq!(record.replica_schema_version, 1);
assert!(record.validate().is_err());
}
}
fn decoded_replica(
@@ -5155,6 +5497,7 @@ mod tests {
.map(|set_index| ScannerPauseBacklogReplicaId {
pool_index: 0,
set_index,
set_id: None,
})
.collect::<Vec<_>>();
replica_record_for_members(stable, committed, &replicas)
@@ -5167,6 +5510,7 @@ mod tests {
ScannerPauseBacklogRetirementReplica {
pool_index: id.pool_index,
set_index: id.set_index,
set_id: id.set_id,
writable: true,
data: Some(serde_json::to_vec(record).expect("native record bytes")),
}
@@ -5294,6 +5638,7 @@ mod tests {
replicas.extend(targets.iter().map(|id| ScannerPauseBacklogRetirementReplica {
pool_index: id.pool_index,
set_index: id.set_index,
set_id: id.set_id,
writable: true,
data: None,
}));
+24 -4
View File
@@ -113,17 +113,37 @@ pub(super) async fn setup_scanner_cycle_store_at_path_with_layout_and_disk_prein
sets_per_pool: usize,
drives_per_set: usize,
preinitialize_disks: bool,
) -> Arc<ECStore> {
setup_scanner_cycle_store_with_pool_order(
root,
seed_usage_baseline,
&(0..pool_count).collect::<Vec<_>>(),
sets_per_pool,
drives_per_set,
preinitialize_disks,
)
.await
}
pub(super) async fn setup_scanner_cycle_store_with_pool_order(
root: &Path,
seed_usage_baseline: bool,
pool_order: &[usize],
sets_per_pool: usize,
drives_per_set: usize,
preinitialize_disks: bool,
) -> Arc<ECStore> {
init_ecstore_config_for_scanner_tests();
let pool_count = pool_order.len();
let mut pools = Vec::with_capacity(pool_count);
for pool_index in 0..pool_count {
for (pool_index, physical_pool) in pool_order.iter().enumerate() {
let mut endpoints = Vec::new();
for set_index in 0..sets_per_pool {
for disk_index in 0..drives_per_set {
let disk_path = if sets_per_pool == 1 {
root.join(format!("pool{pool_index}/disk{disk_index}"))
root.join(format!("pool{physical_pool}/disk{disk_index}"))
} else {
root.join(format!("pool{pool_index}/set{set_index}/disk{disk_index}"))
root.join(format!("pool{physical_pool}/set{set_index}/disk{disk_index}"))
};
tokio::fs::create_dir_all(&disk_path)
.await
@@ -144,7 +164,7 @@ pub(super) async fn setup_scanner_cycle_store_at_path_with_layout_and_disk_prein
cmd_line: if pool_count == 1 && sets_per_pool == 1 {
"scanner-cycle-metrics".to_string()
} else {
format!("scanner-cycle-metrics-pool-{pool_index}")
format!("scanner-cycle-metrics-pool-{physical_pool}")
},
platform: format!("OS: {} | Arch: {}", std::env::consts::OS, std::env::consts::ARCH),
});
+33 -20
View File
@@ -425,7 +425,10 @@ enum FolderScanSource {
#[derive(Clone, Copy, PartialEq, Eq)]
enum FolderScanMode {
Namespace,
ErasureCandidate { reuse_namespace: bool },
ErasureCandidate {
reuse_namespace: bool,
parent_has_metadata: bool,
},
}
#[derive(Debug, PartialEq, Eq)]
@@ -2065,7 +2068,9 @@ impl FolderScanner {
if candidate_unclassified {
// Apply namespace reuse only after the current listing
// proves there is no direct erasure part.
if let FolderScanMode::ErasureCandidate { reuse_namespace: true } = mode
if let FolderScanMode::ErasureCandidate {
reuse_namespace: true, ..
} = mode
&& !into.compacted
&& self.old_cache.find(&folder.name).is_some()
{
@@ -2079,10 +2084,23 @@ impl FolderScanner {
return Ok(FolderScanOutcome::Namespace);
}
candidate_unclassified = false;
if !self.budget.try_start_directory() {
return Err(ScannerError::Other("Operation cancelled".to_string()));
// An empty UUID directory beside object metadata has
// no namespace work. Its listing was already charged;
// nonempty namespaces still need directory admission.
let empty_object_child = matches!(
mode,
FolderScanMode::ErasureCandidate {
parent_has_metadata: true,
..
}
) && deferred_metadata.is_none()
&& queued_folders.is_empty();
if !empty_object_child {
if !self.budget.try_start_directory() {
return Err(ScannerError::Other("Operation cancelled".to_string()));
}
wait_for_scanner_io(&ctx, self.sleeper.sleep_folder_preset()).await?;
}
wait_for_scanner_io(&ctx, self.sleeper.sleep_folder_preset()).await?;
}
let Some(entry) = deferred_metadata.take() else { break };
(entry, true)
@@ -2552,20 +2570,6 @@ impl FolderScanner {
return Err(ScannerError::Other("Operation cancelled".to_string()));
}
if found_object_metadata && self.is_erasure_mode {
// If we found an object in erasure mode, we skip subdirs (only datadirs)...
debug!(
target: "rustfs::scanner::folder",
event = EVENT_SCANNER_FOLDER_STATE,
component = LOG_COMPONENT_SCANNER,
subsystem = LOG_SUBSYSTEM_FOLDER,
folder = %folder.name,
state = "erasure_object_found",
"Scanner folder descent stopped after erasure object"
);
break;
}
// Checkpointed sweeps need child entries to validate their resume
// frontier (#8292). Outside that, also compact a wide bucket root:
// excluding it left #8961's ~170k top-level prefixes unbounded until
@@ -2654,6 +2658,7 @@ impl FolderScanner {
let mut child_mode = if self.is_erasure_mode && is_erasure_candidate(&folder_item.name) {
FolderScanMode::ErasureCandidate {
reuse_namespace: reuse_covered,
parent_has_metadata: found_object_metadata,
}
} else {
FolderScanMode::Namespace
@@ -2721,7 +2726,10 @@ impl FolderScanner {
}
// Sampling may skip namespace work, never the
// classification that protects its parent object.
child_mode = FolderScanMode::ErasureCandidate { reuse_namespace: true };
child_mode = FolderScanMode::ErasureCandidate {
reuse_namespace: true,
parent_has_metadata: found_object_metadata,
};
} else {
folder_item.object_heal_prob_div = data_usage_update_dir_cycles();
}
@@ -2816,6 +2824,11 @@ impl FolderScanner {
updates.cache.delete_recursive(&h);
}
into.children.remove(&clone_cache_value(&h.0));
// Object keys can also be prefixes. A classified payload
// directory must not hide their other namespace children.
if found_object_metadata {
continue;
}
// Earlier namespace siblings may have completed, but none
// can certify coverage of this unresolved parent object.
self.coverage_gap |= forward_sweep;
@@ -183,6 +183,117 @@ fn charged_ops(root: &std::path::Path) -> u64 {
status.charged_ops
}
#[tokio::test]
#[serial]
async fn erasure_object_metadata_preserves_child_object_namespaces() {
for (corrupt_parent, cached_parent) in [(false, false), (true, false), (false, true)] {
let (mut scanner, root) = build_test_scanner().await;
let _guard = TestGuard::new(60, 100, &mut scanner, root.clone());
scanner.old_cache.info.name = "bucket".into();
scanner.new_cache.info.name = "bucket".into();
scanner.is_erasure_mode = true;
bind_budget(&mut scanner, &root, 1_000_000);
let namespace = format!("parent/{}/nested", Uuid::new_v4());
for object in ["parent", "parent/child", namespace.as_str()] {
write_test_object_metadata_bytes(&root, "bucket", object, &metadata_for_object_version("bucket", object, None)).await;
}
let payload = root.join("bucket/parent").join(Uuid::new_v4().to_string());
tokio::fs::create_dir_all(&payload)
.await
.expect("create parent payload directory");
tokio::fs::write(payload.join("part.1"), b"shard")
.await
.expect("create actual payload evidence");
if corrupt_parent {
tokio::fs::write(root.join("bucket/parent/xl.meta"), b"")
.await
.expect("corrupt only the parent metadata");
}
if cached_parent {
scanner.old_cache.replace("bucket", "", DataUsageEntry::default());
scanner.old_cache.replace(
"bucket/parent",
"bucket",
DataUsageEntry {
objects: 1,
size: 1,
compacted: true,
..Default::default()
},
);
let interval = data_usage_update_dir_cycles();
scanner.old_cache.info.next_cycle = u64::from(
(0..interval)
.find(|cycle| hash_path("bucket/parent").mod_(*cycle, interval))
.expect("one normal sampling cycle revisits the old parent-only cache"),
);
}
let mut into = DataUsageEntry::default();
scanner
.scan_folder(
CancellationToken::new(),
CachedFolder {
name: "bucket".into(),
parent: None,
object_heal_prob_div: 1,
},
&mut into,
)
.await
.expect("scan object and child namespaces independently");
let total = scanner.new_cache.size_recursive("bucket").expect("retain bucket accounting");
assert_eq!(total.objects, if corrupt_parent { 2 } else { 3 });
assert_eq!(total.size, if corrupt_parent { 2 } else { 3 });
assert_eq!(total.failed_objects, usize::from(corrupt_parent));
assert_eq!(
scanner.new_cache.info.failed_objects.len(),
usize::from(corrupt_parent),
"payload evidence must not invent missing parent metadata"
);
}
}
#[tokio::test]
#[serial]
async fn erasure_object_prefix_children_keep_the_object_budget() {
let (mut scanner, root) = build_test_scanner().await;
let _guard = TestGuard::new(60, 100, &mut scanner, root.clone());
scanner.old_cache.info.name = "bucket".into();
scanner.new_cache.info.name = "bucket".into();
scanner.is_erasure_mode = true;
bind_budget(&mut scanner, &root, 1_000_000);
for object in ["parent", "parent/child", "parent/other"] {
write_test_object_metadata(&root, "bucket", object).await;
}
let visits = metadata_visits::observe(&mut scanner);
let parent = CancellationToken::new();
let budget = ScannerCycleBudget::new_with_progress_tracking(
&parent,
crate::scanner_budget::ScannerCycleBudgetConfig {
max_objects: Some(2),
..Default::default()
},
);
scanner.budget = budget.clone();
let result = scanner
.scan_folder(
budget.token(),
CachedFolder {
name: "bucket".into(),
parent: None,
object_heal_prob_div: 1,
},
&mut DataUsageEntry::default(),
)
.await;
assert!(result.is_err(), "parent metadata cannot hide children or bypass their budget");
assert_eq!(budget.reason(), Some(crate::scanner_budget::ScannerCycleBudgetReason::Objects));
assert_eq!(budget.progress().0, 2);
assert_eq!(visits.load(Ordering::Relaxed), 2);
}
#[tokio::test]
#[serial]
async fn erasure_directory_probe_reaches_eof_across_budgeted_batches() {
@@ -204,7 +315,10 @@ async fn erasure_directory_probe_reaches_eof_across_budgeted_batches() {
ctx,
folder,
&mut DataUsageEntry::default(),
FolderScanMode::ErasureCandidate { reuse_namespace: false },
FolderScanMode::ErasureCandidate {
reuse_namespace: false,
parent_has_metadata: false,
},
false,
)
.await
@@ -271,7 +385,10 @@ async fn scanner_directory_probe_budget_wait_cancels_without_rate_delay() {
ctx.clone(),
folder,
&mut DataUsageEntry::default(),
FolderScanMode::ErasureCandidate { reuse_namespace: false },
FolderScanMode::ErasureCandidate {
reuse_namespace: false,
parent_has_metadata: false,
},
false,
)
.await;
@@ -311,7 +428,10 @@ async fn scanner_directory_probe_cancels_during_entry_budget_wait() {
ctx.clone(),
folder,
&mut DataUsageEntry::default(),
FolderScanMode::ErasureCandidate { reuse_namespace: false },
FolderScanMode::ErasureCandidate {
reuse_namespace: false,
parent_has_metadata: false,
},
false,
)
.await;
@@ -382,7 +502,16 @@ async fn erasure_candidate_defers_metadata_until_part_classification_finishes()
scanner.budget = ScannerCycleBudget::new_with_progress_tracking(&ctx, Default::default());
let mut into = DataUsageEntry::default();
let outcome = scanner
.scan_folder_inner(ctx, folder, &mut into, FolderScanMode::ErasureCandidate { reuse_namespace: false }, false)
.scan_folder_inner(
ctx,
folder,
&mut into,
FolderScanMode::ErasureCandidate {
reuse_namespace: false,
parent_has_metadata: false,
},
false,
)
.await
.expect("classify candidate");
assert_eq!(outcome, FolderScanOutcome::ErasureData);
@@ -594,7 +723,10 @@ async fn erasure_candidate_part_symlink_is_not_data_evidence() {
CancellationToken::new(),
folder,
&mut DataUsageEntry::default(),
FolderScanMode::ErasureCandidate { reuse_namespace: false },
FolderScanMode::ErasureCandidate {
reuse_namespace: false,
parent_has_metadata: false,
},
false,
)
.await
+162
View File
@@ -735,4 +735,166 @@ mod tests {
}
}
}
#[test]
fn highwayhash_simd_matches_portable_scalar_and_chunked_slices_across_unaligned_boundaries() {
use highway::PortableHash;
for &size in &[
0usize,
1,
7,
15,
31,
32,
33,
63,
64,
65,
127,
128,
129,
255,
257,
1024,
4096 + 17,
64 * 1024,
] {
let payload = generate_compat_test_data(size);
let mut h_simd = HighwayHasher::new(MAGIC_HIGHWAY_HASH256_PARSED_KEY);
h_simd.append(&payload);
let simd_digest = u8x32_from_u64x4(h_simd.finalize256());
let mut h_scalar = PortableHash::new(MAGIC_HIGHWAY_HASH256_PARSED_KEY);
h_scalar.append(&payload);
let scalar_digest = u8x32_from_u64x4(h_scalar.finalize256());
assert_eq!(
simd_digest, scalar_digest,
"HighwayHasher SIMD must match PortableHash scalar byte-for-byte at {size}B"
);
for &chunk_size in &[17usize, 61, 4096] {
if size > 4096 && chunk_size < 61 {
continue;
}
let slices: Vec<&[u8]> = payload.chunks(chunk_size).collect();
for algo in [
HashAlgorithm::HighwayHash256S,
HashAlgorithm::HighwayHash256SLegacy,
HashAlgorithm::SHA256,
HashAlgorithm::BLAKE2b512,
HashAlgorithm::Md5,
] {
let contig_out = algo.hash_encode(&payload);
let sliced_out = algo.hash_encode_slices(slices.iter().copied());
assert_eq!(
contig_out.as_ref(),
sliced_out.as_ref(),
"{algo:?} contiguous vs {chunk_size}B sliced digest mismatch at {size}B"
);
}
}
}
}
#[test]
#[ignore = "utility throughput microbenchmark; run via cargo test -p rustfs-utils --lib --features hash --release -- --ignored --nocapture"]
fn microbench_highwayhash_simd_vs_portable_scalar_and_hash_algorithms() {
use highway::PortableHash;
use std::hint::black_box;
use std::time::Instant;
for &(size, iters) in &[(1024usize, 200usize), (64 * 1024, 40), (1024 * 1024, 8)] {
let payload = generate_compat_test_data(size);
// 1. Compare HighwayHasher (runtime SIMD: AVX2/SSE4.1/NEON) against PortableHash scalar reference.
let t_simd = Instant::now();
let mut simd_digest = [0u8; 32];
for _ in 0..iters {
let mut h = HighwayHasher::new(MAGIC_HIGHWAY_HASH256_PARSED_KEY);
h.append(black_box(&payload));
simd_digest = u8x32_from_u64x4(h.finalize256());
black_box(&simd_digest);
}
let simd_elapsed = t_simd.elapsed();
let t_scalar = Instant::now();
let mut scalar_digest = [0u8; 32];
for _ in 0..iters {
let mut h = PortableHash::new(MAGIC_HIGHWAY_HASH256_PARSED_KEY);
h.append(black_box(&payload));
scalar_digest = u8x32_from_u64x4(h.finalize256());
black_box(&scalar_digest);
}
let scalar_elapsed = t_scalar.elapsed();
assert_eq!(
simd_digest, scalar_digest,
"HighwayHasher SIMD must match PortableHash scalar byte-for-byte at {size}B"
);
let total_mib = (size * iters) as f64 / (1024.0 * 1024.0);
let simd_mibs = total_mib / simd_elapsed.as_secs_f64().max(1e-9);
let scalar_mibs = total_mib / scalar_elapsed.as_secs_f64().max(1e-9);
println!(
"[microbench][utils::hash] size={size}B HighwayHash256S SIMD={simd_mibs:.1} MiB/s vs PortableScalar={scalar_mibs:.1} MiB/s (speedup={:.2}x)",
simd_mibs / scalar_mibs.max(1e-9)
);
// 2. Compare contiguous hash_encode vs 4 KiB chunked hash_encode_slices across all bitrot algorithms.
let slices: Vec<&[u8]> = payload.chunks(4096).collect();
for algo in [
HashAlgorithm::HighwayHash256S,
HashAlgorithm::HighwayHash256SLegacy,
HashAlgorithm::SHA256,
HashAlgorithm::BLAKE2b512,
HashAlgorithm::Md5,
] {
let t_contig = Instant::now();
let mut contig_out = Vec::new();
for _ in 0..iters {
contig_out = algo.hash_encode(black_box(&payload)).as_ref().to_vec();
black_box(&contig_out);
}
let contig_elapsed = t_contig.elapsed();
let t_sliced = Instant::now();
let mut sliced_out = Vec::new();
for _ in 0..iters {
sliced_out = algo.hash_encode_slices(black_box(slices.iter().copied())).as_ref().to_vec();
black_box(&sliced_out);
}
let sliced_elapsed = t_sliced.elapsed();
assert_eq!(contig_out, sliced_out, "{algo:?} contiguous vs 4KiB sliced digest mismatch at {size}B");
let contig_mibs = total_mib / contig_elapsed.as_secs_f64().max(1e-9);
let sliced_mibs = total_mib / sliced_elapsed.as_secs_f64().max(1e-9);
println!(
"[microbench][utils::hash] size={size}B algo={algo:?} contiguous={contig_mibs:.1} MiB/s vs sliced(4KiB)={sliced_mibs:.1} MiB/s"
);
}
}
// 3. Key-routing microbenchmark: crc_hash (CRC32) vs sip_hash (SipHash-1-3).
let keys: Vec<String> = (0..1024)
.map(|i| format!("bucket/prefix-{i}/object-key-{i}.parquet"))
.collect();
let t_crc = Instant::now();
let mut crc_acc = 0usize;
for key in &keys {
crc_acc ^= crc_hash(black_box(key), 16);
}
let crc_ns = t_crc.elapsed().as_nanos();
let t_sip = Instant::now();
let mut sip_acc = 0usize;
for key in &keys {
sip_acc ^= sip_hash(black_box(key), 16, &[0u8; 16]);
}
black_box(sip_acc);
let sip_ns = t_sip.elapsed().as_nanos();
assert!(black_box(crc_acc) < 16 && black_box(sip_acc) < 16);
println!("[microbench][utils::hash] 1024 object keys: crc_hash={crc_ns}ns total, sip_hash={sip_ns}ns total");
}
}
+2
View File
@@ -8,6 +8,8 @@ collection:
## Operations
- [Candidate release pipeline](operations/release-pipeline.md): frozen inputs, staged acceptance, immutable publication and recovery.
- [Sampled local batch CPU diagnostics](operations/local-batch-cpu-diagnostics.md) — default-off request-correlated allocation and worker wall/CPU diagnosis.
- [Retained MRF responsibilities](operations/mrf-responsibilities.md) — unknown bucket identities, bounded probes, and unreadable checkpoint recovery.
@@ -72,6 +72,8 @@ Ordinary publication rechecks the selected pool against the durable pool metadat
Running, queued, failed, canceled, and completed decommission states exclude the source from new ordinary publication, including new multipart uploads. Previously created multipart uploads retain their drain path while the source remains non-terminal; terminal source states reject further multipart publication. Failed and canceled entries become writable for new ordinary publication only after an allowed clear operation removes that state. This check does not change repair admission or the separate fence for operations that only release capacity.
Bucket creation and deletion remove stale bucket usage through namespace maintenance, independently of the scanner publication pause. Failed or canceled decommission state alone does not reject that cleanup. The update checks the logical snapshot revision while holding the fixed and physical object locks, then conditionally publishes to a writable pool; a suspended source remains available for decommission to reconcile against the newer target. Cleanup preserves the scanner epoch and cycle, and retains the bucket namespace and movement guards until its physical writes finish even if the request is canceled. Namespace lease loss still fences the write. Scanner publication continues to require its existing movement admission.
For mixed batch deletes, only the pools selected to receive new delete markers are publication targets. Exact-version deletions on other pools remain protected by the same pool metadata read fence, without treating the retiring source or an unrelated reserved target as a destination for those markers.
### Shared Capacity On Healthy Targets
@@ -143,6 +145,10 @@ Existing capacity-owner and mutation checks reconcile the pending intent before
source cleanup; the replica exception does not clear an unknown intent, rewrite
the native target, or change the scanner's committed-membership selection.
Backlog membership also carries an optional `set_id`, the first disk-slot UUID from the erasure set's persisted format. This identity survives pool removal, reordering, and disk replacement; `pool_index` and `set_index` remain routing locations. Readers resolve bound members by that UUID before checking the existing commit and rollback proofs. A new pool at a removed member's old index cannot impersonate that member. New publications bind every member, while positional records remain readable and gain identities only after their existing authority is verified. Replica schema 1 is retained; 1.0.4 readers ignore the additive identity field and still use positional membership.
Finish upgrading all scanner-capable nodes before changing pool membership. An older writer can replace a bound record with a positional one. If a legacy record was already stranded by pool reindexing, the missing identity cannot be guessed safely: restore the known topology in which its proof was readable, then let the updated writer bind the membership before changing it again. Do not delete or reset the backlog ledger to bypass this check. Partial writes, canceled callers, missing members, and conflicting authority remain subject to the native CAS and recovery rules above.
## MinIO Divergence Decisions
Behavior that is close to MinIO but not byte-for-byte identical. Changing either decision requires an operator compatibility note and updated characterization tests.
+1 -1
View File
@@ -130,7 +130,7 @@ These behaviors are client-visible and differ from AWS S3 or MinIO. They bound h
| Bucket tag key or value outside the AWS tag character class (`[\p{L}\p{N}\p{Z}+\-=._:/@]`) | `400 InvalidTag` on `PutBucketTagging` | AWS enforces the same class; MinIO is more permissive | AWS parity. Legacy stored tags still load. |
| `CopyObject` with the default `x-amz-tagging-directive: COPY` from a source that carries tags | `403 AccessDenied` unless the caller also holds `s3:GetObjectTagging` (or `s3:GetObjectVersionTagging`) on the source; the destination's `s3:RequestObjectTag/*` conditions are evaluated against the copied tag set | AWS and MinIO copy the tags with `s3:GetObject` alone | A copy would otherwise disclose source tags to a principal that may not read them, and source tags could satisfy destination conditions the caller cannot assert (rustfs/backlog#3034). Use `REPLACE` with an explicit `x-amz-tagging` to copy without the tagging grant. |
| `ListObjectVersions` / `ListObjectsV2` with the include-deleted extension | Requires `s3:ListBucketVersions` | Vendor extension | Delete markers and non-current versions are version-listing data. |
| `DeleteObject` of a missing key that names a prefix | `204 NoContent` when the recovery probe of the prefix fits the request budget (`DELETE_RECOVERY_MAX_DISCOVERY_ENTRIES`, 4096 entries per drive, one entry per 64 plain children); `503 SlowDown` beyond that, deterministically, and for the duration of a rolling upgrade while a drive in the set still runs a release without the bounded recovery RPC | `204 NoContent` | The probe looks for interrupted delete transactions under the prefix and is bounded so a hostile prefix cannot pin the request. Prefixes with more than 262,144 direct children per drive, or roughly 1,400 to 4,096 UUID-named children, exceed the bound (rustfs/backlog#3038 residual). |
| `DeleteObject` of a missing key that names a prefix | `204 NoContent` when the recovery probe of the prefix fits its per-drive allowance (`DELETE_RECOVERY_MAX_DISCOVERY_ENTRIES`, 262,144 entries: one per UUID-named direct child, one per 64 other names), its per-call time limit (a third of the drive timeout, at most 10 seconds) and the request budget (`DeleteRecoveryWorkBudget`, shared by every key and drive of the request); `503 SlowDown` beyond that, and for the duration of a rolling upgrade while a drive in the set still runs a release without the bounded recovery RPC | `204 NoContent` | The probe looks for interrupted delete transactions of the key and is bounded so a hostile prefix cannot pin the request. A folder completes with up to 262,144 UUID-named direct children or about 16.7 million other direct children per drive when the drive can walk them within the time limit (a warm cache walks roughly 25,000 UUID-named children per second; a cold HDD far fewer); inside a UUID-named child only journal files and data-directory markers are examined, so a child key's own `xl.meta`, data directories or children cost 1/64 each. |
| `PutObject` / `UploadPart` body delivering less than 64 KiB per 60 seconds of server wait | `400 RequestTimeout` (the minimum-progress floor, logged as `upload_body_progress_stalled`); independent of `RUSTFS_HTTP_REQUEST_BODY_READ_TIMEOUT`, including when that timeout is 0 | No minimum upload rate | A trickle upload otherwise pins upload admission and its buffers indefinitely; roughly 1.1 KiB/s is below any practical client. |
| First write after `SetBucketQuota`, or after the quota changes, on a bucket without an authoritative usage ledger | `503 SlowDown` (retryable) until a background scan of the bucket publishes the exact ledger; a failed scan retries after 30 seconds | AWS has no bucket quota; MinIO admits with unknown usage | Quota admission uses only the authoritative ledger (scanner reporting is informational, see `reporting_retirement` tests), and the request must not hold its bucket metadata guard across a whole-bucket listing. The window is one bucket listing long. |
+79
View File
@@ -0,0 +1,79 @@
# Candidate acceptance and release publication
RustFS builds the final product version before publication. A candidate is identified by its Actions run and `<version>-preview.<run_number>`; this identifier is not a Git tag or the executable's displayed version. Failed candidates consume no public version. A public tag is created only after the complete artifact manifest is approved.
## Entry points
| Workflow | Input | Result | Public writes |
|---|---|---|---|
| `release-candidate.yml` | Exact final version and full source SHA | Four Linux archives, native checks, upgrade results, Linux manifest | None |
| `release-complete.yml` | Successful candidate run, accepted Linux manifest SHA-256, backlog acceptance report | Desktop archives, DEB/RPM, OCI images, Helm chart, full certification and sealed manifest | None |
| `release-publish.yml` | Successful completion run and approved final manifest SHA-256 | Annotated tag, GitHub Release, R2 files, registry images, stable pointers and Helm index | Explicit publication |
The builder workflows (`build.yml`, `package.yml`, `docker.yml`, `helm-package.yml`) are reusable stages. Only `release-publish.yml` has release write credentials. Development builds retain their artifact names and download channel. Daily development builds now select Linux only; macOS/Windows development artifacts are refreshed by an explicit manual build, and both remain mandatory for a completed release candidate. Tag pushes no longer start a second compilation.
## Freeze and build
Select a reviewed source commit whose Cargo version equals the intended final version. Finish exact-source CI before dispatching the candidate. The candidate workflow itself must run from upstream main. Unrelated main commits do not require rebuilding an accepted source.
```bash
gh workflow run release-candidate.yml --ref main \
-f version='<version>' -f source_sha='<full-source-sha>'
```
The initial job records the source and workflow SHAs, release recipe, dependency lock, exact stable Rust version, Console tag/asset/SHA-256, CLI assets, previous stable release and validation plan. All platforms consume that record; none independently follows `latest`. The source commit must contain the same reviewed release recipe as the dispatcher. Relevant recipe changes require a new candidate; unrelated source changes on main do not invalidate the frozen record.
Linux is built first. Each archived executable must display the final version, expected target, source commit and compiler version, then start natively, serve the Console, and pass an authenticated S3 bucket/object round trip. The upgrade lane runs the existing pinned compatibility matrix plus the stable version frozen at candidate creation. It compiles the harness and runs the accepted executable; it does not rebuild a substitute server. A zero-test invocation fails.
The Linux manifest is partial evidence and cannot authorize publication. Download the exact artifacts for field acceptance. Record the source, manifest digest, run URL and actual tested binary digests in backlog. Required field acceptance includes Scanner/Heal and multi-pool behavior, major features, memory, high-concurrency 503s, performance against the agreed baseline, previous-version data integrity and upgrade recovery. The agreed stability duration is a release-specific requirement, not an implicit workflow default.
## Complete the accepted candidate
After the Linux report passes, dispatch `release-complete.yml` with the exact digest printed by the candidate run. The `release-linux-acceptance` environment is the operator gate. Configure its protection and allowed main branch in repository settings; a typed digest and report identify what is being accepted, while the workflow independently revalidates automated evidence.
```bash
gh workflow run release-complete.yml --ref main \
-f candidate_run_id='<run-id>' \
-f linux_manifest_sha256='<accepted-linux-manifest-sha256>' \
-f acceptance_report='https://github.com/rustfs/backlog/issues/<issue>#issuecomment-<comment>'
```
macOS and Windows now build and run the same native checks. Packages and images wrap the accepted Linux binaries. DEB/RPM extraction must reproduce the recorded binary digest. Both OCI variants are checked on amd64 and arm64 and scanned before sealing. Vulnerability scans retain the repository's report-only policy; scanner execution errors still fail the stage. Image OS packages are resolved during packaging, then the actual resulting OCI bytes are accepted and preserved; this is not a promise of reproducible package-manager output.
The Helm chart is tested and packaged before sealing, with chart and application versions equal to the final product version. Full certification requires every platform and upgrade case from the original plan. The seal includes artifact hashes, raw evidence, source/input identities, acceptance report, checksums, SBOM and provenance. Successful Linux receipts remain valid while desktop results are added because each receipt binds its own exact archive and executable, the frozen plan, candidate identity and trusted producer.
Native smoke checks are not a substitute for Console UI and rc acceptance. Complete the release skill's UI/client checks against these exact artifacts, including desktop checks applicable to the release, before asking for final publication approval.
## Approve and publish
Review the complete run and final `release-manifest.json`. Human approval binds its SHA-256, source and version. A changed manifest requires renewed approval. No source version bump, compilation, package generation or image build occurs during publication.
```bash
gh workflow run release-publish.yml --ref main \
-f complete_run_id='<successful-completion-run-id>' \
-f manifest_sha256='<approved-final-manifest-sha256>'
```
The publisher downloads the artifact from the exact successful main workflow run, checks GitHub's artifact digest, recomputes file hashes and reruns complete certification. It creates an annotated tag whose message binds the source and manifest. An existing tag or asset with different bytes is a hard failure; tags are never moved and uploads never clobber assets.
Versioned GitHub assets, R2 objects and Docker Hub/GHCR/Quay manifests are uploaded and read back before the Release becomes public. OCI promotion uses `--preserve-digests`; registry conversion fails rather than silently accepting a new image. R2 creation uses atomic `If-None-Match: *`. The small evidence bundle persists with the public Release after Actions artifacts expire. Provenance is descriptive, not a claim of cryptographic attestation.
Publication uses a per-version lock and a shared channel lock, both without cancelling an active run. Stable channels advance only after versioned delivery succeeds. GitHub Latest acts as a monotonic marker during partial channel updates, so an older retry cannot downgrade Docker tags, R2 aliases or installer pointers. Prereleases do not advance stable channels. Helm publishes the already accepted chart after versioned images exist. GitHub's legacy `*-latest.zip` download names remain aliases of the accepted bytes; they become default downloads only when GitHub Latest advances.
Cross-registry publication is not atomic. A channel job can fail after some verified destinations advance. Resume the same run/seal: matching destinations are reused and conflicting ones stop the workflow. Retrying never recompiles. Do not claim complete publication until the channel receipt and all public endpoints are verified.
## Failure and retention
- Infrastructure failure: rerun failed jobs in the same run. Frozen inputs, successful platform/package/image artifacts and completed seals are reused. Missing or expired artifacts fail closed.
- Product or build-input change before tagging: create a new candidate run and repeat affected acceptance. Do not mutate the old candidate.
- Failure after tagging: resume publication of exactly that seal. A code fix requires a new patch version; never move the tag.
- New main commits: retain evidence for unchanged source, recipe and inputs. Recheck only affected boundaries; do not restart the entire matrix because an unrelated PR merged.
- Artifacts are retained for 30 days while awaiting acceptance. Public-repository Actions artifacts and staging identifiers are not confidential storage. Preserve important failure evidence in task-owned storage and backlog.
## Migration and workflow maintenance
Finish any in-flight legacy tag release, including its downstream Docker/package/Helm jobs, before enabling this cutover on main. Existing tags and Releases retain their identities. Do not feed a legacy public preview binary into the new promotion path: it embeds a different version and was not built from the frozen-input contract.
All workflow responsibilities, triggers and schedule ownership remain registered in `.github/scheduled-validations.json`. The migration does not re-enable workflows disabled in GitHub. PR checks, cache warming, nightly development builds, functional chains, lab tests and monitoring retain their distinct purposes; shared source or cache does not make different feature/platform tests redundant. The watchdog allocates a runner only for relevant unsuccessful scheduled/chained runs, while its registry still determines ownership.
Run `actionlint`, the release contract suite and the workflow registry validator after changes. The Security Audit workflow also exercises actual GitHub artifact transport, DEB/RPM package extraction, both multi-platform OCI builders, local registry promotion and archive scanning. Native packaging jobs run only for relevant changes or a manual audit. Their payloads are explicit transport fixtures, not product release acceptance. See the [release skill](../../.agents/skills/rustfs-release-publish/SKILL.md) for field acceptance, human confirmation and post-release follow-up.
+1
View File
@@ -101,6 +101,7 @@ Scheduled lanes never block a PR. Their workflow-local gate fails the run. The v
| `coverage.yml` (weekly) | `coverage` | report-only trend; lcov and JSON artifact | yes (disabled) | `make coverage` |
| `runner-hygiene.yml` (monthly) | `check-ephemerality` | runner ephemerality | yes | dispatch |
| `e2e-upgrade.yml` (weekly, and on version tags) | `upgrade` (serial cases sharing one harness build) | upgrade and rollback gate; server logs | yes | `python3 scripts/e2e_binary.py build`, then the job's `python3 scripts/e2e_binary.py run -- cargo test --locked -p e2e_test` commands with `RUSTFS_UPGRADE_SOURCE_BINARY` pointing at the pinned previous release (`UPGRADE_SOURCE_VERSION`) |
| `reference-parity.yml` (nightly, and on PRs that change the lane's own files) | `plan`, `build-candidate`, `s3tests` (one job per target in `.config/reference-parity-targets.toml`), `compare` | report-only parity gate: fails on any per-node-id s3-tests difference between a branch candidate and the release pinned in `.config/reference-binary.toml` that `.config/reference-parity-tolerated.txt` does not list, and on a stale tolerated entry; per-target JUnit, logs and `diff.md`/`diff.json` | yes | `DEPLOY_MODE=binary RUSTFS_BINARY=<binary> MAXFAIL=0 XDIST=4 ./scripts/s3-tests/run.sh` once per target, then `python3 scripts/s3-tests/reference_parity.py compare` as in the `compare` job |
| `oidc-keycloak.yml` (weekly) | `oidc-keycloak-live` | live OIDC gate | yes | see the PR row |
| `targets-integration.yml` (nightly) | `targets-live` | live target gate; container logs | yes (disabled) | see the PR row |
| `gateway-integration-sync.yml` (weekdays) | `sync` | merges `main` into `gateway/integration` as a `--no-ff` merge commit; a conflicting merge is aborted and opens a `sync/<source>-<date>` pull request listing the conflicting files, which keeps the run green; a rejected push or missing pull request fails it | yes | dispatch, optionally with `source_ref` |
+2
View File
@@ -8,6 +8,8 @@ The classifier reads the pull request base revision's planner and policy. It com
The committed policy is `enforced`: ordinary Rust changes select their reverse dependency closure plus the fixed core inventory, and unrelated changes can omit the Connect specialization. Package-owned fixtures, non-Rust inputs, build scripts, manifests, unknown paths and non-PR events retain full validation. Existing documentation and isolated E2E scopes remain conservative. Changing a PR's own policy cannot authorize skipping its base revision's requirements. A base planner predating the current workflow contract clears all narrower outputs and selects the full matrix. Set the policy back to `shadow` to retain proposed selections while restoring full ordinary-code execution; do not remove core inventories or negative checks as a rollback.
Stable installation-version alignment has one content-based exception in enforced mode: literal version fields in `flake.nix`, `helm/rustfs/Chart.yaml` and `rustfs.spec`, plus a standard prepended RPM release note, can use the documentation scope. Quick Checks, Typos, CLA and the required aggregate remain. The planner compares both complete Git blobs and requires ordinary non-executable files; changes to build logic, CLI pins, chart settings, RPM macros or older changelog entries retain the full gate. New/deleted files, mode changes, prerelease syntax, unavailable history, shadow rollout and explicit full-path policy overrides also retain full validation. Product-code changes in the same PR retain their normal dependency and core coverage, and main pushes, merge groups, scheduled and manual runs still validate the complete candidate. This exception does not establish artifact availability or authorize a release; post-release checks still verify the published target before updating defaults.
`Test and Lint` remains the required aggregate name. Every verification producer and consumer is a direct dependency. The verifier accepts only the effective plan's exact expected success/skipped outcomes; failure, cancellation, missing results, extra jobs or a plan for a different candidate fail. Quick Checks and CLA retain their separate required names.
## Build and execute
+1 -1
View File
@@ -77,7 +77,7 @@
rustfs = rustPlatform.buildRustPackage {
pname = "rustfs";
version = "1.0.4";
version = "1.0.5";
src = ./.;
+2 -2
View File
@@ -2,8 +2,8 @@ apiVersion: v2
name: rustfs
description: RustFS helm chart to deploy RustFS on kubernetes cluster.
type: application
version: "1.0.4"
appVersion: "1.0.4"
version: "1.0.5"
appVersion: "1.0.5"
home: https://rustfs.com
icon: https://media.sys.truenas.net/apps/rustfs/icons/icon.svg
maintainers:
+4 -1
View File
@@ -1,7 +1,7 @@
%global _enable_debug_packages 0
%global _empty_manifest_terminate_build 0
Name: rustfs
Version: 1.0.4
Version: 1.0.5
Release: 1
Summary: High-performance distributed object storage for MinIO alternative
@@ -57,6 +57,9 @@ install %_builddir/%{name}-%{version}/target/%_arch/%_arch-unknown-linux-gnu/rel
%_bindir/rustfs
%changelog
* Wed Nov 11 2026 overtrue <anzhengchao@gmail.com>
- Update RPM package to RustFS 1.0.5
* Mon Oct 26 2026 overtrue <anzhengchao@gmail.com>
- Update RPM package to RustFS 1.0.4
+96 -27
View File
@@ -254,11 +254,15 @@ impl DefaultAdminUsecase {
/// absent from `buckets_usage` — absent already means "unknown" on the
/// wire, distinct from a present zero — and the snapshot is marked
/// non-converged so clients can tell it is not the whole namespace. A
/// snapshot that is structurally incomplete, or that measured nothing the
/// namespace still contains, carries no usable information and is dropped
/// exactly as before.
fn narrow_data_usage_snapshot_to_measured_buckets(info: &mut DataUsageInfo, buckets: impl IntoIterator<Item = String>) {
if !info.is_complete_bucket_usage_snapshot() && !info.is_valid_partial_snapshot() {
/// snapshot that is structurally incomplete (`snapshot_measured` is false),
/// or that measured nothing the namespace still contains, carries no usable
/// information and is dropped exactly as before.
fn narrow_data_usage_snapshot_to_measured_buckets(
info: &mut DataUsageInfo,
snapshot_measured: bool,
buckets: impl IntoIterator<Item = String>,
) {
if !snapshot_measured {
*info = DataUsageInfo::default();
return;
}
@@ -275,13 +279,31 @@ impl DefaultAdminUsecase {
let covers_namespace = current_buckets.len() == info.buckets_usage.len();
info.buckets_count = info.buckets_usage.len() as u64;
if !covers_namespace {
// Some bucket in the namespace was never measured, so the totals
// below are a floor rather than the converged truth.
if !covers_namespace || !info.is_complete_bucket_usage_snapshot() {
// Some bucket in the namespace was never measured, or holds
// mutations no scan has reconciled yet, so the totals below are
// not the converged truth.
info.usage_snapshot_converged = Some(false);
}
}
/// Overlay single-process counts on a persisted snapshot and narrow it
/// for display.
///
/// Whether the snapshot measured anything is decided before the overlay.
/// The overlay clears `usage_snapshot_complete` when it holds counts it
/// cannot reconcile with the snapshot, such as writes into a bucket that
/// has no verified baseline. That makes the totals unconverged, not the
/// scanned buckets unusable; treating it as structural incompleteness
/// blanked usage for every bucket for as long as one such bucket kept
/// receiving writes, because each write restarts its scanner fence.
async fn present_data_usage_snapshot(mut info: DataUsageInfo, buckets: impl IntoIterator<Item = String>) -> DataUsageInfo {
let snapshot_measured = info.is_complete_bucket_usage_snapshot() || info.is_valid_partial_snapshot();
apply_bucket_usage_memory_overlay(&mut info).await;
Self::narrow_data_usage_snapshot_to_measured_buckets(&mut info, snapshot_measured, buckets);
info
}
async fn rebalance_status_snapshot(store: &ECStore) -> AdminUsecaseResult<Option<RebalanceMeta>> {
store.rebalance_status_meta_snapshot().await.map_err(|err| {
error!("refresh rebalance metadata for pool status failed: {:?}", err);
@@ -323,8 +345,7 @@ impl DefaultAdminUsecase {
/// This path never triggers a live full-version listing
/// (rustfs/backlog#1306); freshness is owned by the scanner.
pub(crate) async fn query_data_usage_info_with_store(store: Arc<ECStore>) -> AdminUsecaseResult<DataUsageInfo> {
let mut info = Self::map_data_usage_load_result(load_admin_data_usage_from_backend_cached(store.clone()).await)?;
apply_bucket_usage_memory_overlay(&mut info).await;
let info = Self::map_data_usage_load_result(load_admin_data_usage_from_backend_cached(store.clone()).await)?;
let buckets = store
.list_bucket(&BucketOptions {
cached: true,
@@ -333,11 +354,7 @@ impl DefaultAdminUsecase {
})
.await
.map_err(|_| Self::app_error(S3ErrorCode::InternalError, "list_bucket failed"))?;
// Serve what the scan measured; buckets it never reached stay absent
// (absent means unknown on the wire, unlike a present zero) and the
// snapshot is flagged non-converged. See rustfs#5806 for why blanking
// the whole document instead was worse than the rule it enforced.
Self::narrow_data_usage_snapshot_to_measured_buckets(&mut info, buckets.into_iter().map(|bucket| bucket.name));
let mut info = Self::present_data_usage_snapshot(info, buckets.into_iter().map(|bucket| bucket.name)).await;
let storage_info = StorageAdminApi::storage_info(store.as_ref()).await;
@@ -876,6 +893,12 @@ mod tests {
});
}
/// Narrow a snapshot no overlay has touched, judging it as persisted.
fn narrow_persisted_snapshot(info: &mut DataUsageInfo, buckets: impl IntoIterator<Item = String>) {
let snapshot_measured = info.is_complete_bucket_usage_snapshot() || info.is_valid_partial_snapshot();
DefaultAdminUsecase::narrow_data_usage_snapshot_to_measured_buckets(info, snapshot_measured, buckets);
}
/// rustfs#5806: a bucket the last scan never reached must not blank the
/// usage the scan *did* measure — that turned every bucket creation into a
/// deployment-wide usage blackout until the next cycle.
@@ -897,10 +920,7 @@ mod tests {
// A brand-new bucket the scan never saw: keep bucket-a, omit bucket-new,
// and say so via the convergence flag.
let mut info = measured("bucket-a");
DefaultAdminUsecase::narrow_data_usage_snapshot_to_measured_buckets(
&mut info,
["bucket-a".to_string(), "bucket-new".to_string(), ".rustfs.sys".to_string()],
);
narrow_persisted_snapshot(&mut info, ["bucket-a".to_string(), "bucket-new".to_string(), ".rustfs.sys".to_string()]);
assert!(info.buckets_usage.contains_key("bucket-a"), "measured usage must survive");
assert!(!info.buckets_usage.contains_key("bucket-new"), "unmeasured bucket stays absent");
assert_eq!(info.usage_snapshot_converged, Some(false));
@@ -909,22 +929,19 @@ mod tests {
// Full coverage leaves the snapshot untouched.
let mut info = measured("bucket-a");
DefaultAdminUsecase::narrow_data_usage_snapshot_to_measured_buckets(
&mut info,
["bucket-a".to_string(), ".rustfs.sys".to_string()],
);
narrow_persisted_snapshot(&mut info, ["bucket-a".to_string(), ".rustfs.sys".to_string()]);
assert_eq!(info.usage_snapshot_converged, None, "a covering snapshot is not downgraded");
assert_eq!(info.buckets_count, 1);
// A bucket deleted since the scan must not linger in the response.
let mut info = measured("bucket-gone");
DefaultAdminUsecase::narrow_data_usage_snapshot_to_measured_buckets(&mut info, ["bucket-live".to_string()]);
narrow_persisted_snapshot(&mut info, ["bucket-live".to_string()]);
assert_eq!(info, DataUsageInfo::default(), "nothing measured is still in the namespace");
// Structurally incomplete snapshots carry no usable information.
let mut info = measured("bucket-a");
info.usage_snapshot_complete = false;
DefaultAdminUsecase::narrow_data_usage_snapshot_to_measured_buckets(&mut info, ["bucket-a".to_string()]);
narrow_persisted_snapshot(&mut info, ["bucket-a".to_string()]);
assert_eq!(info, DataUsageInfo::default());
// A structurally valid partial admin view can still carry useful
@@ -945,7 +962,7 @@ mod tests {
tombstone: false,
segment_invalidation_proof: None,
}];
DefaultAdminUsecase::narrow_data_usage_snapshot_to_measured_buckets(&mut info, ["bucket-a".to_string()]);
narrow_persisted_snapshot(&mut info, ["bucket-a".to_string()]);
assert_eq!(info.usage_snapshot_converged, Some(false));
assert_eq!(info.buckets_count, 1);
assert_eq!(info.objects_total_count, 7);
@@ -957,11 +974,63 @@ mod tests {
buckets_count: 0,
..Default::default()
};
DefaultAdminUsecase::narrow_data_usage_snapshot_to_measured_buckets(&mut info, [".rustfs.sys".to_string()]);
narrow_persisted_snapshot(&mut info, [".rustfs.sys".to_string()]);
assert_eq!(info.usage_snapshot_converged, None);
assert_eq!(info.buckets_count, 0);
}
/// A write into a bucket without a verified baseline leaves process-local
/// counts the overlay cannot reconcile, so it clears
/// `usage_snapshot_complete`. That must flag the response non-converged,
/// not erase every bucket the persisted scan measured: under continuous
/// writes the scanner fence never clears and the blackout never ends.
#[tokio::test]
async fn unverified_bucket_writes_do_not_blank_measured_usage() {
let measured_bucket = "admin-usage-measured";
let unverified_bucket = "admin-usage-unverified";
let mut persisted = DataUsageInfo {
last_update: Some(std::time::SystemTime::now() - std::time::Duration::from_secs(60)),
usage_snapshot_complete: true,
buckets_count: 1,
..Default::default()
};
persisted.buckets_usage.insert(
measured_bucket.to_string(),
rustfs_data_usage::BucketUsageInfo {
objects_count: 4_000,
versions_count: 4_000,
size: 8_192_000,
..Default::default()
},
);
persisted.bucket_sizes.insert(measured_bucket.to_string(), 8_192_000);
persisted.calculate_totals();
assert!(persisted.is_complete_bucket_usage_snapshot());
crate::app::storage_api::test::data_usage::record_bucket_object_write_memory(unverified_bucket, None, 1_024).await;
let info = DefaultAdminUsecase::present_data_usage_snapshot(
persisted,
[measured_bucket.to_string(), unverified_bucket.to_string()],
)
.await;
assert_eq!(
info.buckets_usage
.get(measured_bucket)
.map(|usage| (usage.objects_count, usage.size)),
Some((4_000, 8_192_000)),
"usage the scanner measured must survive an unrelated unverified bucket"
);
assert!(
!info.buckets_usage.contains_key(unverified_bucket),
"a bucket without a verified baseline stays absent (unknown)"
);
assert_eq!(info.usage_snapshot_converged, Some(false));
assert!(!info.is_complete_bucket_usage_snapshot());
assert_eq!(info.objects_total_count, 4_000);
}
#[tokio::test]
async fn execute_query_storage_info_returns_internal_error_when_store_uninitialized() {
let usecase = DefaultAdminUsecase::without_context();
+83 -12
View File
@@ -17,6 +17,7 @@
use super::*;
use crate::app::storage_api::object_usecase::DeleteObjectCondition;
use s3s::dto::ObjectIdentifier;
use std::borrow::Cow;
use std::collections::HashSet;
async fn authorize_recursive_delete<T>(
@@ -81,6 +82,19 @@ async fn authorize_recursive_delete<T>(
result
}
// XML body keys bypass s3s's URL-path normalization. Match its private slash
// normalizer without URL-decoding or resolving dot segments in client keys.
fn normalize_delete_object_key(key: &str) -> Cow<'_, str> {
if !key.starts_with('/') {
return Cow::Borrowed(key);
}
let mut normalized = key.split('/').filter(|part| !part.is_empty()).collect::<Vec<_>>().join("/");
if key.ends_with('/') {
normalized.push('/');
}
Cow::Owned(normalized)
}
fn successful_delete_audit_objects(
delete: &s3s::dto::Delete,
successful_results: impl IntoIterator<Item = bool>,
@@ -90,7 +104,9 @@ fn successful_delete_audit_objects(
.iter()
.zip(successful_results)
.filter(|(_, successful)| *successful)
.map(|(requested, _)| AuditObjectVersion::new(requested.key.clone(), requested.version_id.clone()))
.map(|(requested, _)| {
AuditObjectVersion::new(normalize_delete_object_key(&requested.key).into_owned(), requested.version_id.clone())
})
.collect()
}
@@ -590,7 +606,10 @@ impl DefaultObjectUsecase {
// Per-key denials are audited once per denied action, in aggregate, after the loop.
let mut bulk_denials = crate::storage::access::BulkDenialAudit::start(&mut req)?;
for (idx, obj_id) in delete.objects.iter().enumerate() {
if let Err(err) = validate_object_key(&obj_id.key, "DeleteObjects") {
// Authorization and every storage side effect must address the same
// normalized object; retain the request key only for response entries.
let key = normalize_delete_object_key(&obj_id.key);
if let Err(err) = validate_object_key(&key, "DeleteObjects") {
delete_results[idx].error = Some(s3s::dto::Error {
code: Some(err.code().as_str().to_string()),
key: Some(obj_id.key.clone()),
@@ -623,7 +642,7 @@ impl DefaultObjectUsecase {
{
let req_info = req_info_mut(&mut req)?;
req_info.bucket = Some(bucket.clone());
req_info.object = Some(obj_id.key.clone());
req_info.object = Some(key.to_string());
req_info.version_id = version_id.clone();
}
@@ -670,7 +689,7 @@ impl DefaultObjectUsecase {
}
}
if let Err(err) = validate_table_catalog_object_mutation(&bucket, &obj_id.key).await {
if let Err(err) = validate_table_catalog_object_mutation(&bucket, &key).await {
delete_results[idx].error = Some(s3s::dto::Error {
code: Some("InvalidRequest".to_string()),
key: Some(obj_id.key.clone()),
@@ -680,9 +699,9 @@ impl DefaultObjectUsecase {
continue;
}
let synthetic_version_id = delete_condition.is_none() && version_id.is_none() && is_dir_object(&obj_id.key);
let synthetic_version_id = delete_condition.is_none() && version_id.is_none() && is_dir_object(&key);
let object = ObjectToDelete {
object_name: obj_id.key.clone(),
object_name: key.into_owned(),
version_id: version_uuid,
delete_condition,
synthetic_version_id,
@@ -910,15 +929,16 @@ impl DefaultObjectUsecase {
let deleted = (!delete.quiet.unwrap_or_default()).then(|| {
delete_results
.iter()
.filter_map(|result| result.delete_object.as_ref().map(|object| (result, object)))
.map(|(result, object)| DeletedObject {
.enumerate()
.filter_map(|(idx, result)| result.delete_object.as_ref().map(|object| (idx, result, object)))
.map(|(idx, result, object)| DeletedObject {
delete_marker: { if object.delete_marker { Some(true) } else { None } },
delete_marker_version_id: delete_marker_response_version_id(
object.delete_marker_version_id,
object.delete_marker && result.version_suspended && result.requested_version_id.is_none(),
result.synthetic_version_id,
),
key: Some(object.object_name.clone()),
key: Some(delete.objects[idx].key.clone()),
version_id: delete_entry_response_version_id(
object.version_id,
object.delete_marker,
@@ -936,7 +956,12 @@ impl DefaultObjectUsecase {
let errors = delete_results
.iter()
.filter_map(|v| v.error.clone())
.enumerate()
.filter_map(|(idx, result)| {
let mut error = result.error.clone()?;
error.key = Some(delete.objects[idx].key.clone());
Some(error)
})
.collect::<Vec<s3s::dto::Error>>();
let output = DeleteObjectsOutput {
deleted,
@@ -2819,11 +2844,57 @@ mod tests {
assert_eq!(err.message(), Some("Not init"));
}
#[test]
fn delete_objects_key_normalization_matches_s3s_path_keys() {
use s3s::path::{
parse_path_style_with_validation_and_normalization, parse_virtual_hosted_style_with_validation_and_normalization,
};
use s3s::validation::AwsNameValidation;
for key in [
"plain",
"/a/b",
"//a/b",
"/dir///sub//file",
"/dir///",
"/",
"///",
"dir//file",
"/literal%2F+名",
"/dir/../file",
"/dir/./file",
"/dir\\file",
"/a\0b",
] {
let normalized = normalize_delete_object_key(key);
let path = parse_path_style_with_validation_and_normalization(
&format!("/test-bucket/{key}"),
&AwsNameValidation::new(),
true,
)
.expect("path-style key should parse before storage validation");
let virtual_host = parse_virtual_hosted_style_with_validation_and_normalization(
Some("test-bucket"),
&format!("/{key}"),
&AwsNameValidation::new(),
true,
)
.expect("virtual-hosted key should parse before storage validation");
assert_eq!(Some(normalized.as_ref()), path.get_object_key(), "path-style mismatch for {key:?}");
assert_eq!(
Some(normalized.as_ref()),
virtual_host.get_object_key(),
"virtual-hosted mismatch for {key:?}"
);
}
assert_eq!(normalize_delete_object_key(""), "");
}
#[test]
fn delete_objects_audit_details_include_only_successful_request_entries() {
let requested = vec![
ObjectIdentifier {
key: "first-key".to_string(),
key: "/first-key".to_string(),
version_id: None,
..Default::default()
},
@@ -2833,7 +2904,7 @@ mod tests {
..Default::default()
},
ObjectIdentifier {
key: "versioned-key".to_string(),
key: "//versioned-key".to_string(),
version_id: Some("requested-version".to_string()),
..Default::default()
},
+2 -2
View File
@@ -14460,8 +14460,8 @@ fn object_mutation_entrypoints_call_reserved_prefix_guard() {
assert_ordered(
method(delete, "fn execute_delete_objects_owned("),
&[
"validate_object_key(&obj_id.key, \"DeleteObjects\")",
"if let Err(err) = validate_table_catalog_object_mutation(&bucket, &obj_id.key).await {",
"validate_object_key(&key, \"DeleteObjects\")",
"if let Err(err) = validate_table_catalog_object_mutation(&bucket, &key).await {",
"continue;\n }",
"authorized_deletes.push(AuthorizedDelete { idx, object });",
],
+8 -3
View File
@@ -37,11 +37,11 @@ their issue closes.
| `check_no_planning_docs.sh` | ci-gate | Blocks committed planning-type documents | ci.yml Quick Checks; `make pre-commit` |
| `check_no_tokio_io_uring.sh` | ci-gate | Keeps tokio's io-uring backend disabled | ci.yml Quick Checks |
| `check_s3s_footprint.sh` | ci-gate | Exact, lower-only baseline on the direct s3s surface ahead of the gateway migration (reads `.config/s3s-footprint-baseline.txt`; `s3_error_lines` leaves out lines whose file imports the macro from `rustfs-s3-types`, rule in the script header); `--mode allowlist` fails on any Rust file referencing s3s outside `.config/s3s-edge-allowlist.txt`, `--dry-run` lists them | ci.yml Quick Checks; `make pre-commit`; `make script-tests` (`--self-test`) |
| `check_test_count_ratchet.sh` | ci-gate | Exact per-crate test-count and `#[ignore]` ratchet against `.config/test-count-baseline.txt`; no crate may lose tests or gain ignores | ci.yml Quick Checks; `make pre-commit` |
| `check_test_count_ratchet.sh` | ci-gate | Per-crate test-count floor and `#[ignore]` ceiling against `.config/test-count-baseline.txt`; no crate may fall below its tests row or rise above its ignored row (extra tests and new crates pass with a note) | ci.yml Quick Checks; `make pre-commit` |
| `check_unsafe_code_allowances.sh` | ci-gate | Unsafe-code allowance ledger guard | ci.yml Quick Checks |
| `layer-dependency-baseline.txt` | ci-gate (data) | Committed baseline consumed by `check_layer_dependencies.sh` | [Architecture guard troubleshooting](../docs/operations/architecture-guard-troubleshooting.md) |
| `static.sh` | ci-gate | Static-build helper executed inside image builds | `Dockerfile.source`, `Dockerfile.decommission-local` |
| `helm_chart_version.sh` | ci-gate | Keeps the Helm chart version in sync with the release | helm-package.yml |
| `helm_chart_version.sh` | dev-tool | Normalizes chart/app versions for installation references | rustfs-release-version-bump skill |
| `test_helm_templates.sh` | ci-gate | Helm template rendering test | helm-package.yml |
## Test & e2e runners
@@ -55,6 +55,11 @@ their issue closes.
| `test_prepare_replacement_migration.py` | dev-tool | Verifies maintenance approval scope, publication, and stopped-writer assertion | Python unittest; same runbook |
| `test_diagnose_scanner_enumeration_restart.py` | dev-tool | Driver report validation and positive convergence oracle tests | Python unittest; same guide |
| `test_object_generation_protocol_model.py` | dev-tool | Bounded single-slot, four-voter promise/accept design model and recovery/retirement examples; no production runtime evidence | `python3 scripts/test_object_generation_protocol_model.py`; [Generation contract](../docs/architecture/unified-object-generation.md#executable-design-model-and-acceptance-mapping) |
| `junit_diff.py` | dev-tool | Case-by-case pass/fail/skip diff of two JUnit XML reports (reference vs candidate stack); exit 1 on any difference or stale `--allow` entry, 2 on unreadable input | `python3 scripts/junit_diff.py a.xml b.xml`; backlog T3.1 |
| `test_junit_diff.py` | ci-gate | Contract tests and fixtures (`scripts/tests/junit_diff/`) for `junit_diff.py` | `make script-tests` |
| `s3-tests/reference_parity.py` | ci-gate | Plans the reference parity targets, lists a run's failures for the single rerun, and compares every candidate's s3-tests JUnit against the pinned reference per node id with a tolerated-difference file | `.github/workflows/reference-parity.yml`; backlog T3.2 |
| `s3-tests/test_reference_parity.py` | ci-gate | Contract tests for `reference_parity.py` | `make script-tests` |
| `test_reference_binary.py` | ci-gate | Contract tests for `.github/actions/reference-binary/reference_binary.py` (pin resolution and verified unpacking) | `make script-tests` |
| `e2e-run.sh` | ci-gate | Boots a rustfs server and runs the `s3s-e2e` black-box conformance tool against it | ci.yml `e2e-tests` jobs; `docs/testing/README.md` |
| `run_ecstore_validation_suite.sh` | dev-tool | ecstore black-box validation suite (`quick`/`full`/`destructive`/`fuzz` profiles) | `docs/testing/README.md`, `docs/testing/ecstore-validation-suite-design.md` |
| `run_e2e_tests.sh` | dev-tool | Local `e2e_test` crate runner (starts a server, applies filters, cleans up) | `crates/e2e_test/README.md` |
@@ -164,7 +169,7 @@ their issue closes.
|---|---|---|---|
| `codemods/` | dev-tool | Scripted, idempotent source migrations for the gateway migration (rustfs/backlog#2734); `s3_error_to_s3_types.py` moves s3s error-type paths onto `rustfs-s3-types` per crate closure and refuses the s3s edge allowlist | `make script-tests` (`--self-test`); usage in each script's docstring |
| `fuzz/` | ci-gate | Unified cargo-fuzz runner and helpers for the `fuzz/` sub-workspace | fuzz.yml; `fuzz/README.md` |
| `release/` | ci-gate | Release creation and DEB/RPM version-normalization helpers | build.yml; package.yml |
| `release/` | ci-gate | Release creation and DEB/RPM version-normalization helpers | build.yml; package.yml; manual-main-packages.yml |
| `s3-tests/` | ci-gate | ceph/s3-tests compatibility harness (allow-lists, patches, report tooling) | ci.yml; e2e-s3tests.yml; `scripts/s3-tests/README.md` |
| `security/` | ci-gate | Workflow-pin enforcement and release supply-chain asset generation | audit.yml; build.yml |
| `table-catalog/` | dev-tool | S3-Tables / pyiceberg validation suite | `docs/architecture/s3-tables-support-matrix.md` |
+44 -24
View File
@@ -3,7 +3,8 @@
# section 1 rule 3; task T0.3 in rustfs/backlog#2738).
#
# No PR may lower the number of tests in any workspace crate, and no PR may
# add #[ignore]: "delete the test to get green" is rejected in CI. The counter
# add #[ignore] without a reviewed baseline change: "delete the test to get
# green" is rejected in CI. The counter
# is deliberately per crate, never a workspace total — a drop in one crate
# must not hide behind an addition in another.
#
@@ -25,26 +26,37 @@
# doc comments that mention `#[test]` out of the count.
#
# The counts are compared row by row with .config/test-count-baseline.txt
# (`<package>\t<tests>\t<ignored>`, one row per member). The baseline is an
# EXACT value, not a floor, in both columns:
# (`<package>\t<tests>\t<ignored>`, one row per member). The baseline is a
# FLOOR on tests and a CEILING on ignored. It used to be an exact value in both
# columns, which made every PR that adds a test edit the same file; on a branch
# that takes many commits a week those PRs conflict with one another. What the
# ratchet protects is that nobody deletes or ignores tests to get green without
# it showing in the diff, and that only needs a one-sided bound:
#
# - tests below the row fails: `crate <name>: <actual> < baseline <n>`.
# Restore the tests. If the removal is intentional, lower the row in the
# same PR and justify it in the PR description, so the reduction is visible
# in the diff instead of disappearing into a green check.
# - tests above the row fails as well, telling the author to raise the row in
# the same PR. An exact baseline cannot drift upward silently, so the next
# deletion is always measured against the true count.
# - ignored above the row fails (the line contains `ignored`); adding
# #[ignore] is not accepted on gateway/integration. Ignored below the row
# fails too, asking for the row to be lowered so that ratchet stays tight.
# - tests above the row passes and prints a note on stdout. Rows are raised
# only by `--update`, in whichever PR chooses to tighten them.
# - ignored above the row fails (the line contains `ignored >`): raise the
# row in the same PR and justify the new #[ignore] in the PR description.
# Ignored below the row passes with a note.
# - a row whose crate is no longer a member fails (deleting a crate deletes
# its tests), and a member without a row fails.
# its tests; remove the row in the same PR).
# - a member without a row passes with a note, so adding a crate does not
# block on this file; the next `--update` picks it up. Until then the new
# crate has no floor.
#
# Failures go to stderr and exit 1. Notes go to stdout and never use the
# failure shapes above (` < baseline`, `ignored >`, `not a workspace member`).
#
# `--update` rewrites the baseline from the current tree. Review the diff it
# produces before committing it; it is the only sanctioned way to change the
# file other than editing the single row a PR touched.
#
# Contract tests: scripts/test_check_test_count_ratchet.sh (make script-tests).
#
# Usage:
# scripts/check_test_count_ratchet.sh # compare against the baseline
# scripts/check_test_count_ratchet.sh --update # regenerate the baseline
@@ -132,10 +144,12 @@ if [[ "$mode" == "update" ]]; then
echo "# One row per workspace member: <package>, tests, ignored, tab-separated."
echo "# tests counts #[test], #[tokio::test], #[tokio::test(...)] and #[test_case(...)]"
echo "# attribute lines under the package directory; ignored counts #[ignore] and"
echo "# #[ignore = \"...\"]. Both columns are exact: CI fails when a crate's tests fall"
echo "# below or rise above its row, or when its ignored count changes. Raise a row in"
echo "# the PR that adds tests; lowering one is a reviewed statement that tests were"
echo "# intentionally removed. Regenerate with:"
echo "# #[ignore = \"...\"]. The tests column is a floor and the ignored column is a"
echo "# ceiling: CI fails when a crate's tests fall below its row or its ignored count"
echo "# rises above it. Extra tests and fewer ignores pass; --update tightens the rows."
echo "# Lowering a tests row or raising an ignored row is a reviewed statement that"
echo "# tests were intentionally removed or ignored; justify it in the PR description."
echo "# Regenerate with:"
echo "# scripts/check_test_count_ratchet.sh --update"
cat "$TMP_DIR/actual"
} >"$BASELINE_FILE"
@@ -165,9 +179,10 @@ if cut -f1 "$TMP_DIR/baseline" | LC_ALL=C sort | uniq -d | grep . >"$TMP_DIR/dup
exit 1
fi
# Every violation is one line. The `<actual> < baseline <n>` and `ignored`
# shapes are the acceptance contract of rustfs/backlog#2738; keep them.
awk -F'\t' '
# Every violation is one line on stderr; every note is one line on stdout. The
# `<actual> < baseline <n>` and `ignored >` shapes are the acceptance contract
# of rustfs/backlog#2738 and the contract tests; notes must never reuse them.
awk -F'\t' -v notes="$TMP_DIR/notes" '
NR == FNR {
baseline_tests[$1] = $2
baseline_ignored[$1] = $3
@@ -176,18 +191,18 @@ NR == FNR {
{
seen[$1] = 1
if (!($1 in baseline_tests)) {
printf "crate %s: no baseline row (%d tests, %d ignored); add it in this PR\n", $1, $2, $3
printf "crate %s: no baseline row (%d tests, %d ignore attributes); run --update to add it\n", $1, $2, $3 >notes
next
}
if ($2 + 0 < baseline_tests[$1] + 0) {
printf "crate %s: %d < baseline %d\n", $1, $2, baseline_tests[$1]
} else if ($2 + 0 > baseline_tests[$1] + 0) {
printf "crate %s: %d > baseline %d; raise the baseline row to %d in this PR\n", $1, $2, baseline_tests[$1], $2
printf "crate %s: %d above baseline %d; run --update to tighten\n", $1, $2, baseline_tests[$1] >notes
}
if ($3 + 0 > baseline_ignored[$1] + 0) {
printf "crate %s: %d ignored > baseline %d; adding #[ignore] is not accepted\n", $1, $3, baseline_ignored[$1]
printf "crate %s: %d ignored > baseline %d; raise the baseline row to %d in this PR and justify the new #[ignore] in the PR description\n", $1, $3, baseline_ignored[$1], $3
} else if ($3 + 0 < baseline_ignored[$1] + 0) {
printf "crate %s: %d ignored < baseline %d; lower the baseline row to %d in this PR\n", $1, $3, baseline_ignored[$1], $3
printf "crate %s: %d ignore attributes below baseline %d; run --update to tighten\n", $1, $3, baseline_ignored[$1] >notes
}
}
END {
@@ -199,14 +214,19 @@ END {
}
' "$TMP_DIR/baseline" "$TMP_DIR/actual" | LC_ALL=C sort >"$TMP_DIR/violations"
if [[ -f "$TMP_DIR/notes" ]]; then
LC_ALL=C sort "$TMP_DIR/notes" | sed 's/^/note: /'
fi
if grep -q . "$TMP_DIR/violations"; then
cat "$TMP_DIR/violations" >&2
echo "error: test-count ratchet violated (rustfs/backlog#2734 section 1 rule 3)." >&2
echo " A crate below its baseline lost tests: restore them. If the removal is" >&2
echo " intentional, lower that row in $BASELINE_FILE in the same PR and" >&2
echo " justify it in the PR description. A crate above its baseline raises its" >&2
echo " row in the same PR ('$0 --update' rewrites the file; review the diff)." >&2
echo " justify it in the PR description. A crate above its ignored baseline" >&2
echo " raises that row in the same PR with a justification in the PR description" >&2
echo " ('$0 --update' rewrites the file; review the diff)." >&2
exit 1
fi
echo "test-count ratchet OK: $crate_count crates, $total_tests tests, $total_ignored ignored match $BASELINE_FILE"
echo "test-count ratchet OK: $crate_count crates, $total_tests tests, $total_ignored ignored; every crate at or above its tests floor and at or below its ignored ceiling in $BASELINE_FILE"
+62 -3
View File
@@ -98,6 +98,58 @@ def git_file(root: Path, revision: str, path: str) -> str:
return git(root, "show", f"{revision}:{path}")
def release_metadata_only(root: Path, base: str, head: str, path: str) -> bool:
"""Recognize literal stable-version edits, never general packaging changes."""
stable = r"(?:0|[1-9][0-9]*)\.(?:0|[1-9][0-9]*)\.(?:0|[1-9][0-9]*)"
marker = "<version>"
patterns = {
"flake.nix": [rf'(?m)^([ \t]*pname = "rustfs";\n[ \t]*version = ")({stable})(";)$'],
"helm/rustfs/Chart.yaml": [rf'(?m)^({key}: ")({stable})(")$' for key in ("version", "appVersion")],
"rustfs.spec": [rf'(?m)^(Version:[ \t]+)({stable})()$'],
}
if path not in patterns or not SHA.fullmatch(base) or not SHA.fullmatch(head):
return False
try:
normalized, versions = [], []
for revision in (base, head):
# A symlink, added/deleted file or executable-mode change is a
# build input even if its apparent text resembles metadata.
entry = git(root, "ls-tree", "-z", revision, "--", path)
if not re.fullmatch(rf"100644 blob [0-9a-f]{{40}}\t{re.escape(path)}\x00", entry):
return False
content, values = git_file(root, revision, path), []
# Existing markers could make edits to distinct fields collapse
# to identical normalized text.
if marker in content:
return False
for pattern in patterns[path]:
matches = list(re.finditer(pattern, content))
if len(matches) != 1:
return False
values.append(matches[0][2])
content = re.sub(pattern, lambda match: match[1] + marker + match[3], content)
if len(set(values)) != 1:
return False
normalized.append(content)
versions.append(values[0])
if path == "rustfs.spec":
old, new = (content.split("%changelog\n") for content in normalized)
if len(old) != 2 or len(new) != 2 or old[0] != new[0]:
return False
if old[1] == new[1]:
return True
# Only the standard prepended release note is non-executable;
# RPM macro syntax and edits to older history retain the full gate.
note = (r"\* (?:Mon|Tue|Wed|Thu|Fri|Sat|Sun) "
r"(?:Jan|Feb|Mar|Apr|May|Jun|Jul|Aug|Sep|Oct|Nov|Dec) [0-9]{2} [0-9]{4} "
r"[A-Za-z0-9 ._-]+ <[A-Za-z0-9._+-]+@[A-Za-z0-9.-]+>\n"
rf"- Update RPM package to RustFS {re.escape(versions[1])}\n\n")
return re.fullmatch(note + re.escape(old[1]), new[1]) is not None
return normalized[0] == normalized[1]
except (OSError, subprocess.CalledProcessError, UnicodeError, ValueError):
return False
def dependency_tables(document: dict):
for key, value in document.items():
if key in ("dependencies", "dev-dependencies", "build-dependencies"):
@@ -283,10 +335,17 @@ def select_plan(event: str, base: str, head: str, ref: str, root: Path, action:
base_packages, base_graph = head_packages, head_graph
reasons.append(f"conservative fallback: {type(error).__name__}")
graph = {name: head_graph.get(name, set()) | base_graph.get(name, set()) for name in set(head_graph) | set(base_graph)}
affected = set()
affected, non_runtime = set(), set()
if not full and policy is not None:
for path in changed:
if documentation_path(path):
non_runtime.add(path)
continue
if (policy["rollout"] == "enforced"
and not any(fnmatch.fnmatchcase(path, pattern) for pattern in policy["full_paths"])
and release_metadata_only(root, base, head, path)):
non_runtime.add(path)
reasons.append(f"literal stable release metadata: {path}")
continue
if path in E2E_SELECTION_FILES:
affected.add("e2e_test")
@@ -308,9 +367,9 @@ def select_plan(event: str, base: str, head: str, ref: str, root: Path, action:
affected.update(owners)
if not affected and not full:
mode = "docs"
reasons.append("all changes are documentation")
reasons.append("all changes are documentation or literal stable release metadata")
elif affected == {"e2e_test"} and not full and not (closure({"e2e_test"}, graph, reverse=True) - {"e2e_test"}) and all(
documentation_path(path) or path in E2E_SELECTION_FILES or
path in non_runtime or path in E2E_SELECTION_FILES or
(path.startswith("crates/e2e_test/src/") and path.endswith(".rs")) for path in changed):
mode = "e2e"
reasons.append("isolated E2E harness changes")
+1 -1
View File
@@ -13,7 +13,7 @@ OUTPUT="$4"
case "$ARCH" in
amd64|arm64) ;;
*) echo "Unsupported rc architecture: $ARCH" >&2; exit 1 ;;
*) echo "Unsupported rc architecture: '$ARCH' (expected amd64 or arm64; an empty value means the caller's TARGETARCH build argument was not set)" >&2; exit 1 ;;
esac
case "$LIBC" in
musl) LIBC_SUFFIX="" ;;
+1 -6
View File
@@ -17,12 +17,7 @@ case "$RAW" in
esac
APP_VERSION="${RAW_TAG#v}"
BETA_NUM=$(printf '%s\n' "$APP_VERSION" | sed -n -E 's/^.*-beta\.([0-9]+)$/\1/p')
if [ -n "$BETA_NUM" ]; then
CHART_VERSION="0.${BETA_NUM}.0"
else
CHART_VERSION="$APP_VERSION"
fi
CHART_VERSION="$APP_VERSION"
if [ -n "${GITHUB_OUTPUT:-}" ]; then
{
+167
View File
@@ -0,0 +1,167 @@
#!/usr/bin/env python3
# Copyright 2026 RustFS Team
#
# Licensed under the Apache License, Version 2.0 (the "License");
# you may not use this file except in compliance with the License.
# You may obtain a copy of the License at
#
# http://www.apache.org/licenses/LICENSE-2.0
#
# Unless required by applicable law or agreed to in writing, software
# distributed under the License is distributed on an "AS IS" BASIS,
# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
# See the License for the specific language governing permissions and
# limitations under the License.
"""Compare two JUnit XML reports case by case.
Cases are aligned on classname + name and reduced to pass / fail / skip. A case
present in only one report is reported as `missing` on the other side. A
`<flakyFailure>` case that finally passed is a pass and is listed separately.
Exit codes: 0 no unallowed differences and no stale allow-list entry; 1 at least
one difference or stale allow-list entry; 2 unreadable or malformed input.
"""
import argparse
import json
import sys
import xml.etree.ElementTree as ET
PASS, FAIL, SKIP, MISSING = "pass", "fail", "skip", "missing"
_RETRY_TAGS = {"flakyFailure", "flakyError", "rerunFailure", "rerunError"}
class InputError(Exception):
"""Input that cannot be trusted as a report; maps to exit code 2."""
def case_id(classname, name):
return f"{classname}::{name}" if classname else name
def parse_report(path):
"""Return ({case_id: state}, [flaky case ids]) for one JUnit file."""
try:
with open(path, "rb") as handle:
data = handle.read()
except OSError as exc:
raise InputError(f"{path}: cannot read: {exc.strerror or exc}") from exc
if not data.strip():
raise InputError(f"{path}: file is empty")
try:
root = ET.fromstring(data)
except ET.ParseError as exc:
raise InputError(f"{path}: malformed XML: {exc}") from exc
if root.tag not in ("testsuites", "testsuite"):
raise InputError(f"{path}: unexpected root element <{root.tag}>")
states, flaky = {}, []
for case in root.iter("testcase"):
name = case.get("name")
if not name:
raise InputError(f"{path}: <testcase> without a name")
cid = case_id(case.get("classname", ""), name)
if cid in states:
raise InputError(f"{path}: duplicate case {cid}")
tags = {child.tag for child in case}
if tags & {"failure", "error"}:
state = FAIL
elif "skipped" in tags:
state = SKIP
else:
state = PASS
states[cid] = state
if state == PASS and tags & _RETRY_TAGS:
flaky.append(cid)
if not states:
raise InputError(f"{path}: report contains no test cases")
return states, sorted(flaky)
def read_allow(path):
try:
with open(path, encoding="utf-8") as handle:
lines = handle.read().splitlines()
except (OSError, UnicodeDecodeError) as exc:
raise InputError(f"{path}: cannot read allow-list: {exc}") from exc
return {line.strip() for line in lines if line.strip() and not line.lstrip().startswith("#")}
def counts(states):
result = {PASS: 0, FAIL: 0, SKIP: 0}
for state in states.values():
result[state] += 1
return result
def compare(a, b, allow):
diffs = []
for cid in sorted(set(a) | set(b)):
sa, sb = a.get(cid, MISSING), b.get(cid, MISSING)
if sa != sb:
diffs.append({"case": cid, "a": sa, "b": sb, "allowed": cid in allow})
stale = sorted(allow - {d["case"] for d in diffs})
return diffs, stale
def render(args, diffs, stale, flaky_a, flaky_b, counts_a, counts_b):
out = []
for label, c in ((args.label_a, counts_a), (args.label_b, counts_b)):
out.append(f"{label}: pass={c[PASS]} fail={c[FAIL]} skip={c[SKIP]}")
for label, flaky in ((args.label_a, flaky_a), (args.label_b, flaky_b)):
if flaky:
out.append(f"\nFlaky in {label} (passed after retry, not a difference):")
out.extend(f" {cid}" for cid in flaky)
if diffs:
width = max(len("case"), *(len(d["case"]) for d in diffs))
wa = max(len(args.label_a), 7)
out.append("")
out.append(f"{'case':<{width}} {args.label_a:<{wa}} {args.label_b}")
for d in diffs:
note = " (allowed)" if d["allowed"] else ""
out.append(f"{d['case']:<{width}} {d['a']:<{wa}} {d['b']}{note}")
if stale:
out.append("\nStale allow-list entries (no matching difference; remove them):")
out.extend(f" {cid}" for cid in stale)
if not diffs and not stale:
out.append("\nNo differences.")
return "\n".join(out)
def main(argv=None):
parser = argparse.ArgumentParser(description=__doc__.splitlines()[0])
parser.add_argument("a", help="first JUnit XML report")
parser.add_argument("b", help="second JUnit XML report")
parser.add_argument("--label-a", default="a")
parser.add_argument("--label-b", default="b")
parser.add_argument("--json", metavar="PATH", help="write a machine-readable summary")
parser.add_argument("--allow", metavar="FILE", help="case ids (one per line) whose difference is tolerated")
args = parser.parse_args(argv)
if args.label_a == args.label_b:
print("error: --label-a and --label-b must differ", file=sys.stderr)
return 2
try:
states_a, flaky_a = parse_report(args.a)
states_b, flaky_b = parse_report(args.b)
allow = read_allow(args.allow) if args.allow else set()
diffs, stale = compare(states_a, states_b, allow)
counts_a, counts_b = counts(states_a), counts(states_b)
if args.json:
summary = {
"labels": {"a": args.label_a, "b": args.label_b},
"counts": {args.label_a: counts_a, args.label_b: counts_b},
"differences": diffs,
"flaky": {args.label_a: flaky_a, args.label_b: flaky_b},
"stale_allow": stale,
}
with open(args.json, "w", encoding="utf-8") as handle:
json.dump(summary, handle, indent=2, sort_keys=True)
handle.write("\n")
except (InputError, OSError) as exc:
print(f"error: {exc}", file=sys.stderr)
return 2
print(render(args, diffs, stale, flaky_a, flaky_b, counts_a, counts_b))
return 1 if stale or any(not d["allowed"] for d in diffs) else 0
if __name__ == "__main__":
sys.exit(main())

Some files were not shown because too many files have changed in this diff Show More