* fix(ci): run the mint lane on ubuntu-latest, which has a Docker daemon
* fix(docker): let BuildKit populate TARGETARCH for the rc download
* fix(docker): derive the rc architecture when TARGETARCH is not set
* test(e2e): expect ServiceUnavailable for degraded ListParts after #9149
PR #9149 made ServiceUnavailable the default API code for storage
read-quorum failures and limited SlowDownRead to GET/HEAD object reads.
ListParts maps its storage error through ApiError::from, so a degraded
ListParts now answers 503 ServiceUnavailable. The two distributed
acceptance cases still expected the previous SlowDownRead code and have
failed every nightly e2e-distributed run since 2026-11-05.
The 503 status and the fail-closed (non-empty-list) assertions are
unchanged; only the error code tracks the intended contract.
* style(e2e): apply rustfmt to the ListParts error-code assertion
* feat(server): add the RUSTFS_S3_STACK selector and dual-stack e2e lane (#8411)
* refactor(server): group the s3s-compat tower layers under legacy_compat (#8410)
Move the ten tower layers that patch s3s behaviour out of
rustfs/src/server/layer.rs into rustfs/src/server/legacy_compat/, one file
per layer with its tests. Each file header names the s3s behaviour it
patches, the gateway mechanism that replaces it (Replaced by:), and the task
that removes it (Removed by: T2.12, rustfs/backlog#2771). The layers keep
their code, their order in external_service_stack!, and their behaviour; no
layer is deleted.
Add server::http::tests::stack_census, which renders the ordered service
chain of the real external_service_stack! expansion and compares it with a
committed constant, so a reorder, insertion or removal is a visible diff.
Route the two s3s error types the moved layers use through a
storage_api::server::legacy_compat shim so the split adds no s3s-importing
file, and lower the s3s footprint baselines to the measured values (files 209
to 208, s3_error! lines 1586 to 1585). Keep the server runtime-source bypass
guard covering the moved code.
Refs rustfs/backlog#2787, rustfs/backlog#2734
* feat(s3-types): add the RustFS-owned S3 error type and s3_error! macro (#8406)
`rustfs-s3-types` now provides `S3ErrorCode` (generated by build.rs from
src/codes.txt, the grep-derived call surface, plus `Custom`), `S3Error`,
`S3Result`, `StdError` and the `s3_error` macro with the same call syntax
as s3s, so the T1.2 codemod is a `use` swap. Statuses are hand-written and
pinned to the s3s behaviour RustFS ships today by an oracle test that uses
s3s as a dev-dependency only. The `From` conversions for the legacy edge
live behind the off-by-default `compat-s3s` feature (the orphan rule
forbids them in rustfs/src/error.rs); `rustfs` enables it.
Refs rustfs/backlog#2742, rustfs/backlog#2734
* ci: add the per-crate test-count ratchet (#8404)
Quick Checks now fails any PR that lowers the number of test attributes
in a workspace crate or adds #[ignore], measured per crate against the
exact rows of .config/test-count-baseline.txt (rustfs/backlog#2734
section 1 rule 3). The counter widens the spec's literal
#[(tokio::)?test] to the tokio attributes that carry arguments and both
spellings of #[test_case(...)], and anchors at line start so
commented-out attributes do not count. The initial baseline covers 52
crates, 21077 tests and 164 ignored.
Refs rustfs/backlog#2738
Refs rustfs/backlog#2734
* test(ecstore): pin the persisted bucket-config XML bytes for 13 families (#8408)
Before any DTO or codec is replaced, pin the exact bytes that
deserialize followed by serialize produces for every persisted
bucket-configuration XML family in its three source shapes (AWS
documentation, MinIO-written, RustFS-written), and the bytes RustFS
writes today for a typed value. Byte equality only: a structural
comparison would let a codec replacement change the persisted encoding
unnoticed.
The generator script refuses to overwrite existing goldens without
--force, refuses on dirty codec paths, cannot report success from its
write mode, and records the generating commit in the fixtures README.
Refs rustfs/backlog#2744, rustfs/backlog#2734
* ci: raise the test-count baseline to the integration tree (#8426)
* ci: make the s3s footprint baseline exact and lower-only (#8403)
* ci: make the s3s footprint baseline exact and lower-only
The s3s footprint guard compared three counters against ceilings held in
the script itself, so a ceiling could sit above the tree and a drift was
fixable by raising it (rustfs/rustfs#5964 raised the s3_error! ceiling
to go green). The three values now live in
.config/s3s-footprint-baseline.txt as files, s3_error_lines and
ecstore_files, the script reads only that file, and a measurement that
differs from its baseline in either direction fails: growth prints
"s3s footprint grew: <measured> > <baseline>", shrinkage prints
"lower the baseline to <measured> in this PR". The counting method is
unchanged; the baseline is the current measurement of
gateway/integration (209 / 1585 / 36). The header records the end state:
once all three reach 0, T4.2 switches the script to failing on any
occurrence.
Refs rustfs/backlog#2739, rustfs/backlog#2734
* ci: add the s3s edge allowlist mode to the footprint guard (#8405)
`scripts/check_s3s_footprint.sh --mode allowlist` fails on every Rust
file that references s3s (`s3s::` or `s3s_sigv4`) and is not matched by
an entry of the new `.config/s3s-edge-allowlist.txt`; `--dry-run` lists
those files and exits 0. The allowlist carries the initial edge from the
spec, and the script rejects an entry that names or matches anything
under crates/ other than crates/e2e_test. The baseline mode and the
argument-less CI invocation are unchanged.
Refs rustfs/backlog#2741, rustfs/backlog#2734
* test(ecstore): reach the bucket-config codec through the test facade (#8428)
* refactor(s3-types): own the HTTP Body and drop s3s from three crates (#8431)
Add rustfs_s3_types::Body, a BoxBody<Bytes, StdError> wrapper with
empty() and From<Bytes | Vec<u8> | String | &'static str>, and the
legacy From conversions to and from s3s::Body behind compat-s3s.
The signer is now generic over the request body: it only reads and
writes the request head, so it needs no body type and its callers keep
theirs. s3-client sends its requests with the owned Body, madmin signs a
bodiless head, and targets drops its two S3Request/S3Response header
wrappers in favour of extract_params_header.
rustfs-signer, rustfs-madmin and rustfs-targets no longer depend on
s3s; the s3s footprint baseline drops from 209 to 202 files.
Refs rustfs/backlog#2747, rustfs/backlog#2734
* refactor(ecstore): move S3ErrorCode to rustfs-s3-types by codemod (#8432)
* test(ecstore): pin the InvalidRange responses of new_getobjectreader
Both error branches of new_getobjectreader build an ErrorResponse with S3ErrorCode::InvalidRange, and no test observed either code. Pin the code, its wire name and the location fields of each branch before the code type moves to rustfs-s3-types (rustfs/backlog#2743), so a changed code on either branch fails a test. The ecstore test-count row rises by the two tests.
* chore(scripts): add the s3s-to-rustfs-s3-types error codemod
Task T1.2 of rustfs/backlog#2734 (rustfs/backlog#2743) moves S3Error, S3ErrorCode, S3Result and s3_error! from s3s to rustfs-s3-types by script, one crate closure per PR. The codemod rewrites use trees (splitting mixed s3s imports) and qualified code paths, never comments or literals, adds the rustfs-s3-types dependency, drops s3s only from a crate that no longer names it, refuses every file of the s3s edge allowlist, takes --hold for a file whose type is fixed outside the closure, and prints a per-crate census. A second run changes nothing. Its self-test runs under make script-tests.
* refactor(ecstore): take S3ErrorCode from rustfs-s3-types
Output of scripts/codemods/s3_error_to_s3_types.py over crates/ecstore, crates/lifecycle, crates/scanner and crates/replication (rustfs/backlog#2743 batches 1 and 2). The ErrorResponse code field, the two InvalidRange constructions in new_getobjectreader and an unused tier import now use rustfs_s3_types::S3ErrorCode; the variant names and wire names are unchanged. lifecycle, scanner and replication reference none of the four error names. warm_backend.rs and warm_backend_s3.rs are held: they compare against rustfs_s3_client::ErrorResponse.code, which stays the s3s type until s3-client moves.
* ci(footprint): count only s3_error! lines still on s3s's macro
s3_error_lines counted every line holding the text `s3_error!`, which
also matches the RustFS-owned `rustfs_s3_types::s3_error!`. A codemod
that moves call sites onto rustfs-s3-types therefore could not lower it
(rustfs/backlog#2743, decision 3; rustfs/rustfs#8432, gotcha 2).
The counter now applies a textual, per-file and per-line rule that errs
on the side of counting (full rule in the script header):
- occurrences spelled `rustfs_s3_types::s3_error!` never count;
- a line holding `s3s::s3_error!` always counts;
- any other `s3_error!` line counts unless its file is cleared, that is,
it imports the macro from rustfs_s3_types by name on a line starting
with the use item (or defines it with `macro_rules! s3_error`) and does
not also bring s3s's macro in (`use s3s::s3_error`, a `use s3s::{..}`
tree naming it or a crate-root `*`, `use s3s::*`, or
`#[macro_use] extern crate s3s`);
- fail closed: a file matching neither form (`use super::*`, a
crate-local re-export, a renamed or glob rustfs_s3_types import, a
commented-out import, a comment-only mention) counts.
Census of the 1585 lines on 96fc06776: 1478 in 64 files that import the
macro from s3s; 107 in 15 files matching neither form (105 through
`use super::*`, 1 qualified `s3s::s3_error!`, 1 comment line); none in
files that import it from rustfs_s3_types or define it. Every line stays
counted, so the baseline remains s3_error_lines=1585 (files=202 and
ecstore_files=36 unchanged). Rewriting rustfs/src/auth.rs's import to
rustfs_s3_types lowers the new counter to 1567; the old one stayed 1585.
`--self-test` (wired into `make script-tests`) copies the script into
22 fixture trees: 6 positive cases and 16 negative ones (lines that must
not count, forms that must not clear a file, and a missing baseline,
missing key or failing classification search that must fail). Each tree
has a control line that must count, and each run gets s3s text on stdin.
Seven cases were red against the old counter. 19 mutants each turn their
case red: old definition, flipped import test, dropped fail-closed
branch, dropped e2e_test exclusion, dropped s3s veto, dropped qualified
s3s rule, dropped rustfs-qualified strip, unanchored rustfs import,
renamed import accepted, nested glob taken as root glob, single-line
s3s trees only, dropped s3s glob forms, dropped macro_use form, dropped
definition clearing, ignored classification rg errors, error-line search
without its path, classification searches without their path, missing
baseline skipped, missing key skipped.
* test(e2e): compare two RUSTFS_S3_STACK servers over every S3 operation (#8435)
Add dual_stack_differential to e2e_test. It starts two rustfs processes
from the same binary, stack A on RUSTFS_S3_STACK=legacy and stack B on
RUSTFS_E2E_STACK_B (default legacy until rustfs/backlog#2734 T2.5),
sends both the same aws-sdk-s3 script covering the 73 methods of
impl S3 for FS (a normal request and a 4xx negative each), and compares
status, error code, an allowlist of response headers and a few typed
body fields. Error message text is never compared. Each run writes
<target>/dual-stack/<run>.json with compared_ops and the (op, field,
a, b) differences; any difference fails the test.
differential_detects_an_injected_difference starts B with
RUSTFS_REGION=eu-west-1 and requires GetBucketLocation to be reported.
The comparator is pure and unit-tested without a server.
Wire the module into the e2e-smoke filter and update the e2e-smoke and
e2e-full selection digests and the e2e_test test-count row.
Refs rustfs/backlog#2740
* feat(app): add the RustFS-owned RequestEnvelope and its s3s constructor (#8437)
* feat(app): add the RustFS-owned RequestEnvelope and its s3s constructor
* ci: classify the s3s edge adapters in storage/s3_compat as interface
* feat(s3-types): add Body::from_stream and move Swift off s3s::Body (#8438)
rustfs/backlog#2747 batch 3. `Body::from_stream` wraps a chunk stream of
unknown length over `StreamBody`/`BoxBody`: one data frame per chunk, the
first error ends the body, the size hint stays unknown, and dropping the
body drops the stream. Parity tests pin it to the s3s `StreamingBlob::wrap`
it replaces, at the `http_body` level and byte for byte through hyper.
The Swift handlers in rustfs-protocols build their responses as the owned
`Body`. `SwiftService` converts them into the wrapped S3 service's response
body type through `From<Body>`, so rustfs keeps serving `s3s::Body` at the
legacy edge unchanged. Request bodies stay Send-only (`UnsyncBoxBody`); the
two storage writers and the DLO/SLO segment streams get `Sync` from
`sync_wrapper::SyncStream`, which is what s3s's `http_body_unsync` mutex
provided before. The DTO and error uses of s3s in protocols are untouched
(T1.5 and T1.2).
* refactor(protocols): move error types to rustfs-s3-types by codemod (#8439)
T1.2 batch 3 (rustfs/backlog#2743): s3select-api, s3select-query and
the error items of protocols move off s3s by scripts/codemods, run with
--hold crates/protocols/src/sftp/errors.rs. That file stays on s3s: it
downcasts the backend error, whose runtime type is still s3s::S3Error.
- rustfs/src/protocols/client.rs: the two webdav session methods now
return rustfs_s3_types::S3Result and convert the s3s error with From.
- codemod: a child module that calls s3_error! through `use super::*`
gets its own `use rustfs_s3_types::s3_error;` once its parent imports
that macro by name; otherwise it is listed as GLOB.
- s3-types: a table test pins code, status and the s3s error the legacy
edge renders for every variant a moved call site names.
- s3_error_lines 1585 -> 1584 (a lifecycle comment reworded).
* test(connect): restore one-second drive request budgets
Apply the exact test-only repair already landed on main in 5972a09cc3. The filesystem probe and production CLI success test request one second and keep their existing one-second duration and parameter checks. All outcome, I/O, error, cleanup and resource-limit assertions remain.
Scope: rustfs/tests/connect_perf_drive.rs only. Patch-id: cb0a1dfc8f68a67727525adea7458e2cee7d09f8.
Validation environment: LANG=en_US.UTF-8, CARGO_BUILD_JOBS=1, CARGO_INCREMENTAL=0, exclusive APFS-cloned CARGO_TARGET_DIR.
Quick Checks: every command from .github/actions/quick-checks/action.yml, with captured output and exit status below.
$ shellcheck --version && actionlint
ShellCheck - shell script analysis tool
version: 0.11.0
license: GNU General Public License, version 3
website: https://www.shellcheck.net
exit_code=0
$ cargo fmt --all --check
exit_code=0
$ ./scripts/check_unsafe_code_allowances.sh
exit_code=0
$ ./scripts/check_layer_dependencies.sh
Layer dependency guard passed (no new reverse dependencies/cycles).
exit_code=0
$ ./scripts/check_architecture_migration_rules.sh
Architecture migration rules passed.
exit_code=0
$ ./scripts/check_logging_guardrails.sh
✅ Logging guardrails check passed
exit_code=0
$ ./scripts/check_error_other_format_ratchet.sh
error(format!) ratchet passed (total call sites: 582).
exit_code=0
$ ./scripts/check_no_tokio_io_uring.sh
exit_code=0
$ ./scripts/check_extension_schema_boundaries.sh
Extension schema boundaries passed.
exit_code=0
$ ./scripts/check_body_cache_whitelist.sh
Body-cache whitelist guard passed.
exit_code=0
$ ./scripts/check_s3s_footprint.sh
s3s footprint OK: files importing s3s is 195 (baseline files=195)
s3s footprint OK: s3s s3_error! lines is 1584 (baseline s3_error_lines=1584)
s3s footprint OK: ecstore files referencing s3s is 36 (baseline ecstore_files=36)
✅ s3s footprint baseline check passed
exit_code=0
$ ./scripts/check_test_count_ratchet.sh
test-count ratchet OK: 52 crates, 21222 tests, 164 ignored match .config/test-count-baseline.txt
exit_code=0
$ ./scripts/check_fips_wording.sh
FIPS wording guard passed (README.md and CHANGELOG.md contain no forbidden claims).
KMS wording guard passed (crates/kms claims no Vault KV2 Transit wrapping).
exit_code=0
$ ./scripts/check_embedded_secrets.sh
Embedded secret guard passed (no private key material or provider credential literal in the tree).
exit_code=0
$ make script-tests
Running script tests...
entrypoint credential policy tests passed
internode grpc A/B bench env tests passed
object batch benchmark enhanced tests passed
PASS package service script contract
hotpath warp A/B gate tests passed
hotpath warp ABBA tests passed
Scanner validation artifacts written to /var/folders/0w/s4wxsn2n4w9b6f3cd4xrz53w0000gn/T/tmp.1pFe6Kl1Yu/out
Scanner validation artifacts written to /var/folders/0w/s4wxsn2n4w9b6f3cd4xrz53w0000gn/T/tmp.1pFe6Kl1Yu/out-missing-pid
.............................
----------------------------------------------------------------------
Ran 29 tests in 181.286s
OK
{"decision":"verified","pending_gates":["G01","G03","G04","G05","G06","G07","G08","G09","G10","G11","G12","G13","G14","P1","P2","P3","P4","R-D","R-E","R-L"],"pending_lanes":["authority-coverage","checkpoint-and-crash","ec8-4-multiset","maintenance-producers","mixed-version-rollback","mrf-responsibility","scheduler-pressure","status-and-outcome"],"rejected_errors":[],"rejected_gate":null,"release_approved":false,"release_schema_capable":true,"schema":1,"verified_gate":"G02"}
{"decision":"verified","pending_gates":["G01","G02","G03","G04","G05","G06","G07","G08","G09","G10","G11","G12","G13","G14","P1","P2","P3","P4","R-D","R-L"],"pending_lanes":["authority-coverage","checkpoint-and-crash","ec8-4-multiset","maintenance-producers","mixed-version-rollback","mrf-responsibility","scheduler-pressure","status-and-outcome"],"rejected_errors":[],"rejected_gate":null,"release_approved":false,"release_schema_capable":true,"schema":1,"verified_gate":"R-E"}
{"decision":"verified","pending_gates":["G01","G03","G04","G05","G06","G07","G08","G09","G10","G11","G12","G13","G14","P1","P2","P3","P4","R-D","R-E","R-L"],"pending_lanes":["authority-coverage","checkpoint-and-crash","ec8-4-multiset","maintenance-producers","mixed-version-rollback","mrf-responsibility","scheduler-pressure","status-and-outcome"],"rejected_errors":[],"rejected_gate":null,"release_approved":false,"release_schema_capable":true,"schema":1,"verified_gate":"G02"}
{"decision":"verified","pending_gates":["G01","G02","G03","G04","G05","G06","G07","G08","G09","G10","G11","G12","G13","G14","P1","P2","P3","P4","R-D","R-L"],"pending_lanes":["authority-coverage","checkpoint-and-crash","ec8-4-multiset","maintenance-producers","mixed-version-rollback","mrf-responsibility","scheduler-pressure","status-and-outcome"],"rejected_errors":[],"rejected_gate":null,"release_approved":false,"release_schema_capable":true,"schema":1,"verified_gate":"R-E"}
{"decision":"verified","pending_gates":["G02","G03","G04","G05","G06","G07","G08","G09","G10","G11","G12","G13","G14","P1","P2","P3","P4","R-D","R-E","R-L"],"pending_lanes":["authority-coverage","checkpoint-and-crash","ec8-4-multiset","maintenance-producers","mixed-version-rollback","mrf-responsibility","scheduler-pressure","status-and-outcome"],"rejected_errors":[],"rejected_gate":null,"release_approved":false,"release_schema_capable":true,"schema":1,"verified_gate":"G01"}
{"decision":"verified","pending_gates":["G01","G02","G04","G05","G06","G07","G08","G09","G10","G11","G12","G13","G14","P1","P2","P3","P4","R-D","R-E","R-L"],"pending_lanes":["authority-coverage","checkpoint-and-crash","ec8-4-multiset","maintenance-producers","mixed-version-rollback","mrf-responsibility","scheduler-pressure","status-and-outcome"],"rejected_errors":[],"rejected_gate":null,"release_approved":false,"release_schema_capable":true,"schema":1,"verified_gate":"G03"}
{"decision":"verified","pending_gates":["G01","G02","G03","G04","G05","G06","G07","G08","G09","G10","G11","G12","G13","G14","P1","P2","P3","P4","R-D","R-E"],"pending_lanes":["authority-coverage","checkpoint-and-crash","ec8-4-multiset","maintenance-producers","mixed-version-rollback","mrf-responsibility","scheduler-pressure","status-and-outcome"],"rejected_errors":[],"rejected_gate":null,"release_approved":false,"release_schema_capable":true,"schema":1,"verified_gate":"R-L"}
{"decision":"verified","pending_gates":["G01","G02","G03","G04","G05","G06","G07","G08","G09","G10","G11","G12","G13","P1","P2","P3","P4","R-D","R-E","R-L"],"pending_lanes":["authority-coverage","checkpoint-and-crash","ec8-4-multiset","maintenance-producers","mixed-version-rollback","mrf-responsibility","scheduler-pressure","status-and-outcome"],"rejected_errors":[],"rejected_gate":null,"release_approved":false,"release_schema_capable":true,"schema":1,"verified_gate":"G14"}
{"decision":"verified","pending_gates":["G01","G02","G03","G04","G05","G06","G07","G08","G09","G10","G11","G12","G13","P1","P2","P3","P4","R-D","R-E","R-L"],"pending_lanes":["authority-coverage","checkpoint-and-crash","ec8-4-multiset","maintenance-producers","mixed-version-rollback","mrf-responsibility","scheduler-pressure","status-and-outcome"],"rejected_errors":[],"rejected_gate":null,"release_approved":false,"release_schema_capable":true,"schema":1,"verified_gate":"G14"}
{"decision":"verified","pending_gates":["G01","G02","G03","G04","G05","G06","G07","G08","G09","G11","G12","G13","G14","P1","P2","P3","P4","R-D","R-E","R-L"],"pending_lanes":["authority-coverage","checkpoint-and-crash","ec8-4-multiset","maintenance-producers","mixed-version-rollback","mrf-responsibility","scheduler-pressure","status-and-outcome"],"rejected_errors":[],"rejected_gate":null,"release_approved":false,"release_schema_capable":true,"schema":1,"verified_gate":"G10"}
{"decision":"verified","pending_gates":["G01","G02","G03","G04","G05","G06","G07","G08","G09","G10","G11","G12","G13","G14","P2","P3","P4","R-D","R-E","R-L"],"pending_lanes":["authority-coverage","checkpoint-and-crash","ec8-4-multiset","maintenance-producers","mixed-version-rollback","mrf-responsibility","scheduler-pressure","status-and-outcome"],"rejected_errors":[],"rejected_gate":null,"release_approved":false,"release_schema_capable":true,"schema":1,"verified_gate":"P1"}
{"decision":"verified","pending_gates":["G01","G02","G03","G04","G05","G06","G07","G08","G09","G10","G11","G12","G13","G14","P1","P3","P4","R-D","R-E","R-L"],"pending_lanes":["authority-coverage","checkpoint-and-crash","ec8-4-multiset","maintenance-producers","mixed-version-rollback","mrf-responsibility","scheduler-pressure","status-and-outcome"],"rejected_errors":[],"rejected_gate":null,"release_approved":false,"release_schema_capable":true,"schema":1,"verified_gate":"P2"}
{"decision":"verified","pending_gates":["G01","G02","G03","G04","G05","G06","G07","G08","G09","G10","G11","G12","G13","G14","P1","P2","P4","R-D","R-E","R-L"],"pending_lanes":["authority-coverage","checkpoint-and-crash","ec8-4-multiset","maintenance-producers","mixed-version-rollback","mrf-responsibility","scheduler-pressure","status-and-outcome"],"rejected_errors":[],"rejected_gate":null,"release_approved":false,"release_schema_capable":true,"schema":1,"verified_gate":"P3"}
{"decision":"verified","pending_gates":["G01","G02","G03","G04","G06","G07","G08","G09","G10","G11","G12","G13","G14","P1","P2","P3","P4","R-D","R-E","R-L"],"pending_lanes":["authority-coverage","checkpoint-and-crash","ec8-4-multiset","maintenance-producers","mixed-version-rollback","mrf-responsibility","scheduler-pressure","status-and-outcome"],"rejected_errors":[],"rejected_gate":null,"release_approved":false,"release_schema_capable":true,"schema":1,"verified_gate":"G05"}
{"decision":"verified","pending_gates":["G01","G02","G03","G04","G05","G07","G08","G09","G10","G11","G12","G13","G14","P1","P2","P3","P4","R-D","R-E","R-L"],"pending_lanes":["authority-coverage","checkpoint-and-crash","ec8-4-multiset","maintenance-producers","mixed-version-rollback","mrf-responsibility","scheduler-pressure","status-and-outcome"],"rejected_errors":[],"rejected_gate":null,"release_approved":false,"release_schema_capable":true,"schema":1,"verified_gate":"G06"}
{"decision":"verified","pending_gates":["G01","G02","G03","G04","G05","G06","G07","G08","G09","G10","G11","G12","G13","G14","P1","P2","P3","P4","R-E","R-L"],"pending_lanes":["authority-coverage","checkpoint-and-crash","ec8-4-multiset","maintenance-producers","mixed-version-rollback","mrf-responsibility","scheduler-pressure","status-and-outcome"],"rejected_errors":[],"rejected_gate":null,"release_approved":false,"release_schema_capable":true,"schema":1,"verified_gate":"R-D"}
Status-and-outcome release descriptor verified: /private/var/folders/0w/s4wxsn2n4w9b6f3cd4xrz53w0000gn/T/tmp8rwn1pe0/descriptor/release-bundle-status-outcome.json
{"decision":"verified","pending_gates":["G01","G02","G03","G04","G06","G07","G08","G09","G10","G11","G12","G13","G14","P1","P2","P3","P4","R-D","R-E","R-L"],"pending_lanes":["authority-coverage","checkpoint-and-crash","ec8-4-multiset","maintenance-producers","mixed-version-rollback","mrf-responsibility","scheduler-pressure","status-and-outcome"],"rejected_errors":[],"rejected_gate":null,"release_approved":false,"release_schema_capable":true,"schema":1,"verified_gate":"G05"}
{"decision":"verified","pending_gates":["G01","G02","G03","G04","G05","G07","G08","G09","G10","G11","G12","G13","G14","P1","P2","P3","P4","R-D","R-E","R-L"],"pending_lanes":["authority-coverage","checkpoint-and-crash","ec8-4-multiset","maintenance-producers","mixed-version-rollback","mrf-responsibility","scheduler-pressure","status-and-outcome"],"rejected_errors":[],"rejected_gate":null,"release_approved":false,"release_schema_capable":true,"schema":1,"verified_gate":"G06"}
{"decision":"verified","pending_gates":["G01","G02","G03","G04","G05","G06","G07","G08","G09","G10","G11","G12","G13","G14","P1","P2","P3","P4","R-E","R-L"],"pending_lanes":["authority-coverage","checkpoint-and-crash","ec8-4-multiset","maintenance-producers","mixed-version-rollback","mrf-responsibility","scheduler-pressure","status-and-outcome"],"rejected_errors":[],"rejected_gate":null,"release_approved":false,"release_schema_capable":true,"schema":1,"verified_gate":"R-D"}
{"decision":"verified","pending_gates":["G01","G02","G03","G04","G05","G06","G08","G09","G10","G11","G12","G13","G14","P1","P2","P3","P4","R-D","R-E","R-L"],"pending_lanes":["authority-coverage","checkpoint-and-crash","ec8-4-multiset","maintenance-producers","mixed-version-rollback","mrf-responsibility","scheduler-pressure","status-and-outcome"],"rejected_errors":[],"rejected_gate":null,"release_approved":false,"release_schema_capable":true,"schema":1,"verified_gate":"G07"}
{"decision":"verified","pending_gates":["G01","G02","G03","G04","G05","G06","G07","G09","G10","G11","G12","G13","G14","P1","P2","P3","P4","R-D","R-E","R-L"],"pending_lanes":["authority-coverage","checkpoint-and-crash","ec8-4-multiset","maintenance-producers","mixed-version-rollback","mrf-responsibility","scheduler-pressure","status-and-outcome"],"rejected_errors":[],"rejected_gate":null,"release_approved":false,"release_schema_capable":true,"schema":1,"verified_gate":"G08"}
{"decision":"verified","pending_gates":["G01","G02","G03","G04","G05","G06","G07","G08","G09","G10","G11","G12","G13","G14","P1","P2","P3","R-D","R-E","R-L"],"pending_lanes":["authority-coverage","checkpoint-and-crash","ec8-4-multiset","maintenance-producers","mixed-version-rollback","mrf-responsibility","scheduler-pressure","status-and-outcome"],"rejected_errors":[],"rejected_gate":null,"release_approved":false,"release_schema_capable":true,"schema":1,"verified_gate":"P4"}
exact 1MiB handoff ABBA tests passed
pinned paired ABBA bench tests passed
==> /var/folders/0w/s4wxsn2n4w9b6f3cd4xrz53w0000gn/T/tmp.mYmuI01hNq/bin/path_containment (-max_total_time=7, -seed=123456789, -artifact_prefix=/var/folders/0w/s4wxsn2n4w9b6f3cd4xrz53w0000gn/T/tmp.mYmuI01hNq/explicit-artifacts/path_containment/, corpus=/private/tmp/gw-orchestrator-2734/ci-drive/fuzz/corpus/path_containment)
-seed=123456789
target=path_containment
seed=123456789
max_total_time=7
git_revision=4d15b816236630b2c42f321ffc9ed3654edc94e3
git_dirty=true
runner_mode=prebuilt
==> /var/folders/0w/s4wxsn2n4w9b6f3cd4xrz53w0000gn/T/tmp.mYmuI01hNq/bin/path_containment (-max_total_time=1, -seed=2322617706, -artifact_prefix=/var/folders/0w/s4wxsn2n4w9b6f3cd4xrz53w0000gn/T/tmp.mYmuI01hNq/auto-artifacts/path_containment/, corpus=/private/tmp/gw-orchestrator-2734/ci-drive/fuzz/corpus/path_containment)
-seed=2322617706
==> cargo +nightly fuzz run path_containment (-max_total_time=9, -seed=987654321, -artifact_prefix=/var/folders/0w/s4wxsn2n4w9b6f3cd4xrz53w0000gn/T/tmp.mYmuI01hNq/cargo-artifacts/path_containment/)
+nightly
fuzz
run
path_containment
-seed=987654321
seed=987654321
runner_mode=cargo-fuzz
fuzz runner seed manifest tests passed
✅ scripts/python_bin.sh resolver checks passed
self-test ok: clean fixture with every exemption present passes
self-test ok: all 11 pattern families fire
self-test ok: moving a pinned source is reported
Embedded secret guard self-test passed.
self-test ok: p1-s3s-import: 'use s3s::s3_error;' counts every line, the rustfs_s3_types module's too (s3_error_lines=3)
self-test ok: p2-s3s-mixed-tree: a multi-line s3s use tree naming s3_error counts (s3_error_lines=3)
self-test ok: p3-s3s-glob: a crate-root glob of s3s, bare or inside a tree, counts (s3_error_lines=5)
self-test ok: p4-macro-use: '#[macro_use] extern crate s3s' counts (s3_error_lines=3)
self-test ok: p5-qualified-s3s: a qualified s3s::s3_error! line counts in a cleared file (s3_error_lines=2)
self-test ok: p6-exact-pass: a baseline equal to the tree passes (exit 0)
self-test ok: n1-rustfs-import: 'use rustfs_s3_types::s3_error;' alone counts nothing (s3_error_lines=1)
self-test ok: n2-rustfs-mixed-tree: a rustfs_s3_types tree plus an s3s tree without the macro counts nothing (s3_error_lines=1)
self-test ok: n3-defines: the file defining the macro counts nothing (s3_error_lines=1)
self-test ok: n4-qualified-rustfs: a qualified rustfs_s3_types::s3_error! line counts nothing (s3_error_lines=1)
self-test ok: n5-renamed-call: a call through a renamed macro is not text-matched (out of scope) (s3_error_lines=1)
self-test ok: n6-comment-string: a comment or string mentioning s3_error! in a cleared file counts nothing (s3_error_lines=1)
self-test ok: n7-e2e-excluded: crates/e2e_test is excluded (s3_error_lines=1)
self-test ok: n8-super-glob: an unrecognised import form ('use super::*;') counts (s3_error_lines=3)
self-test ok: n9-facade: a crate-local re-export counts (s3_error_lines=2)
self-test ok: n10-renamed-import: a renamed rustfs_s3_types import does not clear s3_error! lines (s3_error_lines=2)
self-test ok: n11-commented-import: a commented-out rustfs_s3_types import does not clear (s3_error_lines=2)
self-test ok: n12-rustfs-glob: a glob import of rustfs_s3_types does not clear (s3_error_lines=2)
self-test ok: n13-comment-only: a comment mentioning s3_error! in a file with no import counts (s3_error_lines=2)
self-test ok: f1-baseline-missing: a missing baseline file fails (exit 1)
self-test ok: f2-key-missing: a baseline without s3_error_lines fails (exit 1)
self-test ok: f3-rg-fails: a failing classification search fails the guard (exit 1)
✅ s3s footprint self-test passed
test_core_gate_rejects_missing_ignored_filtered_and_corrupt_inputs (__main__.SelfTests.test_core_gate_rejects_missing_ignored_filtered_and_corrupt_inputs) ... ok
test_e2e_requires_registration (__main__.SelfTests.test_e2e_requires_registration) ... ok
test_fuzz_runtime_uses_matrix_target (__main__.SelfTests.test_fuzz_runtime_uses_matrix_target) ... ok
test_ilm_lane_keeps_the_measured_cargo_build_budget (__main__.SelfTests.test_ilm_lane_keeps_the_measured_cargo_build_budget) ... ok
test_profile_listing_binds_platform_digest (__main__.SelfTests.test_profile_listing_binds_platform_digest) ... ok
test_profile_listing_enforces_selection (__main__.SelfTests.test_profile_listing_enforces_selection) ... ok
test_quick_checks_commands_propagate_failure (__main__.SelfTests.test_quick_checks_commands_propagate_failure) ... ok
test_quick_checks_rejects_caller_and_execution_bypasses (__main__.SelfTests.test_quick_checks_rejects_caller_and_execution_bypasses) ... ok
test_runner_readiness_fails_closed (__main__.SelfTests.test_runner_readiness_fails_closed) ... ok
test_s3_runner_rejects_unbounded_failure_locals (__main__.SelfTests.test_s3_runner_rejects_unbounded_failure_locals) ... ok
test_scanner_heal_begin_requires_embedded_source_provenance (__main__.SelfTests.test_scanner_heal_begin_requires_embedded_source_provenance) ... ok
test_scanner_heal_case_does_not_approve_pending_release (__main__.SelfTests.test_scanner_heal_case_does_not_approve_pending_release) ... ok
test_scanner_heal_case_lane_is_required_for_runner_profile (__main__.SelfTests.test_scanner_heal_case_lane_is_required_for_runner_profile) ... ok
test_scanner_heal_case_only_schema_cannot_approve_release (__main__.SelfTests.test_scanner_heal_case_only_schema_cannot_approve_release) ... ok
test_scanner_heal_crash_case_rejects_restart_oracle (__main__.SelfTests.test_scanner_heal_crash_case_rejects_restart_oracle) ... ok
test_scanner_heal_ec84_case_rejects_wrong_erasure_geometry (__main__.SelfTests.test_scanner_heal_ec84_case_rejects_wrong_erasure_geometry) ... ok
test_scanner_heal_finish_collects_oracles_from_registry (__main__.SelfTests.test_scanner_heal_finish_collects_oracles_from_registry) ... ok
test_scanner_heal_multipool_case_accepts_deferred_optional_outage_write (__main__.SelfTests.test_scanner_heal_multipool_case_accepts_deferred_optional_outage_write) ... ok
test_scanner_heal_pending_gate_cannot_map_to_implemented_lane (__main__.SelfTests.test_scanner_heal_pending_gate_cannot_map_to_implemented_lane) ... ok
test_scanner_heal_receipts_reject_reuse_and_missing_builds (__main__.SelfTests.test_scanner_heal_receipts_reject_reuse_and_missing_builds) ... ok
test_scanner_heal_rejects_boolean_fractional_and_out_of_geometry_integers (__main__.SelfTests.test_scanner_heal_rejects_boolean_fractional_and_out_of_geometry_integers) ... ok
test_scanner_heal_rejects_broken_execution_and_artifacts (__main__.SelfTests.test_scanner_heal_rejects_broken_execution_and_artifacts) ... ok
test_scanner_heal_rejects_copied_junit_and_wrong_suite_build (__main__.SelfTests.test_scanner_heal_rejects_copied_junit_and_wrong_suite_build) ... ok
test_scanner_heal_release_bundle_accepts_complete_measured_evidence (__main__.SelfTests.test_scanner_heal_release_bundle_accepts_complete_measured_evidence) ... ok
test_scanner_heal_release_bundle_assembler_copies_measured_artifacts (__main__.SelfTests.test_scanner_heal_release_bundle_assembler_copies_measured_artifacts) ... ok
test_scanner_heal_release_bundle_assembler_merges_measured_descriptors (__main__.SelfTests.test_scanner_heal_release_bundle_assembler_merges_measured_descriptors) ... ok
test_scanner_heal_release_bundle_assembler_rejects_duplicate_descriptor_gate (__main__.SelfTests.test_scanner_heal_release_bundle_assembler_rejects_duplicate_descriptor_gate) ... ok
test_scanner_heal_release_bundle_assembler_rejects_fixture_descriptor (__main__.SelfTests.test_scanner_heal_release_bundle_assembler_rejects_fixture_descriptor) ... ok
test_scanner_heal_release_bundle_assembler_rejects_hash_mismatch (__main__.SelfTests.test_scanner_heal_release_bundle_assembler_rejects_hash_mismatch) ... ok
test_scanner_heal_release_bundle_assembler_rejects_mixed_descriptor_revisions (__main__.SelfTests.test_scanner_heal_release_bundle_assembler_rejects_mixed_descriptor_revisions) ... ok
test_scanner_heal_release_bundle_enforces_topology_duration_profile_and_versions (__main__.SelfTests.test_scanner_heal_release_bundle_enforces_topology_duration_profile_and_versions) ... ok
test_scanner_heal_release_bundle_fixture_writer_is_not_release_evidence (__main__.SelfTests.test_scanner_heal_release_bundle_fixture_writer_is_not_release_evidence) ... ok
test_scanner_heal_release_bundle_gate_accepts_single_measured_gate (__main__.SelfTests.test_scanner_heal_release_bundle_gate_accepts_single_measured_gate) ... ok
test_scanner_heal_release_bundle_gate_rejects_missing_field (__main__.SelfTests.test_scanner_heal_release_bundle_gate_rejects_missing_field) ... ok
test_scanner_heal_release_bundle_gate_rejects_unknown_gate (__main__.SelfTests.test_scanner_heal_release_bundle_gate_rejects_unknown_gate) ... ok
test_scanner_heal_release_bundle_rejects_synthetic_or_missing_fields (__main__.SelfTests.test_scanner_heal_release_bundle_rejects_synthetic_or_missing_fields) ... ok
test_scanner_heal_release_bundle_requires_domain_evidence (__main__.SelfTests.test_scanner_heal_release_bundle_requires_domain_evidence) ... ok
test_scanner_heal_release_bundle_requires_field_provenance (__main__.SelfTests.test_scanner_heal_release_bundle_requires_field_provenance) ... ok
test_scanner_heal_release_bundle_requires_json_artifact_provenance (__main__.SelfTests.test_scanner_heal_release_bundle_requires_json_artifact_provenance) ... ok
test_scanner_heal_release_bundle_requires_mixed_version_field_roles (__main__.SelfTests.test_scanner_heal_release_bundle_requires_mixed_version_field_roles) ... ok
test_scanner_heal_release_bundle_requires_same_gate_measurement_window (__main__.SelfTests.test_scanner_heal_release_bundle_requires_same_gate_measurement_window) ... ok
test_scanner_heal_release_consumes_multiple_registry_oracles (__main__.SelfTests.test_scanner_heal_release_consumes_multiple_registry_oracles) ... ok
test_scanner_heal_release_matrix_lanes_remain_pending_without_real_evidence (__main__.SelfTests.test_scanner_heal_release_matrix_lanes_remain_pending_without_real_evidence) ... ok
test_scanner_heal_release_requirements_cannot_be_cleared (__main__.SelfTests.test_scanner_heal_release_requirements_cannot_be_cleared) ... ok
test_scanner_heal_release_status_rejects_focused_case_run (__main__.SelfTests.test_scanner_heal_release_status_rejects_focused_case_run) ... ok
test_scanner_heal_release_status_rejects_synthetic_case (__main__.SelfTests.test_scanner_heal_release_status_rejects_synthetic_case) ... ok
test_scanner_heal_required_evidence_fields_cannot_be_removed (__main__.SelfTests.test_scanner_heal_required_evidence_fields_cannot_be_removed) ... ok
test_scheduled_alerts_require_completion_watchdog (__main__.SelfTests.test_scheduled_alerts_require_completion_watchdog) ... ok
test_update_profile_selection_changes_only_requested_platform (__main__.SelfTests.test_update_profile_selection_changes_only_requested_platform) ... ok
test_validate_includes_quick_checks (__main__.SelfTests.test_validate_includes_quick_checks) ... ok
test_vault_tests_require_cross_process_serialization (__main__.SelfTests.test_vault_tests_require_cross_process_serialization) ... ok
test_workflow_readiness_requires_dependency_probe (__main__.SelfTests.test_workflow_readiness_requires_dependency_probe) ... ok
----------------------------------------------------------------------
Ran 52 tests in 14.301s
OK
e2e-full selection OK: 1 tests, sha256=29e30e6ed2ed38d9e5b0a7cdd9536abf5c719f7021a02b578ae1381563a3c273
...................................
----------------------------------------------------------------------
Ran 35 tests in 1.161s
OK
...................
----------------------------------------------------------------------
Ran 19 tests in 17.397s
OK
..........
----------------------------------------------------------------------
Ran 10 tests in 0.183s
OK
test_actual_selector_bootstrap_uses_base_policy_and_fails_closed (__main__.SelfTests.test_actual_selector_bootstrap_uses_base_policy_and_fails_closed) ... ok
test_documentation_paths_do_not_hide_build_or_fixture_changes (__main__.SelfTests.test_documentation_paths_do_not_hide_build_or_fixture_changes) ... ok
test_e2e_scope_keeps_production_and_shared_configuration_full (__main__.SelfTests.test_e2e_scope_keeps_production_and_shared_configuration_full) ... ok
test_e2e_shortcut_rejects_new_direct_renamed_and_target_dependencies (__main__.SelfTests.test_e2e_shortcut_rejects_new_direct_renamed_and_target_dependencies) ... ok
test_e2e_shortcut_traverses_implicit_path_dependencies (__main__.SelfTests.test_e2e_shortcut_traverses_implicit_path_dependencies) ... ok
test_event_contract_requires_complete_candidate_and_optional_lanes (__main__.SelfTests.test_event_contract_requires_complete_candidate_and_optional_lanes) ... ok
test_every_wrong_result_missing_job_or_selection_fails_closed (__main__.SelfTests.test_every_wrong_result_missing_job_or_selection_fails_closed) ... ok
test_full_e2e_gate_preserves_workflow_branch_and_event_scope (__main__.SelfTests.test_full_e2e_gate_preserves_workflow_branch_and_event_scope) ... ok
test_git_range_includes_deleted_source_and_rename_origins (__main__.SelfTests.test_git_range_includes_deleted_source_and_rename_origins) ... ok
test_job_ids_and_display_names_cannot_hide_validation (__main__.SelfTests.test_job_ids_and_display_names_cannot_hide_validation) ... ok
test_report_only_stack_lane_mirrors_e2e_smoke_outside_the_aggregate (__main__.SelfTests.test_report_only_stack_lane_mirrors_e2e_smoke_outside_the_aggregate) ... ok
test_repository_wiring_and_missing_dependency_regression (__main__.SelfTests.test_repository_wiring_and_missing_dependency_regression) ... ok
test_verify_command_preserves_failures (__main__.SelfTests.test_verify_command_preserves_failures) ... ok
----------------------------------------------------------------------
Ran 13 tests in 3.280s
OK
security coverage self-test passed
test_api_failures_preserve_other_evidence_and_never_enter_grace (__main__.SelfTests.test_api_failures_preserve_other_evidence_and_never_enter_grace) ... ok
test_attempt_outcome_does_not_replace_recent_success (__main__.SelfTests.test_attempt_outcome_does_not_replace_recent_success) ... ok
test_cancelled_attempt_cannot_refresh_expired_success (__main__.SelfTests.test_cancelled_attempt_cannot_refresh_expired_success) ... ok
test_cli_requires_the_repository_default_branch (__main__.SelfTests.test_cli_requires_the_repository_default_branch) ... ok
test_config_rejects_duplicate_and_invalid_entries (__main__.SelfTests.test_config_rejects_duplicate_and_invalid_entries) ... ok
test_freshness_boundaries (__main__.SelfTests.test_freshness_boundaries) ... ok
test_grace_requires_two_successful_queries_with_no_history (__main__.SelfTests.test_grace_requires_two_successful_queries_with_no_history) ... ok
test_invalid_api_evidence_fails_closed (__main__.SelfTests.test_invalid_api_evidence_fails_closed) ... ok
test_report_retains_every_workflow (__main__.SelfTests.test_report_retains_every_workflow) ... ok
test_requests_filter_schedule_default_branch_and_success_on_server (__main__.SelfTests.test_requests_filter_schedule_default_branch_and_success_on_server) ... ok
----------------------------------------------------------------------
Ran 10 tests in 0.051s
OK
..............................
----------------------------------------------------------------------
Ran 30 tests in 22.354s
OK
..........
----------------------------------------------------------------------
Ran 10 tests in 2.969s
OK
.................................
----------------------------------------------------------------------
Ran 33 tests in 0.160s
OK
Online functional runner labels: pf-testing
.......
----------------------------------------------------------------------
Ran 7 tests in 0.002s
OK
.......
----------------------------------------------------------------------
Ran 7 tests in 0.001s
OK
....
----------------------------------------------------------------------
Ran 4 tests in 0.128s
OK
...
----------------------------------------------------------------------
Ran 3 tests in 0.547s
OK
PASS: cold stampede script self-test
exit_code=0
$ python3 ./scripts/test_object_generation_protocol_model.py
test_a_recovery_barrier_cannot_choose_an_empty_value (__main__.ObjectGenerationProtocolModel.test_a_recovery_barrier_cannot_choose_an_empty_value) ... ok
test_all_bounded_two_proposer_interleavings_choose_at_most_one_value (__main__.ObjectGenerationProtocolModel.test_all_bounded_two_proposer_interleavings_choose_at_most_one_value) ... ok
test_an_unavailable_decision_quorum_cannot_choose_a_candidate (__main__.ObjectGenerationProtocolModel.test_an_unavailable_decision_quorum_cannot_choose_a_candidate) ... ok
test_counter_allocation_requires_persistence_and_is_consumed_once (__main__.ObjectGenerationProtocolModel.test_counter_allocation_requires_persistence_and_is_consumed_once) ... ok
test_counter_restart_burns_persisted_allocations_before_or_after_issue (__main__.ObjectGenerationProtocolModel.test_counter_restart_burns_persisted_allocations_before_or_after_issue) ... ok
test_crash_after_promise_before_reply_does_not_restore_an_older_ballot (__main__.ObjectGenerationProtocolModel.test_crash_after_promise_before_reply_does_not_restore_an_older_ballot) ... ok
test_duplicate_or_unconfigured_voters_cannot_form_a_quorum (__main__.ObjectGenerationProtocolModel.test_duplicate_or_unconfigured_voters_cannot_form_a_quorum) ... ok
test_empty_or_unobserved_minority_slot_can_retry_without_allocating_another_revision (__main__.ObjectGenerationProtocolModel.test_empty_or_unobserved_minority_slot_can_retry_without_allocating_another_revision) ... ok
test_equal_ballot_cannot_accept_different_bytes_and_lower_ballot_is_rejected (__main__.ObjectGenerationProtocolModel.test_equal_ballot_cannot_accept_different_bytes_and_lower_ballot_is_rejected) ... ok
test_forgetting_tombstone_authority_allows_an_aba_create (__main__.ObjectGenerationProtocolModel.test_forgetting_tombstone_authority_allows_an_aba_create) ... ok
test_issuing_before_persistence_reuses_a_ballot_after_restart (__main__.ObjectGenerationProtocolModel.test_issuing_before_persistence_reuses_a_ballot_after_restart) ... ok
test_lost_ack_and_restart_preserve_the_chosen_operation (__main__.ObjectGenerationProtocolModel.test_lost_ack_and_restart_preserve_the_chosen_operation) ... ok
test_minority_accepted_work_is_adopted_and_finished (__main__.ObjectGenerationProtocolModel.test_minority_accepted_work_is_adopted_and_finished) ... ok
test_omitting_highest_accepted_adoption_has_a_double_choice_counterexample (__main__.ObjectGenerationProtocolModel.test_omitting_highest_accepted_adoption_has_a_double_choice_counterexample) ... ok
test_quorum_choice_does_not_require_an_uncontacted_disk_to_learn_it (__main__.ObjectGenerationProtocolModel.test_quorum_choice_does_not_require_an_uncontacted_disk_to_learn_it) ... ok
test_recovery_failure_or_stale_expected_head_does_not_allocate_a_successor (__main__.ObjectGenerationProtocolModel.test_recovery_failure_or_stale_expected_head_does_not_allocate_a_successor) ... ok
test_replacing_voters_with_empty_state_can_choose_a_conflicting_value (__main__.ObjectGenerationProtocolModel.test_replacing_voters_with_empty_state_can_choose_a_conflicting_value) ... ok
test_skipping_predecessor_recovery_chooses_a_broken_revision_chain (__main__.ObjectGenerationProtocolModel.test_skipping_predecessor_recovery_chooses_a_broken_revision_chain) ... ok
test_stale_publication_rollback_and_gc_cannot_change_the_winner (__main__.ObjectGenerationProtocolModel.test_stale_publication_rollback_and_gc_cannot_change_the_winner) ... ok
test_superseded_operation_identity_cannot_be_reused_for_different_input (__main__.ObjectGenerationProtocolModel.test_superseded_operation_identity_cannot_be_reused_for_different_input) ... ok
test_superseded_operation_retry_returns_its_original_successor (__main__.ObjectGenerationProtocolModel.test_superseded_operation_retry_returns_its_original_successor) ... ok
test_tombstone_and_null_marker_payloads_do_not_reset_authority_revision (__main__.ObjectGenerationProtocolModel.test_tombstone_and_null_marker_payloads_do_not_reset_authority_revision) ... ok
test_tombstone_rejects_a_delayed_never_present_create (__main__.ObjectGenerationProtocolModel.test_tombstone_rejects_a_delayed_never_present_create) ... ok
test_unlearned_predecessor_is_recovered_before_the_second_slot (__main__.ObjectGenerationProtocolModel.test_unlearned_predecessor_is_recovered_before_the_second_slot) ... ok
----------------------------------------------------------------------
Ran 24 tests in 0.958s
OK
model: states=51023, transitions=167520, terminal=732, chosen_candidates=2
negative control: double-choice trace=((0, 'prepare', 0), (0, 'prepare', 1), (0, 'prepare', 2), (0, 'accept', 0), (0, 'accept', 1), (0, 'accept', 2), (1, 'prepare', 0), (1, 'prepare', 1), (1, 'prepare', 2), (1, 'accept', 0), (1, 'accept', 1), (1, 'accept', 2))
multi-slot round schedules: 640
exit_code=0
$ python3 ./scripts/check_test_wiring.py
OK: e2e modules, runner selection, fuzz matrices, profiles, scheduled alerts, and Quick Checks are wired
exit_code=0
$ ./scripts/check_no_planning_docs.sh
exit_code=0
$ ./scripts/check_uring_lane_lib_only.sh
OK: no 'uring_'-matching test functions under crates/ecstore/tests; --lib is safe
exit_code=0
Initial complete gate
$ cargo nextest run --profile ci -p rustfs --test connect_perf_drive --no-fail-fast
Nextest run ID e07fc5b9-c6c8-4cdd-ae61-54839bf24437 with nextest profile: ci
Starting 10 tests across 1 binary
PASS [ 0.016s] rustfs::connect_perf_drive cancellation_and_single_collector_leave_no_scratch
PASS [ 0.011s] rustfs::connect_perf_drive cleanup_failure_overrides_cancellation_and_remains_attributed
PASS [ 0.009s] rustfs::connect_perf_drive consent_and_resource_budgets_fail_before_scratch_io
PASS [ 0.009s] rustfs::connect_perf_drive invalid_roots_and_existing_scratch_fail_without_clobbering
PASS [ 16.950s] rustfs::connect_perf_drive production_cli_measures_and_signs_exact_binary_provenance
PASS [ 4.025s] rustfs::connect_perf_drive production_cli_rejects_missing_l1_consent_and_limits_before_artifacts
PASS [ 0.008s] rustfs::connect_perf_drive read_only_root_and_full_drive_errors_are_classified
PASS [ 0.015s] rustfs::connect_perf_drive real_local_write_read_is_measured_and_scratch_is_removed
PASS [ 0.007s] rustfs::connect_perf_drive runtime_timeout_and_full_drive_use_schema_valid_failed_reason
PASS [ 0.035s] rustfs::connect_perf_drive successful_result_has_signed_bounded_private_offline_export
────────────
Summary [ 21.087s] 10 tests run: 10 passed, 0 skipped
exit_code=0
Restored complete gate
$ cargo nextest run --profile ci -p rustfs --test connect_perf_drive --no-fail-fast
Nextest run ID 233b7307-5361-4eb7-94d5-b384f17d2d6a with nextest profile: ci
Starting 10 tests across 1 binary
PASS [ 0.009s] rustfs::connect_perf_drive cancellation_and_single_collector_leave_no_scratch
PASS [ 0.008s] rustfs::connect_perf_drive cleanup_failure_overrides_cancellation_and_remains_attributed
PASS [ 0.008s] rustfs::connect_perf_drive consent_and_resource_budgets_fail_before_scratch_io
PASS [ 0.008s] rustfs::connect_perf_drive invalid_roots_and_existing_scratch_fail_without_clobbering
PASS [ 19.518s] rustfs::connect_perf_drive production_cli_measures_and_signs_exact_binary_provenance
PASS [ 4.146s] rustfs::connect_perf_drive production_cli_rejects_missing_l1_consent_and_limits_before_artifacts
PASS [ 0.009s] rustfs::connect_perf_drive read_only_root_and_full_drive_errors_are_classified
PASS [ 0.018s] rustfs::connect_perf_drive real_local_write_read_is_measured_and_scratch_is_removed
PASS [ 0.009s] rustfs::connect_perf_drive runtime_timeout_and_full_drive_use_schema_valid_failed_reason
PASS [ 0.054s] rustfs::connect_perf_drive successful_result_has_signed_bounded_private_offline_export
────────────
Summary [ 23.787s] 10 tests run: 10 passed, 0 skipped
exit_code=0
Mutation M1: success_measurement reports max(actual elapsed, requested budget + 1), forcing 1001 ms for the one-second request. The retained duration assertion at rustfs/tests/connect_perf_drive.rs:105 fails.
$ cargo nextest run --profile ci -p rustfs --test connect_perf_drive --no-fail-fast -E 'test(real_local_write_read_is_measured_and_scratch_is_removed)'
info: experimental features enabled: setup-scripts
Compiling rustfs v1.0.1 (/private/tmp/gw-orchestrator-2734/ci-drive/rustfs)
warning: linker stderr: ld: __eh_frame section too large (max 16MB) to encode dwarf unwind offsets in compact unwind table, performance of exception handling might be affected
|
= note: `#[warn(linker_messages)]` on by default
warning: `rustfs` (bin "rustfs") generated 1 warning
warning: `rustfs` (bin "rustfs-cli") generated 1 warning (1 duplicate)
Finished `test` profile [unoptimized + debuginfo] target(s) in 7m 06s
────────────
Nextest run ID 9f618777-7bf6-4cbc-8803-dd7f50f449ed with nextest profile: ci
Starting 1 test across 1 binary (9 tests skipped)
FAIL [ 0.016s] rustfs::connect_perf_drive real_local_write_read_is_measured_and_scratch_is_removed
stdout ───
running 1 test
test real_local_write_read_is_measured_and_scratch_is_removed ... FAILED
failures:
failures:
real_local_write_read_is_measured_and_scratch_is_removed
test result: FAILED. 0 passed; 1 failed; 0 ignored; 0 measured; 9 filtered out; finished in 0.01s
stderr ───
thread 'real_local_write_read_is_measured_and_scratch_is_removed' (106986846) panicked at rustfs/tests/connect_perf_drive.rs:105:5:
assertion failed: (1..=1_000).contains(&data.duration_millis)
note: run with `RUST_BACKTRACE=1` environment variable to display a backtrace
────────────
Summary [ 0.017s] 1 test run: 0 passed, 1 failed, 9 skipped
FAIL [ 0.016s] rustfs::connect_perf_drive real_local_write_read_is_measured_and_scratch_is_removed
error: test run failed
exit_code=100
Mutation M2: target_parameters reports requested duration + 1. The in-process assertion at rustfs/tests/connect_perf_drive.rs:111 and CLI JSON assertion at rustfs/tests/connect_perf_drive.rs:527 both reject 1001 against 1000.
$ cargo nextest run --profile ci -p rustfs --test connect_perf_drive --no-fail-fast -E 'test(real_local_write_read_is_measured_and_scratch_is_removed) | test(production_cli_measures_and_signs_exact_binary_provenance)'
info: experimental features enabled: setup-scripts
Compiling rustfs v1.0.1 (/private/tmp/gw-orchestrator-2734/ci-drive/rustfs)
warning: linker stderr: ld: __eh_frame section too large (max 16MB) to encode dwarf unwind offsets in compact unwind table, performance of exception handling might be affected
|
= note: `#[warn(linker_messages)]` on by default
warning: `rustfs` (bin "rustfs") generated 1 warning
warning: `rustfs` (bin "rustfs-cli") generated 1 warning (1 duplicate)
Finished `test` profile [unoptimized + debuginfo] target(s) in 7m 20s
────────────
Nextest run ID b73d46ab-c910-4e1e-8063-7fc7e332ec15 with nextest profile: ci
Starting 2 tests across 1 binary (8 tests skipped)
FAIL [ 13.968s] rustfs::connect_perf_drive production_cli_measures_and_signs_exact_binary_provenance
stdout ───
running 1 test
test production_cli_measures_and_signs_exact_binary_provenance ... FAILED
failures:
failures:
production_cli_measures_and_signs_exact_binary_provenance
test result: FAILED. 0 passed; 1 failed; 0 ignored; 0 measured; 9 filtered out; finished in 13.96s
stderr ───
thread 'production_cli_measures_and_signs_exact_binary_provenance' (107015462) panicked at rustfs/tests/connect_perf_drive.rs:527:5:
assertion `left == right` failed
left: Number(1001)
right: 1000
note: run with `RUST_BACKTRACE=1` environment variable to display a backtrace
FAIL [ 0.015s] rustfs::connect_perf_drive real_local_write_read_is_measured_and_scratch_is_removed
stdout ───
running 1 test
test real_local_write_read_is_measured_and_scratch_is_removed ... FAILED
failures:
failures:
real_local_write_read_is_measured_and_scratch_is_removed
test result: FAILED. 0 passed; 1 failed; 0 ignored; 0 measured; 9 filtered out; finished in 0.01s
stderr ───
thread 'real_local_write_read_is_measured_and_scratch_is_removed' (107016667) panicked at rustfs/tests/connect_perf_drive.rs:111:5:
assertion `left == right` failed
left: 1001
right: 1000
note: run with `RUST_BACKTRACE=1` environment variable to display a backtrace
────────────
Summary [ 13.985s] 2 tests run: 0 passed, 2 failed, 8 skipped
FAIL [ 13.968s] rustfs::connect_perf_drive production_cli_measures_and_signs_exact_binary_provenance
FAIL [ 0.015s] rustfs::connect_perf_drive real_local_write_read_is_measured_and_scratch_is_removed
error: test run failed
exit_code=100
Transient production mutations were restored byte-for-byte before the complete green rerun. Original/restored SHA256: 8d389e6d8836c041aab8c6a93e56addd65ba3f70b33264e1c74f639d0572a2da.
$ git diff --check
exit_code=0
$ git diff --cached --check
exit_code=0
Independent orchestrator acceptance
Tested code tree: e67c72fb01
Tested tree: 16fbbd2ea4
shellcheck --version && actionlint -> exit 0
cargo fmt --all --check -> exit 0
./scripts/check_unsafe_code_allowances.sh -> exit 0
./scripts/check_layer_dependencies.sh -> exit 0
./scripts/check_architecture_migration_rules.sh -> exit 0
./scripts/check_logging_guardrails.sh -> exit 0
./scripts/check_error_other_format_ratchet.sh -> exit 0
./scripts/check_no_tokio_io_uring.sh -> exit 0
./scripts/check_extension_schema_boundaries.sh -> exit 0
./scripts/check_body_cache_whitelist.sh -> exit 0
./scripts/check_s3s_footprint.sh -> exit 0
./scripts/check_test_count_ratchet.sh -> exit 0
./scripts/check_fips_wording.sh -> exit 0
./scripts/check_embedded_secrets.sh -> exit 0
make script-tests -> exit 0
python3 ./scripts/test_object_generation_protocol_model.py -> exit 0
python3 ./scripts/check_test_wiring.py -> exit 0
./scripts/check_no_planning_docs.sh -> exit 0
./scripts/check_uring_lane_lib_only.sh -> exit 0
$ LANG=en_US.UTF-8 CARGO_BUILD_JOBS=1 CARGO_INCREMENTAL=0 cargo nextest run --profile ci -p rustfs --test connect_perf_drive --no-fail-fast
info: experimental features enabled: setup-scripts
warning: linker stderr: ld: __eh_frame section too large (max 16MB) to encode dwarf unwind offsets in compact unwind table, performance of exception handling might be affected
|
= note: `#[warn(linker_messages)]` on by default
warning: `rustfs` (bin "rustfs-cli") generated 1 warning
warning: `rustfs` (bin "rustfs") generated 1 warning (1 duplicate)
Finished `test` profile [unoptimized + debuginfo] target(s) in 0.69s
────────────
Nextest run ID cd1edf4c-f1c0-456d-ad1a-76cc00ce05c9 with nextest profile: ci
Starting 10 tests across 1 binary
PASS [ 0.012s] rustfs::connect_perf_drive cancellation_and_single_collector_leave_no_scratch
PASS [ 0.010s] rustfs::connect_perf_drive cleanup_failure_overrides_cancellation_and_remains_attributed
PASS [ 0.010s] rustfs::connect_perf_drive consent_and_resource_budgets_fail_before_scratch_io
PASS [ 0.009s] rustfs::connect_perf_drive invalid_roots_and_existing_scratch_fail_without_clobbering
PASS [ 20.328s] rustfs::connect_perf_drive production_cli_measures_and_signs_exact_binary_provenance
PASS [ 5.527s] rustfs::connect_perf_drive production_cli_rejects_missing_l1_consent_and_limits_before_artifacts
PASS [ 0.013s] rustfs::connect_perf_drive read_only_root_and_full_drive_errors_are_classified
PASS [ 0.019s] rustfs::connect_perf_drive real_local_write_read_is_measured_and_scratch_is_removed
PASS [ 0.011s] rustfs::connect_perf_drive runtime_timeout_and_full_drive_use_schema_valid_failed_reason
PASS [ 0.049s] rustfs::connect_perf_drive successful_result_has_signed_bounded_private_offline_export
────────────
Summary [ 25.991s] 10 tests run: 10 passed, 0 skipped
exit_code=0
All 54 tracked Cargo.toml files parse without duplicate keys. git diff --check passed. The production source remains unchanged; the two deliberate implementation mutations above were restored before this complete green run.
* fix(test): compare legacy body wire parity as raw bytes
Lossy UTF-8 comparison concealed a changed binary response byte.
Compare the already captured Vec<u8> values directly.
Validation:
- Binary mutant: migrated payload 0x84 vs legacy 0x83; old comparator
passed 1/1, raw comparator failed 0/1 (exit 100); restored green.
- Non-Swift body-drain mutant: untouched-body delegation test failed
with empty vs payload (exit 100); restored handler without a diff.
- cargo nextest run --profile ci -p rustfs-s3-types -p rustfs-protocols
--features rustfs-protocols/swift --no-fail-fast:
543 tests run, 543 passed, 0 skipped after both mutants restored.
- cargo fmt --all --check and git diff --check passed.
Orchestrator acceptance (LANG=en_US.UTF-8):
cargo nextest run --profile ci -p rustfs-s3-types -p rustfs-protocols --features rustfs-protocols/swift --no-fail-fast
Summary [2.445s] 543 tests run: 543 passed, 0 skipped.
All shared quick checks on the final code tree:
shellcheck --version && actionlint -> exit 0
cargo fmt --all --check -> exit 0
./scripts/check_unsafe_code_allowances.sh -> exit 0
./scripts/check_layer_dependencies.sh -> exit 0
./scripts/check_architecture_migration_rules.sh -> exit 0
./scripts/check_logging_guardrails.sh -> exit 0
./scripts/check_error_other_format_ratchet.sh -> exit 0
./scripts/check_no_tokio_io_uring.sh -> exit 0
./scripts/check_extension_schema_boundaries.sh -> exit 0
./scripts/check_body_cache_whitelist.sh -> exit 0
./scripts/check_s3s_footprint.sh -> exit 0
./scripts/check_test_count_ratchet.sh -> exit 0
./scripts/check_fips_wording.sh -> exit 0
./scripts/check_embedded_secrets.sh -> exit 0
make script-tests -> exit 0
python3 ./scripts/test_object_generation_protocol_model.py -> exit 0
python3 ./scripts/check_test_wiring.py -> exit 0
./scripts/check_no_planning_docs.sh -> exit 0
./scripts/check_uring_lane_lib_only.sh -> exit 0
Final independent acceptance after the drive-test CI repair
Parent: a99d8f4f89
Tested code tree: 93cbd96bf0
Tested tree: 075ae5161c2b62a96812bc1e3c904a7663b4f2f8
shellcheck --version && actionlint -> exit 0
cargo fmt --all --check -> exit 0
./scripts/check_unsafe_code_allowances.sh -> exit 0
./scripts/check_layer_dependencies.sh -> exit 0
./scripts/check_architecture_migration_rules.sh -> exit 0
./scripts/check_logging_guardrails.sh -> exit 0
./scripts/check_error_other_format_ratchet.sh -> exit 0
./scripts/check_no_tokio_io_uring.sh -> exit 0
./scripts/check_extension_schema_boundaries.sh -> exit 0
./scripts/check_body_cache_whitelist.sh -> exit 0
./scripts/check_s3s_footprint.sh -> exit 0
./scripts/check_test_count_ratchet.sh -> exit 0
./scripts/check_fips_wording.sh -> exit 0
./scripts/check_embedded_secrets.sh -> exit 0
make script-tests -> exit 0
python3 ./scripts/test_object_generation_protocol_model.py -> exit 0
python3 ./scripts/check_test_wiring.py -> exit 0
./scripts/check_no_planning_docs.sh -> exit 0
./scripts/check_uring_lane_lib_only.sh -> exit 0
$ LANG=en_US.UTF-8 CARGO_BUILD_JOBS=1 CARGO_INCREMENTAL=0 cargo nextest run --profile ci -p rustfs-s3-types -p rustfs-protocols --features rustfs-protocols/swift --no-fail-fast
Finished `test` profile [unoptimized + debuginfo] target(s) in 10m 08s
Nextest run ID 5c7ff102-ec6d-479a-86e0-810859b28bbf with nextest profile: ci
Starting 543 tests across 8 binaries
Summary [ 6.729s] 543 tests run: 543 passed, 0 skipped
exit_code=0
The only source change is the raw-byte wire comparator. Cargo manifests and the ratchet baselines remain unchanged from the independently validated parent. The invalid-byte and consumed-request mutations remain recorded above; their implementation changes were restored.
* fix: repair integration checks and IPC test readiness
Wait for the RPC/locks test sockets to pass the existing owner and mode
checks inside the original 5s readiness window and 10ms poll interval.
Use the existing admin S3 carrier for the new recovery and quota error
calls. Regenerate the test-count baseline for main's 118 existing tests
across seven crates. Pair the private pagination helper's raw-scan inputs
and remove the redundant Copy clone from the retirement event.
Keep the original assertions, responses, authorization, locks and storage
operations. The pagination body and raw-scan expressions at all five calls
are byte-identical; reinserting only the Copy clone restores the entire
base heal file. All ignored baseline values and s3s counts stay unchanged.
Validation base: c47432e353
Validation tree: f081084bc5
Patch SHA-256: 2dcf0028691bfb2772edfb38e262829b04f4151832dcce823c9dd9f7434b52de
Trace production-prefix SHA-256: fd994f218a21429d62866f4efb8eee1ee57b58f4a51865aa35d05a54ef1ccb45
All commands used LANG=en_US.UTF-8, CARGO_BUILD_JOBS=1,
CARGO_INCREMENTAL=0 and CARGO_NET_OFFLINE=true in one exclusive target.
Every gate retained the same tree and six source hashes before and after.
54 Cargo.toml files parse without duplicate keys.
cargo clippy -p rustfs --all-targets -- -D warnings
Finished `dev` profile [unoptimized + debuginfo] target(s) in 12m 50s
Exit: 0. Wall time: 770.855s.
The original 19 quick-check steps:
shellcheck --version && actionlint -> exit 0
cargo fmt --all --check -> exit 0
./scripts/check_unsafe_code_allowances.sh -> exit 0
./scripts/check_layer_dependencies.sh -> exit 0
./scripts/check_architecture_migration_rules.sh -> exit 0
./scripts/check_logging_guardrails.sh -> exit 0
./scripts/check_error_other_format_ratchet.sh -> exit 0
./scripts/check_no_tokio_io_uring.sh -> exit 0
./scripts/check_extension_schema_boundaries.sh -> exit 0
./scripts/check_body_cache_whitelist.sh -> exit 0
./scripts/check_s3s_footprint.sh -> exit 0
./scripts/check_test_count_ratchet.sh -> exit 0
./scripts/check_fips_wording.sh -> exit 0
./scripts/check_embedded_secrets.sh -> exit 0
make script-tests -> exit 0
python3 ./scripts/test_object_generation_protocol_model.py -> exit 0
python3 ./scripts/check_test_wiring.py -> exit 0
./scripts/check_no_planning_docs.sh -> exit 0
./scripts/check_uring_lane_lib_only.sh -> exit 0
Wall time: 286.337s.
cargo nextest run --profile ci -p rustfs --no-fail-fast
Summary [ 217.017s] 5382 tests run: 5382 passed (4 leaky), 40 skipped
Exit: 0. Wall time: 1324.414s.
cargo nextest run --profile ci -p rustfs-s3-types -p rustfs-protocols --features rustfs-protocols/swift --no-fail-fast
Summary [ 2.189s] 543 tests run: 543 passed, 0 skipped
Exit: 0. Wall time: 123.192s.
cargo nextest run --profile ci -p rustfs-ecstore list_objects_paginate --no-fail-fast
Summary [ 0.170s] 8 tests run: 8 passed, 5921 skipped
Exit: 0. Wall time: 261.324s.
cargo nextest run --profile ci -p rustfs-heal durable_mrf_anchor_retires_only_after_exact_bucket_generation_is_deleted --no-fail-fast
Summary [ 0.805s] 1 test run: 1 passed, 835 skipped
Exit: 0. Wall time: 199.801s.
Baseline regeneration:
./scripts/check_test_count_ratchet.sh --update
wrote .config/test-count-baseline.txt: 52 crates, 21340 tests, 164 ignored
Test-only increases: ecstore48, heal3, iam14, io-metrics2, obs3, policy14,
rustfs34. Existing ignored values and all other rows are byte-identical.
s3s footprint OK: files importing s3s is 195 (baseline files=195)
s3s footprint OK: s3s s3_error! lines is 1584 (baseline s3_error_lines=1584)
s3s footprint OK: ecstore files referencing s3s is 36 (baseline ecstore_files=36)
Historical runtime red controls on f622 (same retained two assertions):
cargo nextest run --profile ci -p rustfs --lib -E 'test(=connect::diagnostics::trace_runtime::tests::local_top_rpc_reads_service_http_completions_and_signs_offline) | test(=connect::diagnostics::trace_runtime::tests::local_top_locks_reads_service_manager_and_signs_offline)' --no-fail-fast
Original path readiness with a temporary 500ms bind/seal gap:
Summary [ 0.863s] 2 tests run: 0 passed, 2 failed, 5113 skipped
Exit: 100.
Sealed readiness in that same gap:
Summary [ 5.621s] 2 tests run: 2 passed, 5113 skipped
Exit: 0.
Compiled M1: replace the helper predicate with path.exists():
Summary [ 0.870s] 2 tests run: 0 passed, 2 failed, 5113 skipped
Exit: 100.
Restore the sealed predicate byte-for-byte:
Summary [ 5.640s] 2 tests run: 2 passed, 5113 skipped
Exit: 0.
Both reds observe mode755 with owner_matches=true and fail at the original
RPC/locks unwraps with StateSecurity. M1 failure locations: trace_runtime.rs
2547:29 and 2713:29. No new assertions were introduced.
cargo nextest run --profile ci -p rustfs --lib -E 'test(connect::diagnostics::trace_runtime::tests::)' --no-fail-fast
Summary [ 20.748s] 28 tests run: 28 passed (1 leaky), 5087 skipped
Exit: 0. This security sweep is historical; the full final unit gate above
covers the actual c474 candidate.
The first disk-stopped compile ran no tests and is excluded from red credit.
All temporary seal instrumentation is removed from the final source.
Final unit cleanup markers: 4 leaky, 40 original skips (exit0). The ld
__eh_frame linker warning also appears verbatim in the original f622 run
and the earlier c474 run; there are no new source warnings.
* fix: remove redundant pending closure in disk test
Pass the capture-free pending future factory as a function item in the existing slow-operation test. This fixes the redundant_closure error at disk_store.rs:2914 that stopped strict Clippy before the feature tests could run.
The single-line test edit preserves the factory's result, warning and timeout behavior. Reversing the replacement reproduces the parent file byte for byte. All 220 assertion lines and the production prefix are unchanged; no tests, assertions, guards, profiles or baselines were added or relaxed.
Original Linux CI evidence (run 37888552268 at 53245be9f):
- RIO job113685483992: strict Clippy exit101, redundant closure at disk_store.rs:2914; feature tests skipped.
- Workspace Test and Lint job113685484058: same strict Clippy exit101; nextest and doc tests skipped.
These are lint failures, not runtime mutation kills or known flakes.
Native macOS arm64 validation on tree7a5bf47e7533b29479bc85207416102f1337c566, LANG=en_US.UTF-8, one build job, CARGO_INCREMENTAL=0 and offline dependencies:
shellcheck --version && actionlint -> exit 0
cargo fmt --all --check -> exit 0
./scripts/check_unsafe_code_allowances.sh -> exit 0
./scripts/check_layer_dependencies.sh -> exit 0
./scripts/check_architecture_migration_rules.sh -> exit 0
./scripts/check_logging_guardrails.sh -> exit 0
./scripts/check_error_other_format_ratchet.sh -> exit 0
./scripts/check_no_tokio_io_uring.sh -> exit 0
./scripts/check_extension_schema_boundaries.sh -> exit 0
./scripts/check_body_cache_whitelist.sh -> exit 0
./scripts/check_s3s_footprint.sh -> exit 0
./scripts/check_test_count_ratchet.sh -> exit 0
./scripts/check_fips_wording.sh -> exit 0
./scripts/check_embedded_secrets.sh -> exit 0
make script-tests -> exit 0
python3 ./scripts/test_object_generation_protocol_model.py -> exit 0
python3 ./scripts/check_test_wiring.py -> exit 0
./scripts/check_no_planning_docs.sh -> exit 0
./scripts/check_uring_lane_lib_only.sh -> exit 0
All 19 checks: exit0, 312.161s.
$ cargo clippy -p rustfs -p rustfs-ecstore --all-targets --features rio-v2 -- -D warnings
Finished `dev` profile [unoptimized + debuginfo] target(s) in 20m 11s
exit0, 1213.210s total.
$ cargo nextest run --profile ci -p rustfs-ecstore --features rio-v2 slow_disk_operation_warns_once_while_pending_and_preserves_result --no-fail-fast
Finished `test` profile [unoptimized + debuginfo] target(s) in 4m 34s
PASS [0.026s] rustfs-ecstore disk::disk_store::tests::slow_disk_operation_warns_once_while_pending_and_preserves_result
Summary [ 0.170s] 1 test run: 1 passed, 5943 skipped
exit0, 281.920s total; 5943 existing filter skips.
warning: linker stderr: ld: __eh_frame section too large (max 16MB) to encode dwarf unwind offsets in compact unwind table, performance of exception handling might be affected
This linker warning is retained; flags, profiles and allowances are unchanged.
54 tracked Cargo.toml files parsed without duplicate keys. git diff --check passed. No new assertions were introduced, so no new runtime mutation is claimed. Native compilation and the existing helper test do not substitute for actual Linux RIO execution; the new integration CI run still needs to recover before the RPC merge.
Preservation and output hashes:
parent file SHA256: 8e59ee745e3aaaa22848b943d3b10f6353b515956282cde1f3af2c394200c9ac
final file SHA256: b69ba3237061b161448ae8b62b615885d5770b98a5b4a2479bcfbd70f1bd1f17
unchanged production prefix SHA256: 7903a6e288252c420231242d7b58a25fa2a7140ea3dd430cbe7f0dde11369dcb
RIO Linux failure log SHA256:421addb9868fd88e4109a922311e54f91f0dfd70862f8ba7fcaba193482ed90b
Workspace Linux failure log SHA256:e0aea8d0f014c022252a9534968e083c98cfc1fd760da445ecd04ec3f522e26a
19 output SHA256:35a54f28d707956ed1edb29d492c4dfe8a9fe753d103b111b5a2f1b5c41c2b98
RIO Clippy output SHA256:d6046e3868f2b3992fe11cdd04e9e98eb50a87d1234d9a70dbfeb154ff61638d
slow-operation output SHA256:605b7b258d2fd09c6ef2ab2ffc90d433d0509480f1e45125d20c823759e83608
Refs rustfs/backlog#2734.
* test(ecstore): isolate tier-free recovery fixture startup
The fixture constructor can take the production recovery lock and wait for 60 seconds before this test installs its queue and seeds metadata. Disable its background recovery only while creating this fixture, then restore the environment before calling the existing production recovery entrypoint.
Only the existing test setup changes. Its production entrypoint, default lock parameters, original assertions, 30-second timeout, cancellation and cleanup remain intact. No tests or assertions were added. Production and store/init.rs are byte-identical to the parent.
Independent acceptance ran on merge tree 8ec358cb42ba9ebb3bcfeafcef048a3f0366355b; the runner header reports parent HEAD, while before/after index and source hashes confirm this candidate. Worker task branch d8055766b8 has the same tree.
Every command used LANG=en_US.UTF-8. Task compilation also used CARGO_BUILD_JOBS=1, CARGO_INCREMENTAL=0, CARGO_NET_OFFLINE=true, an independent target, and empty RUSTC_WRAPPER/RUSTC_WORKSPACE_WRAPPER to keep compilers in its owned process group without changing repository configuration.
Command:
shellcheck --version && actionlint
Exit 0; observed output excerpt:
version: 0.11.0
Log SHA-256: fc9eec16b4191d058a295813ca9afbe9d2b37b59975da860170726be162a3924
Command:
cargo fmt --all --check
Exit 0; observed output excerpt:
(no stdout or stderr)
Log SHA-256: e3b0c44298fc1c149afbf4c8996fb92427ae41e4649b934ca495991b7852b855
Command:
./scripts/check_unsafe_code_allowances.sh
Exit 0; observed output excerpt:
(no stdout or stderr)
Log SHA-256: e3b0c44298fc1c149afbf4c8996fb92427ae41e4649b934ca495991b7852b855
Command:
./scripts/check_layer_dependencies.sh
Exit 0; observed output excerpt:
Layer dependency guard passed (no new reverse dependencies/cycles).
Log SHA-256: 69c6cb1896c0586b28d1167380d3260d99d8017a4265c8df5a2348f289e93a12
Command:
./scripts/check_architecture_migration_rules.sh
Exit 0; observed output excerpt:
Architecture migration rules passed.
Log SHA-256: 28790e70d070607ed4de0704f292a65030be92ec224ef106d4b72fd8296f896e
Command:
./scripts/check_logging_guardrails.sh
Exit 0; observed output excerpt:
✅ Logging guardrails check passed
Log SHA-256: 7adf41abff8ed8f7617decf86498888ddde78937099dd9ce47f6d359dfc2d317
Command:
./scripts/check_error_other_format_ratchet.sh
Exit 0; observed output excerpt:
error(format!) ratchet passed (total call sites: 579).
Log SHA-256: 66172aef18f150ba0605af122e26ffcf91f6dd86f43e38b05e2e982ce8d783ee
Command:
./scripts/check_no_tokio_io_uring.sh
Exit 0; observed output excerpt:
(no stdout or stderr)
Log SHA-256: e3b0c44298fc1c149afbf4c8996fb92427ae41e4649b934ca495991b7852b855
Command:
./scripts/check_extension_schema_boundaries.sh
Exit 0; observed output excerpt:
Extension schema boundaries passed.
Log SHA-256: f13d0acd0e2ce5d9815241b12b4dab7d01cc11b9b65bcf6dfb7812c878532a51
Command:
./scripts/check_body_cache_whitelist.sh
Exit 0; observed output excerpt:
Body-cache whitelist guard passed.
Log SHA-256: 6b828847b0383950ee6eef9cd846d0c322c945e47efd4c7d1ad881178810d6bb
Command:
./scripts/check_s3s_footprint.sh
Exit 0; observed output excerpt:
s3s footprint OK: files importing s3s is 195 (baseline files=195)
s3s footprint OK: s3s s3_error! lines is 1584 (baseline s3_error_lines=1584)
s3s footprint OK: ecstore files referencing s3s is 36 (baseline ecstore_files=36)
✅ s3s footprint baseline check passed
Log SHA-256: 47f53d04ef645618563fe1f22abc64667f19078f1974feedcca46e4f2c016844
Command:
./scripts/check_test_count_ratchet.sh
Exit 0; observed output excerpt:
test-count ratchet OK: 52 crates, 21340 tests, 164 ignored match .config/test-count-baseline.txt
Log SHA-256: 618a5e330e933de0c6ff484546df401457024a88bb5986d11d588794e33dab60
Command:
./scripts/check_fips_wording.sh
Exit 0; observed output excerpt:
FIPS wording guard passed (README.md and CHANGELOG.md contain no forbidden claims).
KMS wording guard passed (crates/kms claims no Vault KV2 Transit wrapping).
Log SHA-256: 99221cc9979c480ec4f46f50d76c52b18c72fe499f12e15e4f4a22a1b8bfb140
Command:
./scripts/check_embedded_secrets.sh
Exit 0; observed output excerpt:
Embedded secret guard passed (no private key material or provider credential literal in the tree).
Log SHA-256: 437a3ceba32094d2901889965ad191e411aad32cdb18bf0b8659e818011c406b
Command:
make script-tests
Exit 0; observed output excerpt:
test_cli_requires_the_repository_default_branch (__main__.SelfTests.test_cli_requires_the_repository_default_branch) ... ok
test_config_rejects_duplicate_and_invalid_entries (__main__.SelfTests.test_config_rejects_duplicate_and_invalid_entries) ... ok
test_freshness_boundaries (__main__.SelfTests.test_freshness_boundaries) ... ok
test_grace_requires_two_successful_queries_with_no_history (__main__.SelfTests.test_grace_requires_two_successful_queries_with_no_history) ... ok
test_invalid_api_evidence_fails_closed (__main__.SelfTests.test_invalid_api_evidence_fails_closed) ... ok
test_report_retains_every_workflow (__main__.SelfTests.test_report_retains_every_workflow) ... ok
test_requests_filter_schedule_default_branch_and_success_on_server (__main__.SelfTests.test_requests_filter_schedule_default_branch_and_success_on_server) ... ok
Ran 10 tests in 0.042s
OK
Ran 30 tests in 20.627s
OK
Ran 10 tests in 2.268s
OK
Ran 33 tests in 0.127s
OK
Ran 7 tests in 0.002s
OK
Ran 7 tests in 0.001s
OK
Ran 4 tests in 0.108s
OK
Ran 3 tests in 0.564s
OK
PASS: cold stampede script self-test
Log SHA-256: 75fda2d069a373c63667b908baaf99e6591aab9099a739b0c4c22ceb6da8d68c
Command:
python3 ./scripts/test_object_generation_protocol_model.py
Exit 0; observed output excerpt:
test_an_unavailable_decision_quorum_cannot_choose_a_candidate (__main__.ObjectGenerationProtocolModel.test_an_unavailable_decision_quorum_cannot_choose_a_candidate) ... ok
test_counter_allocation_requires_persistence_and_is_consumed_once (__main__.ObjectGenerationProtocolModel.test_counter_allocation_requires_persistence_and_is_consumed_once) ... ok
test_counter_restart_burns_persisted_allocations_before_or_after_issue (__main__.ObjectGenerationProtocolModel.test_counter_restart_burns_persisted_allocations_before_or_after_issue) ... ok
test_crash_after_promise_before_reply_does_not_restore_an_older_ballot (__main__.ObjectGenerationProtocolModel.test_crash_after_promise_before_reply_does_not_restore_an_older_ballot) ... ok
test_duplicate_or_unconfigured_voters_cannot_form_a_quorum (__main__.ObjectGenerationProtocolModel.test_duplicate_or_unconfigured_voters_cannot_form_a_quorum) ... ok
test_empty_or_unobserved_minority_slot_can_retry_without_allocating_another_revision (__main__.ObjectGenerationProtocolModel.test_empty_or_unobserved_minority_slot_can_retry_without_allocating_another_revision) ... ok
test_equal_ballot_cannot_accept_different_bytes_and_lower_ballot_is_rejected (__main__.ObjectGenerationProtocolModel.test_equal_ballot_cannot_accept_different_bytes_and_lower_ballot_is_rejected) ... ok
test_forgetting_tombstone_authority_allows_an_aba_create (__main__.ObjectGenerationProtocolModel.test_forgetting_tombstone_authority_allows_an_aba_create) ... ok
test_issuing_before_persistence_reuses_a_ballot_after_restart (__main__.ObjectGenerationProtocolModel.test_issuing_before_persistence_reuses_a_ballot_after_restart) ... ok
test_lost_ack_and_restart_preserve_the_chosen_operation (__main__.ObjectGenerationProtocolModel.test_lost_ack_and_restart_preserve_the_chosen_operation) ... ok
test_minority_accepted_work_is_adopted_and_finished (__main__.ObjectGenerationProtocolModel.test_minority_accepted_work_is_adopted_and_finished) ... ok
test_omitting_highest_accepted_adoption_has_a_double_choice_counterexample (__main__.ObjectGenerationProtocolModel.test_omitting_highest_accepted_adoption_has_a_double_choice_counterexample) ... ok
test_quorum_choice_does_not_require_an_uncontacted_disk_to_learn_it (__main__.ObjectGenerationProtocolModel.test_quorum_choice_does_not_require_an_uncontacted_disk_to_learn_it) ... ok
test_recovery_failure_or_stale_expected_head_does_not_allocate_a_successor (__main__.ObjectGenerationProtocolModel.test_recovery_failure_or_stale_expected_head_does_not_allocate_a_successor) ... ok
test_replacing_voters_with_empty_state_can_choose_a_conflicting_value (__main__.ObjectGenerationProtocolModel.test_replacing_voters_with_empty_state_can_choose_a_conflicting_value) ... ok
test_skipping_predecessor_recovery_chooses_a_broken_revision_chain (__main__.ObjectGenerationProtocolModel.test_skipping_predecessor_recovery_chooses_a_broken_revision_chain) ... ok
test_stale_publication_rollback_and_gc_cannot_change_the_winner (__main__.ObjectGenerationProtocolModel.test_stale_publication_rollback_and_gc_cannot_change_the_winner) ... ok
test_superseded_operation_identity_cannot_be_reused_for_different_input (__main__.ObjectGenerationProtocolModel.test_superseded_operation_identity_cannot_be_reused_for_different_input) ... ok
test_superseded_operation_retry_returns_its_original_successor (__main__.ObjectGenerationProtocolModel.test_superseded_operation_retry_returns_its_original_successor) ... ok
test_tombstone_and_null_marker_payloads_do_not_reset_authority_revision (__main__.ObjectGenerationProtocolModel.test_tombstone_and_null_marker_payloads_do_not_reset_authority_revision) ... ok
test_tombstone_rejects_a_delayed_never_present_create (__main__.ObjectGenerationProtocolModel.test_tombstone_rejects_a_delayed_never_present_create) ... ok
test_unlearned_predecessor_is_recovered_before_the_second_slot (__main__.ObjectGenerationProtocolModel.test_unlearned_predecessor_is_recovered_before_the_second_slot) ... ok
Ran 24 tests in 0.929s
OK
Log SHA-256: 0b4be926efb0609d497ae58930340153f01957b606206cbbdb3e0d47eb29d3a2
Command:
python3 ./scripts/check_test_wiring.py
Exit 0; observed output excerpt:
OK: e2e modules, runner selection, fuzz matrices, profiles, scheduled alerts, and Quick Checks are wired
Log SHA-256: 21f676f458451525535ab0ef73d6a9c0d9774e16d76a4d19e396ad34936dd39c
Command:
./scripts/check_no_planning_docs.sh
Exit 0; observed output excerpt:
(no stdout or stderr)
Log SHA-256: e3b0c44298fc1c149afbf4c8996fb92427ae41e4649b934ca495991b7852b855
Command:
./scripts/check_uring_lane_lib_only.sh
Exit 0; observed output excerpt:
OK: no 'uring_'-matching test functions under crates/ecstore/tests; --lib is safe
Log SHA-256: 43f3f851e27ec2ab9e34d6b05fefdede38af992699c473904bfe6f53a46d52e3
Command:
cargo nextest run --profile ci -p rustfs-ecstore --lib --features rio-v2 -E 'test(=bucket::lifecycle::bucket_lifecycle_ops::tests::tier_free_version_recovery_production_entrypoint_enqueues_seeded_item)' --no-fail-fast
Exit 0; wall 234.399s.
Observed output:
Finished `test` profile [unoptimized + debuginfo] target(s) in 3m 49s
Starting 1 test across 1 binary (5871 tests skipped)
SETUP PASS [ 0.006s] ecstore-base-stack: sh -c 'echo RUST_MIN_STACK=4194304 >> "$NEXTEST_ENV"'
PASS [ 0.440s] rustfs-ecstore bucket::lifecycle::bucket_lifecycle_ops::tests::tier_free_version_recovery_production_entrypoint_enqueues_seeded_item
Summary [ 0.586s] 1 test run: 1 passed, 5871 skipped
Log SHA-256: 34e973f86e8dc2102c1a0e483e0ac802ce7df3d1b40a603932f62c0805c4de27
Command:
cargo nextest run --profile ci -p rustfs-ecstore --lib --features rio-v2 -E 'test(=bucket::lifecycle::bucket_lifecycle_ops::tests::tier_free_version_recovery_eight_coordinators_hold_one_leader_and_take_over) | test(=store::bucket::tests::creation_commit_backfill_runs_under_the_config_write_fence) | test(=store::bucket::tests::orphaned_bucket_recovery_requires_every_volume_absent_and_is_retryable)' --no-fail-fast
Exit 0; wall 4.010s.
Observed output:
Finished `test` profile [unoptimized + debuginfo] target(s) in 0.43s
Starting 3 tests across 1 binary (5869 tests skipped)
SETUP PASS [ 0.005s] ecstore-base-stack: sh -c 'echo RUST_MIN_STACK=4194304 >> "$NEXTEST_ENV"'
PASS [ 0.068s] rustfs-ecstore bucket::lifecycle::bucket_lifecycle_ops::tests::tier_free_version_recovery_eight_coordinators_hold_one_leader_and_take_over
PASS [ 0.637s] rustfs-ecstore store::bucket::tests::creation_commit_backfill_runs_under_the_config_write_fence
PASS [ 0.846s] rustfs-ecstore store::bucket::tests::orphaned_bucket_recovery_requires_every_volume_absent_and_is_retryable
Summary [ 0.987s] 3 tests run: 3 passed, 5869 skipped
Log SHA-256: b5da2a927ff18ca9aa217762f7186a6d4371bd9e8feedd7b77940e10b64dffac
Command:
cargo clippy -p rustfs -p rustfs-ecstore --all-targets --features rio-v2 -- -D warnings
Exit 0; wall 1084.508s.
Observed output:
Finished `dev` profile [unoptimized + debuginfo] target(s) in 18m 03s
Log SHA-256: 28f98be7970faaa71cd118da1da4e6e2e3d23b9dda7e6449eaa7a37e741260b5
Compiled mutation command (unchanged for each red and restored run):
cargo nextest run --profile ci -p rustfs-ecstore --lib --features rio-v2 -E 'test(=bucket::lifecycle::bucket_lifecycle_ops::tests::tier_free_version_recovery_production_entrypoint_enqueues_seeded_item)' --no-fail-fast
M1: reverse only this test-local constructor flag from false to true. The mutant compiled, then failed the original recovered-item queue assertion after 30.481s. Restoring the entire observed candidate passed in 0.460s. Red log 55bf3b1535529ac0bcfb8071f9b56f9f40631007a85c315367e7e93786d65293; restored log 204d5fbe4574a7917f5e740a795d864fe28a7e7fb5e33d27b0a9cf68e2f17f52.
M2: replace only the actual awaited enqueue with false, consuming its original inputs. The mutant compiled, then failed the same original queue assertion after 30.516s. Restoring the entire observed candidate passed in 0.457s. Red log 89186330413215e231e2b06bf013cc31097ea8f4a106d91421a36d6c8986a7fd; restored log e848b42d2302401afa5dbcd9e37bf76d2e56a96ab6de5a488513b997f53c5e9b.
All temporary observation and mutation code was removed before final acceptance. Historical baseline and diagnostic failures remain recorded; they are not mutation credit. The separate Object Lock intent-retry diagnostic still fails, and these native macOS checks do not establish Linux CI recovery.
* refactor(rpc): serve internode rpc bodies with the owned Body (#8441)
* test(rpc): pin the wire bytes of every internode rpc response body
rustfs/backlog#2747 batch 4, step 1 of 2, on the s3s bodies the rpc paths
serve today. Each response body builder in storage/rpc/http_service.rs is
served through hyper's HTTP/1.1 server and the raw bytes the peer reads are
compared with goldens captured from this commit: status, content-length or
chunked framing, chunk boundaries, the terminating chunk, HEAD, and the cut
connection a failure after data produces. The capability responses carry a
per-process proof, so their length header and payload are checked against
the body they encode instead.
The read_file_stream response builder moves into read_file_stream_response
so the test reaches the same code the handler runs; behaviour is unchanged.
* refactor(rpc): serve internode rpc bodies with the owned Body
rustfs/backlog#2747 batch 4, step 2 of 2. The internode rpc HTTP paths
build their responses with rustfs_s3_types::Body: Body::from_stream
replaces Body::from(StreamingBlob::wrap(..)) at the six stream sites, and
the in-memory and empty bodies keep their constructors. InternodeRpcService
converts its responses into the wrapped service's response body through
From<Body>, so the server keeps composing it with the s3s service as
before. The wire-byte goldens from the previous commit pass unchanged;
storage/rpc no longer names s3s.
* fix(rpc): simplify wire test response builder type
Name the boxed response builder used by the capability wire test so
all-target Clippy can check it without a type-complexity warning.
Validation (LANG=en_US.UTF-8, CARGO_INCREMENTAL=0):
- cargo clippy -p rustfs --all-targets -- -D warnings: exit 0.
- cargo nextest run --profile ci -p rustfs --lib -E 'test(/storage::rpc::/)': 279 tests run, 279 passed, 4842 skipped.
- All 19 quick-check commands, including script-tests, both Python scripts and cargo fmt --all --check: exit 0.
- Footprint baseline regenerated: files=194, s3_error_lines=1584, ecstore_files=36; test-count baseline regenerated: 52 crates, 21228 tests, 164 ignored.
- All 54 Cargo.toml files parse without duplicate keys.
No assertion was added or changed by this lint fix. R1-R8 are retained
in the existing PR verification record.
* fix(ecstore): resume fenced Object Lock creation intents
* fix(ecstore): bound creation retry futures on the stack
* fix(ecstore): require native proof for creation retries
* fix(ecstore): bound bucket creation stack usage (#8466)
* fix(ecstore): retain creation fences through rollback
Keep lifecycle, metadata-transaction, and namespace guards around rollback and metadata cleanup. Preserve the original publication error when a guard is lost. Build the timestamp retry fixture through the real interrupted producer so it checks a valid pending creation.
Validated source tree: 4e8e207999.
Two independent source reviews found no blocking regression.
Validation (LANG=en_US.UTF-8):
- bash -c 'shellcheck --version && actionlint' — exit 0
- cargo fmt --all --check — exit 0
- ./scripts/check_unsafe_code_allowances.sh — exit 0
- ./scripts/check_layer_dependencies.sh — exit 0
- ./scripts/check_architecture_migration_rules.sh — exit 0
- ./scripts/check_logging_guardrails.sh — exit 0
- ./scripts/check_error_other_format_ratchet.sh — exit 0
- ./scripts/check_no_tokio_io_uring.sh — exit 0
- ./scripts/check_extension_schema_boundaries.sh — exit 0
- ./scripts/check_body_cache_whitelist.sh — exit 0
- ./scripts/check_s3s_footprint.sh — exit 0
- ./scripts/check_test_count_ratchet.sh — exit 0
- ./scripts/check_fips_wording.sh — exit 0
- ./scripts/check_embedded_secrets.sh — exit 0
- make script-tests — exit 0
- python3 ./scripts/test_object_generation_protocol_model.py — exit 0
- python3 ./scripts/check_test_wiring.py — exit 0
- ./scripts/check_no_planning_docs.sh — exit 0
- ./scripts/check_uring_lane_lib_only.sh — exit 0
- cargo nextest run --profile ci -p rustfs-ecstore --lib --locked --no-fail-fast — 5894 passed, 0 skipped
- cargo check -p rustfs-ecstore --all-targets --locked — exit 0
- cargo nextest run --profile ci -p rustfs --lib --locked --no-fail-fast -E 'test(=admin::handlers::table_catalog::tests::cold_strong_registration_completes_while_other_bucket_backup_waits) | test(=admin::handlers::table_catalog::tests::commit_backend_borrowed_put_preserves_owner_authorization_and_precondition) | test(=admin::handlers::table_catalog::tests::disable_table_bucket_preserves_objects_and_can_be_reenabled) | test(=admin::handlers::table_catalog::tests::load_table_handler_negotiates_vended_credentials_without_breaking_metadata_only_callers) | test(=admin::handlers::table_catalog::tests::namespace_property_handler_updates_object_backed_catalog_and_maps_errors) | test(=app::object::get::tests::execute_get_object_resumes_from_relocated_pool_without_splicing_body) | test(=table_catalog::store::tests::catalog_writer_loss_prevents_recovery_storage_commit)' — 7 passed
- cargo nextest run --profile ci -p rustfs --lib --locked --no-fail-fast -E 'test(unsupported_remote_disk_)' — 2 passed
Additional fault-injection evidence remains incomplete. Timestamp M1 reached its original assertion; its restore run is being collected. M2–M5 and the three planned rollback faults have not run. An automated risk check interrupted the timestamp proof task; no assertions or validation settings were weakened.
* fix(ecstore): box nested bucket rollback state
Keep the guarded rollback future behind a heap allocation so bucket creation does not store its nested state inline. The rollback body, all three fence references and pending-proof revocation order are unchanged.
The same three original Linux tests overflowed on both e611 CI attempts. Local format, ecstore test compilation and all 19 quick-check steps pass; the original Linux feature lanes still need to validate this candidate.
* test(ecstore): isolate rollback guard loss fixtures
Run the three rollback metadata-cleanup cases in separate test processes so reused /proc/self/fd aliases from earlier cases cannot invalidate their creation witnesses. Keep the original fixture, guards, assertions and io_uring test selection, and verify that each child runs the intended case successfully.
Update the generated test-count baseline for the added macro arm.
Validation: cargo check -p rustfs-ecstore --tests; cargo fmt --all --check; all 19 quick-check commands, including make script-tests and the two Python checks. Linux io_uring acceptance and fault controls remain pending.
* refactor(replication): move the replication config onto gateway types (#8443)
* wip: T1.4 replication family
* refactor(replication): move the replication config onto gateway types
The replication configuration family leaves s3s: rustfs-replication no longer depends on s3s, and ECStore bucket metadata, the scanner and the rustfs app/admin layers hold rustfs_gateway_types::persistence::PersistedReplicationConfiguration. The persisted XML goes through the gateway persistence codec behind a new BucketConfigXml trait (the 39 byte goldens pass with unchanged fixtures); the legacy s3s edge converts once in storage/ecfs.rs through storage/s3_compat/replication.rs; the scanner data usage cache keeps its MessagePack bytes through a serde mirror pinned by bytes recorded from s3s 5761ddfe. rustfs-gateway-types 0.32.0 is pinned once (rev 772d3385). Refs rustfs/backlog#2745.
* test: use S3 serializer for list response tests
* fix(ci): restore integration test compilation
Gate the diagnostic test import with its feature, initialize the conditional
MetadataWriteStore fields explicitly, and decode the scanner replication fixture
through its existing cache boundary. Preserve the original fixture and assertions.
Validation was run on tree 544acb80c0 at 6003d4b8.
LANG=en_US.UTF-8; no source changes follow validation.
Quick checks: 19/19 passed.
- shellcheck --version && actionlint: exit 0
- cargo fmt --all --check: exit 0
- ./scripts/check_unsafe_code_allowances.sh: exit 0
- ./scripts/check_layer_dependencies.sh: exit 0
- ./scripts/check_architecture_migration_rules.sh: exit 0
- ./scripts/check_logging_guardrails.sh: exit 0
- ./scripts/check_error_other_format_ratchet.sh: exit 0
- ./scripts/check_no_tokio_io_uring.sh: exit 0
- ./scripts/check_extension_schema_boundaries.sh: exit 0
- ./scripts/check_body_cache_whitelist.sh: exit 0
- ./scripts/check_s3s_footprint.sh: exit 0
- ./scripts/check_test_count_ratchet.sh: exit 0
- ./scripts/check_fips_wording.sh: exit 0
- ./scripts/check_embedded_secrets.sh: exit 0
- make script-tests: exit 0
- python3 ./scripts/test_object_generation_protocol_model.py: exit 0
- python3 ./scripts/check_test_wiring.py: exit 0
- ./scripts/check_no_planning_docs.sh: exit 0
- ./scripts/check_uring_lane_lib_only.sh: exit 0
Darwin validation:
- cargo clippy --locked -p rustfs-ecstore -p rustfs-scanner --all-targets -- -D warnings: exit 0
- cargo nextest run --locked -p rustfs-ecstore -p rustfs-scanner --lib -E 'test(config::com::tests::save_config_preserves_retryable_system_volume_errors) | test(scanner_io::tests::replication_maintenance_refreshes_compacted_stats_across_cycles_and_retries)' --test-threads 1 --no-fail-fast: exit 0, 2/2 passed
- cargo clippy --locked -p rustfs-ecstore -p rustfs-scanner --all-targets --features rustfs-ecstore/diagnostic-read-version-caller,rustfs-scanner/diagnostic-read-version-caller -- -D warnings: exit 101
- cargo nextest run --locked -p rustfs-ecstore -p rustfs-scanner --lib --features rustfs-ecstore/diagnostic-read-version-caller,rustfs-scanner/diagnostic-read-version-caller -E 'test(config::com::tests::save_config_preserves_retryable_system_volume_errors) | test(scanner_io::tests::replication_maintenance_refreshes_compacted_stats_across_cycles_and_retries) | test(cluster::rpc::runtime_sources::tests::) | test(config::com::tests::diagnostic_config_write_retains_origin_and_api_for_results_and_cancellation)' --test-threads 1 --no-fail-fast: exit 0, 14/14 passed
Diagnostic strict Clippy remains red on four unchanged production lints:
runtime_sources.rs if_same_then_else and three redundant_locals reports in
set_disk/core/metadata_read.rs and set_disk/read.rs. Their source ranges and
feature wiring match the base; the base was not compiled separately.
Linux aarch64 validation (Rust/Clippy 1.99.0, UID 501, jobs=1):
- cargo clippy --all-targets -- -D warnings: exit 0; Finished dev profile in 68m 12s
- cargo clippy -p rustfs -p rustfs-ecstore --all-targets --features rio-v2 -- -D warnings: exit 0; Finished dev profile in 49m 45s
Source HEAD/tree and clean status matched before and after both commands.
No compiler or stack overrides were used. protoc 35.1 was installed from its
verified ARM64 release; flatc was absent and not invoked by the workspace build scripts.
The local environment differs from hosted CI.
* fix(ecstore): recover delete batches with missing versions
Treat an explicit non-null version as a no-op when it is absent from both recovery sources. Keep rejecting missing null versions, delete-marker mismatches and conflicting local generations.
Add returning-disk and partial-receipt regressions, extend journal boundary checks, and regenerate the ecstore test-count baseline. Production faults exercise the 24 added assertion sites and 7 semantic checks: 33 faults failed at the expected sites and passed after full restoration.
Validation (LANG=en_US.UTF-8, one build job):
cargo nextest run --locked -p rustfs-ecstore --lib -E 'test(delete_recovery) | test(quorum_delete_cleanup_coordinator_binds_single_and_batch) | test(crash_delete_retry_batch_reclaims_inline_and_reserved_external)' --test-threads 1 --no-fail-fast
Summary [ 34.580s] 69 tests run: 69 passed, 6058 skipped; exit 0
cargo clippy --locked -p rustfs-ecstore --all-targets -- -D warnings
Finished `dev` profile [unoptimized + debuginfo] target(s) in 5m 37s; exit 0
All 19 final Quick Checks completed with exit 0:
- shellcheck --version && actionlint
- cargo fmt --all --check
- ./scripts/check_unsafe_code_allowances.sh
- ./scripts/check_layer_dependencies.sh
- ./scripts/check_architecture_migration_rules.sh
- ./scripts/check_logging_guardrails.sh
- ./scripts/check_error_other_format_ratchet.sh
- ./scripts/check_no_tokio_io_uring.sh
- ./scripts/check_extension_schema_boundaries.sh
- ./scripts/check_body_cache_whitelist.sh
- ./scripts/check_s3s_footprint.sh
- ./scripts/check_test_count_ratchet.sh
- ./scripts/check_fips_wording.sh
- ./scripts/check_embedded_secrets.sh
- make script-tests
- python3 ./scripts/test_object_generation_protocol_model.py
- python3 ./scripts/check_test_wiring.py
- ./scripts/check_no_planning_docs.sh
- ./scripts/check_uring_lane_lib_only.sh
test-count ratchet OK: 52 crates, 22009 tests, 165 ignored match .config/test-count-baseline.txt
PASS: cold stampede script self-test
model: states=51023, transitions=167520, terminal=732, chosen_candidates=2
multi-slot round schedules: 640
OK: e2e modules, runner selection, fuzz matrices, profiles, scheduled alerts, and Quick Checks are wired
* fix(ecstore): retry tier commit intent lock contention
Retry transient namespace lock timeouts within the existing three-attempt
CAS budget. Reload and verify the original mutation identity after
contention, and preserve the final typed error when attempts are exhausted.
Cover save/read contention, exhausted retries, replacement identities and
CAS reconciliation without changing the existing timeout or lock policy.
Validation:
- cargo nextest run --locked -p rustfs-ecstore --features test-util --lib -E 'test(tier_mutation)' --test-threads 1 --no-fail-fast: 52 passed.
- cargo nextest run --locked -p rustfs-ecstore --features test-util --lib -E 'test(intent_advance_) | test(=services::tier::tier::tests::caller_cancellation_after_durable_prepare_does_not_hide_the_mutation)' --test-threads 1 --no-fail-fast: 5 passed.
- cargo clippy --locked -p rustfs-ecstore --all-targets -- -D warnings: passed.
- cargo clippy --locked -p rustfs-ecstore --all-targets --features test-util -- -D warnings: passed.
- 27 mutation pairs failed at the intended assertions and passed after restoring the production code.
- bash scripts/check_test_count_ratchet.sh --update, then bash scripts/check_test_count_ratchet.sh: 52 crates, 22013 tests, 165 ignored; ecstore increases from 6581 to 6585.
* refactor(lifecycle): replace s3s domain types with Gateway DTOs
Keep lifecycle rules, restore requests, Object Lock, and scanner cached tags in domain types. Convert at the existing HTTP boundary while preserving legacy XML, scanner cache bytes, date ordering, and tombstones.
Linux test validation before the final import-path-only adjustment:
cargo nextest run -p rustfs-ecstore bucket_config_goldens
39 passed; 6211 skipped by selector.
cargo nextest run --lib -p rustfs -E 'test(/test_zero_rule_lifecycle_tombstone_recognition/) | test(/test_local_lifecycle_staleness_axis_selection/) | test(/test_lifecycle_staleness_preserves_legacy_submillisecond_axis/) | test(/test_merge_incoming_lifecycle/) | test(/test_lifecycle_expiry_statement_matrix/) | test(/test_lifecycle_expiry_subset_xml/) | test(/put_bucket_lifecycle_validation_errors_keep_their_s3_code/) | test(/execute_put_bucket_lifecycle_configuration_rejects_missing_configuration/) | test(/execute_restore_object_rejects_missing_restore_request/) | test(/execute_restore_object_rejects_select_type/) | test(/execute_restore_object_maps_failures_to_typed_s3_errors/) | test(/execute_restore_object_returns_internal_error_when_store_uninitialized/) | test(/restore_request_conversion_preserves_legacy_output_values/) | test(/restore_request_conversion_preserves_presence_and_raw_enums/) | test(/restore_request_conversion_distinguishes_absent_and_empty_containers/)'
25 passed; 5318 skipped by selector.
cargo nextest run --profile ci -p rustfs-ecstore -p rustfs-lifecycle -p rustfs-replication -p rustfs-scanner
7872 passed; 21 skipped under the existing CI profile.
Previously verified mutation controls: 39 production faults, 41 RED/GREEN pairs.
cargo clippy --locked -p rustfs -p rustfs-ecstore -p rustfs-lifecycle -p rustfs-scanner --all-targets -- -D warnings
Passed in 40m 45s before the import-path adjustment, then in 1m 59s on
the final source with warnings denied. The final adjustment imports the
existing helper through the object domain facade; fmt and the architecture
check also passed.
Part of rustfs/backlog#2745. The remaining ecstore configuration families follow separately.
* refactor(admin): move error types to rustfs-s3-types (#8440)
* refactor(protocols): move error types to rustfs-s3-types by codemod
T1.2 batch 3 (rustfs/backlog#2743): s3select-api, s3select-query and
the error items of protocols move off s3s by scripts/codemods, run with
--hold crates/protocols/src/sftp/errors.rs. That file stays on s3s: it
downcasts the backend error, whose runtime type is still s3s::S3Error.
- rustfs/src/protocols/client.rs: the two webdav session methods now
return rustfs_s3_types::S3Result and convert the s3s error with From.
- codemod: a child module that calls s3_error! through `use super::*`
gets its own `use rustfs_s3_types::s3_error;` once its parent imports
that macro by name; otherwise it is listed as GLOB.
- s3-types: a table test pins code, status and the s3s error the legacy
edge renders for every variant a moved call site names.
- s3_error_lines 1585 -> 1584 (a lifecycle comment reworded).
* refactor(admin): move error types to rustfs-s3-types (WIP, no build)
T1.2 batch 4 (rustfs/backlog#2743), work in progress: rustfs check
still fails with 16 errors (listed in the PR body).
- codemod: --facade PATH treats a crate-local re-export of s3s's error
items like s3s for the four names (9 new self-tests, 9 mutants red).
- codemod output for rustfs/src/admin and rustfs/src/site_replication
(--facade crate::storage_api::site_replication::s3); 6 glob children
get an explicit rustfs_s3_types::s3_error import.
- admin/router.rs: the Operation trait returns rustfs_s3_types::S3Result.
- error.rs: From<ApiError> builds the rustfs-s3-types error; the s3s
conversion is derived from it.
* fix(admin): convert error types at the s3s boundary
Batch 4 of T1.2 left the admin and site_replication modules returning
rustfs_s3_types::S3Error while their callers and callees on the storage
and auth side still speak s3s::S3Error. Convert at each edge with the
existing From impls in rustfs-s3-types (compat_s3s.rs) instead of
changing any code, message or status:
- admin/auth.rs, admin/handlers/object_zip_download.rs,
admin/handlers/table_catalog/mod.rs: map the s3s result returned by
check_key_valid / authorize_request / authorize_internal_object_request
into the admin error type with .map_err(Into::into).
- storage/ecfs.rs: the s3s-facing side maps the admin site_replication
gate results the other way.
- admin/handlers/pools.rs: ApiError.code stays s3s::S3ErrorCode, so the
rustfs code is converted with .into() when the struct is built, in
production and in the two tests that build or compare it.
- admin/handlers/heal.rs, admin/runtime_sources.rs: the router returns
s3s errors (S3Route impl); the expected rustfs code is converted with
.into() before the comparison.
- admin/handlers/table_catalog/mod.rs: iceberg_rest_error takes
&'static str so the Cow<'static, str> custom code borrows it; every
caller already passes an ICEBERG_ERROR_* constant or a &'static str.
- admin/handlers/quota.rs: drop a now same-type .into() after s3_error!.
- storage_api.rs, site_replication/mod.rs: drop re-exports and an import
that no longer have users (clippy -D warnings).
cargo check -p rustfs --all-targets and cargo clippy -p rustfs
--all-targets -- -D warnings are clean; the s3s footprint baseline is
unchanged at files=172, s3_error_lines=303, ecstore_files=28.
Refs rustfs/backlog#2743, rustfs/backlog#2734.
* refactor(ecstore): move bucket configs onto gateway persisted types (#8709)
* refactor(ecstore): move bucket configs onto gateway persisted types
Finish the ECStore batch of T1.4 (rustfs/backlog#2745). The remaining
eleven bucket configuration families (Website, CORS, Tagging, Versioning,
ObjectLock, Encryption, Notification, Logging, Accelerate, RequestPayment,
PublicAccessBlock) now read and write their stored XML through the
rustfs-gateway-types persistence and cors_tagging codecs instead of the
s3s codec, so the legacy s3s bucket-config codec and its
s3gate-metadata-xml compat marker are removed.
- ecstore: BucketConfigXml is implemented by the gateway codec for every
family; metadata, metadata_sys, versioning, object lock, replication and
lifecycle consumers take the persisted types. Region moves to
rustfs_s3_types::Region, and the ETag conversion to s3s moves to the
rustfs edge with its tests. Only warm_backend.rs and warm_backend_s3.rs
still reference s3s, held for T1.5 (rustfs/backlog#2746).
- scanner: object lock cache, item actions and data usage use the
persisted types.
- rustfs: the legacy S3 edge converts each family between s3s DTOs and the
persisted types in storage/storage_api.rs by writing one codec's XML and
reading it with the other; Tagging converts field by field.
- Bump the rustfs-gateway-types pin to 21283e87.
The 39 persisted-byte goldens are unchanged and pass, so a document the
old codec wrote reads back and re-encodes to the same bytes. s3s footprint:
files 181 -> 146, ecstore_files 28 -> 2, s3_error_lines unchanged.
* test(app): prepare the PutBucketEncryption config from the request
The uninitialized-store test passed an empty persisted configuration, which the use case now validates before it reaches the store check, so it answered MalformedXML instead of InternalError. Convert the request's configuration the way the S3 edge does.
* style(app): format the PutBucketEncryption test
* refactor(s3-client): move remote error codes and DTO enums off s3s (#8783)
Part of T1.5 (rustfs/backlog#2746), batch 1.
* refactor(s3select): run queries on gateway DTO request shapes (#8794)
* refactor(s3select): run queries on an engine request built from gateway DTO shapes
Part of T1.5 (rustfs/backlog#2746), batch 2.
* fix(s3select): keep the select request conversion in the app layer
The layer guard forbids app -> storage; the conversion moves from storage/s3_compat/select.rs to app/select_request.rs unchanged.
* fix(heal): retry transient metadata conflicts (#8812)
* fix(heal): retry transient metadata conflicts
* chore: update heal test count baseline
* test: reserve nextest capacity for progress probes (#8818)
* test(e2e): use a non-default region for the differential negative control
Since main's region-aware CreateBucket and GetBucketLocation (#8739, #8743), us-east-1 answers an empty LocationConstraint and 200 for an owned re-create, so the control no longer reported its own region. Also refresh the e2e-full linux selection from the CI listing (682 tests).
* refactor(protocols): move StorageBackend to gateway types (#8825)
* refactor(protocols): move storage backend to gateway types
* test: update e2e test count baseline
* fix: preserve copy source keys and unknown lengths
* test: update protocols test count baseline
* fix(protocols): preserve typed SFTP backend errors
* test(protocols): use gateway list input in policy tests
* fix(protocols): clear clippy failures after T1.5 merge (#8831)
fix(protocols): clear clippy failures after merge
* fix(ecstore): finalize restore metadata after local copy-back (#8836)
* fix(ecstore): finalize restore metadata after local copy-back
* test(ecstore): ratchet added restore finalization cases
* fix(ecstore): keep restore copyback compatible with older peers
* fix(ecstore): gate restore copyback on fleet compatibility
* fix(ecstore): retain transition identity during restore copyback
* fix(ecstore): route committed restore copyback locally
* fix restore copyback cleanup
* fix(ecstore): honor restore probe test fallback
* refactor(storage): migrate option errors to domain types (#8839)
* refactor(s3): migrate private multipart validation errors (#8844)
* refactor(auth): derive policy conditions from AuthzFacts (#8442)
* refactor(auth): extract authorization facts
* test(e2e): update smoke selection baseline
* test(e2e): update full selection baseline
* refactor(admin): migrate health and prefix usage to owned requests (#8887)
* refactor(admin): add owned health router boundary
* refactor(admin): migrate prefix usage to owned request
* refactor(admin): complete the owned admin request boundary (#8949)
* refactor(admin): complete the owned admin request boundary
Every type implementing the owned admin `Operation` now serves the legacy
router through one blanket impl that converts the s3s request and response
at that single edge, so handlers register with `AdminOperation(&Handler {})`
and need no adapter statics; `LegacyAdminAdapter` is removed.
`AdminRequest` gains the accessors handlers read today (`remote_addr`,
`client_info`, `request_context`, `req_info`, `req_info_mut`,
`install_req_info`, `envelope_mut`, `take_body`), the request helpers gain
envelope forms (`authenticate_envelope_credentials`,
`object_store_from_envelope`, the table-catalog `*_from_envelope` helpers,
`KmsAdminAudit::from_envelope`, `AccountAuditContext::from_envelope`,
`CallerIdentity::resolve_envelope`), `read_compatible_admin_body` accepts
either body type, and the owned `Body` gains `store_all_limited` and
`Default`. `admin::operation::test_support::TestRequest` builds owned
requests for handler tests. Unused accessors carry a per-method dead-code
expectation that each migrating handler batch removes.
No handler behavior changes; the remaining migrations follow in batches.
* chore(admin): add the T1.12 admin handler codemod
The script rewrites an admin handler file from the s3s request and response
types to the owned boundary (imports, signatures, request field and helper
calls, body access, response construction, and test request literals) and
leaves a TODO comment at every site it cannot rewrite mechanically. It is
idempotent and is removed once T1.12 completes.
* refactor(admin): migrate replication and KMS handlers to owned requests (#8951)
Move the T1.12 batch 2 admin handlers (site_replication, kms_keys, event,
replication, diagnostics, on_demand_migration, audit, account, kms_rekey,
kms_key_lifecycle, object_data_cache, tls_debug, integrity) from the s3s
request/response types to the owned AdminRequest/AdminResponse boundary.
Handler logic, authorization order, error codes and response bytes are
unchanged; the request/response plumbing now goes through the envelope
accessors and helpers introduced by the owned boundary.
The shared response helpers (utils::json_response, utils::empty_response,
handlers::admin_json_response, target_descriptor::build_json_response,
profile::authorize_profile_request) and the site replication repair
executor still answer with the s3s types, so the migrated files carry
owned-boundary forms of those helpers and convert the repair response at
its call edge. The streaming console-log body and the two devnull drains
move from the s3s byte stream to the owned Body stream/frame API.
Drop the dead_code expectations for the owned accessors and envelope
helpers this batch starts using, and lower the s3s footprint baseline to
the measured 99 files.
Part of rustfs/backlog#2788 (rustfs/backlog#2734 T1.12).
* refactor(admin): add owned forms of the shared admin response helpers
Handler batches were copying `json_response`, `empty_response`,
`admin_json_response`, and `build_json_response` into each migrated file
because the shared helpers still return the s3s response. Provide owned
forms next to each legacy helper (`owned_json_response`,
`owned_empty_response`, `owned_admin_json_response`,
`owned_build_json_response`, `authorize_profile_envelope`) with parity
tests against the legacy shape, derive `Debug` for `AdminResponse` so
tests can use `expect_err`, and fix three codemod rules: strip owned
string conversions in test credentials, annotate non-literal URI parses,
and leave a rebound request parameter immutable.
* refactor(admin): use the shared owned response helpers in batch 2 (#8954)
Replace the per-file owned-boundary copies of json_response,
empty_response, admin_json_response, build_json_response and
authorize_profile_request in the batch 2 admin handlers with the shared
owned_* helpers and authorize_profile_envelope that the boundary branch
now provides, and restore expect_err in the handler tests now that
AdminResponse implements Debug; the `.err().expect(..)` stand-ins were
failing the base branch's clippy step (clippy::err_expect under
-D warnings). The site replication repair call edge
keeps its conversion because the repair executor still answers with the
s3s response type.
Drop the dead_code expectations on the shared owned helpers this batch
now uses in non-test code.
Part of rustfs/backlog#2788 (rustfs/backlog#2734 T1.12).
* fix(admin): preserve legacy body limit error responses
* refactor(admin): move policy, ILM and batch 3 handlers to owned requests (#8956)
Batch 3 of T1.12 (rustfs/backlog#2788, parent rustfs/backlog#2734) moves
14 admin handler files to AdminRequest/AdminResponse and the owned
Operation trait: ilm_transition, policies, idp_compat, config_admin,
plugins_instances, profile_admin, group, heal, object_zip_download,
kms_key_metadata, durability, inspect_archive, sts and
gateway_key_inventory. Handler logic, authorization order, error codes
and response bytes are unchanged; test request literals use
test_support::TestRequest.
The handlers use the shared owned helpers (owned_json_response,
owned_admin_json_response, owned_build_json_response,
authorize_profile_envelope). The dead_code expectations on the
AdminRequest accessors and RequestEnvelope::uri_mut they now exercise
are removed. The legacy target_descriptor::build_json_response keeps
only its equivalence test caller and carries the matching expectation
until it retires.
object_zip_download builds the s3s request the S3 gate still reads from
the owned request facts; heal authenticates through
authenticate_envelope_credentials; the trace, inspect-archive and zip
bodies use Body::from_stream. extract_plugin_instance_filters takes the
URI so extensions.rs keeps calling it, and the two idp_compat
delegations stay on the legacy trait until their delegates migrate.
The s3s footprint baseline drops from 99 to 90 files.
* test(admin): anchor the replication diff body contract on the call
The source-contract test looked for `let body = read_compatible_admin_body(`
on one line; rustfmt wrapped that binding after the handler moved to the
owned request boundary, so the marker was no longer found. Anchor on the
call itself so formatting cannot change what the contract checks.
* refactor(admin): migrate table catalog and batch 1 to owned requests (#8955)
* fix(admin): preserve catalog body limit response bytes
* refactor(admin): migrate identity and cluster handlers to owned requests (#8966)
* refactor(admin): migrate identity and cluster handlers to owned requests
Move the T1.12 batch 4 admin handlers (user, service_account, oidc, tier,
system, pools, quota, kms_dynamic, kms_backup, rebalance, module_switch,
bucket_meta, cluster_snapshot, realtime, bucket_recovery, profile) from the
s3s request/response types to the owned AdminRequest/AdminResponse
boundary. All 90 impl Operation blocks now take AdminRequest and return
AdminResponse; handler logic, authorization order, error codes and response
bytes are unchanged. The handlers use the shared owned helpers
(owned_json_response, owned_admin_json_response, authorize_profile_envelope,
the envelope-form runtime sources and audit constructors) instead of local
copies.
profile.rs drops the s3s forms of authorize_profile_request and
profile_not_implemented_response now that profile_admin.rs is on the
envelope gate too; profile_not_implemented_response answers with an
AdminResponse. The realtime metrics stream moves from the s3s ByteStream
adapter to Body::from_stream, and the rebalance idle-404 document still
renders through s3s::S3Error::to_http_response with its body converted at
the edge.
The idp_compat delegators ImportIamV2 and AddServiceAccountLdap move to the
owned trait as well: with their targets on both traits, the method-call
delegation became ambiguous between the owned impl and the router blanket
impl, so they now take the owned request directly.
KmsAdminAudit::from_request loses its last production caller with this batch
and is deleted; its comparison test now asserts the envelope facts directly.
Lower the s3s footprint baseline to the measured 68 files.
Part of rustfs/backlog#2788 (rustfs/backlog#2734 T1.12).
* refactor(admin): remove the legacy admin helpers batch 4 superseded
Batch 4 moved the last production callers of the s3s-typed admin helpers
to their owned forms, which left `admin_json_response`, `json_response`,
`authenticate_request`, `authorize_admin_request`, `object_store_from_req`
and `object_store_from_extensions` unused, along with the `Body`,
`S3Request` and `S3Response` re-exports of the admin `s3` facade. Remove
them. The owned forms keep their documentation; the behavior tests of the
removed `json_response` now exercise `owned_json_response`, and the
compat-prefix test asserts the owned response directly.
Two tests existed only to compare a removed legacy helper with its owned
twin (`owned_json_response_matches_the_legacy_shape`) or to cover the
removed legacy gate (`authorize_admin_request_without_credentials_is_rejected`);
they go with the helpers, and the owned equivalents stay. The rustfs
test-count row drops from 5957 to 5955 and the s3s footprint from 68 to 66
files.
* refactor(admin): drop the legacy target descriptor JSON response helper (#8968)
`target_descriptor::build_json_response` returned the s3s response and
had no production caller left once the event, audit and plugin instance
handlers moved to `owned_build_json_response`; it survived only for the
legacy/owned equivalence test. Remove it, give the owned helper its own
documentation, and make the test assert the owned response directly:
status, `application/json`, the echoed `x-request-id` when present and
only then, and the body bytes. The file no longer imports s3s, so the
footprint baseline drops from 66 to 65 files.
* test(ecstore): pin native history on cancellation under the cancellable queue
main moved the local write path onto the disk blocking queue, which skips an
operation once its waiter is dropped, so a borrowed write cancelled while it is
queued no longer creates its file. The integration-only native publication
test still expected the queued open to run. Assert the new execution model
instead (the open never runs) while keeping its safety claims: the Started
record is taken before queueing and the cancelled publication never regrants
native history.
* chore(gateway): raise test-count rows for tests merged from main
* test(s3select): use the owned JSON input types in merged planner tests
* fix(ecstore): scope the merged direct-writer test hook to test builds
* fix(ecstore): create parents before admitted truncate opens
Align the heal tests with removal-disabled preservation and local-only outcome
fields, and refresh the e2e-full selection for the synced tests.
* chore(gateway): update test-count baseline after main sync
Main #9241 removed the node pool-metadata readiness cache with 8 tests
and added 6, so the rustfs row drops from 6440 to 6438. The ecstore and
s3select rows rise for tests added by #9237, #9239 and #9241.
* feat(gateway-adapter): add IAM-backed CredentialProvider
Add the ring-2 crate rustfs-gateway-adapter with IamCredentials, the
gateway CredentialProvider answered by RustFS IAM (rustfs/backlog#2760,
credentials half; PolicySource is deferred).
- Pin rustfs-gateway at the existing gateway rev with
default-features = false so the server crate stays out of the build.
- IamKeyLookup is a narrow IAM trait implemented for IamSys so the
adapter is testable without an in-memory IAM backend.
- Root credentials are injected at construction and resolve before IAM
is ready.
- Mapping: keystone: prefix -> NotFound (deliberate change from the
legacy empty secret); IAM not ready or not initialized -> Unavailable;
unknown key or NoSuch{User,ServiceAccount,TempAccount,Account} ->
NotFound; other IAM errors -> Backend; invalid or "off" principals
resolve disabled; STS credentials carry a SessionBinding, and an STS
credential without an expiry is a Backend fault rather than a
never-ending session.
- No authorization in lookup; no Debug on secret-holding types; IAM
error text is dropped after classification.
* feat(scripts): add junit_diff.py for dual-stack JUnit comparison (#9275)
* feat(gateway-adapter): add IAM-backed PolicySource
Add IamPolicySource, the gateway PolicySource answered by RustFS IAM
(rustfs/backlog#2760, policy half). One snapshot per request carries a
RustfsPrincipal: principal kind, account, effective username, parent
user, credential groups, live group memberships, token claims and
is_owner.
- Anonymous callers get an empty snapshot without asking IAM.
- IAM not ready, not initialized or failing is PolicyError::unavailable,
never an empty snapshot.
- Unknown, disabled and session-kind-mismatched principals are refused
with constant labels; unreadable tokens are malformed.
- Claims are read from the stored token (root secret for STS, the
account's own secret for service accounts), and is_owner mirrors the
legacy has_root_access rule.
- RustfsPrincipal holds no secret or session token and has no Debug.
* ci(gateway): make the test-count ratchet a floor, not exact (#9278)
ci(gateway): make the test-count ratchet a floor instead of an exact baseline
Tests below a row still fail; tests above it, a member without a row and ignores below the row now pass with a note. Ignores above a row still fail. Adds contract tests wired into make script-tests.
* feat(gateway-adapter): add host, bucket-owner and request-id adapters
* fix(gateway-adapter): avoid ecstore path in doc comment
* fix(gateway-adapter): keep the s3s path pattern out of docs
* chore(gateway): record ignored microbenchmarks merged from main (#9290)
chore(gateway): record the ignored microbenchmarks merged from main
main added three throughput microbenchmarks that are ignored by design
(#9217): one each in rustfs-checksums, rustfs-rio and rustfs-utils. The
test-count ratchet treats a higher ignored count as a failure until the
row is raised, so raise those three rows to 1 and tighten the test rows
to the merged tree with --update.
* test(gateway-adapter): make the test carrier Clone for Extensions::insert
* fix(gateway-adapter): share live root and refuse short secrets
Address the security review of the T2.1 adapters (rustfs/backlog#2760).
- Add RootCredentials (validated: root secret and STS signing key at
least MIN_SECRET_KEY_LEN bytes), the RootSource trait and the settable
RootCell. IamAdapters::new is now the only constructor of
IamCredentials and IamPolicySource, so both read one live root source
per request; no root source answers unavailable.
- A looked-up identity whose secret is shorter than 8 bytes is NotFound
in the credential provider and DISABLED_PRINCIPAL in the policy
source, whatever the lookup reports as valid.
- STS claims are read with the STS signing key from the root source
instead of assuming it equals the root secret.
- Cover the off-status-alone branch, the OIDC owner exclusion, service
accounts of root, the name live groups are read under, and another
session's token.
* ci(s3-tests): add reference parity lane against pinned 1.0.5 (#9288)
* fix(gateway-adapter): effective secret length and unsplittable pair
Address the confirmation review of the T2.1 adapters
(rustfs/backlog#2760).
- One rule for every secret: its length after trailing NUL bytes are
removed must be at least MIN_SECRET_KEY_LEN. HMAC zero-pads its key,
so trailing NULs add nothing; interior NULs count. Applied to the root
secret, the STS signing key and every looked-up credential.
- IamAdapters keeps both halves private and hands them to a builder
together through the consuming install(builder, make_authenticator);
compile-fail doctests show a mixed pair cannot be assembled.
- Any Fn() -> Option<Arc<RootCredentials>> is a RootSource, so the
binary reads its live root accessor directly; RootCell is now a
test-only helper.
* test(gateway-adapter): use bucket names the rules accept in host tests
* refactor(gateway-adapter): never refuse on request id; owner is the reported constant
* docs(gateway-adapter): state what install guarantees about the provider handle
---------
Co-authored-by: rustfs-cla-assistant[bot] <273121464+rustfs-cla-assistant[bot]@users.noreply.github.com>
Dispatch-only workflow that compiles the current main commit with a
caller-supplied version and uploads zip, deb, and rpm artifacts.
Co-authored-by: hector <majinghe@users.noreply.github.com>
* feat(server): add the RUSTFS_S3_STACK selector and dual-stack e2e lane (#8411)
* refactor(server): group the s3s-compat tower layers under legacy_compat (#8410)
Move the ten tower layers that patch s3s behaviour out of
rustfs/src/server/layer.rs into rustfs/src/server/legacy_compat/, one file
per layer with its tests. Each file header names the s3s behaviour it
patches, the gateway mechanism that replaces it (Replaced by:), and the task
that removes it (Removed by: T2.12, rustfs/backlog#2771). The layers keep
their code, their order in external_service_stack!, and their behaviour; no
layer is deleted.
Add server::http::tests::stack_census, which renders the ordered service
chain of the real external_service_stack! expansion and compares it with a
committed constant, so a reorder, insertion or removal is a visible diff.
Route the two s3s error types the moved layers use through a
storage_api::server::legacy_compat shim so the split adds no s3s-importing
file, and lower the s3s footprint baselines to the measured values (files 209
to 208, s3_error! lines 1586 to 1585). Keep the server runtime-source bypass
guard covering the moved code.
Refs rustfs/backlog#2787, rustfs/backlog#2734
* feat(s3-types): add the RustFS-owned S3 error type and s3_error! macro (#8406)
`rustfs-s3-types` now provides `S3ErrorCode` (generated by build.rs from
src/codes.txt, the grep-derived call surface, plus `Custom`), `S3Error`,
`S3Result`, `StdError` and the `s3_error` macro with the same call syntax
as s3s, so the T1.2 codemod is a `use` swap. Statuses are hand-written and
pinned to the s3s behaviour RustFS ships today by an oracle test that uses
s3s as a dev-dependency only. The `From` conversions for the legacy edge
live behind the off-by-default `compat-s3s` feature (the orphan rule
forbids them in rustfs/src/error.rs); `rustfs` enables it.
Refs rustfs/backlog#2742, rustfs/backlog#2734
* ci: add the per-crate test-count ratchet (#8404)
Quick Checks now fails any PR that lowers the number of test attributes
in a workspace crate or adds #[ignore], measured per crate against the
exact rows of .config/test-count-baseline.txt (rustfs/backlog#2734
section 1 rule 3). The counter widens the spec's literal
#[(tokio::)?test] to the tokio attributes that carry arguments and both
spellings of #[test_case(...)], and anchors at line start so
commented-out attributes do not count. The initial baseline covers 52
crates, 21077 tests and 164 ignored.
Refs rustfs/backlog#2738
Refs rustfs/backlog#2734
* test(ecstore): pin the persisted bucket-config XML bytes for 13 families (#8408)
Before any DTO or codec is replaced, pin the exact bytes that
deserialize followed by serialize produces for every persisted
bucket-configuration XML family in its three source shapes (AWS
documentation, MinIO-written, RustFS-written), and the bytes RustFS
writes today for a typed value. Byte equality only: a structural
comparison would let a codec replacement change the persisted encoding
unnoticed.
The generator script refuses to overwrite existing goldens without
--force, refuses on dirty codec paths, cannot report success from its
write mode, and records the generating commit in the fixtures README.
Refs rustfs/backlog#2744, rustfs/backlog#2734
* ci: raise the test-count baseline to the integration tree (#8426)
* ci: make the s3s footprint baseline exact and lower-only (#8403)
* ci: make the s3s footprint baseline exact and lower-only
The s3s footprint guard compared three counters against ceilings held in
the script itself, so a ceiling could sit above the tree and a drift was
fixable by raising it (rustfs/rustfs#5964 raised the s3_error! ceiling
to go green). The three values now live in
.config/s3s-footprint-baseline.txt as files, s3_error_lines and
ecstore_files, the script reads only that file, and a measurement that
differs from its baseline in either direction fails: growth prints
"s3s footprint grew: <measured> > <baseline>", shrinkage prints
"lower the baseline to <measured> in this PR". The counting method is
unchanged; the baseline is the current measurement of
gateway/integration (209 / 1585 / 36). The header records the end state:
once all three reach 0, T4.2 switches the script to failing on any
occurrence.
Refs rustfs/backlog#2739, rustfs/backlog#2734
* ci: add the s3s edge allowlist mode to the footprint guard (#8405)
`scripts/check_s3s_footprint.sh --mode allowlist` fails on every Rust
file that references s3s (`s3s::` or `s3s_sigv4`) and is not matched by
an entry of the new `.config/s3s-edge-allowlist.txt`; `--dry-run` lists
those files and exits 0. The allowlist carries the initial edge from the
spec, and the script rejects an entry that names or matches anything
under crates/ other than crates/e2e_test. The baseline mode and the
argument-less CI invocation are unchanged.
Refs rustfs/backlog#2741, rustfs/backlog#2734
* test(ecstore): reach the bucket-config codec through the test facade (#8428)
* refactor(s3-types): own the HTTP Body and drop s3s from three crates (#8431)
Add rustfs_s3_types::Body, a BoxBody<Bytes, StdError> wrapper with
empty() and From<Bytes | Vec<u8> | String | &'static str>, and the
legacy From conversions to and from s3s::Body behind compat-s3s.
The signer is now generic over the request body: it only reads and
writes the request head, so it needs no body type and its callers keep
theirs. s3-client sends its requests with the owned Body, madmin signs a
bodiless head, and targets drops its two S3Request/S3Response header
wrappers in favour of extract_params_header.
rustfs-signer, rustfs-madmin and rustfs-targets no longer depend on
s3s; the s3s footprint baseline drops from 209 to 202 files.
Refs rustfs/backlog#2747, rustfs/backlog#2734
* refactor(ecstore): move S3ErrorCode to rustfs-s3-types by codemod (#8432)
* test(ecstore): pin the InvalidRange responses of new_getobjectreader
Both error branches of new_getobjectreader build an ErrorResponse with S3ErrorCode::InvalidRange, and no test observed either code. Pin the code, its wire name and the location fields of each branch before the code type moves to rustfs-s3-types (rustfs/backlog#2743), so a changed code on either branch fails a test. The ecstore test-count row rises by the two tests.
* chore(scripts): add the s3s-to-rustfs-s3-types error codemod
Task T1.2 of rustfs/backlog#2734 (rustfs/backlog#2743) moves S3Error, S3ErrorCode, S3Result and s3_error! from s3s to rustfs-s3-types by script, one crate closure per PR. The codemod rewrites use trees (splitting mixed s3s imports) and qualified code paths, never comments or literals, adds the rustfs-s3-types dependency, drops s3s only from a crate that no longer names it, refuses every file of the s3s edge allowlist, takes --hold for a file whose type is fixed outside the closure, and prints a per-crate census. A second run changes nothing. Its self-test runs under make script-tests.
* refactor(ecstore): take S3ErrorCode from rustfs-s3-types
Output of scripts/codemods/s3_error_to_s3_types.py over crates/ecstore, crates/lifecycle, crates/scanner and crates/replication (rustfs/backlog#2743 batches 1 and 2). The ErrorResponse code field, the two InvalidRange constructions in new_getobjectreader and an unused tier import now use rustfs_s3_types::S3ErrorCode; the variant names and wire names are unchanged. lifecycle, scanner and replication reference none of the four error names. warm_backend.rs and warm_backend_s3.rs are held: they compare against rustfs_s3_client::ErrorResponse.code, which stays the s3s type until s3-client moves.
* ci(footprint): count only s3_error! lines still on s3s's macro
s3_error_lines counted every line holding the text `s3_error!`, which
also matches the RustFS-owned `rustfs_s3_types::s3_error!`. A codemod
that moves call sites onto rustfs-s3-types therefore could not lower it
(rustfs/backlog#2743, decision 3; rustfs/rustfs#8432, gotcha 2).
The counter now applies a textual, per-file and per-line rule that errs
on the side of counting (full rule in the script header):
- occurrences spelled `rustfs_s3_types::s3_error!` never count;
- a line holding `s3s::s3_error!` always counts;
- any other `s3_error!` line counts unless its file is cleared, that is,
it imports the macro from rustfs_s3_types by name on a line starting
with the use item (or defines it with `macro_rules! s3_error`) and does
not also bring s3s's macro in (`use s3s::s3_error`, a `use s3s::{..}`
tree naming it or a crate-root `*`, `use s3s::*`, or
`#[macro_use] extern crate s3s`);
- fail closed: a file matching neither form (`use super::*`, a
crate-local re-export, a renamed or glob rustfs_s3_types import, a
commented-out import, a comment-only mention) counts.
Census of the 1585 lines on 96fc06776: 1478 in 64 files that import the
macro from s3s; 107 in 15 files matching neither form (105 through
`use super::*`, 1 qualified `s3s::s3_error!`, 1 comment line); none in
files that import it from rustfs_s3_types or define it. Every line stays
counted, so the baseline remains s3_error_lines=1585 (files=202 and
ecstore_files=36 unchanged). Rewriting rustfs/src/auth.rs's import to
rustfs_s3_types lowers the new counter to 1567; the old one stayed 1585.
`--self-test` (wired into `make script-tests`) copies the script into
22 fixture trees: 6 positive cases and 16 negative ones (lines that must
not count, forms that must not clear a file, and a missing baseline,
missing key or failing classification search that must fail). Each tree
has a control line that must count, and each run gets s3s text on stdin.
Seven cases were red against the old counter. 19 mutants each turn their
case red: old definition, flipped import test, dropped fail-closed
branch, dropped e2e_test exclusion, dropped s3s veto, dropped qualified
s3s rule, dropped rustfs-qualified strip, unanchored rustfs import,
renamed import accepted, nested glob taken as root glob, single-line
s3s trees only, dropped s3s glob forms, dropped macro_use form, dropped
definition clearing, ignored classification rg errors, error-line search
without its path, classification searches without their path, missing
baseline skipped, missing key skipped.
* test(e2e): compare two RUSTFS_S3_STACK servers over every S3 operation (#8435)
Add dual_stack_differential to e2e_test. It starts two rustfs processes
from the same binary, stack A on RUSTFS_S3_STACK=legacy and stack B on
RUSTFS_E2E_STACK_B (default legacy until rustfs/backlog#2734 T2.5),
sends both the same aws-sdk-s3 script covering the 73 methods of
impl S3 for FS (a normal request and a 4xx negative each), and compares
status, error code, an allowlist of response headers and a few typed
body fields. Error message text is never compared. Each run writes
<target>/dual-stack/<run>.json with compared_ops and the (op, field,
a, b) differences; any difference fails the test.
differential_detects_an_injected_difference starts B with
RUSTFS_REGION=eu-west-1 and requires GetBucketLocation to be reported.
The comparator is pure and unit-tested without a server.
Wire the module into the e2e-smoke filter and update the e2e-smoke and
e2e-full selection digests and the e2e_test test-count row.
Refs rustfs/backlog#2740
* feat(app): add the RustFS-owned RequestEnvelope and its s3s constructor (#8437)
* feat(app): add the RustFS-owned RequestEnvelope and its s3s constructor
* ci: classify the s3s edge adapters in storage/s3_compat as interface
* feat(s3-types): add Body::from_stream and move Swift off s3s::Body (#8438)
rustfs/backlog#2747 batch 3. `Body::from_stream` wraps a chunk stream of
unknown length over `StreamBody`/`BoxBody`: one data frame per chunk, the
first error ends the body, the size hint stays unknown, and dropping the
body drops the stream. Parity tests pin it to the s3s `StreamingBlob::wrap`
it replaces, at the `http_body` level and byte for byte through hyper.
The Swift handlers in rustfs-protocols build their responses as the owned
`Body`. `SwiftService` converts them into the wrapped S3 service's response
body type through `From<Body>`, so rustfs keeps serving `s3s::Body` at the
legacy edge unchanged. Request bodies stay Send-only (`UnsyncBoxBody`); the
two storage writers and the DLO/SLO segment streams get `Sync` from
`sync_wrapper::SyncStream`, which is what s3s's `http_body_unsync` mutex
provided before. The DTO and error uses of s3s in protocols are untouched
(T1.5 and T1.2).
* refactor(protocols): move error types to rustfs-s3-types by codemod (#8439)
T1.2 batch 3 (rustfs/backlog#2743): s3select-api, s3select-query and
the error items of protocols move off s3s by scripts/codemods, run with
--hold crates/protocols/src/sftp/errors.rs. That file stays on s3s: it
downcasts the backend error, whose runtime type is still s3s::S3Error.
- rustfs/src/protocols/client.rs: the two webdav session methods now
return rustfs_s3_types::S3Result and convert the s3s error with From.
- codemod: a child module that calls s3_error! through `use super::*`
gets its own `use rustfs_s3_types::s3_error;` once its parent imports
that macro by name; otherwise it is listed as GLOB.
- s3-types: a table test pins code, status and the s3s error the legacy
edge renders for every variant a moved call site names.
- s3_error_lines 1585 -> 1584 (a lifecycle comment reworded).
* test(connect): restore one-second drive request budgets
Apply the exact test-only repair already landed on main in 5972a09cc3. The filesystem probe and production CLI success test request one second and keep their existing one-second duration and parameter checks. All outcome, I/O, error, cleanup and resource-limit assertions remain.
Scope: rustfs/tests/connect_perf_drive.rs only. Patch-id: cb0a1dfc8f68a67727525adea7458e2cee7d09f8.
Validation environment: LANG=en_US.UTF-8, CARGO_BUILD_JOBS=1, CARGO_INCREMENTAL=0, exclusive APFS-cloned CARGO_TARGET_DIR.
Quick Checks: every command from .github/actions/quick-checks/action.yml, with captured output and exit status below.
$ shellcheck --version && actionlint
ShellCheck - shell script analysis tool
version: 0.11.0
license: GNU General Public License, version 3
website: https://www.shellcheck.net
exit_code=0
$ cargo fmt --all --check
exit_code=0
$ ./scripts/check_unsafe_code_allowances.sh
exit_code=0
$ ./scripts/check_layer_dependencies.sh
Layer dependency guard passed (no new reverse dependencies/cycles).
exit_code=0
$ ./scripts/check_architecture_migration_rules.sh
Architecture migration rules passed.
exit_code=0
$ ./scripts/check_logging_guardrails.sh
✅ Logging guardrails check passed
exit_code=0
$ ./scripts/check_error_other_format_ratchet.sh
error(format!) ratchet passed (total call sites: 582).
exit_code=0
$ ./scripts/check_no_tokio_io_uring.sh
exit_code=0
$ ./scripts/check_extension_schema_boundaries.sh
Extension schema boundaries passed.
exit_code=0
$ ./scripts/check_body_cache_whitelist.sh
Body-cache whitelist guard passed.
exit_code=0
$ ./scripts/check_s3s_footprint.sh
s3s footprint OK: files importing s3s is 195 (baseline files=195)
s3s footprint OK: s3s s3_error! lines is 1584 (baseline s3_error_lines=1584)
s3s footprint OK: ecstore files referencing s3s is 36 (baseline ecstore_files=36)
✅ s3s footprint baseline check passed
exit_code=0
$ ./scripts/check_test_count_ratchet.sh
test-count ratchet OK: 52 crates, 21222 tests, 164 ignored match .config/test-count-baseline.txt
exit_code=0
$ ./scripts/check_fips_wording.sh
FIPS wording guard passed (README.md and CHANGELOG.md contain no forbidden claims).
KMS wording guard passed (crates/kms claims no Vault KV2 Transit wrapping).
exit_code=0
$ ./scripts/check_embedded_secrets.sh
Embedded secret guard passed (no private key material or provider credential literal in the tree).
exit_code=0
$ make script-tests
Running script tests...
entrypoint credential policy tests passed
internode grpc A/B bench env tests passed
object batch benchmark enhanced tests passed
PASS package service script contract
hotpath warp A/B gate tests passed
hotpath warp ABBA tests passed
Scanner validation artifacts written to /var/folders/0w/s4wxsn2n4w9b6f3cd4xrz53w0000gn/T/tmp.1pFe6Kl1Yu/out
Scanner validation artifacts written to /var/folders/0w/s4wxsn2n4w9b6f3cd4xrz53w0000gn/T/tmp.1pFe6Kl1Yu/out-missing-pid
.............................
----------------------------------------------------------------------
Ran 29 tests in 181.286s
OK
{"decision":"verified","pending_gates":["G01","G03","G04","G05","G06","G07","G08","G09","G10","G11","G12","G13","G14","P1","P2","P3","P4","R-D","R-E","R-L"],"pending_lanes":["authority-coverage","checkpoint-and-crash","ec8-4-multiset","maintenance-producers","mixed-version-rollback","mrf-responsibility","scheduler-pressure","status-and-outcome"],"rejected_errors":[],"rejected_gate":null,"release_approved":false,"release_schema_capable":true,"schema":1,"verified_gate":"G02"}
{"decision":"verified","pending_gates":["G01","G02","G03","G04","G05","G06","G07","G08","G09","G10","G11","G12","G13","G14","P1","P2","P3","P4","R-D","R-L"],"pending_lanes":["authority-coverage","checkpoint-and-crash","ec8-4-multiset","maintenance-producers","mixed-version-rollback","mrf-responsibility","scheduler-pressure","status-and-outcome"],"rejected_errors":[],"rejected_gate":null,"release_approved":false,"release_schema_capable":true,"schema":1,"verified_gate":"R-E"}
{"decision":"verified","pending_gates":["G01","G03","G04","G05","G06","G07","G08","G09","G10","G11","G12","G13","G14","P1","P2","P3","P4","R-D","R-E","R-L"],"pending_lanes":["authority-coverage","checkpoint-and-crash","ec8-4-multiset","maintenance-producers","mixed-version-rollback","mrf-responsibility","scheduler-pressure","status-and-outcome"],"rejected_errors":[],"rejected_gate":null,"release_approved":false,"release_schema_capable":true,"schema":1,"verified_gate":"G02"}
{"decision":"verified","pending_gates":["G01","G02","G03","G04","G05","G06","G07","G08","G09","G10","G11","G12","G13","G14","P1","P2","P3","P4","R-D","R-L"],"pending_lanes":["authority-coverage","checkpoint-and-crash","ec8-4-multiset","maintenance-producers","mixed-version-rollback","mrf-responsibility","scheduler-pressure","status-and-outcome"],"rejected_errors":[],"rejected_gate":null,"release_approved":false,"release_schema_capable":true,"schema":1,"verified_gate":"R-E"}
{"decision":"verified","pending_gates":["G02","G03","G04","G05","G06","G07","G08","G09","G10","G11","G12","G13","G14","P1","P2","P3","P4","R-D","R-E","R-L"],"pending_lanes":["authority-coverage","checkpoint-and-crash","ec8-4-multiset","maintenance-producers","mixed-version-rollback","mrf-responsibility","scheduler-pressure","status-and-outcome"],"rejected_errors":[],"rejected_gate":null,"release_approved":false,"release_schema_capable":true,"schema":1,"verified_gate":"G01"}
{"decision":"verified","pending_gates":["G01","G02","G04","G05","G06","G07","G08","G09","G10","G11","G12","G13","G14","P1","P2","P3","P4","R-D","R-E","R-L"],"pending_lanes":["authority-coverage","checkpoint-and-crash","ec8-4-multiset","maintenance-producers","mixed-version-rollback","mrf-responsibility","scheduler-pressure","status-and-outcome"],"rejected_errors":[],"rejected_gate":null,"release_approved":false,"release_schema_capable":true,"schema":1,"verified_gate":"G03"}
{"decision":"verified","pending_gates":["G01","G02","G03","G04","G05","G06","G07","G08","G09","G10","G11","G12","G13","G14","P1","P2","P3","P4","R-D","R-E"],"pending_lanes":["authority-coverage","checkpoint-and-crash","ec8-4-multiset","maintenance-producers","mixed-version-rollback","mrf-responsibility","scheduler-pressure","status-and-outcome"],"rejected_errors":[],"rejected_gate":null,"release_approved":false,"release_schema_capable":true,"schema":1,"verified_gate":"R-L"}
{"decision":"verified","pending_gates":["G01","G02","G03","G04","G05","G06","G07","G08","G09","G10","G11","G12","G13","P1","P2","P3","P4","R-D","R-E","R-L"],"pending_lanes":["authority-coverage","checkpoint-and-crash","ec8-4-multiset","maintenance-producers","mixed-version-rollback","mrf-responsibility","scheduler-pressure","status-and-outcome"],"rejected_errors":[],"rejected_gate":null,"release_approved":false,"release_schema_capable":true,"schema":1,"verified_gate":"G14"}
{"decision":"verified","pending_gates":["G01","G02","G03","G04","G05","G06","G07","G08","G09","G10","G11","G12","G13","P1","P2","P3","P4","R-D","R-E","R-L"],"pending_lanes":["authority-coverage","checkpoint-and-crash","ec8-4-multiset","maintenance-producers","mixed-version-rollback","mrf-responsibility","scheduler-pressure","status-and-outcome"],"rejected_errors":[],"rejected_gate":null,"release_approved":false,"release_schema_capable":true,"schema":1,"verified_gate":"G14"}
{"decision":"verified","pending_gates":["G01","G02","G03","G04","G05","G06","G07","G08","G09","G11","G12","G13","G14","P1","P2","P3","P4","R-D","R-E","R-L"],"pending_lanes":["authority-coverage","checkpoint-and-crash","ec8-4-multiset","maintenance-producers","mixed-version-rollback","mrf-responsibility","scheduler-pressure","status-and-outcome"],"rejected_errors":[],"rejected_gate":null,"release_approved":false,"release_schema_capable":true,"schema":1,"verified_gate":"G10"}
{"decision":"verified","pending_gates":["G01","G02","G03","G04","G05","G06","G07","G08","G09","G10","G11","G12","G13","G14","P2","P3","P4","R-D","R-E","R-L"],"pending_lanes":["authority-coverage","checkpoint-and-crash","ec8-4-multiset","maintenance-producers","mixed-version-rollback","mrf-responsibility","scheduler-pressure","status-and-outcome"],"rejected_errors":[],"rejected_gate":null,"release_approved":false,"release_schema_capable":true,"schema":1,"verified_gate":"P1"}
{"decision":"verified","pending_gates":["G01","G02","G03","G04","G05","G06","G07","G08","G09","G10","G11","G12","G13","G14","P1","P3","P4","R-D","R-E","R-L"],"pending_lanes":["authority-coverage","checkpoint-and-crash","ec8-4-multiset","maintenance-producers","mixed-version-rollback","mrf-responsibility","scheduler-pressure","status-and-outcome"],"rejected_errors":[],"rejected_gate":null,"release_approved":false,"release_schema_capable":true,"schema":1,"verified_gate":"P2"}
{"decision":"verified","pending_gates":["G01","G02","G03","G04","G05","G06","G07","G08","G09","G10","G11","G12","G13","G14","P1","P2","P4","R-D","R-E","R-L"],"pending_lanes":["authority-coverage","checkpoint-and-crash","ec8-4-multiset","maintenance-producers","mixed-version-rollback","mrf-responsibility","scheduler-pressure","status-and-outcome"],"rejected_errors":[],"rejected_gate":null,"release_approved":false,"release_schema_capable":true,"schema":1,"verified_gate":"P3"}
{"decision":"verified","pending_gates":["G01","G02","G03","G04","G06","G07","G08","G09","G10","G11","G12","G13","G14","P1","P2","P3","P4","R-D","R-E","R-L"],"pending_lanes":["authority-coverage","checkpoint-and-crash","ec8-4-multiset","maintenance-producers","mixed-version-rollback","mrf-responsibility","scheduler-pressure","status-and-outcome"],"rejected_errors":[],"rejected_gate":null,"release_approved":false,"release_schema_capable":true,"schema":1,"verified_gate":"G05"}
{"decision":"verified","pending_gates":["G01","G02","G03","G04","G05","G07","G08","G09","G10","G11","G12","G13","G14","P1","P2","P3","P4","R-D","R-E","R-L"],"pending_lanes":["authority-coverage","checkpoint-and-crash","ec8-4-multiset","maintenance-producers","mixed-version-rollback","mrf-responsibility","scheduler-pressure","status-and-outcome"],"rejected_errors":[],"rejected_gate":null,"release_approved":false,"release_schema_capable":true,"schema":1,"verified_gate":"G06"}
{"decision":"verified","pending_gates":["G01","G02","G03","G04","G05","G06","G07","G08","G09","G10","G11","G12","G13","G14","P1","P2","P3","P4","R-E","R-L"],"pending_lanes":["authority-coverage","checkpoint-and-crash","ec8-4-multiset","maintenance-producers","mixed-version-rollback","mrf-responsibility","scheduler-pressure","status-and-outcome"],"rejected_errors":[],"rejected_gate":null,"release_approved":false,"release_schema_capable":true,"schema":1,"verified_gate":"R-D"}
Status-and-outcome release descriptor verified: /private/var/folders/0w/s4wxsn2n4w9b6f3cd4xrz53w0000gn/T/tmp8rwn1pe0/descriptor/release-bundle-status-outcome.json
{"decision":"verified","pending_gates":["G01","G02","G03","G04","G06","G07","G08","G09","G10","G11","G12","G13","G14","P1","P2","P3","P4","R-D","R-E","R-L"],"pending_lanes":["authority-coverage","checkpoint-and-crash","ec8-4-multiset","maintenance-producers","mixed-version-rollback","mrf-responsibility","scheduler-pressure","status-and-outcome"],"rejected_errors":[],"rejected_gate":null,"release_approved":false,"release_schema_capable":true,"schema":1,"verified_gate":"G05"}
{"decision":"verified","pending_gates":["G01","G02","G03","G04","G05","G07","G08","G09","G10","G11","G12","G13","G14","P1","P2","P3","P4","R-D","R-E","R-L"],"pending_lanes":["authority-coverage","checkpoint-and-crash","ec8-4-multiset","maintenance-producers","mixed-version-rollback","mrf-responsibility","scheduler-pressure","status-and-outcome"],"rejected_errors":[],"rejected_gate":null,"release_approved":false,"release_schema_capable":true,"schema":1,"verified_gate":"G06"}
{"decision":"verified","pending_gates":["G01","G02","G03","G04","G05","G06","G07","G08","G09","G10","G11","G12","G13","G14","P1","P2","P3","P4","R-E","R-L"],"pending_lanes":["authority-coverage","checkpoint-and-crash","ec8-4-multiset","maintenance-producers","mixed-version-rollback","mrf-responsibility","scheduler-pressure","status-and-outcome"],"rejected_errors":[],"rejected_gate":null,"release_approved":false,"release_schema_capable":true,"schema":1,"verified_gate":"R-D"}
{"decision":"verified","pending_gates":["G01","G02","G03","G04","G05","G06","G08","G09","G10","G11","G12","G13","G14","P1","P2","P3","P4","R-D","R-E","R-L"],"pending_lanes":["authority-coverage","checkpoint-and-crash","ec8-4-multiset","maintenance-producers","mixed-version-rollback","mrf-responsibility","scheduler-pressure","status-and-outcome"],"rejected_errors":[],"rejected_gate":null,"release_approved":false,"release_schema_capable":true,"schema":1,"verified_gate":"G07"}
{"decision":"verified","pending_gates":["G01","G02","G03","G04","G05","G06","G07","G09","G10","G11","G12","G13","G14","P1","P2","P3","P4","R-D","R-E","R-L"],"pending_lanes":["authority-coverage","checkpoint-and-crash","ec8-4-multiset","maintenance-producers","mixed-version-rollback","mrf-responsibility","scheduler-pressure","status-and-outcome"],"rejected_errors":[],"rejected_gate":null,"release_approved":false,"release_schema_capable":true,"schema":1,"verified_gate":"G08"}
{"decision":"verified","pending_gates":["G01","G02","G03","G04","G05","G06","G07","G08","G09","G10","G11","G12","G13","G14","P1","P2","P3","R-D","R-E","R-L"],"pending_lanes":["authority-coverage","checkpoint-and-crash","ec8-4-multiset","maintenance-producers","mixed-version-rollback","mrf-responsibility","scheduler-pressure","status-and-outcome"],"rejected_errors":[],"rejected_gate":null,"release_approved":false,"release_schema_capable":true,"schema":1,"verified_gate":"P4"}
exact 1MiB handoff ABBA tests passed
pinned paired ABBA bench tests passed
==> /var/folders/0w/s4wxsn2n4w9b6f3cd4xrz53w0000gn/T/tmp.mYmuI01hNq/bin/path_containment (-max_total_time=7, -seed=123456789, -artifact_prefix=/var/folders/0w/s4wxsn2n4w9b6f3cd4xrz53w0000gn/T/tmp.mYmuI01hNq/explicit-artifacts/path_containment/, corpus=/private/tmp/gw-orchestrator-2734/ci-drive/fuzz/corpus/path_containment)
-seed=123456789
target=path_containment
seed=123456789
max_total_time=7
git_revision=4d15b816236630b2c42f321ffc9ed3654edc94e3
git_dirty=true
runner_mode=prebuilt
==> /var/folders/0w/s4wxsn2n4w9b6f3cd4xrz53w0000gn/T/tmp.mYmuI01hNq/bin/path_containment (-max_total_time=1, -seed=2322617706, -artifact_prefix=/var/folders/0w/s4wxsn2n4w9b6f3cd4xrz53w0000gn/T/tmp.mYmuI01hNq/auto-artifacts/path_containment/, corpus=/private/tmp/gw-orchestrator-2734/ci-drive/fuzz/corpus/path_containment)
-seed=2322617706
==> cargo +nightly fuzz run path_containment (-max_total_time=9, -seed=987654321, -artifact_prefix=/var/folders/0w/s4wxsn2n4w9b6f3cd4xrz53w0000gn/T/tmp.mYmuI01hNq/cargo-artifacts/path_containment/)
+nightly
fuzz
run
path_containment
-seed=987654321
seed=987654321
runner_mode=cargo-fuzz
fuzz runner seed manifest tests passed
✅ scripts/python_bin.sh resolver checks passed
self-test ok: clean fixture with every exemption present passes
self-test ok: all 11 pattern families fire
self-test ok: moving a pinned source is reported
Embedded secret guard self-test passed.
self-test ok: p1-s3s-import: 'use s3s::s3_error;' counts every line, the rustfs_s3_types module's too (s3_error_lines=3)
self-test ok: p2-s3s-mixed-tree: a multi-line s3s use tree naming s3_error counts (s3_error_lines=3)
self-test ok: p3-s3s-glob: a crate-root glob of s3s, bare or inside a tree, counts (s3_error_lines=5)
self-test ok: p4-macro-use: '#[macro_use] extern crate s3s' counts (s3_error_lines=3)
self-test ok: p5-qualified-s3s: a qualified s3s::s3_error! line counts in a cleared file (s3_error_lines=2)
self-test ok: p6-exact-pass: a baseline equal to the tree passes (exit 0)
self-test ok: n1-rustfs-import: 'use rustfs_s3_types::s3_error;' alone counts nothing (s3_error_lines=1)
self-test ok: n2-rustfs-mixed-tree: a rustfs_s3_types tree plus an s3s tree without the macro counts nothing (s3_error_lines=1)
self-test ok: n3-defines: the file defining the macro counts nothing (s3_error_lines=1)
self-test ok: n4-qualified-rustfs: a qualified rustfs_s3_types::s3_error! line counts nothing (s3_error_lines=1)
self-test ok: n5-renamed-call: a call through a renamed macro is not text-matched (out of scope) (s3_error_lines=1)
self-test ok: n6-comment-string: a comment or string mentioning s3_error! in a cleared file counts nothing (s3_error_lines=1)
self-test ok: n7-e2e-excluded: crates/e2e_test is excluded (s3_error_lines=1)
self-test ok: n8-super-glob: an unrecognised import form ('use super::*;') counts (s3_error_lines=3)
self-test ok: n9-facade: a crate-local re-export counts (s3_error_lines=2)
self-test ok: n10-renamed-import: a renamed rustfs_s3_types import does not clear s3_error! lines (s3_error_lines=2)
self-test ok: n11-commented-import: a commented-out rustfs_s3_types import does not clear (s3_error_lines=2)
self-test ok: n12-rustfs-glob: a glob import of rustfs_s3_types does not clear (s3_error_lines=2)
self-test ok: n13-comment-only: a comment mentioning s3_error! in a file with no import counts (s3_error_lines=2)
self-test ok: f1-baseline-missing: a missing baseline file fails (exit 1)
self-test ok: f2-key-missing: a baseline without s3_error_lines fails (exit 1)
self-test ok: f3-rg-fails: a failing classification search fails the guard (exit 1)
✅ s3s footprint self-test passed
test_core_gate_rejects_missing_ignored_filtered_and_corrupt_inputs (__main__.SelfTests.test_core_gate_rejects_missing_ignored_filtered_and_corrupt_inputs) ... ok
test_e2e_requires_registration (__main__.SelfTests.test_e2e_requires_registration) ... ok
test_fuzz_runtime_uses_matrix_target (__main__.SelfTests.test_fuzz_runtime_uses_matrix_target) ... ok
test_ilm_lane_keeps_the_measured_cargo_build_budget (__main__.SelfTests.test_ilm_lane_keeps_the_measured_cargo_build_budget) ... ok
test_profile_listing_binds_platform_digest (__main__.SelfTests.test_profile_listing_binds_platform_digest) ... ok
test_profile_listing_enforces_selection (__main__.SelfTests.test_profile_listing_enforces_selection) ... ok
test_quick_checks_commands_propagate_failure (__main__.SelfTests.test_quick_checks_commands_propagate_failure) ... ok
test_quick_checks_rejects_caller_and_execution_bypasses (__main__.SelfTests.test_quick_checks_rejects_caller_and_execution_bypasses) ... ok
test_runner_readiness_fails_closed (__main__.SelfTests.test_runner_readiness_fails_closed) ... ok
test_s3_runner_rejects_unbounded_failure_locals (__main__.SelfTests.test_s3_runner_rejects_unbounded_failure_locals) ... ok
test_scanner_heal_begin_requires_embedded_source_provenance (__main__.SelfTests.test_scanner_heal_begin_requires_embedded_source_provenance) ... ok
test_scanner_heal_case_does_not_approve_pending_release (__main__.SelfTests.test_scanner_heal_case_does_not_approve_pending_release) ... ok
test_scanner_heal_case_lane_is_required_for_runner_profile (__main__.SelfTests.test_scanner_heal_case_lane_is_required_for_runner_profile) ... ok
test_scanner_heal_case_only_schema_cannot_approve_release (__main__.SelfTests.test_scanner_heal_case_only_schema_cannot_approve_release) ... ok
test_scanner_heal_crash_case_rejects_restart_oracle (__main__.SelfTests.test_scanner_heal_crash_case_rejects_restart_oracle) ... ok
test_scanner_heal_ec84_case_rejects_wrong_erasure_geometry (__main__.SelfTests.test_scanner_heal_ec84_case_rejects_wrong_erasure_geometry) ... ok
test_scanner_heal_finish_collects_oracles_from_registry (__main__.SelfTests.test_scanner_heal_finish_collects_oracles_from_registry) ... ok
test_scanner_heal_multipool_case_accepts_deferred_optional_outage_write (__main__.SelfTests.test_scanner_heal_multipool_case_accepts_deferred_optional_outage_write) ... ok
test_scanner_heal_pending_gate_cannot_map_to_implemented_lane (__main__.SelfTests.test_scanner_heal_pending_gate_cannot_map_to_implemented_lane) ... ok
test_scanner_heal_receipts_reject_reuse_and_missing_builds (__main__.SelfTests.test_scanner_heal_receipts_reject_reuse_and_missing_builds) ... ok
test_scanner_heal_rejects_boolean_fractional_and_out_of_geometry_integers (__main__.SelfTests.test_scanner_heal_rejects_boolean_fractional_and_out_of_geometry_integers) ... ok
test_scanner_heal_rejects_broken_execution_and_artifacts (__main__.SelfTests.test_scanner_heal_rejects_broken_execution_and_artifacts) ... ok
test_scanner_heal_rejects_copied_junit_and_wrong_suite_build (__main__.SelfTests.test_scanner_heal_rejects_copied_junit_and_wrong_suite_build) ... ok
test_scanner_heal_release_bundle_accepts_complete_measured_evidence (__main__.SelfTests.test_scanner_heal_release_bundle_accepts_complete_measured_evidence) ... ok
test_scanner_heal_release_bundle_assembler_copies_measured_artifacts (__main__.SelfTests.test_scanner_heal_release_bundle_assembler_copies_measured_artifacts) ... ok
test_scanner_heal_release_bundle_assembler_merges_measured_descriptors (__main__.SelfTests.test_scanner_heal_release_bundle_assembler_merges_measured_descriptors) ... ok
test_scanner_heal_release_bundle_assembler_rejects_duplicate_descriptor_gate (__main__.SelfTests.test_scanner_heal_release_bundle_assembler_rejects_duplicate_descriptor_gate) ... ok
test_scanner_heal_release_bundle_assembler_rejects_fixture_descriptor (__main__.SelfTests.test_scanner_heal_release_bundle_assembler_rejects_fixture_descriptor) ... ok
test_scanner_heal_release_bundle_assembler_rejects_hash_mismatch (__main__.SelfTests.test_scanner_heal_release_bundle_assembler_rejects_hash_mismatch) ... ok
test_scanner_heal_release_bundle_assembler_rejects_mixed_descriptor_revisions (__main__.SelfTests.test_scanner_heal_release_bundle_assembler_rejects_mixed_descriptor_revisions) ... ok
test_scanner_heal_release_bundle_enforces_topology_duration_profile_and_versions (__main__.SelfTests.test_scanner_heal_release_bundle_enforces_topology_duration_profile_and_versions) ... ok
test_scanner_heal_release_bundle_fixture_writer_is_not_release_evidence (__main__.SelfTests.test_scanner_heal_release_bundle_fixture_writer_is_not_release_evidence) ... ok
test_scanner_heal_release_bundle_gate_accepts_single_measured_gate (__main__.SelfTests.test_scanner_heal_release_bundle_gate_accepts_single_measured_gate) ... ok
test_scanner_heal_release_bundle_gate_rejects_missing_field (__main__.SelfTests.test_scanner_heal_release_bundle_gate_rejects_missing_field) ... ok
test_scanner_heal_release_bundle_gate_rejects_unknown_gate (__main__.SelfTests.test_scanner_heal_release_bundle_gate_rejects_unknown_gate) ... ok
test_scanner_heal_release_bundle_rejects_synthetic_or_missing_fields (__main__.SelfTests.test_scanner_heal_release_bundle_rejects_synthetic_or_missing_fields) ... ok
test_scanner_heal_release_bundle_requires_domain_evidence (__main__.SelfTests.test_scanner_heal_release_bundle_requires_domain_evidence) ... ok
test_scanner_heal_release_bundle_requires_field_provenance (__main__.SelfTests.test_scanner_heal_release_bundle_requires_field_provenance) ... ok
test_scanner_heal_release_bundle_requires_json_artifact_provenance (__main__.SelfTests.test_scanner_heal_release_bundle_requires_json_artifact_provenance) ... ok
test_scanner_heal_release_bundle_requires_mixed_version_field_roles (__main__.SelfTests.test_scanner_heal_release_bundle_requires_mixed_version_field_roles) ... ok
test_scanner_heal_release_bundle_requires_same_gate_measurement_window (__main__.SelfTests.test_scanner_heal_release_bundle_requires_same_gate_measurement_window) ... ok
test_scanner_heal_release_consumes_multiple_registry_oracles (__main__.SelfTests.test_scanner_heal_release_consumes_multiple_registry_oracles) ... ok
test_scanner_heal_release_matrix_lanes_remain_pending_without_real_evidence (__main__.SelfTests.test_scanner_heal_release_matrix_lanes_remain_pending_without_real_evidence) ... ok
test_scanner_heal_release_requirements_cannot_be_cleared (__main__.SelfTests.test_scanner_heal_release_requirements_cannot_be_cleared) ... ok
test_scanner_heal_release_status_rejects_focused_case_run (__main__.SelfTests.test_scanner_heal_release_status_rejects_focused_case_run) ... ok
test_scanner_heal_release_status_rejects_synthetic_case (__main__.SelfTests.test_scanner_heal_release_status_rejects_synthetic_case) ... ok
test_scanner_heal_required_evidence_fields_cannot_be_removed (__main__.SelfTests.test_scanner_heal_required_evidence_fields_cannot_be_removed) ... ok
test_scheduled_alerts_require_completion_watchdog (__main__.SelfTests.test_scheduled_alerts_require_completion_watchdog) ... ok
test_update_profile_selection_changes_only_requested_platform (__main__.SelfTests.test_update_profile_selection_changes_only_requested_platform) ... ok
test_validate_includes_quick_checks (__main__.SelfTests.test_validate_includes_quick_checks) ... ok
test_vault_tests_require_cross_process_serialization (__main__.SelfTests.test_vault_tests_require_cross_process_serialization) ... ok
test_workflow_readiness_requires_dependency_probe (__main__.SelfTests.test_workflow_readiness_requires_dependency_probe) ... ok
----------------------------------------------------------------------
Ran 52 tests in 14.301s
OK
e2e-full selection OK: 1 tests, sha256=29e30e6ed2ed38d9e5b0a7cdd9536abf5c719f7021a02b578ae1381563a3c273
...................................
----------------------------------------------------------------------
Ran 35 tests in 1.161s
OK
...................
----------------------------------------------------------------------
Ran 19 tests in 17.397s
OK
..........
----------------------------------------------------------------------
Ran 10 tests in 0.183s
OK
test_actual_selector_bootstrap_uses_base_policy_and_fails_closed (__main__.SelfTests.test_actual_selector_bootstrap_uses_base_policy_and_fails_closed) ... ok
test_documentation_paths_do_not_hide_build_or_fixture_changes (__main__.SelfTests.test_documentation_paths_do_not_hide_build_or_fixture_changes) ... ok
test_e2e_scope_keeps_production_and_shared_configuration_full (__main__.SelfTests.test_e2e_scope_keeps_production_and_shared_configuration_full) ... ok
test_e2e_shortcut_rejects_new_direct_renamed_and_target_dependencies (__main__.SelfTests.test_e2e_shortcut_rejects_new_direct_renamed_and_target_dependencies) ... ok
test_e2e_shortcut_traverses_implicit_path_dependencies (__main__.SelfTests.test_e2e_shortcut_traverses_implicit_path_dependencies) ... ok
test_event_contract_requires_complete_candidate_and_optional_lanes (__main__.SelfTests.test_event_contract_requires_complete_candidate_and_optional_lanes) ... ok
test_every_wrong_result_missing_job_or_selection_fails_closed (__main__.SelfTests.test_every_wrong_result_missing_job_or_selection_fails_closed) ... ok
test_full_e2e_gate_preserves_workflow_branch_and_event_scope (__main__.SelfTests.test_full_e2e_gate_preserves_workflow_branch_and_event_scope) ... ok
test_git_range_includes_deleted_source_and_rename_origins (__main__.SelfTests.test_git_range_includes_deleted_source_and_rename_origins) ... ok
test_job_ids_and_display_names_cannot_hide_validation (__main__.SelfTests.test_job_ids_and_display_names_cannot_hide_validation) ... ok
test_report_only_stack_lane_mirrors_e2e_smoke_outside_the_aggregate (__main__.SelfTests.test_report_only_stack_lane_mirrors_e2e_smoke_outside_the_aggregate) ... ok
test_repository_wiring_and_missing_dependency_regression (__main__.SelfTests.test_repository_wiring_and_missing_dependency_regression) ... ok
test_verify_command_preserves_failures (__main__.SelfTests.test_verify_command_preserves_failures) ... ok
----------------------------------------------------------------------
Ran 13 tests in 3.280s
OK
security coverage self-test passed
test_api_failures_preserve_other_evidence_and_never_enter_grace (__main__.SelfTests.test_api_failures_preserve_other_evidence_and_never_enter_grace) ... ok
test_attempt_outcome_does_not_replace_recent_success (__main__.SelfTests.test_attempt_outcome_does_not_replace_recent_success) ... ok
test_cancelled_attempt_cannot_refresh_expired_success (__main__.SelfTests.test_cancelled_attempt_cannot_refresh_expired_success) ... ok
test_cli_requires_the_repository_default_branch (__main__.SelfTests.test_cli_requires_the_repository_default_branch) ... ok
test_config_rejects_duplicate_and_invalid_entries (__main__.SelfTests.test_config_rejects_duplicate_and_invalid_entries) ... ok
test_freshness_boundaries (__main__.SelfTests.test_freshness_boundaries) ... ok
test_grace_requires_two_successful_queries_with_no_history (__main__.SelfTests.test_grace_requires_two_successful_queries_with_no_history) ... ok
test_invalid_api_evidence_fails_closed (__main__.SelfTests.test_invalid_api_evidence_fails_closed) ... ok
test_report_retains_every_workflow (__main__.SelfTests.test_report_retains_every_workflow) ... ok
test_requests_filter_schedule_default_branch_and_success_on_server (__main__.SelfTests.test_requests_filter_schedule_default_branch_and_success_on_server) ... ok
----------------------------------------------------------------------
Ran 10 tests in 0.051s
OK
..............................
----------------------------------------------------------------------
Ran 30 tests in 22.354s
OK
..........
----------------------------------------------------------------------
Ran 10 tests in 2.969s
OK
.................................
----------------------------------------------------------------------
Ran 33 tests in 0.160s
OK
Online functional runner labels: pf-testing
.......
----------------------------------------------------------------------
Ran 7 tests in 0.002s
OK
.......
----------------------------------------------------------------------
Ran 7 tests in 0.001s
OK
....
----------------------------------------------------------------------
Ran 4 tests in 0.128s
OK
...
----------------------------------------------------------------------
Ran 3 tests in 0.547s
OK
PASS: cold stampede script self-test
exit_code=0
$ python3 ./scripts/test_object_generation_protocol_model.py
test_a_recovery_barrier_cannot_choose_an_empty_value (__main__.ObjectGenerationProtocolModel.test_a_recovery_barrier_cannot_choose_an_empty_value) ... ok
test_all_bounded_two_proposer_interleavings_choose_at_most_one_value (__main__.ObjectGenerationProtocolModel.test_all_bounded_two_proposer_interleavings_choose_at_most_one_value) ... ok
test_an_unavailable_decision_quorum_cannot_choose_a_candidate (__main__.ObjectGenerationProtocolModel.test_an_unavailable_decision_quorum_cannot_choose_a_candidate) ... ok
test_counter_allocation_requires_persistence_and_is_consumed_once (__main__.ObjectGenerationProtocolModel.test_counter_allocation_requires_persistence_and_is_consumed_once) ... ok
test_counter_restart_burns_persisted_allocations_before_or_after_issue (__main__.ObjectGenerationProtocolModel.test_counter_restart_burns_persisted_allocations_before_or_after_issue) ... ok
test_crash_after_promise_before_reply_does_not_restore_an_older_ballot (__main__.ObjectGenerationProtocolModel.test_crash_after_promise_before_reply_does_not_restore_an_older_ballot) ... ok
test_duplicate_or_unconfigured_voters_cannot_form_a_quorum (__main__.ObjectGenerationProtocolModel.test_duplicate_or_unconfigured_voters_cannot_form_a_quorum) ... ok
test_empty_or_unobserved_minority_slot_can_retry_without_allocating_another_revision (__main__.ObjectGenerationProtocolModel.test_empty_or_unobserved_minority_slot_can_retry_without_allocating_another_revision) ... ok
test_equal_ballot_cannot_accept_different_bytes_and_lower_ballot_is_rejected (__main__.ObjectGenerationProtocolModel.test_equal_ballot_cannot_accept_different_bytes_and_lower_ballot_is_rejected) ... ok
test_forgetting_tombstone_authority_allows_an_aba_create (__main__.ObjectGenerationProtocolModel.test_forgetting_tombstone_authority_allows_an_aba_create) ... ok
test_issuing_before_persistence_reuses_a_ballot_after_restart (__main__.ObjectGenerationProtocolModel.test_issuing_before_persistence_reuses_a_ballot_after_restart) ... ok
test_lost_ack_and_restart_preserve_the_chosen_operation (__main__.ObjectGenerationProtocolModel.test_lost_ack_and_restart_preserve_the_chosen_operation) ... ok
test_minority_accepted_work_is_adopted_and_finished (__main__.ObjectGenerationProtocolModel.test_minority_accepted_work_is_adopted_and_finished) ... ok
test_omitting_highest_accepted_adoption_has_a_double_choice_counterexample (__main__.ObjectGenerationProtocolModel.test_omitting_highest_accepted_adoption_has_a_double_choice_counterexample) ... ok
test_quorum_choice_does_not_require_an_uncontacted_disk_to_learn_it (__main__.ObjectGenerationProtocolModel.test_quorum_choice_does_not_require_an_uncontacted_disk_to_learn_it) ... ok
test_recovery_failure_or_stale_expected_head_does_not_allocate_a_successor (__main__.ObjectGenerationProtocolModel.test_recovery_failure_or_stale_expected_head_does_not_allocate_a_successor) ... ok
test_replacing_voters_with_empty_state_can_choose_a_conflicting_value (__main__.ObjectGenerationProtocolModel.test_replacing_voters_with_empty_state_can_choose_a_conflicting_value) ... ok
test_skipping_predecessor_recovery_chooses_a_broken_revision_chain (__main__.ObjectGenerationProtocolModel.test_skipping_predecessor_recovery_chooses_a_broken_revision_chain) ... ok
test_stale_publication_rollback_and_gc_cannot_change_the_winner (__main__.ObjectGenerationProtocolModel.test_stale_publication_rollback_and_gc_cannot_change_the_winner) ... ok
test_superseded_operation_identity_cannot_be_reused_for_different_input (__main__.ObjectGenerationProtocolModel.test_superseded_operation_identity_cannot_be_reused_for_different_input) ... ok
test_superseded_operation_retry_returns_its_original_successor (__main__.ObjectGenerationProtocolModel.test_superseded_operation_retry_returns_its_original_successor) ... ok
test_tombstone_and_null_marker_payloads_do_not_reset_authority_revision (__main__.ObjectGenerationProtocolModel.test_tombstone_and_null_marker_payloads_do_not_reset_authority_revision) ... ok
test_tombstone_rejects_a_delayed_never_present_create (__main__.ObjectGenerationProtocolModel.test_tombstone_rejects_a_delayed_never_present_create) ... ok
test_unlearned_predecessor_is_recovered_before_the_second_slot (__main__.ObjectGenerationProtocolModel.test_unlearned_predecessor_is_recovered_before_the_second_slot) ... ok
----------------------------------------------------------------------
Ran 24 tests in 0.958s
OK
model: states=51023, transitions=167520, terminal=732, chosen_candidates=2
negative control: double-choice trace=((0, 'prepare', 0), (0, 'prepare', 1), (0, 'prepare', 2), (0, 'accept', 0), (0, 'accept', 1), (0, 'accept', 2), (1, 'prepare', 0), (1, 'prepare', 1), (1, 'prepare', 2), (1, 'accept', 0), (1, 'accept', 1), (1, 'accept', 2))
multi-slot round schedules: 640
exit_code=0
$ python3 ./scripts/check_test_wiring.py
OK: e2e modules, runner selection, fuzz matrices, profiles, scheduled alerts, and Quick Checks are wired
exit_code=0
$ ./scripts/check_no_planning_docs.sh
exit_code=0
$ ./scripts/check_uring_lane_lib_only.sh
OK: no 'uring_'-matching test functions under crates/ecstore/tests; --lib is safe
exit_code=0
Initial complete gate
$ cargo nextest run --profile ci -p rustfs --test connect_perf_drive --no-fail-fast
Nextest run ID e07fc5b9-c6c8-4cdd-ae61-54839bf24437 with nextest profile: ci
Starting 10 tests across 1 binary
PASS [ 0.016s] rustfs::connect_perf_drive cancellation_and_single_collector_leave_no_scratch
PASS [ 0.011s] rustfs::connect_perf_drive cleanup_failure_overrides_cancellation_and_remains_attributed
PASS [ 0.009s] rustfs::connect_perf_drive consent_and_resource_budgets_fail_before_scratch_io
PASS [ 0.009s] rustfs::connect_perf_drive invalid_roots_and_existing_scratch_fail_without_clobbering
PASS [ 16.950s] rustfs::connect_perf_drive production_cli_measures_and_signs_exact_binary_provenance
PASS [ 4.025s] rustfs::connect_perf_drive production_cli_rejects_missing_l1_consent_and_limits_before_artifacts
PASS [ 0.008s] rustfs::connect_perf_drive read_only_root_and_full_drive_errors_are_classified
PASS [ 0.015s] rustfs::connect_perf_drive real_local_write_read_is_measured_and_scratch_is_removed
PASS [ 0.007s] rustfs::connect_perf_drive runtime_timeout_and_full_drive_use_schema_valid_failed_reason
PASS [ 0.035s] rustfs::connect_perf_drive successful_result_has_signed_bounded_private_offline_export
────────────
Summary [ 21.087s] 10 tests run: 10 passed, 0 skipped
exit_code=0
Restored complete gate
$ cargo nextest run --profile ci -p rustfs --test connect_perf_drive --no-fail-fast
Nextest run ID 233b7307-5361-4eb7-94d5-b384f17d2d6a with nextest profile: ci
Starting 10 tests across 1 binary
PASS [ 0.009s] rustfs::connect_perf_drive cancellation_and_single_collector_leave_no_scratch
PASS [ 0.008s] rustfs::connect_perf_drive cleanup_failure_overrides_cancellation_and_remains_attributed
PASS [ 0.008s] rustfs::connect_perf_drive consent_and_resource_budgets_fail_before_scratch_io
PASS [ 0.008s] rustfs::connect_perf_drive invalid_roots_and_existing_scratch_fail_without_clobbering
PASS [ 19.518s] rustfs::connect_perf_drive production_cli_measures_and_signs_exact_binary_provenance
PASS [ 4.146s] rustfs::connect_perf_drive production_cli_rejects_missing_l1_consent_and_limits_before_artifacts
PASS [ 0.009s] rustfs::connect_perf_drive read_only_root_and_full_drive_errors_are_classified
PASS [ 0.018s] rustfs::connect_perf_drive real_local_write_read_is_measured_and_scratch_is_removed
PASS [ 0.009s] rustfs::connect_perf_drive runtime_timeout_and_full_drive_use_schema_valid_failed_reason
PASS [ 0.054s] rustfs::connect_perf_drive successful_result_has_signed_bounded_private_offline_export
────────────
Summary [ 23.787s] 10 tests run: 10 passed, 0 skipped
exit_code=0
Mutation M1: success_measurement reports max(actual elapsed, requested budget + 1), forcing 1001 ms for the one-second request. The retained duration assertion at rustfs/tests/connect_perf_drive.rs:105 fails.
$ cargo nextest run --profile ci -p rustfs --test connect_perf_drive --no-fail-fast -E 'test(real_local_write_read_is_measured_and_scratch_is_removed)'
info: experimental features enabled: setup-scripts
Compiling rustfs v1.0.1 (/private/tmp/gw-orchestrator-2734/ci-drive/rustfs)
warning: linker stderr: ld: __eh_frame section too large (max 16MB) to encode dwarf unwind offsets in compact unwind table, performance of exception handling might be affected
|
= note: `#[warn(linker_messages)]` on by default
warning: `rustfs` (bin "rustfs") generated 1 warning
warning: `rustfs` (bin "rustfs-cli") generated 1 warning (1 duplicate)
Finished `test` profile [unoptimized + debuginfo] target(s) in 7m 06s
────────────
Nextest run ID 9f618777-7bf6-4cbc-8803-dd7f50f449ed with nextest profile: ci
Starting 1 test across 1 binary (9 tests skipped)
FAIL [ 0.016s] rustfs::connect_perf_drive real_local_write_read_is_measured_and_scratch_is_removed
stdout ───
running 1 test
test real_local_write_read_is_measured_and_scratch_is_removed ... FAILED
failures:
failures:
real_local_write_read_is_measured_and_scratch_is_removed
test result: FAILED. 0 passed; 1 failed; 0 ignored; 0 measured; 9 filtered out; finished in 0.01s
stderr ───
thread 'real_local_write_read_is_measured_and_scratch_is_removed' (106986846) panicked at rustfs/tests/connect_perf_drive.rs:105:5:
assertion failed: (1..=1_000).contains(&data.duration_millis)
note: run with `RUST_BACKTRACE=1` environment variable to display a backtrace
────────────
Summary [ 0.017s] 1 test run: 0 passed, 1 failed, 9 skipped
FAIL [ 0.016s] rustfs::connect_perf_drive real_local_write_read_is_measured_and_scratch_is_removed
error: test run failed
exit_code=100
Mutation M2: target_parameters reports requested duration + 1. The in-process assertion at rustfs/tests/connect_perf_drive.rs:111 and CLI JSON assertion at rustfs/tests/connect_perf_drive.rs:527 both reject 1001 against 1000.
$ cargo nextest run --profile ci -p rustfs --test connect_perf_drive --no-fail-fast -E 'test(real_local_write_read_is_measured_and_scratch_is_removed) | test(production_cli_measures_and_signs_exact_binary_provenance)'
info: experimental features enabled: setup-scripts
Compiling rustfs v1.0.1 (/private/tmp/gw-orchestrator-2734/ci-drive/rustfs)
warning: linker stderr: ld: __eh_frame section too large (max 16MB) to encode dwarf unwind offsets in compact unwind table, performance of exception handling might be affected
|
= note: `#[warn(linker_messages)]` on by default
warning: `rustfs` (bin "rustfs") generated 1 warning
warning: `rustfs` (bin "rustfs-cli") generated 1 warning (1 duplicate)
Finished `test` profile [unoptimized + debuginfo] target(s) in 7m 20s
────────────
Nextest run ID b73d46ab-c910-4e1e-8063-7fc7e332ec15 with nextest profile: ci
Starting 2 tests across 1 binary (8 tests skipped)
FAIL [ 13.968s] rustfs::connect_perf_drive production_cli_measures_and_signs_exact_binary_provenance
stdout ───
running 1 test
test production_cli_measures_and_signs_exact_binary_provenance ... FAILED
failures:
failures:
production_cli_measures_and_signs_exact_binary_provenance
test result: FAILED. 0 passed; 1 failed; 0 ignored; 0 measured; 9 filtered out; finished in 13.96s
stderr ───
thread 'production_cli_measures_and_signs_exact_binary_provenance' (107015462) panicked at rustfs/tests/connect_perf_drive.rs:527:5:
assertion `left == right` failed
left: Number(1001)
right: 1000
note: run with `RUST_BACKTRACE=1` environment variable to display a backtrace
FAIL [ 0.015s] rustfs::connect_perf_drive real_local_write_read_is_measured_and_scratch_is_removed
stdout ───
running 1 test
test real_local_write_read_is_measured_and_scratch_is_removed ... FAILED
failures:
failures:
real_local_write_read_is_measured_and_scratch_is_removed
test result: FAILED. 0 passed; 1 failed; 0 ignored; 0 measured; 9 filtered out; finished in 0.01s
stderr ───
thread 'real_local_write_read_is_measured_and_scratch_is_removed' (107016667) panicked at rustfs/tests/connect_perf_drive.rs:111:5:
assertion `left == right` failed
left: 1001
right: 1000
note: run with `RUST_BACKTRACE=1` environment variable to display a backtrace
────────────
Summary [ 13.985s] 2 tests run: 0 passed, 2 failed, 8 skipped
FAIL [ 13.968s] rustfs::connect_perf_drive production_cli_measures_and_signs_exact_binary_provenance
FAIL [ 0.015s] rustfs::connect_perf_drive real_local_write_read_is_measured_and_scratch_is_removed
error: test run failed
exit_code=100
Transient production mutations were restored byte-for-byte before the complete green rerun. Original/restored SHA256: 8d389e6d8836c041aab8c6a93e56addd65ba3f70b33264e1c74f639d0572a2da.
$ git diff --check
exit_code=0
$ git diff --cached --check
exit_code=0
Independent orchestrator acceptance
Tested code tree: e67c72fb01
Tested tree: 16fbbd2ea4
shellcheck --version && actionlint -> exit 0
cargo fmt --all --check -> exit 0
./scripts/check_unsafe_code_allowances.sh -> exit 0
./scripts/check_layer_dependencies.sh -> exit 0
./scripts/check_architecture_migration_rules.sh -> exit 0
./scripts/check_logging_guardrails.sh -> exit 0
./scripts/check_error_other_format_ratchet.sh -> exit 0
./scripts/check_no_tokio_io_uring.sh -> exit 0
./scripts/check_extension_schema_boundaries.sh -> exit 0
./scripts/check_body_cache_whitelist.sh -> exit 0
./scripts/check_s3s_footprint.sh -> exit 0
./scripts/check_test_count_ratchet.sh -> exit 0
./scripts/check_fips_wording.sh -> exit 0
./scripts/check_embedded_secrets.sh -> exit 0
make script-tests -> exit 0
python3 ./scripts/test_object_generation_protocol_model.py -> exit 0
python3 ./scripts/check_test_wiring.py -> exit 0
./scripts/check_no_planning_docs.sh -> exit 0
./scripts/check_uring_lane_lib_only.sh -> exit 0
$ LANG=en_US.UTF-8 CARGO_BUILD_JOBS=1 CARGO_INCREMENTAL=0 cargo nextest run --profile ci -p rustfs --test connect_perf_drive --no-fail-fast
info: experimental features enabled: setup-scripts
warning: linker stderr: ld: __eh_frame section too large (max 16MB) to encode dwarf unwind offsets in compact unwind table, performance of exception handling might be affected
|
= note: `#[warn(linker_messages)]` on by default
warning: `rustfs` (bin "rustfs-cli") generated 1 warning
warning: `rustfs` (bin "rustfs") generated 1 warning (1 duplicate)
Finished `test` profile [unoptimized + debuginfo] target(s) in 0.69s
────────────
Nextest run ID cd1edf4c-f1c0-456d-ad1a-76cc00ce05c9 with nextest profile: ci
Starting 10 tests across 1 binary
PASS [ 0.012s] rustfs::connect_perf_drive cancellation_and_single_collector_leave_no_scratch
PASS [ 0.010s] rustfs::connect_perf_drive cleanup_failure_overrides_cancellation_and_remains_attributed
PASS [ 0.010s] rustfs::connect_perf_drive consent_and_resource_budgets_fail_before_scratch_io
PASS [ 0.009s] rustfs::connect_perf_drive invalid_roots_and_existing_scratch_fail_without_clobbering
PASS [ 20.328s] rustfs::connect_perf_drive production_cli_measures_and_signs_exact_binary_provenance
PASS [ 5.527s] rustfs::connect_perf_drive production_cli_rejects_missing_l1_consent_and_limits_before_artifacts
PASS [ 0.013s] rustfs::connect_perf_drive read_only_root_and_full_drive_errors_are_classified
PASS [ 0.019s] rustfs::connect_perf_drive real_local_write_read_is_measured_and_scratch_is_removed
PASS [ 0.011s] rustfs::connect_perf_drive runtime_timeout_and_full_drive_use_schema_valid_failed_reason
PASS [ 0.049s] rustfs::connect_perf_drive successful_result_has_signed_bounded_private_offline_export
────────────
Summary [ 25.991s] 10 tests run: 10 passed, 0 skipped
exit_code=0
All 54 tracked Cargo.toml files parse without duplicate keys. git diff --check passed. The production source remains unchanged; the two deliberate implementation mutations above were restored before this complete green run.
* fix(test): compare legacy body wire parity as raw bytes
Lossy UTF-8 comparison concealed a changed binary response byte.
Compare the already captured Vec<u8> values directly.
Validation:
- Binary mutant: migrated payload 0x84 vs legacy 0x83; old comparator
passed 1/1, raw comparator failed 0/1 (exit 100); restored green.
- Non-Swift body-drain mutant: untouched-body delegation test failed
with empty vs payload (exit 100); restored handler without a diff.
- cargo nextest run --profile ci -p rustfs-s3-types -p rustfs-protocols
--features rustfs-protocols/swift --no-fail-fast:
543 tests run, 543 passed, 0 skipped after both mutants restored.
- cargo fmt --all --check and git diff --check passed.
Orchestrator acceptance (LANG=en_US.UTF-8):
cargo nextest run --profile ci -p rustfs-s3-types -p rustfs-protocols --features rustfs-protocols/swift --no-fail-fast
Summary [2.445s] 543 tests run: 543 passed, 0 skipped.
All shared quick checks on the final code tree:
shellcheck --version && actionlint -> exit 0
cargo fmt --all --check -> exit 0
./scripts/check_unsafe_code_allowances.sh -> exit 0
./scripts/check_layer_dependencies.sh -> exit 0
./scripts/check_architecture_migration_rules.sh -> exit 0
./scripts/check_logging_guardrails.sh -> exit 0
./scripts/check_error_other_format_ratchet.sh -> exit 0
./scripts/check_no_tokio_io_uring.sh -> exit 0
./scripts/check_extension_schema_boundaries.sh -> exit 0
./scripts/check_body_cache_whitelist.sh -> exit 0
./scripts/check_s3s_footprint.sh -> exit 0
./scripts/check_test_count_ratchet.sh -> exit 0
./scripts/check_fips_wording.sh -> exit 0
./scripts/check_embedded_secrets.sh -> exit 0
make script-tests -> exit 0
python3 ./scripts/test_object_generation_protocol_model.py -> exit 0
python3 ./scripts/check_test_wiring.py -> exit 0
./scripts/check_no_planning_docs.sh -> exit 0
./scripts/check_uring_lane_lib_only.sh -> exit 0
Final independent acceptance after the drive-test CI repair
Parent: a99d8f4f89
Tested code tree: 93cbd96bf0
Tested tree: 075ae5161c2b62a96812bc1e3c904a7663b4f2f8
shellcheck --version && actionlint -> exit 0
cargo fmt --all --check -> exit 0
./scripts/check_unsafe_code_allowances.sh -> exit 0
./scripts/check_layer_dependencies.sh -> exit 0
./scripts/check_architecture_migration_rules.sh -> exit 0
./scripts/check_logging_guardrails.sh -> exit 0
./scripts/check_error_other_format_ratchet.sh -> exit 0
./scripts/check_no_tokio_io_uring.sh -> exit 0
./scripts/check_extension_schema_boundaries.sh -> exit 0
./scripts/check_body_cache_whitelist.sh -> exit 0
./scripts/check_s3s_footprint.sh -> exit 0
./scripts/check_test_count_ratchet.sh -> exit 0
./scripts/check_fips_wording.sh -> exit 0
./scripts/check_embedded_secrets.sh -> exit 0
make script-tests -> exit 0
python3 ./scripts/test_object_generation_protocol_model.py -> exit 0
python3 ./scripts/check_test_wiring.py -> exit 0
./scripts/check_no_planning_docs.sh -> exit 0
./scripts/check_uring_lane_lib_only.sh -> exit 0
$ LANG=en_US.UTF-8 CARGO_BUILD_JOBS=1 CARGO_INCREMENTAL=0 cargo nextest run --profile ci -p rustfs-s3-types -p rustfs-protocols --features rustfs-protocols/swift --no-fail-fast
Finished `test` profile [unoptimized + debuginfo] target(s) in 10m 08s
Nextest run ID 5c7ff102-ec6d-479a-86e0-810859b28bbf with nextest profile: ci
Starting 543 tests across 8 binaries
Summary [ 6.729s] 543 tests run: 543 passed, 0 skipped
exit_code=0
The only source change is the raw-byte wire comparator. Cargo manifests and the ratchet baselines remain unchanged from the independently validated parent. The invalid-byte and consumed-request mutations remain recorded above; their implementation changes were restored.
* fix: repair integration checks and IPC test readiness
Wait for the RPC/locks test sockets to pass the existing owner and mode
checks inside the original 5s readiness window and 10ms poll interval.
Use the existing admin S3 carrier for the new recovery and quota error
calls. Regenerate the test-count baseline for main's 118 existing tests
across seven crates. Pair the private pagination helper's raw-scan inputs
and remove the redundant Copy clone from the retirement event.
Keep the original assertions, responses, authorization, locks and storage
operations. The pagination body and raw-scan expressions at all five calls
are byte-identical; reinserting only the Copy clone restores the entire
base heal file. All ignored baseline values and s3s counts stay unchanged.
Validation base: c47432e353
Validation tree: f081084bc5
Patch SHA-256: 2dcf0028691bfb2772edfb38e262829b04f4151832dcce823c9dd9f7434b52de
Trace production-prefix SHA-256: fd994f218a21429d62866f4efb8eee1ee57b58f4a51865aa35d05a54ef1ccb45
All commands used LANG=en_US.UTF-8, CARGO_BUILD_JOBS=1,
CARGO_INCREMENTAL=0 and CARGO_NET_OFFLINE=true in one exclusive target.
Every gate retained the same tree and six source hashes before and after.
54 Cargo.toml files parse without duplicate keys.
cargo clippy -p rustfs --all-targets -- -D warnings
Finished `dev` profile [unoptimized + debuginfo] target(s) in 12m 50s
Exit: 0. Wall time: 770.855s.
The original 19 quick-check steps:
shellcheck --version && actionlint -> exit 0
cargo fmt --all --check -> exit 0
./scripts/check_unsafe_code_allowances.sh -> exit 0
./scripts/check_layer_dependencies.sh -> exit 0
./scripts/check_architecture_migration_rules.sh -> exit 0
./scripts/check_logging_guardrails.sh -> exit 0
./scripts/check_error_other_format_ratchet.sh -> exit 0
./scripts/check_no_tokio_io_uring.sh -> exit 0
./scripts/check_extension_schema_boundaries.sh -> exit 0
./scripts/check_body_cache_whitelist.sh -> exit 0
./scripts/check_s3s_footprint.sh -> exit 0
./scripts/check_test_count_ratchet.sh -> exit 0
./scripts/check_fips_wording.sh -> exit 0
./scripts/check_embedded_secrets.sh -> exit 0
make script-tests -> exit 0
python3 ./scripts/test_object_generation_protocol_model.py -> exit 0
python3 ./scripts/check_test_wiring.py -> exit 0
./scripts/check_no_planning_docs.sh -> exit 0
./scripts/check_uring_lane_lib_only.sh -> exit 0
Wall time: 286.337s.
cargo nextest run --profile ci -p rustfs --no-fail-fast
Summary [ 217.017s] 5382 tests run: 5382 passed (4 leaky), 40 skipped
Exit: 0. Wall time: 1324.414s.
cargo nextest run --profile ci -p rustfs-s3-types -p rustfs-protocols --features rustfs-protocols/swift --no-fail-fast
Summary [ 2.189s] 543 tests run: 543 passed, 0 skipped
Exit: 0. Wall time: 123.192s.
cargo nextest run --profile ci -p rustfs-ecstore list_objects_paginate --no-fail-fast
Summary [ 0.170s] 8 tests run: 8 passed, 5921 skipped
Exit: 0. Wall time: 261.324s.
cargo nextest run --profile ci -p rustfs-heal durable_mrf_anchor_retires_only_after_exact_bucket_generation_is_deleted --no-fail-fast
Summary [ 0.805s] 1 test run: 1 passed, 835 skipped
Exit: 0. Wall time: 199.801s.
Baseline regeneration:
./scripts/check_test_count_ratchet.sh --update
wrote .config/test-count-baseline.txt: 52 crates, 21340 tests, 164 ignored
Test-only increases: ecstore48, heal3, iam14, io-metrics2, obs3, policy14,
rustfs34. Existing ignored values and all other rows are byte-identical.
s3s footprint OK: files importing s3s is 195 (baseline files=195)
s3s footprint OK: s3s s3_error! lines is 1584 (baseline s3_error_lines=1584)
s3s footprint OK: ecstore files referencing s3s is 36 (baseline ecstore_files=36)
Historical runtime red controls on f622 (same retained two assertions):
cargo nextest run --profile ci -p rustfs --lib -E 'test(=connect::diagnostics::trace_runtime::tests::local_top_rpc_reads_service_http_completions_and_signs_offline) | test(=connect::diagnostics::trace_runtime::tests::local_top_locks_reads_service_manager_and_signs_offline)' --no-fail-fast
Original path readiness with a temporary 500ms bind/seal gap:
Summary [ 0.863s] 2 tests run: 0 passed, 2 failed, 5113 skipped
Exit: 100.
Sealed readiness in that same gap:
Summary [ 5.621s] 2 tests run: 2 passed, 5113 skipped
Exit: 0.
Compiled M1: replace the helper predicate with path.exists():
Summary [ 0.870s] 2 tests run: 0 passed, 2 failed, 5113 skipped
Exit: 100.
Restore the sealed predicate byte-for-byte:
Summary [ 5.640s] 2 tests run: 2 passed, 5113 skipped
Exit: 0.
Both reds observe mode755 with owner_matches=true and fail at the original
RPC/locks unwraps with StateSecurity. M1 failure locations: trace_runtime.rs
2547:29 and 2713:29. No new assertions were introduced.
cargo nextest run --profile ci -p rustfs --lib -E 'test(connect::diagnostics::trace_runtime::tests::)' --no-fail-fast
Summary [ 20.748s] 28 tests run: 28 passed (1 leaky), 5087 skipped
Exit: 0. This security sweep is historical; the full final unit gate above
covers the actual c474 candidate.
The first disk-stopped compile ran no tests and is excluded from red credit.
All temporary seal instrumentation is removed from the final source.
Final unit cleanup markers: 4 leaky, 40 original skips (exit0). The ld
__eh_frame linker warning also appears verbatim in the original f622 run
and the earlier c474 run; there are no new source warnings.
* fix: remove redundant pending closure in disk test
Pass the capture-free pending future factory as a function item in the existing slow-operation test. This fixes the redundant_closure error at disk_store.rs:2914 that stopped strict Clippy before the feature tests could run.
The single-line test edit preserves the factory's result, warning and timeout behavior. Reversing the replacement reproduces the parent file byte for byte. All 220 assertion lines and the production prefix are unchanged; no tests, assertions, guards, profiles or baselines were added or relaxed.
Original Linux CI evidence (run 37888552268 at 53245be9f):
- RIO job113685483992: strict Clippy exit101, redundant closure at disk_store.rs:2914; feature tests skipped.
- Workspace Test and Lint job113685484058: same strict Clippy exit101; nextest and doc tests skipped.
These are lint failures, not runtime mutation kills or known flakes.
Native macOS arm64 validation on tree7a5bf47e7533b29479bc85207416102f1337c566, LANG=en_US.UTF-8, one build job, CARGO_INCREMENTAL=0 and offline dependencies:
shellcheck --version && actionlint -> exit 0
cargo fmt --all --check -> exit 0
./scripts/check_unsafe_code_allowances.sh -> exit 0
./scripts/check_layer_dependencies.sh -> exit 0
./scripts/check_architecture_migration_rules.sh -> exit 0
./scripts/check_logging_guardrails.sh -> exit 0
./scripts/check_error_other_format_ratchet.sh -> exit 0
./scripts/check_no_tokio_io_uring.sh -> exit 0
./scripts/check_extension_schema_boundaries.sh -> exit 0
./scripts/check_body_cache_whitelist.sh -> exit 0
./scripts/check_s3s_footprint.sh -> exit 0
./scripts/check_test_count_ratchet.sh -> exit 0
./scripts/check_fips_wording.sh -> exit 0
./scripts/check_embedded_secrets.sh -> exit 0
make script-tests -> exit 0
python3 ./scripts/test_object_generation_protocol_model.py -> exit 0
python3 ./scripts/check_test_wiring.py -> exit 0
./scripts/check_no_planning_docs.sh -> exit 0
./scripts/check_uring_lane_lib_only.sh -> exit 0
All 19 checks: exit0, 312.161s.
$ cargo clippy -p rustfs -p rustfs-ecstore --all-targets --features rio-v2 -- -D warnings
Finished `dev` profile [unoptimized + debuginfo] target(s) in 20m 11s
exit0, 1213.210s total.
$ cargo nextest run --profile ci -p rustfs-ecstore --features rio-v2 slow_disk_operation_warns_once_while_pending_and_preserves_result --no-fail-fast
Finished `test` profile [unoptimized + debuginfo] target(s) in 4m 34s
PASS [0.026s] rustfs-ecstore disk::disk_store::tests::slow_disk_operation_warns_once_while_pending_and_preserves_result
Summary [ 0.170s] 1 test run: 1 passed, 5943 skipped
exit0, 281.920s total; 5943 existing filter skips.
warning: linker stderr: ld: __eh_frame section too large (max 16MB) to encode dwarf unwind offsets in compact unwind table, performance of exception handling might be affected
This linker warning is retained; flags, profiles and allowances are unchanged.
54 tracked Cargo.toml files parsed without duplicate keys. git diff --check passed. No new assertions were introduced, so no new runtime mutation is claimed. Native compilation and the existing helper test do not substitute for actual Linux RIO execution; the new integration CI run still needs to recover before the RPC merge.
Preservation and output hashes:
parent file SHA256: 8e59ee745e3aaaa22848b943d3b10f6353b515956282cde1f3af2c394200c9ac
final file SHA256: b69ba3237061b161448ae8b62b615885d5770b98a5b4a2479bcfbd70f1bd1f17
unchanged production prefix SHA256: 7903a6e288252c420231242d7b58a25fa2a7140ea3dd430cbe7f0dde11369dcb
RIO Linux failure log SHA256:421addb9868fd88e4109a922311e54f91f0dfd70862f8ba7fcaba193482ed90b
Workspace Linux failure log SHA256:e0aea8d0f014c022252a9534968e083c98cfc1fd760da445ecd04ec3f522e26a
19 output SHA256:35a54f28d707956ed1edb29d492c4dfe8a9fe753d103b111b5a2f1b5c41c2b98
RIO Clippy output SHA256:d6046e3868f2b3992fe11cdd04e9e98eb50a87d1234d9a70dbfeb154ff61638d
slow-operation output SHA256:605b7b258d2fd09c6ef2ab2ffc90d433d0509480f1e45125d20c823759e83608
Refs rustfs/backlog#2734.
* test(ecstore): isolate tier-free recovery fixture startup
The fixture constructor can take the production recovery lock and wait for 60 seconds before this test installs its queue and seeds metadata. Disable its background recovery only while creating this fixture, then restore the environment before calling the existing production recovery entrypoint.
Only the existing test setup changes. Its production entrypoint, default lock parameters, original assertions, 30-second timeout, cancellation and cleanup remain intact. No tests or assertions were added. Production and store/init.rs are byte-identical to the parent.
Independent acceptance ran on merge tree 8ec358cb42ba9ebb3bcfeafcef048a3f0366355b; the runner header reports parent HEAD, while before/after index and source hashes confirm this candidate. Worker task branch d8055766b8 has the same tree.
Every command used LANG=en_US.UTF-8. Task compilation also used CARGO_BUILD_JOBS=1, CARGO_INCREMENTAL=0, CARGO_NET_OFFLINE=true, an independent target, and empty RUSTC_WRAPPER/RUSTC_WORKSPACE_WRAPPER to keep compilers in its owned process group without changing repository configuration.
Command:
shellcheck --version && actionlint
Exit 0; observed output excerpt:
version: 0.11.0
Log SHA-256: fc9eec16b4191d058a295813ca9afbe9d2b37b59975da860170726be162a3924
Command:
cargo fmt --all --check
Exit 0; observed output excerpt:
(no stdout or stderr)
Log SHA-256: e3b0c44298fc1c149afbf4c8996fb92427ae41e4649b934ca495991b7852b855
Command:
./scripts/check_unsafe_code_allowances.sh
Exit 0; observed output excerpt:
(no stdout or stderr)
Log SHA-256: e3b0c44298fc1c149afbf4c8996fb92427ae41e4649b934ca495991b7852b855
Command:
./scripts/check_layer_dependencies.sh
Exit 0; observed output excerpt:
Layer dependency guard passed (no new reverse dependencies/cycles).
Log SHA-256: 69c6cb1896c0586b28d1167380d3260d99d8017a4265c8df5a2348f289e93a12
Command:
./scripts/check_architecture_migration_rules.sh
Exit 0; observed output excerpt:
Architecture migration rules passed.
Log SHA-256: 28790e70d070607ed4de0704f292a65030be92ec224ef106d4b72fd8296f896e
Command:
./scripts/check_logging_guardrails.sh
Exit 0; observed output excerpt:
✅ Logging guardrails check passed
Log SHA-256: 7adf41abff8ed8f7617decf86498888ddde78937099dd9ce47f6d359dfc2d317
Command:
./scripts/check_error_other_format_ratchet.sh
Exit 0; observed output excerpt:
error(format!) ratchet passed (total call sites: 579).
Log SHA-256: 66172aef18f150ba0605af122e26ffcf91f6dd86f43e38b05e2e982ce8d783ee
Command:
./scripts/check_no_tokio_io_uring.sh
Exit 0; observed output excerpt:
(no stdout or stderr)
Log SHA-256: e3b0c44298fc1c149afbf4c8996fb92427ae41e4649b934ca495991b7852b855
Command:
./scripts/check_extension_schema_boundaries.sh
Exit 0; observed output excerpt:
Extension schema boundaries passed.
Log SHA-256: f13d0acd0e2ce5d9815241b12b4dab7d01cc11b9b65bcf6dfb7812c878532a51
Command:
./scripts/check_body_cache_whitelist.sh
Exit 0; observed output excerpt:
Body-cache whitelist guard passed.
Log SHA-256: 6b828847b0383950ee6eef9cd846d0c322c945e47efd4c7d1ad881178810d6bb
Command:
./scripts/check_s3s_footprint.sh
Exit 0; observed output excerpt:
s3s footprint OK: files importing s3s is 195 (baseline files=195)
s3s footprint OK: s3s s3_error! lines is 1584 (baseline s3_error_lines=1584)
s3s footprint OK: ecstore files referencing s3s is 36 (baseline ecstore_files=36)
✅ s3s footprint baseline check passed
Log SHA-256: 47f53d04ef645618563fe1f22abc64667f19078f1974feedcca46e4f2c016844
Command:
./scripts/check_test_count_ratchet.sh
Exit 0; observed output excerpt:
test-count ratchet OK: 52 crates, 21340 tests, 164 ignored match .config/test-count-baseline.txt
Log SHA-256: 618a5e330e933de0c6ff484546df401457024a88bb5986d11d588794e33dab60
Command:
./scripts/check_fips_wording.sh
Exit 0; observed output excerpt:
FIPS wording guard passed (README.md and CHANGELOG.md contain no forbidden claims).
KMS wording guard passed (crates/kms claims no Vault KV2 Transit wrapping).
Log SHA-256: 99221cc9979c480ec4f46f50d76c52b18c72fe499f12e15e4f4a22a1b8bfb140
Command:
./scripts/check_embedded_secrets.sh
Exit 0; observed output excerpt:
Embedded secret guard passed (no private key material or provider credential literal in the tree).
Log SHA-256: 437a3ceba32094d2901889965ad191e411aad32cdb18bf0b8659e818011c406b
Command:
make script-tests
Exit 0; observed output excerpt:
test_cli_requires_the_repository_default_branch (__main__.SelfTests.test_cli_requires_the_repository_default_branch) ... ok
test_config_rejects_duplicate_and_invalid_entries (__main__.SelfTests.test_config_rejects_duplicate_and_invalid_entries) ... ok
test_freshness_boundaries (__main__.SelfTests.test_freshness_boundaries) ... ok
test_grace_requires_two_successful_queries_with_no_history (__main__.SelfTests.test_grace_requires_two_successful_queries_with_no_history) ... ok
test_invalid_api_evidence_fails_closed (__main__.SelfTests.test_invalid_api_evidence_fails_closed) ... ok
test_report_retains_every_workflow (__main__.SelfTests.test_report_retains_every_workflow) ... ok
test_requests_filter_schedule_default_branch_and_success_on_server (__main__.SelfTests.test_requests_filter_schedule_default_branch_and_success_on_server) ... ok
Ran 10 tests in 0.042s
OK
Ran 30 tests in 20.627s
OK
Ran 10 tests in 2.268s
OK
Ran 33 tests in 0.127s
OK
Ran 7 tests in 0.002s
OK
Ran 7 tests in 0.001s
OK
Ran 4 tests in 0.108s
OK
Ran 3 tests in 0.564s
OK
PASS: cold stampede script self-test
Log SHA-256: 75fda2d069a373c63667b908baaf99e6591aab9099a739b0c4c22ceb6da8d68c
Command:
python3 ./scripts/test_object_generation_protocol_model.py
Exit 0; observed output excerpt:
test_an_unavailable_decision_quorum_cannot_choose_a_candidate (__main__.ObjectGenerationProtocolModel.test_an_unavailable_decision_quorum_cannot_choose_a_candidate) ... ok
test_counter_allocation_requires_persistence_and_is_consumed_once (__main__.ObjectGenerationProtocolModel.test_counter_allocation_requires_persistence_and_is_consumed_once) ... ok
test_counter_restart_burns_persisted_allocations_before_or_after_issue (__main__.ObjectGenerationProtocolModel.test_counter_restart_burns_persisted_allocations_before_or_after_issue) ... ok
test_crash_after_promise_before_reply_does_not_restore_an_older_ballot (__main__.ObjectGenerationProtocolModel.test_crash_after_promise_before_reply_does_not_restore_an_older_ballot) ... ok
test_duplicate_or_unconfigured_voters_cannot_form_a_quorum (__main__.ObjectGenerationProtocolModel.test_duplicate_or_unconfigured_voters_cannot_form_a_quorum) ... ok
test_empty_or_unobserved_minority_slot_can_retry_without_allocating_another_revision (__main__.ObjectGenerationProtocolModel.test_empty_or_unobserved_minority_slot_can_retry_without_allocating_another_revision) ... ok
test_equal_ballot_cannot_accept_different_bytes_and_lower_ballot_is_rejected (__main__.ObjectGenerationProtocolModel.test_equal_ballot_cannot_accept_different_bytes_and_lower_ballot_is_rejected) ... ok
test_forgetting_tombstone_authority_allows_an_aba_create (__main__.ObjectGenerationProtocolModel.test_forgetting_tombstone_authority_allows_an_aba_create) ... ok
test_issuing_before_persistence_reuses_a_ballot_after_restart (__main__.ObjectGenerationProtocolModel.test_issuing_before_persistence_reuses_a_ballot_after_restart) ... ok
test_lost_ack_and_restart_preserve_the_chosen_operation (__main__.ObjectGenerationProtocolModel.test_lost_ack_and_restart_preserve_the_chosen_operation) ... ok
test_minority_accepted_work_is_adopted_and_finished (__main__.ObjectGenerationProtocolModel.test_minority_accepted_work_is_adopted_and_finished) ... ok
test_omitting_highest_accepted_adoption_has_a_double_choice_counterexample (__main__.ObjectGenerationProtocolModel.test_omitting_highest_accepted_adoption_has_a_double_choice_counterexample) ... ok
test_quorum_choice_does_not_require_an_uncontacted_disk_to_learn_it (__main__.ObjectGenerationProtocolModel.test_quorum_choice_does_not_require_an_uncontacted_disk_to_learn_it) ... ok
test_recovery_failure_or_stale_expected_head_does_not_allocate_a_successor (__main__.ObjectGenerationProtocolModel.test_recovery_failure_or_stale_expected_head_does_not_allocate_a_successor) ... ok
test_replacing_voters_with_empty_state_can_choose_a_conflicting_value (__main__.ObjectGenerationProtocolModel.test_replacing_voters_with_empty_state_can_choose_a_conflicting_value) ... ok
test_skipping_predecessor_recovery_chooses_a_broken_revision_chain (__main__.ObjectGenerationProtocolModel.test_skipping_predecessor_recovery_chooses_a_broken_revision_chain) ... ok
test_stale_publication_rollback_and_gc_cannot_change_the_winner (__main__.ObjectGenerationProtocolModel.test_stale_publication_rollback_and_gc_cannot_change_the_winner) ... ok
test_superseded_operation_identity_cannot_be_reused_for_different_input (__main__.ObjectGenerationProtocolModel.test_superseded_operation_identity_cannot_be_reused_for_different_input) ... ok
test_superseded_operation_retry_returns_its_original_successor (__main__.ObjectGenerationProtocolModel.test_superseded_operation_retry_returns_its_original_successor) ... ok
test_tombstone_and_null_marker_payloads_do_not_reset_authority_revision (__main__.ObjectGenerationProtocolModel.test_tombstone_and_null_marker_payloads_do_not_reset_authority_revision) ... ok
test_tombstone_rejects_a_delayed_never_present_create (__main__.ObjectGenerationProtocolModel.test_tombstone_rejects_a_delayed_never_present_create) ... ok
test_unlearned_predecessor_is_recovered_before_the_second_slot (__main__.ObjectGenerationProtocolModel.test_unlearned_predecessor_is_recovered_before_the_second_slot) ... ok
Ran 24 tests in 0.929s
OK
Log SHA-256: 0b4be926efb0609d497ae58930340153f01957b606206cbbdb3e0d47eb29d3a2
Command:
python3 ./scripts/check_test_wiring.py
Exit 0; observed output excerpt:
OK: e2e modules, runner selection, fuzz matrices, profiles, scheduled alerts, and Quick Checks are wired
Log SHA-256: 21f676f458451525535ab0ef73d6a9c0d9774e16d76a4d19e396ad34936dd39c
Command:
./scripts/check_no_planning_docs.sh
Exit 0; observed output excerpt:
(no stdout or stderr)
Log SHA-256: e3b0c44298fc1c149afbf4c8996fb92427ae41e4649b934ca495991b7852b855
Command:
./scripts/check_uring_lane_lib_only.sh
Exit 0; observed output excerpt:
OK: no 'uring_'-matching test functions under crates/ecstore/tests; --lib is safe
Log SHA-256: 43f3f851e27ec2ab9e34d6b05fefdede38af992699c473904bfe6f53a46d52e3
Command:
cargo nextest run --profile ci -p rustfs-ecstore --lib --features rio-v2 -E 'test(=bucket::lifecycle::bucket_lifecycle_ops::tests::tier_free_version_recovery_production_entrypoint_enqueues_seeded_item)' --no-fail-fast
Exit 0; wall 234.399s.
Observed output:
Finished `test` profile [unoptimized + debuginfo] target(s) in 3m 49s
Starting 1 test across 1 binary (5871 tests skipped)
SETUP PASS [ 0.006s] ecstore-base-stack: sh -c 'echo RUST_MIN_STACK=4194304 >> "$NEXTEST_ENV"'
PASS [ 0.440s] rustfs-ecstore bucket::lifecycle::bucket_lifecycle_ops::tests::tier_free_version_recovery_production_entrypoint_enqueues_seeded_item
Summary [ 0.586s] 1 test run: 1 passed, 5871 skipped
Log SHA-256: 34e973f86e8dc2102c1a0e483e0ac802ce7df3d1b40a603932f62c0805c4de27
Command:
cargo nextest run --profile ci -p rustfs-ecstore --lib --features rio-v2 -E 'test(=bucket::lifecycle::bucket_lifecycle_ops::tests::tier_free_version_recovery_eight_coordinators_hold_one_leader_and_take_over) | test(=store::bucket::tests::creation_commit_backfill_runs_under_the_config_write_fence) | test(=store::bucket::tests::orphaned_bucket_recovery_requires_every_volume_absent_and_is_retryable)' --no-fail-fast
Exit 0; wall 4.010s.
Observed output:
Finished `test` profile [unoptimized + debuginfo] target(s) in 0.43s
Starting 3 tests across 1 binary (5869 tests skipped)
SETUP PASS [ 0.005s] ecstore-base-stack: sh -c 'echo RUST_MIN_STACK=4194304 >> "$NEXTEST_ENV"'
PASS [ 0.068s] rustfs-ecstore bucket::lifecycle::bucket_lifecycle_ops::tests::tier_free_version_recovery_eight_coordinators_hold_one_leader_and_take_over
PASS [ 0.637s] rustfs-ecstore store::bucket::tests::creation_commit_backfill_runs_under_the_config_write_fence
PASS [ 0.846s] rustfs-ecstore store::bucket::tests::orphaned_bucket_recovery_requires_every_volume_absent_and_is_retryable
Summary [ 0.987s] 3 tests run: 3 passed, 5869 skipped
Log SHA-256: b5da2a927ff18ca9aa217762f7186a6d4371bd9e8feedd7b77940e10b64dffac
Command:
cargo clippy -p rustfs -p rustfs-ecstore --all-targets --features rio-v2 -- -D warnings
Exit 0; wall 1084.508s.
Observed output:
Finished `dev` profile [unoptimized + debuginfo] target(s) in 18m 03s
Log SHA-256: 28f98be7970faaa71cd118da1da4e6e2e3d23b9dda7e6449eaa7a37e741260b5
Compiled mutation command (unchanged for each red and restored run):
cargo nextest run --profile ci -p rustfs-ecstore --lib --features rio-v2 -E 'test(=bucket::lifecycle::bucket_lifecycle_ops::tests::tier_free_version_recovery_production_entrypoint_enqueues_seeded_item)' --no-fail-fast
M1: reverse only this test-local constructor flag from false to true. The mutant compiled, then failed the original recovered-item queue assertion after 30.481s. Restoring the entire observed candidate passed in 0.460s. Red log 55bf3b1535529ac0bcfb8071f9b56f9f40631007a85c315367e7e93786d65293; restored log 204d5fbe4574a7917f5e740a795d864fe28a7e7fb5e33d27b0a9cf68e2f17f52.
M2: replace only the actual awaited enqueue with false, consuming its original inputs. The mutant compiled, then failed the same original queue assertion after 30.516s. Restoring the entire observed candidate passed in 0.457s. Red log 89186330413215e231e2b06bf013cc31097ea8f4a106d91421a36d6c8986a7fd; restored log e848b42d2302401afa5dbcd9e37bf76d2e56a96ab6de5a488513b997f53c5e9b.
All temporary observation and mutation code was removed before final acceptance. Historical baseline and diagnostic failures remain recorded; they are not mutation credit. The separate Object Lock intent-retry diagnostic still fails, and these native macOS checks do not establish Linux CI recovery.
* refactor(rpc): serve internode rpc bodies with the owned Body (#8441)
* test(rpc): pin the wire bytes of every internode rpc response body
rustfs/backlog#2747 batch 4, step 1 of 2, on the s3s bodies the rpc paths
serve today. Each response body builder in storage/rpc/http_service.rs is
served through hyper's HTTP/1.1 server and the raw bytes the peer reads are
compared with goldens captured from this commit: status, content-length or
chunked framing, chunk boundaries, the terminating chunk, HEAD, and the cut
connection a failure after data produces. The capability responses carry a
per-process proof, so their length header and payload are checked against
the body they encode instead.
The read_file_stream response builder moves into read_file_stream_response
so the test reaches the same code the handler runs; behaviour is unchanged.
* refactor(rpc): serve internode rpc bodies with the owned Body
rustfs/backlog#2747 batch 4, step 2 of 2. The internode rpc HTTP paths
build their responses with rustfs_s3_types::Body: Body::from_stream
replaces Body::from(StreamingBlob::wrap(..)) at the six stream sites, and
the in-memory and empty bodies keep their constructors. InternodeRpcService
converts its responses into the wrapped service's response body through
From<Body>, so the server keeps composing it with the s3s service as
before. The wire-byte goldens from the previous commit pass unchanged;
storage/rpc no longer names s3s.
* fix(rpc): simplify wire test response builder type
Name the boxed response builder used by the capability wire test so
all-target Clippy can check it without a type-complexity warning.
Validation (LANG=en_US.UTF-8, CARGO_INCREMENTAL=0):
- cargo clippy -p rustfs --all-targets -- -D warnings: exit 0.
- cargo nextest run --profile ci -p rustfs --lib -E 'test(/storage::rpc::/)': 279 tests run, 279 passed, 4842 skipped.
- All 19 quick-check commands, including script-tests, both Python scripts and cargo fmt --all --check: exit 0.
- Footprint baseline regenerated: files=194, s3_error_lines=1584, ecstore_files=36; test-count baseline regenerated: 52 crates, 21228 tests, 164 ignored.
- All 54 Cargo.toml files parse without duplicate keys.
No assertion was added or changed by this lint fix. R1-R8 are retained
in the existing PR verification record.
* fix(ecstore): resume fenced Object Lock creation intents
* fix(ecstore): bound creation retry futures on the stack
* fix(ecstore): require native proof for creation retries
* fix(ecstore): bound bucket creation stack usage (#8466)
* fix(ecstore): retain creation fences through rollback
Keep lifecycle, metadata-transaction, and namespace guards around rollback and metadata cleanup. Preserve the original publication error when a guard is lost. Build the timestamp retry fixture through the real interrupted producer so it checks a valid pending creation.
Validated source tree: 4e8e207999.
Two independent source reviews found no blocking regression.
Validation (LANG=en_US.UTF-8):
- bash -c 'shellcheck --version && actionlint' — exit 0
- cargo fmt --all --check — exit 0
- ./scripts/check_unsafe_code_allowances.sh — exit 0
- ./scripts/check_layer_dependencies.sh — exit 0
- ./scripts/check_architecture_migration_rules.sh — exit 0
- ./scripts/check_logging_guardrails.sh — exit 0
- ./scripts/check_error_other_format_ratchet.sh — exit 0
- ./scripts/check_no_tokio_io_uring.sh — exit 0
- ./scripts/check_extension_schema_boundaries.sh — exit 0
- ./scripts/check_body_cache_whitelist.sh — exit 0
- ./scripts/check_s3s_footprint.sh — exit 0
- ./scripts/check_test_count_ratchet.sh — exit 0
- ./scripts/check_fips_wording.sh — exit 0
- ./scripts/check_embedded_secrets.sh — exit 0
- make script-tests — exit 0
- python3 ./scripts/test_object_generation_protocol_model.py — exit 0
- python3 ./scripts/check_test_wiring.py — exit 0
- ./scripts/check_no_planning_docs.sh — exit 0
- ./scripts/check_uring_lane_lib_only.sh — exit 0
- cargo nextest run --profile ci -p rustfs-ecstore --lib --locked --no-fail-fast — 5894 passed, 0 skipped
- cargo check -p rustfs-ecstore --all-targets --locked — exit 0
- cargo nextest run --profile ci -p rustfs --lib --locked --no-fail-fast -E 'test(=admin::handlers::table_catalog::tests::cold_strong_registration_completes_while_other_bucket_backup_waits) | test(=admin::handlers::table_catalog::tests::commit_backend_borrowed_put_preserves_owner_authorization_and_precondition) | test(=admin::handlers::table_catalog::tests::disable_table_bucket_preserves_objects_and_can_be_reenabled) | test(=admin::handlers::table_catalog::tests::load_table_handler_negotiates_vended_credentials_without_breaking_metadata_only_callers) | test(=admin::handlers::table_catalog::tests::namespace_property_handler_updates_object_backed_catalog_and_maps_errors) | test(=app::object::get::tests::execute_get_object_resumes_from_relocated_pool_without_splicing_body) | test(=table_catalog::store::tests::catalog_writer_loss_prevents_recovery_storage_commit)' — 7 passed
- cargo nextest run --profile ci -p rustfs --lib --locked --no-fail-fast -E 'test(unsupported_remote_disk_)' — 2 passed
Additional fault-injection evidence remains incomplete. Timestamp M1 reached its original assertion; its restore run is being collected. M2–M5 and the three planned rollback faults have not run. An automated risk check interrupted the timestamp proof task; no assertions or validation settings were weakened.
* fix(ecstore): box nested bucket rollback state
Keep the guarded rollback future behind a heap allocation so bucket creation does not store its nested state inline. The rollback body, all three fence references and pending-proof revocation order are unchanged.
The same three original Linux tests overflowed on both e611 CI attempts. Local format, ecstore test compilation and all 19 quick-check steps pass; the original Linux feature lanes still need to validate this candidate.
* test(ecstore): isolate rollback guard loss fixtures
Run the three rollback metadata-cleanup cases in separate test processes so reused /proc/self/fd aliases from earlier cases cannot invalidate their creation witnesses. Keep the original fixture, guards, assertions and io_uring test selection, and verify that each child runs the intended case successfully.
Update the generated test-count baseline for the added macro arm.
Validation: cargo check -p rustfs-ecstore --tests; cargo fmt --all --check; all 19 quick-check commands, including make script-tests and the two Python checks. Linux io_uring acceptance and fault controls remain pending.
* refactor(replication): move the replication config onto gateway types (#8443)
* wip: T1.4 replication family
* refactor(replication): move the replication config onto gateway types
The replication configuration family leaves s3s: rustfs-replication no longer depends on s3s, and ECStore bucket metadata, the scanner and the rustfs app/admin layers hold rustfs_gateway_types::persistence::PersistedReplicationConfiguration. The persisted XML goes through the gateway persistence codec behind a new BucketConfigXml trait (the 39 byte goldens pass with unchanged fixtures); the legacy s3s edge converts once in storage/ecfs.rs through storage/s3_compat/replication.rs; the scanner data usage cache keeps its MessagePack bytes through a serde mirror pinned by bytes recorded from s3s 5761ddfe. rustfs-gateway-types 0.32.0 is pinned once (rev 772d3385). Refs rustfs/backlog#2745.
* test: use S3 serializer for list response tests
* fix(ci): restore integration test compilation
Gate the diagnostic test import with its feature, initialize the conditional
MetadataWriteStore fields explicitly, and decode the scanner replication fixture
through its existing cache boundary. Preserve the original fixture and assertions.
Validation was run on tree 544acb80c0 at 6003d4b8.
LANG=en_US.UTF-8; no source changes follow validation.
Quick checks: 19/19 passed.
- shellcheck --version && actionlint: exit 0
- cargo fmt --all --check: exit 0
- ./scripts/check_unsafe_code_allowances.sh: exit 0
- ./scripts/check_layer_dependencies.sh: exit 0
- ./scripts/check_architecture_migration_rules.sh: exit 0
- ./scripts/check_logging_guardrails.sh: exit 0
- ./scripts/check_error_other_format_ratchet.sh: exit 0
- ./scripts/check_no_tokio_io_uring.sh: exit 0
- ./scripts/check_extension_schema_boundaries.sh: exit 0
- ./scripts/check_body_cache_whitelist.sh: exit 0
- ./scripts/check_s3s_footprint.sh: exit 0
- ./scripts/check_test_count_ratchet.sh: exit 0
- ./scripts/check_fips_wording.sh: exit 0
- ./scripts/check_embedded_secrets.sh: exit 0
- make script-tests: exit 0
- python3 ./scripts/test_object_generation_protocol_model.py: exit 0
- python3 ./scripts/check_test_wiring.py: exit 0
- ./scripts/check_no_planning_docs.sh: exit 0
- ./scripts/check_uring_lane_lib_only.sh: exit 0
Darwin validation:
- cargo clippy --locked -p rustfs-ecstore -p rustfs-scanner --all-targets -- -D warnings: exit 0
- cargo nextest run --locked -p rustfs-ecstore -p rustfs-scanner --lib -E 'test(config::com::tests::save_config_preserves_retryable_system_volume_errors) | test(scanner_io::tests::replication_maintenance_refreshes_compacted_stats_across_cycles_and_retries)' --test-threads 1 --no-fail-fast: exit 0, 2/2 passed
- cargo clippy --locked -p rustfs-ecstore -p rustfs-scanner --all-targets --features rustfs-ecstore/diagnostic-read-version-caller,rustfs-scanner/diagnostic-read-version-caller -- -D warnings: exit 101
- cargo nextest run --locked -p rustfs-ecstore -p rustfs-scanner --lib --features rustfs-ecstore/diagnostic-read-version-caller,rustfs-scanner/diagnostic-read-version-caller -E 'test(config::com::tests::save_config_preserves_retryable_system_volume_errors) | test(scanner_io::tests::replication_maintenance_refreshes_compacted_stats_across_cycles_and_retries) | test(cluster::rpc::runtime_sources::tests::) | test(config::com::tests::diagnostic_config_write_retains_origin_and_api_for_results_and_cancellation)' --test-threads 1 --no-fail-fast: exit 0, 14/14 passed
Diagnostic strict Clippy remains red on four unchanged production lints:
runtime_sources.rs if_same_then_else and three redundant_locals reports in
set_disk/core/metadata_read.rs and set_disk/read.rs. Their source ranges and
feature wiring match the base; the base was not compiled separately.
Linux aarch64 validation (Rust/Clippy 1.99.0, UID 501, jobs=1):
- cargo clippy --all-targets -- -D warnings: exit 0; Finished dev profile in 68m 12s
- cargo clippy -p rustfs -p rustfs-ecstore --all-targets --features rio-v2 -- -D warnings: exit 0; Finished dev profile in 49m 45s
Source HEAD/tree and clean status matched before and after both commands.
No compiler or stack overrides were used. protoc 35.1 was installed from its
verified ARM64 release; flatc was absent and not invoked by the workspace build scripts.
The local environment differs from hosted CI.
* fix(ecstore): recover delete batches with missing versions
Treat an explicit non-null version as a no-op when it is absent from both recovery sources. Keep rejecting missing null versions, delete-marker mismatches and conflicting local generations.
Add returning-disk and partial-receipt regressions, extend journal boundary checks, and regenerate the ecstore test-count baseline. Production faults exercise the 24 added assertion sites and 7 semantic checks: 33 faults failed at the expected sites and passed after full restoration.
Validation (LANG=en_US.UTF-8, one build job):
cargo nextest run --locked -p rustfs-ecstore --lib -E 'test(delete_recovery) | test(quorum_delete_cleanup_coordinator_binds_single_and_batch) | test(crash_delete_retry_batch_reclaims_inline_and_reserved_external)' --test-threads 1 --no-fail-fast
Summary [ 34.580s] 69 tests run: 69 passed, 6058 skipped; exit 0
cargo clippy --locked -p rustfs-ecstore --all-targets -- -D warnings
Finished `dev` profile [unoptimized + debuginfo] target(s) in 5m 37s; exit 0
All 19 final Quick Checks completed with exit 0:
- shellcheck --version && actionlint
- cargo fmt --all --check
- ./scripts/check_unsafe_code_allowances.sh
- ./scripts/check_layer_dependencies.sh
- ./scripts/check_architecture_migration_rules.sh
- ./scripts/check_logging_guardrails.sh
- ./scripts/check_error_other_format_ratchet.sh
- ./scripts/check_no_tokio_io_uring.sh
- ./scripts/check_extension_schema_boundaries.sh
- ./scripts/check_body_cache_whitelist.sh
- ./scripts/check_s3s_footprint.sh
- ./scripts/check_test_count_ratchet.sh
- ./scripts/check_fips_wording.sh
- ./scripts/check_embedded_secrets.sh
- make script-tests
- python3 ./scripts/test_object_generation_protocol_model.py
- python3 ./scripts/check_test_wiring.py
- ./scripts/check_no_planning_docs.sh
- ./scripts/check_uring_lane_lib_only.sh
test-count ratchet OK: 52 crates, 22009 tests, 165 ignored match .config/test-count-baseline.txt
PASS: cold stampede script self-test
model: states=51023, transitions=167520, terminal=732, chosen_candidates=2
multi-slot round schedules: 640
OK: e2e modules, runner selection, fuzz matrices, profiles, scheduled alerts, and Quick Checks are wired
* fix(ecstore): retry tier commit intent lock contention
Retry transient namespace lock timeouts within the existing three-attempt
CAS budget. Reload and verify the original mutation identity after
contention, and preserve the final typed error when attempts are exhausted.
Cover save/read contention, exhausted retries, replacement identities and
CAS reconciliation without changing the existing timeout or lock policy.
Validation:
- cargo nextest run --locked -p rustfs-ecstore --features test-util --lib -E 'test(tier_mutation)' --test-threads 1 --no-fail-fast: 52 passed.
- cargo nextest run --locked -p rustfs-ecstore --features test-util --lib -E 'test(intent_advance_) | test(=services::tier::tier::tests::caller_cancellation_after_durable_prepare_does_not_hide_the_mutation)' --test-threads 1 --no-fail-fast: 5 passed.
- cargo clippy --locked -p rustfs-ecstore --all-targets -- -D warnings: passed.
- cargo clippy --locked -p rustfs-ecstore --all-targets --features test-util -- -D warnings: passed.
- 27 mutation pairs failed at the intended assertions and passed after restoring the production code.
- bash scripts/check_test_count_ratchet.sh --update, then bash scripts/check_test_count_ratchet.sh: 52 crates, 22013 tests, 165 ignored; ecstore increases from 6581 to 6585.
* refactor(lifecycle): replace s3s domain types with Gateway DTOs
Keep lifecycle rules, restore requests, Object Lock, and scanner cached tags in domain types. Convert at the existing HTTP boundary while preserving legacy XML, scanner cache bytes, date ordering, and tombstones.
Linux test validation before the final import-path-only adjustment:
cargo nextest run -p rustfs-ecstore bucket_config_goldens
39 passed; 6211 skipped by selector.
cargo nextest run --lib -p rustfs -E 'test(/test_zero_rule_lifecycle_tombstone_recognition/) | test(/test_local_lifecycle_staleness_axis_selection/) | test(/test_lifecycle_staleness_preserves_legacy_submillisecond_axis/) | test(/test_merge_incoming_lifecycle/) | test(/test_lifecycle_expiry_statement_matrix/) | test(/test_lifecycle_expiry_subset_xml/) | test(/put_bucket_lifecycle_validation_errors_keep_their_s3_code/) | test(/execute_put_bucket_lifecycle_configuration_rejects_missing_configuration/) | test(/execute_restore_object_rejects_missing_restore_request/) | test(/execute_restore_object_rejects_select_type/) | test(/execute_restore_object_maps_failures_to_typed_s3_errors/) | test(/execute_restore_object_returns_internal_error_when_store_uninitialized/) | test(/restore_request_conversion_preserves_legacy_output_values/) | test(/restore_request_conversion_preserves_presence_and_raw_enums/) | test(/restore_request_conversion_distinguishes_absent_and_empty_containers/)'
25 passed; 5318 skipped by selector.
cargo nextest run --profile ci -p rustfs-ecstore -p rustfs-lifecycle -p rustfs-replication -p rustfs-scanner
7872 passed; 21 skipped under the existing CI profile.
Previously verified mutation controls: 39 production faults, 41 RED/GREEN pairs.
cargo clippy --locked -p rustfs -p rustfs-ecstore -p rustfs-lifecycle -p rustfs-scanner --all-targets -- -D warnings
Passed in 40m 45s before the import-path adjustment, then in 1m 59s on
the final source with warnings denied. The final adjustment imports the
existing helper through the object domain facade; fmt and the architecture
check also passed.
Part of rustfs/backlog#2745. The remaining ecstore configuration families follow separately.
* refactor(admin): move error types to rustfs-s3-types (#8440)
* refactor(protocols): move error types to rustfs-s3-types by codemod
T1.2 batch 3 (rustfs/backlog#2743): s3select-api, s3select-query and
the error items of protocols move off s3s by scripts/codemods, run with
--hold crates/protocols/src/sftp/errors.rs. That file stays on s3s: it
downcasts the backend error, whose runtime type is still s3s::S3Error.
- rustfs/src/protocols/client.rs: the two webdav session methods now
return rustfs_s3_types::S3Result and convert the s3s error with From.
- codemod: a child module that calls s3_error! through `use super::*`
gets its own `use rustfs_s3_types::s3_error;` once its parent imports
that macro by name; otherwise it is listed as GLOB.
- s3-types: a table test pins code, status and the s3s error the legacy
edge renders for every variant a moved call site names.
- s3_error_lines 1585 -> 1584 (a lifecycle comment reworded).
* refactor(admin): move error types to rustfs-s3-types (WIP, no build)
T1.2 batch 4 (rustfs/backlog#2743), work in progress: rustfs check
still fails with 16 errors (listed in the PR body).
- codemod: --facade PATH treats a crate-local re-export of s3s's error
items like s3s for the four names (9 new self-tests, 9 mutants red).
- codemod output for rustfs/src/admin and rustfs/src/site_replication
(--facade crate::storage_api::site_replication::s3); 6 glob children
get an explicit rustfs_s3_types::s3_error import.
- admin/router.rs: the Operation trait returns rustfs_s3_types::S3Result.
- error.rs: From<ApiError> builds the rustfs-s3-types error; the s3s
conversion is derived from it.
* fix(admin): convert error types at the s3s boundary
Batch 4 of T1.2 left the admin and site_replication modules returning
rustfs_s3_types::S3Error while their callers and callees on the storage
and auth side still speak s3s::S3Error. Convert at each edge with the
existing From impls in rustfs-s3-types (compat_s3s.rs) instead of
changing any code, message or status:
- admin/auth.rs, admin/handlers/object_zip_download.rs,
admin/handlers/table_catalog/mod.rs: map the s3s result returned by
check_key_valid / authorize_request / authorize_internal_object_request
into the admin error type with .map_err(Into::into).
- storage/ecfs.rs: the s3s-facing side maps the admin site_replication
gate results the other way.
- admin/handlers/pools.rs: ApiError.code stays s3s::S3ErrorCode, so the
rustfs code is converted with .into() when the struct is built, in
production and in the two tests that build or compare it.
- admin/handlers/heal.rs, admin/runtime_sources.rs: the router returns
s3s errors (S3Route impl); the expected rustfs code is converted with
.into() before the comparison.
- admin/handlers/table_catalog/mod.rs: iceberg_rest_error takes
&'static str so the Cow<'static, str> custom code borrows it; every
caller already passes an ICEBERG_ERROR_* constant or a &'static str.
- admin/handlers/quota.rs: drop a now same-type .into() after s3_error!.
- storage_api.rs, site_replication/mod.rs: drop re-exports and an import
that no longer have users (clippy -D warnings).
cargo check -p rustfs --all-targets and cargo clippy -p rustfs
--all-targets -- -D warnings are clean; the s3s footprint baseline is
unchanged at files=172, s3_error_lines=303, ecstore_files=28.
Refs rustfs/backlog#2743, rustfs/backlog#2734.
* refactor(ecstore): move bucket configs onto gateway persisted types (#8709)
* refactor(ecstore): move bucket configs onto gateway persisted types
Finish the ECStore batch of T1.4 (rustfs/backlog#2745). The remaining
eleven bucket configuration families (Website, CORS, Tagging, Versioning,
ObjectLock, Encryption, Notification, Logging, Accelerate, RequestPayment,
PublicAccessBlock) now read and write their stored XML through the
rustfs-gateway-types persistence and cors_tagging codecs instead of the
s3s codec, so the legacy s3s bucket-config codec and its
s3gate-metadata-xml compat marker are removed.
- ecstore: BucketConfigXml is implemented by the gateway codec for every
family; metadata, metadata_sys, versioning, object lock, replication and
lifecycle consumers take the persisted types. Region moves to
rustfs_s3_types::Region, and the ETag conversion to s3s moves to the
rustfs edge with its tests. Only warm_backend.rs and warm_backend_s3.rs
still reference s3s, held for T1.5 (rustfs/backlog#2746).
- scanner: object lock cache, item actions and data usage use the
persisted types.
- rustfs: the legacy S3 edge converts each family between s3s DTOs and the
persisted types in storage/storage_api.rs by writing one codec's XML and
reading it with the other; Tagging converts field by field.
- Bump the rustfs-gateway-types pin to 21283e87.
The 39 persisted-byte goldens are unchanged and pass, so a document the
old codec wrote reads back and re-encodes to the same bytes. s3s footprint:
files 181 -> 146, ecstore_files 28 -> 2, s3_error_lines unchanged.
* test(app): prepare the PutBucketEncryption config from the request
The uninitialized-store test passed an empty persisted configuration, which the use case now validates before it reaches the store check, so it answered MalformedXML instead of InternalError. Convert the request's configuration the way the S3 edge does.
* style(app): format the PutBucketEncryption test
* refactor(s3-client): move remote error codes and DTO enums off s3s (#8783)
Part of T1.5 (rustfs/backlog#2746), batch 1.
* refactor(s3select): run queries on gateway DTO request shapes (#8794)
* refactor(s3select): run queries on an engine request built from gateway DTO shapes
Part of T1.5 (rustfs/backlog#2746), batch 2.
* fix(s3select): keep the select request conversion in the app layer
The layer guard forbids app -> storage; the conversion moves from storage/s3_compat/select.rs to app/select_request.rs unchanged.
* fix(heal): retry transient metadata conflicts (#8812)
* fix(heal): retry transient metadata conflicts
* chore: update heal test count baseline
* test: reserve nextest capacity for progress probes (#8818)
* test(e2e): use a non-default region for the differential negative control
Since main's region-aware CreateBucket and GetBucketLocation (#8739, #8743), us-east-1 answers an empty LocationConstraint and 200 for an owned re-create, so the control no longer reported its own region. Also refresh the e2e-full linux selection from the CI listing (682 tests).
* refactor(protocols): move StorageBackend to gateway types (#8825)
* refactor(protocols): move storage backend to gateway types
* test: update e2e test count baseline
* fix: preserve copy source keys and unknown lengths
* test: update protocols test count baseline
* fix(protocols): preserve typed SFTP backend errors
* test(protocols): use gateway list input in policy tests
* fix(protocols): clear clippy failures after T1.5 merge (#8831)
fix(protocols): clear clippy failures after merge
* fix(ecstore): finalize restore metadata after local copy-back (#8836)
* fix(ecstore): finalize restore metadata after local copy-back
* test(ecstore): ratchet added restore finalization cases
* fix(ecstore): keep restore copyback compatible with older peers
* fix(ecstore): gate restore copyback on fleet compatibility
* fix(ecstore): retain transition identity during restore copyback
* fix(ecstore): route committed restore copyback locally
* fix restore copyback cleanup
* fix(ecstore): honor restore probe test fallback
* refactor(storage): migrate option errors to domain types (#8839)
* refactor(s3): migrate private multipart validation errors (#8844)
* refactor(auth): derive policy conditions from AuthzFacts (#8442)
* refactor(auth): extract authorization facts
* test(e2e): update smoke selection baseline
* test(e2e): update full selection baseline
* refactor(admin): migrate health and prefix usage to owned requests (#8887)
* refactor(admin): add owned health router boundary
* refactor(admin): migrate prefix usage to owned request
* refactor(admin): complete the owned admin request boundary (#8949)
* refactor(admin): complete the owned admin request boundary
Every type implementing the owned admin `Operation` now serves the legacy
router through one blanket impl that converts the s3s request and response
at that single edge, so handlers register with `AdminOperation(&Handler {})`
and need no adapter statics; `LegacyAdminAdapter` is removed.
`AdminRequest` gains the accessors handlers read today (`remote_addr`,
`client_info`, `request_context`, `req_info`, `req_info_mut`,
`install_req_info`, `envelope_mut`, `take_body`), the request helpers gain
envelope forms (`authenticate_envelope_credentials`,
`object_store_from_envelope`, the table-catalog `*_from_envelope` helpers,
`KmsAdminAudit::from_envelope`, `AccountAuditContext::from_envelope`,
`CallerIdentity::resolve_envelope`), `read_compatible_admin_body` accepts
either body type, and the owned `Body` gains `store_all_limited` and
`Default`. `admin::operation::test_support::TestRequest` builds owned
requests for handler tests. Unused accessors carry a per-method dead-code
expectation that each migrating handler batch removes.
No handler behavior changes; the remaining migrations follow in batches.
* chore(admin): add the T1.12 admin handler codemod
The script rewrites an admin handler file from the s3s request and response
types to the owned boundary (imports, signatures, request field and helper
calls, body access, response construction, and test request literals) and
leaves a TODO comment at every site it cannot rewrite mechanically. It is
idempotent and is removed once T1.12 completes.
* refactor(admin): migrate replication and KMS handlers to owned requests (#8951)
Move the T1.12 batch 2 admin handlers (site_replication, kms_keys, event,
replication, diagnostics, on_demand_migration, audit, account, kms_rekey,
kms_key_lifecycle, object_data_cache, tls_debug, integrity) from the s3s
request/response types to the owned AdminRequest/AdminResponse boundary.
Handler logic, authorization order, error codes and response bytes are
unchanged; the request/response plumbing now goes through the envelope
accessors and helpers introduced by the owned boundary.
The shared response helpers (utils::json_response, utils::empty_response,
handlers::admin_json_response, target_descriptor::build_json_response,
profile::authorize_profile_request) and the site replication repair
executor still answer with the s3s types, so the migrated files carry
owned-boundary forms of those helpers and convert the repair response at
its call edge. The streaming console-log body and the two devnull drains
move from the s3s byte stream to the owned Body stream/frame API.
Drop the dead_code expectations for the owned accessors and envelope
helpers this batch starts using, and lower the s3s footprint baseline to
the measured 99 files.
Part of rustfs/backlog#2788 (rustfs/backlog#2734 T1.12).
* refactor(admin): add owned forms of the shared admin response helpers
Handler batches were copying `json_response`, `empty_response`,
`admin_json_response`, and `build_json_response` into each migrated file
because the shared helpers still return the s3s response. Provide owned
forms next to each legacy helper (`owned_json_response`,
`owned_empty_response`, `owned_admin_json_response`,
`owned_build_json_response`, `authorize_profile_envelope`) with parity
tests against the legacy shape, derive `Debug` for `AdminResponse` so
tests can use `expect_err`, and fix three codemod rules: strip owned
string conversions in test credentials, annotate non-literal URI parses,
and leave a rebound request parameter immutable.
* refactor(admin): use the shared owned response helpers in batch 2 (#8954)
Replace the per-file owned-boundary copies of json_response,
empty_response, admin_json_response, build_json_response and
authorize_profile_request in the batch 2 admin handlers with the shared
owned_* helpers and authorize_profile_envelope that the boundary branch
now provides, and restore expect_err in the handler tests now that
AdminResponse implements Debug; the `.err().expect(..)` stand-ins were
failing the base branch's clippy step (clippy::err_expect under
-D warnings). The site replication repair call edge
keeps its conversion because the repair executor still answers with the
s3s response type.
Drop the dead_code expectations on the shared owned helpers this batch
now uses in non-test code.
Part of rustfs/backlog#2788 (rustfs/backlog#2734 T1.12).
* fix(admin): preserve legacy body limit error responses
* refactor(admin): move policy, ILM and batch 3 handlers to owned requests (#8956)
Batch 3 of T1.12 (rustfs/backlog#2788, parent rustfs/backlog#2734) moves
14 admin handler files to AdminRequest/AdminResponse and the owned
Operation trait: ilm_transition, policies, idp_compat, config_admin,
plugins_instances, profile_admin, group, heal, object_zip_download,
kms_key_metadata, durability, inspect_archive, sts and
gateway_key_inventory. Handler logic, authorization order, error codes
and response bytes are unchanged; test request literals use
test_support::TestRequest.
The handlers use the shared owned helpers (owned_json_response,
owned_admin_json_response, owned_build_json_response,
authorize_profile_envelope). The dead_code expectations on the
AdminRequest accessors and RequestEnvelope::uri_mut they now exercise
are removed. The legacy target_descriptor::build_json_response keeps
only its equivalence test caller and carries the matching expectation
until it retires.
object_zip_download builds the s3s request the S3 gate still reads from
the owned request facts; heal authenticates through
authenticate_envelope_credentials; the trace, inspect-archive and zip
bodies use Body::from_stream. extract_plugin_instance_filters takes the
URI so extensions.rs keeps calling it, and the two idp_compat
delegations stay on the legacy trait until their delegates migrate.
The s3s footprint baseline drops from 99 to 90 files.
* test(admin): anchor the replication diff body contract on the call
The source-contract test looked for `let body = read_compatible_admin_body(`
on one line; rustfmt wrapped that binding after the handler moved to the
owned request boundary, so the marker was no longer found. Anchor on the
call itself so formatting cannot change what the contract checks.
* refactor(admin): migrate table catalog and batch 1 to owned requests (#8955)
* fix(admin): preserve catalog body limit response bytes
* refactor(admin): migrate identity and cluster handlers to owned requests (#8966)
* refactor(admin): migrate identity and cluster handlers to owned requests
Move the T1.12 batch 4 admin handlers (user, service_account, oidc, tier,
system, pools, quota, kms_dynamic, kms_backup, rebalance, module_switch,
bucket_meta, cluster_snapshot, realtime, bucket_recovery, profile) from the
s3s request/response types to the owned AdminRequest/AdminResponse
boundary. All 90 impl Operation blocks now take AdminRequest and return
AdminResponse; handler logic, authorization order, error codes and response
bytes are unchanged. The handlers use the shared owned helpers
(owned_json_response, owned_admin_json_response, authorize_profile_envelope,
the envelope-form runtime sources and audit constructors) instead of local
copies.
profile.rs drops the s3s forms of authorize_profile_request and
profile_not_implemented_response now that profile_admin.rs is on the
envelope gate too; profile_not_implemented_response answers with an
AdminResponse. The realtime metrics stream moves from the s3s ByteStream
adapter to Body::from_stream, and the rebalance idle-404 document still
renders through s3s::S3Error::to_http_response with its body converted at
the edge.
The idp_compat delegators ImportIamV2 and AddServiceAccountLdap move to the
owned trait as well: with their targets on both traits, the method-call
delegation became ambiguous between the owned impl and the router blanket
impl, so they now take the owned request directly.
KmsAdminAudit::from_request loses its last production caller with this batch
and is deleted; its comparison test now asserts the envelope facts directly.
Lower the s3s footprint baseline to the measured 68 files.
Part of rustfs/backlog#2788 (rustfs/backlog#2734 T1.12).
* refactor(admin): remove the legacy admin helpers batch 4 superseded
Batch 4 moved the last production callers of the s3s-typed admin helpers
to their owned forms, which left `admin_json_response`, `json_response`,
`authenticate_request`, `authorize_admin_request`, `object_store_from_req`
and `object_store_from_extensions` unused, along with the `Body`,
`S3Request` and `S3Response` re-exports of the admin `s3` facade. Remove
them. The owned forms keep their documentation; the behavior tests of the
removed `json_response` now exercise `owned_json_response`, and the
compat-prefix test asserts the owned response directly.
Two tests existed only to compare a removed legacy helper with its owned
twin (`owned_json_response_matches_the_legacy_shape`) or to cover the
removed legacy gate (`authorize_admin_request_without_credentials_is_rejected`);
they go with the helpers, and the owned equivalents stay. The rustfs
test-count row drops from 5957 to 5955 and the s3s footprint from 68 to 66
files.
* refactor(admin): drop the legacy target descriptor JSON response helper (#8968)
`target_descriptor::build_json_response` returned the s3s response and
had no production caller left once the event, audit and plugin instance
handlers moved to `owned_build_json_response`; it survived only for the
legacy/owned equivalence test. Remove it, give the owned helper its own
documentation, and make the test assert the owned response directly:
status, `application/json`, the echoed `x-request-id` when present and
only then, and the body bytes. The file no longer imports s3s, so the
footprint baseline drops from 66 to 65 files.
* test(ecstore): pin native history on cancellation under the cancellable queue
main moved the local write path onto the disk blocking queue, which skips an
operation once its waiter is dropped, so a borrowed write cancelled while it is
queued no longer creates its file. The integration-only native publication
test still expected the queued open to run. Assert the new execution model
instead (the open never runs) while keeping its safety claims: the Started
record is taken before queueing and the cancelled publication never regrants
native history.
* chore(gateway): raise test-count rows for tests merged from main
* test(s3select): use the owned JSON input types in merged planner tests
* fix(ecstore): scope the merged direct-writer test hook to test builds
* fix(ecstore): create parents before admitted truncate opens
Align the heal tests with removal-disabled preservation and local-only outcome
fields, and refresh the e2e-full selection for the synced tests.
* chore(gateway): update test-count baseline after main sync
Main #9241 removed the node pool-metadata readiness cache with 8 tests
and added 6, so the rustfs row drops from 6440 to 6438. The ecstore and
s3select rows rise for tests added by #9237, #9239 and #9241.
* feat(gateway-adapter): add IAM-backed CredentialProvider
Add the ring-2 crate rustfs-gateway-adapter with IamCredentials, the
gateway CredentialProvider answered by RustFS IAM (rustfs/backlog#2760,
credentials half; PolicySource is deferred).
- Pin rustfs-gateway at the existing gateway rev with
default-features = false so the server crate stays out of the build.
- IamKeyLookup is a narrow IAM trait implemented for IamSys so the
adapter is testable without an in-memory IAM backend.
- Root credentials are injected at construction and resolve before IAM
is ready.
- Mapping: keystone: prefix -> NotFound (deliberate change from the
legacy empty secret); IAM not ready or not initialized -> Unavailable;
unknown key or NoSuch{User,ServiceAccount,TempAccount,Account} ->
NotFound; other IAM errors -> Backend; invalid or "off" principals
resolve disabled; STS credentials carry a SessionBinding, and an STS
credential without an expiry is a Backend fault rather than a
never-ending session.
- No authorization in lookup; no Debug on secret-holding types; IAM
error text is dropped after classification.
* feat(scripts): add junit_diff.py for dual-stack JUnit comparison (#9275)
* feat(gateway-adapter): add IAM-backed PolicySource
Add IamPolicySource, the gateway PolicySource answered by RustFS IAM
(rustfs/backlog#2760, policy half). One snapshot per request carries a
RustfsPrincipal: principal kind, account, effective username, parent
user, credential groups, live group memberships, token claims and
is_owner.
- Anonymous callers get an empty snapshot without asking IAM.
- IAM not ready, not initialized or failing is PolicyError::unavailable,
never an empty snapshot.
- Unknown, disabled and session-kind-mismatched principals are refused
with constant labels; unreadable tokens are malformed.
- Claims are read from the stored token (root secret for STS, the
account's own secret for service accounts), and is_owner mirrors the
legacy has_root_access rule.
- RustfsPrincipal holds no secret or session token and has no Debug.
* ci(gateway): make the test-count ratchet a floor, not exact (#9278)
ci(gateway): make the test-count ratchet a floor instead of an exact baseline
Tests below a row still fail; tests above it, a member without a row and ignores below the row now pass with a note. Ignores above a row still fail. Adds contract tests wired into make script-tests.
* chore(gateway): record ignored microbenchmarks merged from main (#9290)
chore(gateway): record the ignored microbenchmarks merged from main
main added three throughput microbenchmarks that are ignored by design
(#9217): one each in rustfs-checksums, rustfs-rio and rustfs-utils. The
test-count ratchet treats a higher ignored count as a failure until the
row is raised, so raise those three rows to 1 and tighten the test rows
to the merged tree with --update.
* fix(gateway-adapter): share live root and refuse short secrets
Address the security review of the T2.1 adapters (rustfs/backlog#2760).
- Add RootCredentials (validated: root secret and STS signing key at
least MIN_SECRET_KEY_LEN bytes), the RootSource trait and the settable
RootCell. IamAdapters::new is now the only constructor of
IamCredentials and IamPolicySource, so both read one live root source
per request; no root source answers unavailable.
- A looked-up identity whose secret is shorter than 8 bytes is NotFound
in the credential provider and DISABLED_PRINCIPAL in the policy
source, whatever the lookup reports as valid.
- STS claims are read with the STS signing key from the root source
instead of assuming it equals the root secret.
- Cover the off-status-alone branch, the OIDC owner exclusion, service
accounts of root, the name live groups are read under, and another
session's token.
* ci(s3-tests): add reference parity lane against pinned 1.0.5 (#9288)
* fix(gateway-adapter): effective secret length and unsplittable pair
Address the confirmation review of the T2.1 adapters
(rustfs/backlog#2760).
- One rule for every secret: its length after trailing NUL bytes are
removed must be at least MIN_SECRET_KEY_LEN. HMAC zero-pads its key,
so trailing NULs add nothing; interior NULs count. Applied to the root
secret, the STS signing key and every looked-up credential.
- IamAdapters keeps both halves private and hands them to a builder
together through the consuming install(builder, make_authenticator);
compile-fail doctests show a mixed pair cannot be assembled.
- Any Fn() -> Option<Arc<RootCredentials>> is a RootSource, so the
binary reads its live root accessor directly; RootCell is now a
test-only helper.
* docs(gateway-adapter): state what install guarantees about the provider handle
---------
Co-authored-by: rustfs-cla-assistant[bot] <273121464+rustfs-cla-assistant[bot]@users.noreply.github.com>
A multipart index marker is recorded before its staging root is published, and a listing prunes markers whose root is gone. The window in between was covered by a five-minute age check, which cannot prove that the publisher's filesystem work has ended: a publisher stalled past it lost its marker, and the upload it then published stayed out of ListMultipartUploads on that drive until restart.
The publisher now holds an in-memory publication pin from before it records the marker until the filesystem work that publishes the root has returned. The pin travels with that work (the blocking closure or the lease owner chain), so it also covers work that outlives a cancelled caller. Pruning reads the pin first and the staging root second, under the marker lease. The age check and the per-write marker touch are removed.
Lands gateway/integration on main: the guard rails and the domain
decoupling for replacing s3s with rustfs/gateway (rustfs/backlog#2734,
Phase 0 and Phase 1). The S3 edge is still s3s and no wire behaviour
change is intended.
- Guard rails: lower-only s3s footprint baseline and edge allow-list,
per-crate test-count floor, 39 byte-level goldens for the persisted
bucket-config XML families, a two-server differential harness over
all 73 S3 operations, a JUnit differ, and a reference-parity lane
against the sha256-pinned 1.0.5 binary.
- Owned types: rustfs-s3-types (S3Error, S3ErrorCode, Body),
RequestEnvelope, AuthzFacts.
- Engine crates off s3s: ecstore, lifecycle, replication, scanner,
protocols, s3select, s3-client, rpc.
- Admin: error types moved; 300 of 314 Operation impls take the owned
AdminRequest.
- Server: the s3s-compat tower layers grouped under
server/legacy_compat; RUSTFS_S3_STACK=legacy|gateway selector, where
gateway refuses to start until the gateway edge exists.
Fixes made on the branch that change ecstore and heal behaviour are
listed in the pull request description. The per-commit history stays
on gateway/integration.
* fix(ecstore): let missing-key deletes complete on UUID-named folders
Deleting a key that does not exist probes the key's directory for
interrupted delete transactions. The probe was bounded at 4096 entries
per drive and charged every entry inside a UUID-named child, so a key
naming a folder of roughly 1,400 or more UUID-named child keys answered
503 SlowDown on every retry.
Inside a UUID-named child only journal files and data-directory markers
can be evidence, so every other name (a child key's own xl.meta, data
directories or children) is now skipped unread at 1/64 like plain
children of the folder. The per-call allowance of the bounded protocol
rises to 262,144 entries per drive; the resumable background step keeps
4096 and the request budget keeps its sizing, so the per-request total
is unchanged. The probe runs on the durable-mutation lane instead of
the one-worker foreground lane, and stops incomplete when its caller
goes away or after a third of the drive timeout (at most 10 s), so a
slow walk answers SlowDown instead of timing out the drive.
* test(ecstore): keep discovery fixtures meaningful under 1/64 charging
Part files inside a data directory are inert to object-scoped discovery
now, so reclaim_orphan_data_dirs_drains_empty_discovery_pages fills its
first step with the referenced data directories of many versions, which
still cost a full entry each. The charging test drives the walk directly
instead of through the wall-clock-limited disk entry point, so a loaded
machine cannot turn it into an incomplete probe.
chore(gateway): record the ignored microbenchmarks merged from main
main added three throughput microbenchmarks that are ignored by design
(#9217): one each in rustfs-checksums, rustfs-rio and rustfs-utils. The
test-count ratchet treats a higher ignored count as a failure until the
row is raised, so raise those three rows to 1 and tighten the test rows
to the merged tree with --update.
ci(gateway): make the test-count ratchet a floor instead of an exact baseline
Tests below a row still fail; tests above it, a member without a row and ignores below the row now pass with a note. Ignores above a row still fail. Adds contract tests wired into make script-tests.
RUSTFS_E2E_REFERENCE_BINARY makes side A of the dual-stack differential a released rustfs binary, started without RUSTFS_S3_STACK. A missing, non-file or non-executable path fails the run instead of falling back to the same-binary mode. The report records which side is the reference and each side's --version output.
Add unaligned-boundary SIMD vs scalar equivalence unit tests and on-demand (#[ignore]) utility-level throughput microbenchmarks across rustfs-utils, rustfs-rio, and rustfs-checksums:
- crates/utils/src/hash.rs: verify HighwayHasher SIMD (AVX2/SSE4.1/NEON) against highway::PortableHash scalar reference and contiguous hash_encode vs odd-chunked hash_encode_slices across unaligned tail boundaries; provide on-demand throughput microbenchmark.
- crates/rio/src/checksum.rs: verify SIMD-accelerated CRC32, CRC32C, and CRC64_NVME against bitwise scalar reference implementations across vector tail boundaries and measure utility-level throughput across all 10 BASE_CHECKSUM_TYPES.
- crates/checksums/src/lib.rs: verify streaming chunked update + finalize + headers() against one-shot digests across all 9 ChecksumAlgorithm variants for unaligned chunk and payload sizes, plus on-demand throughput microbenchmark.
The single-process usage overlay clears usage_snapshot_complete when it
holds counts it cannot reconcile with the persisted snapshot, for
example writes into a bucket that has no verified baseline. The admin
datausageinfo path then judged the overlaid document as structurally
incomplete and replaced it with an empty one, hiding every bucket the
scanner had measured. Because each write restarts that bucket's scanner
fence, a continuously written bucket kept the whole response empty
indefinitely.
Decide whether the persisted snapshot is usable before the overlay is
applied, and report overlay-induced incompleteness as a non-converged
snapshot instead, as is already done for buckets the scan never reached
(rustfs#5806).
Use the larger GitHub-hosted macOS runner for the macos-aarch64
release target. The target id, triple, and other platforms stay the same.
Co-authored-by: hector <majinghe@users.noreply.github.com>
* fix(ecstore): allow bucket creation after failed decommission
* fix(scanner): preserve backlog membership across pool reindexing
* test(scanner): cover partial publication after pool reindexing
Main #9241 removed the node pool-metadata readiness cache with 8 tests
and added 6, so the rustfs row drops from 6440 to 6438. The ecstore and
s3select rows rise for tests added by #9237, #9239 and #9241.
When a fused recoverable delete is interrupted after publishing the transaction directory and before updating xl.meta, xl.meta and <transaction>/xl.meta.bkp remain hardlinks to the same inode. Under POSIX rename semantics, renaming xl.meta.bkp over xl.meta during rollback succeeds as a no-op without unlinking xl.meta.bkp, which leaves xl.meta.bkp in the transaction directory after journal removal and strands the transaction in DeleteRecoveryState::LegacyAmbiguous.
Unlink xl.meta.bkp after syncing the restored object directory and before syncing the rollback directory and removing journal markers, and extend delete_recovery_fused_prepare_checkpoint_reopens_before_head_publication to preserve the hardlink and verify rollback completion and retry.
* fix(ecstore): share immutable pool metadata authority reads
* fix(ecstore): read pool metadata authority without the save gate
Health inferred pool metadata write authority from acquiring the local
authority gate within 100 ms. Sharing the gate among read-only
admissions still left held writers, and writers queued behind a reader
parked on the pool.bin fence, turning into cluster-write 503s.
Each writer now publishes the authority it leaves before releasing the
gate, and health reads that published state without acquiring it.
Transaction latches stay shared with the published copy; recovery
clears the latch and its failure under one lock so inspection never
observes a half-cleared state. The inspection-timeout path that can no
longer occur (check_timed_out, the check_timeout admin state, the
pool_metadata_check_timeout reason, the node timeout cache and its
metric) is removed.
* fix(rpc): index nonce expiry by signed validity
* fix(rpc): size replay cache from allocated storage layout
* fix(rpc): make replay admission atomic and release expired tables
* fix(rpc): preserve replay admission refusal at the S3 boundary
Port main's changes into integration's relocated code: the XML compatibility
body bound and presigned-guard tests into legacy_compat, the policy-condition
fixes (#9150, #9159, #9163) into auth facts and the owned admin request API,
and record native publications on main's new fused delete-recovery paths.
* ci: align full e2e selection with retired catalog test
* ci: verify pinned tools through GitHub release asset API
* ci: fetch pinned typos through the release asset API
* ci: download pinned actionlint through the release asset API
* ci: ensure GitHub CLI exists before pinned mc download
* ci: budget full quick checks after tool installation
* feat(observability): record unary read RPC durations
Record complete server-side durations for ReadMultiple and BatchReadVersion, including error and cancellation paths.\n\nCo-Authored-By: heihutu <heihutu@gmail.com>\nCo-Authored-By: zhi22915 <qiuzgang@gmail.com>
* test(observability): cover unary duration handlers
Exercise ReadMultiple and BatchReadVersion missing-disk paths and assert operation duration histograms.\n\nCo-Authored-By: heihutu <heihutu@gmail.com>\nCo-Authored-By: zhi22915 <qiuzgang@gmail.com>
* fix(observability): cache unary read RPC duration handles
Reuse cached duration histograms for ReadMultiple and BatchReadVersion once
the stable server label is available. Preserve the existing ReadVersion
handle and the fallback for other operations and backends.
Add an integration regression for handle reuse, startup fallback, labels,
and recorded values using the production cache path.
Co-Authored-By: heihutu <heihutu@gmail.com>
Co-Authored-By: zhi22915 <qiuzgang@gmail.com>
* fix(observability): separate server duration from client RTT
Record unary ReadMultiple and BatchReadVersion handler lifetimes in a
separate server duration histogram while preserving existing client RTT
series and allocation-free cached handles for both measurements.
Cover client/server sample separation, cache reuse, startup and backend
fallbacks, and missing-disk handlers. Document the new server metric.
Co-Authored-By: heihutu <heihutu@gmail.com>
Co-Authored-By: zhi22915 <qiuzgang@gmail.com>
---------
Co-authored-by: heihutu <heihutu@gmail.com>
Co-authored-by: zhi22915 <qiuzgang@gmail.com>
Co-authored-by: Chris An <anzhengchao@gmail.com>
* fix(table-catalog): reject storage credentials for catalog access
* fix(table-catalog): avoid growing the s3s error macro footprint
Use the existing S3Error constructor for the storage-credential rejection
while preserving its AccessDenied code and message. Keep the merged PR
within the current s3s footprint baseline without relaxing the guard.
Co-Authored-By: heihutu <heihutu@gmail.com>
Co-Authored-By: zhi22915 <qiuzgang@gmail.com>
---------
Co-authored-by: Hauser Zhi <housemecn@gmail.com>
Co-authored-by: heihutu <heihutu@gmail.com>
Co-authored-by: zhi22915 <qiuzgang@gmail.com>
Co-authored-by: Chris An <anzhengchao@gmail.com>
Use a direct response future for originless requests and forward through
the polled-ready service. Nominate every retained CORS header input and
transfer evaluated headers without rebuilding the map.
Co-authored-by: heihutu <heihutu@gmail.com>
Co-authored-by: zhi22915 <qiuzgang@gmail.com>
* fix(ecstore): prevent cached usage snapshot rollback
Order refreshed snapshots by their authoritative lineage before replacing cached usage. Share the rule with early bucket refreshes while retaining quota baselines, invalidation fencing, legitimate deletions, and clock recovery.
Co-Authored-By: heihutu <heihutu@gmail.com>
Co-Authored-By: zhi22915 <qiuzgang@gmail.com>
* fix(server): skip trace propagation for disabled RPC spans
Avoid extracting and assigning a remote parent when RPC tracing is disabled after authentication and admission. Cover disabled logging and the production HTTP span propagation path.
Co-Authored-By: heihutu <heihutu@gmail.com>
Co-Authored-By: zhi22915 <qiuzgang@gmail.com>
* fix(ecstore): preserve usage ordering across refresh recovery
Retain the last good snapshot independently of negative-cache status. Order observations within the same authoritative baseline by their own coordinated epoch and cycle before wall time, including clock rollback and recovery. Capture persisted baseline identity without cloning its bucket maps.
Co-Authored-By: heihutu <heihutu@gmail.com>
Co-Authored-By: zhi22915 <qiuzgang@gmail.com>
* refactor(server): centralize request trace propagation
Let the HTTP span factory own parent extraction and assignment. Use actual span filtering, preserve span-only subscribers, and remove repeated tracing work from RPC authentication.
Co-Authored-By: heihutu <heihutu@gmail.com>
Co-Authored-By: zhi22915 <qiuzgang@gmail.com>
---------
Co-authored-by: heihutu <heihutu@gmail.com>
Co-authored-by: zhi22915 <qiuzgang@gmail.com>
* fix(s3): retain protected multipart ETags in object listings
Record encrypted part ETag provenance, aggregate it at read quorum, and attest completed multipart ETags for keyless listings. Preserve legacy fingerprint protection and test versioned SSE-KMS and SSE-C upload, listing, and replay paths.
Co-Authored-By: heihutu <heihutu@gmail.com>
Co-Authored-By: zhi22915 <qiuzgang@gmail.com>
* perf(s3): stream multipart ETag aggregation
Restore the preserved ETag fast path when selected parts lack protected provenance. Stream native part digests through a fixed stack buffer, preserve legacy hex-or-raw semantics, and strengthen exact-output and override regression coverage.
Co-Authored-By: heihutu <heihutu@gmail.com>
Co-Authored-By: zhi22915 <qiuzgang@gmail.com>
* fix(s3): preserve encrypted ETag proof during data movement
Capture quorum-backed source proof before multipart completion metadata updates, require exact preserved ETag and version identity, and retain proof only when publication keeps the source version. Cover real two-pool rebalance, null versions, legacy sources, and injected or rewritten identities.
Co-Authored-By: heihutu <heihutu@gmail.com>
Co-Authored-By: zhi22915 <qiuzgang@gmail.com>
* fix(s3): retain encrypted multipart ETag proof on restore
---------
Co-authored-by: heihutu <heihutu@gmail.com>
Co-authored-by: zhi22915 <qiuzgang@gmail.com>
Co-authored-by: overtrue <anzhengchao@gmail.com>
* test(storage): reproduce HTTPS rejoin readiness in Docker
Compare cluster-only and node-ready/write gates on immutable native binaries. Preserve first-failure and latest-acknowledged-content evidence across recovery, capture lock diagnostics before restart, and limit cleanup to verified owned resources.
Co-Authored-By: heihutu <heihutu@gmail.com>
Co-Authored-By: zhi22915 <qiuzgang@gmail.com>
* fix(testing): unify Docker repro ownership and controls
Use typed resource absence and one verified container inventory for staging, partial setup and retained nodes. Reject inherited experiment overrides before creating resources and remove duplicated state.
Co-Authored-By: heihutu <heihutu@gmail.com>
Co-Authored-By: zhi22915 <qiuzgang@gmail.com>
* fix(testing): guard staging volumes and interrupted capture
Verify volume ownership before privileged staging and disable implicit image-content copying. Preserve failed evidence and continue owned finalization when capture or cleanup is interrupted.
Co-Authored-By: heihutu <heihutu@gmail.com>
Co-Authored-By: zhi22915 <qiuzgang@gmail.com>
* fix(testing): capture live rejoin logs and verify trace coverage
Read live tmpfs logs inside the verified container and use one explicit stdout sink for diagnostics. Record bounded capture provenance and fail diagnostic coverage when acquisition, guard-drop or physical-release records are missing, without masking HTTP failures.
Co-Authored-By: heihutu <heihutu@gmail.com>
Co-Authored-By: zhi22915 <qiuzgang@gmail.com>
* fix(testing): reject inherited aliases and finalize on SIGTERM
Reject inherited MINIO compatibility settings before Docker resource creation. Route SIGTERM through interrupt-aware finalization and restore the previous handler after each run. Cover real process signals and failure preservation.
Co-Authored-By: heihutu <heihutu@gmail.com>
Co-Authored-By: zhi22915 <qiuzgang@gmail.com>
* fix(testing): normalize legacy controls and termination
Reject inherited MINIO compatibility aliases in neutral runtime images. Bridge run-mode SIGTERM into the existing interruption finalization and restore the caller signal handler on return or exception.
Co-Authored-By: heihutu <heihutu@gmail.com>
Co-Authored-By: zhi22915 <qiuzgang@gmail.com>
---------
Co-authored-by: heihutu <heihutu@gmail.com>
Co-authored-by: zhi22915 <qiuzgang@gmail.com>
Apply cache variation at the shared response boundary, preserve existing
Vary selectors, and reuse parsed global policy and compact CORS inputs.
Co-authored-by: heihutu <heihutu@gmail.com>
Co-authored-by: zhi22915 <qiuzgang@gmail.com>
* fix(scanner): bound wide bucket-root fanout memory (#8961)
Root prefixes were excluded from ordinary compaction and force_compact only
flattened the root above a hardcoded 250k child ceiling, so buckets with
tens of thousands of top-level prefixes kept one cache entry each after
#8971. Compact wide roots outside checkpointed forward sweeps, and honor
force_compact's limit for root fanout.
* fix(scanner): keep root fanout on interrupted force_compact
Interrupted sweeps seal a resume frontier after force_compact. Applying
the ordinary 10k root limit there flattened frontier-named children and
made the next cycle replay completed prefixes. Use the emergency FORCE
ceiling on the interrupted path; successful completions still honor the
ordinary limit. Drop clippy assertions_on_constants in the new tests.
Signed-off-by: Chris An <anzhengchao@gmail.com>
* test(scanner): seal reloaded frontier after interrupted compaction
Const assertions keep the wide-root preconditions without tripping
clippy::assertions_on_constants. The interrupted and finished sweep
paths share one limit helper, and a cache wider than the ordinary
10k limit still seals its frontier child after reload.
Co-authored-by: Chris An <overtrue@users.noreply.github.com>
* fix(scanner): isolate unreadable children during root compaction
* fix(scanner): isolate compacted namespace child errors
* fix(scanner): preserve compacted scan accounting and coverage
Commit isolated namespace child usage only after successful traversal, while retaining completed accounting on cancellation and resource deferral. Synchronize reduced root totals before the cancellation-prone flush boundary.
Treat skipped namespaces as incomplete coverage and invalidate legacy sampling baselines until a fresh verification pass succeeds. Keep provisional usage out of progress updates and retain the upstream symlink regression.
Co-Authored-By: heihutu <heihutu@gmail.com>
Co-Authored-By: zhi22915 <qiuzgang@gmail.com>
* fix(scanner): preserve legacy retry coverage
* test(scanner): regress cancel overwrite after wide-root reduce
houseme's 714f590 already refreshes the root accumulator from the
compacted cache entry before flush_pacing. Add a unit regression that
simulates the interrupt replace_hashed path so a future regression
cannot reintroduce dangling child links over the aggregate.
Signed-off-by: overtrue <anzhengchao@gmail.com>
---------
Signed-off-by: Chris An <anzhengchao@gmail.com>
Signed-off-by: overtrue <anzhengchao@gmail.com>
Co-authored-by: Cursor Agent <cursoragent@cursor.com>
Co-authored-by: Chris An <overtrue@users.noreply.github.com>
Co-authored-by: Hauser Zhi <housemecn@gmail.com>
Co-authored-by: heihutu <heihutu@gmail.com>
Co-authored-by: zhi22915 <qiuzgang@gmail.com>
* fix(container): explain log directory initialization failures
Report failed log directory creation with ownership guidance and a Compose quote hint while preserving configured values. Add a corrected example and real entrypoint regression coverage.
Co-Authored-By: heihutu <heihutu@gmail.com>
Co-Authored-By: zhi22915 <qiuzgang@gmail.com>
* refactor(container): unify startup directory preparation
Share directory creation, diagnostics, and best-effort ownership across data and log paths. Skip delimiter and regex subprocesses for ordinary volume paths without introducing a second parser. Expand real-entrypoint coverage for ownership, failures, delimiters, distributed ranges, and setup command counts.
Co-Authored-By: heihutu <heihutu@gmail.com>
Co-Authored-By: zhi22915 <qiuzgang@gmail.com>
---------
Co-authored-by: heihutu <heihutu@gmail.com>
Co-authored-by: zhi22915 <qiuzgang@gmail.com>
Record end-to-end durations for ReadAll, WriteAll, ReadMultiple, and BatchReadVersion alongside existing request, error, and payload metrics. This makes direct unary payload ABBA results observable through the internode operation duration family.
Co-authored-by: zhi22915 <qiuzgang@gmail.com>
Release the per-peer prewarm deduplication marker when the best-effort dial fails, allowing a later RemoteDisk construction to retry the connection warmup without changing lazy connect fallback behavior.
Co-authored-by: zhi22915 <qiuzgang@gmail.com>
SFTP subdirectory mkdir and rmdir sent the on-disk __XLDIR__ marker
through the public S3 API. After the suffix was reserved, those calls
were rejected as invalid client keys, so ordinary directory operations
failed while bucket mkdir still succeeded.
Send the logical trailing-slash key and let storage encode the marker.
Co-authored-by: Cursor Agent <cursoragent@cursor.com>
Co-authored-by: Chris An <overtrue@users.noreply.github.com>
* test(server): cover recovered-node admission boundaries
Document backend readiness and write-availability checks during rejoin.
Extend the existing regression through the production middleware order
without changing runtime admission or cluster-quorum semantics.
Co-Authored-By: heihutu <heihutu@gmail.com>
Co-Authored-By: zhi22915 <qiuzgang@gmail.com>
* fix(server): bound rejoin waits and preserve recovery ownership
Enforce startup and unreachable-grace bounds while consuming peer replies.
Release admission when every set has its required peer majority and cancel
surplus read-only probes without carrying votes between rounds.
Bound local readiness observations and polling sleeps. Retain initialized
IAM across preparation failures, install the context once, and retry only
publication after readiness timeouts. Share preparation with inline startup.
Co-Authored-By: heihutu <heihutu@gmail.com>
Co-Authored-By: zhi22915 <qiuzgang@gmail.com>
* perf(server): make peer admission folding incremental
Compile peer-to-set memberships once and retain compact per-edge votes.
Consume each reply once, discard its payload, and update only affected
set counters instead of rescanning all sets and retained responses.
Recheck the captured deadline after polling and folding a reply so late
work cannot admit the node or clear an expired unreachable grace period.
Co-Authored-By: heihutu <heihutu@gmail.com>
Co-Authored-By: zhi22915 <qiuzgang@gmail.com>
---------
Co-authored-by: heihutu <heihutu@gmail.com>
Co-authored-by: zhi22915 <qiuzgang@gmail.com>
* fix(ecstore,heal,select): bound storage-layer work driven by client input
Scan findings in the erasure store, heal, file metadata and S3 Select
layers, grouped because they share the disk walk, the multipart staging
area, delete recovery and the select snapshot lease:
- Multipart: upload ids are canonicalized before they name a staging path
or a lock key (backlog#3027); trailing-slash keys are encoded like every
other handler (backlog#3055); unlisted parts are removed at completion
(backlog#3106); per-bucket uploads are found through a signed per-disk
index instead of a scan of the shared staging area (backlog#3121);
encrypted part ETags carry a per-upload secret so a write-only user
cannot confirm plaintext guesses (backlog#3044).
- Delete and recovery: recovery walks are bounded per request
(backlog#3038); a delete marker on a key whose xl.meta path is a
directory, and sidecar names used as child keys, no longer wedge the
journal (backlog#3035, backlog#3105); over-quota PUTs no longer list the
bucket under the ledger lock (backlog#3128).
- Listing: scan_dir sends metadata without inline data (backlog#3052),
counts delete markers against max-keys (backlog#3108), and rejects a
marker that carries the cursor tag (backlog#3059).
- Heal: bucket metadata and heal records live in separate namespaces so
DeleteBucket cannot erase a drive rebuild (backlog#3039), a deleted
bucket is skipped instead of failing the rebuild (backlog#3040), and
directory objects are encoded on readback (backlog#3061).
- Restore and tiering: RestoreObject preserves tags and the version's
modification time (backlog#3073, backlog#3074); Snowball transport
headers are not forwarded to the tier (backlog#3090); an epoch
modification time round-trips through xl.meta (backlog#3063).
- Locks: PutObject takes its bucket locks after the body is staged
(backlog#3131); SelectObjectContent releases its namespace locks once
the snapshot is taken (backlog#3132).
- RPC and peers: the v3 signature check consumes the v2 nonce
(backlog#3129); a bucket-metadata reload timeout no longer marks the
peer offline (backlog#3124); replication target requests carry an
operation deadline (backlog#3102); the scanner resumes its interrupted
wait after a lifecycle write (backlog#3118).
- S3 Select: bounded schema inference, column cap and budgeted CSV
batches, Parquet dictionary bounds and pre-planner projection checks
(backlog#3075, backlog#3076, backlog#3077, backlog#3078).
* test(e2e): seed legacy duplicate bucket tags offline for the OPA contract
* fix(s3): authorize what handlers store and bound request-driven work
Scan findings in the S3 request layer (access hooks, object use cases,
HTTP server layers), grouped because they share the authorization
condition builder, the PutObject admission path and the response
layers. Builds on the storage-layer batch for the bounded recovery and
multipart APIs it calls.
Authorization and policy conditions:
- copy source is percent-decoded and version-canonical before the
s3:x-amz-copy-source condition (backlog#3135); CopyObject authorizes
the tag set it copies (backlog#3034); HeadObject with the replication
check header still authorizes s3:GetObject (backlog#3047); the
x-rustfs-source-version-id header feeds s3:versionid (backlog#3072);
recursive force-delete with a REPLICA header authorizes every
descendant as a delete (backlog#3070); include-deleted listings require
s3:ListBucketVersions (backlog#3099); the metadata listing route keeps
table isolation and uses the same prefix it authorizes (backlog#3053,
backlog#3054); If-Match probing needs read permission (backlog#3097);
Object Lambda checks s3:GetObjectVersion (backlog#3058); replication
proxy fallback keeps ExistingObjectTag denies (backlog#3089); HeadObject
returns lock and tag headers only with permission (backlog#3113);
IPv4 clients on the dual-stack listener match IPv4 SourceIp conditions
(backlog#3049); bucket policies are parsed once and cached
(backlog#3032); keystone: access keys no longer verify with an empty
secret (backlog#3043).
Admission and resource bounds:
- GetObject releases disk-read permits when the client stops reading
(backlog#3023); PutObject takes buffer and foreground admission only
after body bytes arrive and bounds the wait (backlog#3031,
backlog#3084); xl.meta size is bounded on the write path
(backlog#3100); XML compat layers stream instead of buffering whole
bodies (backlog#3094); tier GETs stream the remote body (backlog#3046); on-demand migration pulls release permits on
unread GETs (backlog#3110); SSE-KMS key policy covers UploadPart and
completion (backlog#3130); MinIO admin decryption runs Argon2id off the
async worker (backlog#3024).
Input validation and leaks:
- replication headers are validated (source ETag shape, original size)
(backlog#3092, backlog#3093); control characters in keys and lifecycle
rule IDs are rejected (backlog#3101, backlog#3117); Snowball member
mtime needs s3:ReplicateObject (backlog#3083); DeleteObjects checks
Content-MD5 and admin/STS reject SigV2 replays with altered query or
body (backlog#3037, backlog#3026); request logs redact presigned
queries and handler spans skip credentials (backlog#3068,
backlog#3107); SSE-C listings and completions withhold plaintext
metadata and checksums (backlog#3120, backlog#3050); CORS resolves the
bucket like s3s does (backlog#3104); PutBucketTagging rejects duplicate
keys (backlog#3126); notifications fire after a client disconnect
(backlog#3127); ListenNotification streams re-check access
(backlog#3119).
* fix(api): drop imports left unused by the base merge
* test(s3): move the KMS multipart denial test to the app layer
The test drives app-layer multipart use cases, so it belongs with them;
living in the SSE module it imported app code from the infra layer and
failed the layer dependency guard. The SSE test helpers it borrows are
exposed through the storage and app facades.
* fix(s3): keep the SSE-C key fingerprint on keyless multipart completion
The backlog#3050 fix withheld both the plaintext checksums and the
customer-key MD5 header from a CompleteMultipartUpload that did not
present the key. The checksums are a digest of the plaintext and stay
hidden; the key MD5 identifies the key only, and the S3 conformance suite
(test_copy_part_enc) expects it on a keyless completion, so it is echoed
again.
* fix(tier): stream GET bodies and document replication budgets
* fix(ecstore,s3): keep the multipart index and cold fill off disk lifetimes
Two lifecycle regressions surfaced once the storage and request batches
ran together against main:
- The multipart index owner cloned the disk's pinned publication root and
bootstrapped on a task of its own. That kept the mount alive past the
last LocalDisk owner (the disk lifecycle allows exactly three holders of
the root FD) and left a stray task behind, which the local-disk and
read-repair tests detect. The owner now keeps only the root path and
reopens it for identity checks, and the bootstrap scan rides the disk's
existing cleanup task, where stopping the disk abandons it and a
reconnect resumes it.
- The disk-read admission wrapper defers releasing its permit until an
in-flight read completes, so a cancelled client cannot widen disk
concurrency. The cold-fill producer, however, drops a read it has
already classified as stalled when its deadline fires, and that permit
stayed pinned for the lifetime of the stuck read. The producer now
abandons the admission explicitly on timeout.
* test(ecstore): pass the index owner to the Linux cleanup-loop test
* fix(s3): release cold-fill admission after watchdog handoff
* fix(s3,ecstore): close the round-1 adversarial review findings
Adversarial review of the combined request- and storage-layer batch found
fixes that fell short of their issues, two behaviour regressions for
legitimate clients and several unrequested costs:
- backlog#3038: a bulk DeleteObjects reserved the whole per-disk
inspection allowance up front for every key in flight and exhausted the
16 Mi request budget on any set wider than eight drives, failing healthy
keys with SlowDown. Calls are now admitted while budget remains and
charged for the entries they actually visited. Object-scoped discovery
charges plain child keys at one entry per 64 names instead of one each,
so deleting a missing key that names a folder prefix completes instead
of answering 503 forever while a hostile prefix stays bounded. A peer
still on the previous binary is reported
as an unsupported participant, the repository's rolling-upgrade
contract, instead of as a budget verdict.
- backlog#3037/#3026: a SigV2 configuration PUT without Content-MD5 was
still replayable with another body; it is refused with MissingContentMD5
while SigV4 keeps the optional-MD5 contract.
- backlog#3072: DeleteObject now resolves x-rustfs-source-version-id before
authorization like PUT and COPY, so a replica delete cannot remove a
version the policy never saw.
- backlog#3121: the multipart index no longer fsyncs a marker and four
directories per multipart metadata write (it is rebuilt on every start),
and ListMultipartUploads discovers roots concurrently.
- backlog#3077: Parquet SelectObjectContent decodes budget-sized batches
instead of one row per batch.
- backlog#3031: a full upload buffer pool hands out an unpooled buffer
instead of refusing the upload; the progress floor already bounds stalls.
- backlog#3023: the GET disk-read idle cut-off is configurable
(RUSTFS_GET_DISK_READ_IDLE_TIMEOUT_SECS, default 120 s) so a paused
client is not truncated after 30 s.
- backlog#3075: JSON Lines schema inference keeps the sample collected at
the 4 MiB bound instead of failing the query.
- backlog#3024: compat admin decryption queues briefly for a slot instead
of refusing the third concurrent request.
- backlog#3113: HEAD skips its permission probes for the bucket owner.
- backlog#3117: control characters in a lifecycle rule ID are rejected on
PutBucketLifecycle as well.
The lock-crate publication FIFO serves backlog#3111 and moves to the
table-catalog PR. Also fixed: a lossy admin error message, an unused
s3-client function, two unused test-util features, a duplicated key
validation, the s3s footprint baseline, and a stale log event name in the
reverse-proxy runbook. The ODM multipart write-back test runs on the large
test stack like its sibling.
* fix(table-catalog): bound maintenance work and isolate table namespaces (#9174)
* fix(table-catalog): bound maintenance work and isolate table namespaces
Scan findings in the Iceberg table catalog:
- Metadata maintenance deleted every unreferenced *.metadata.json in a
table's directory without checking whether a renamed table in another
namespace still owned those files (backlog#3060); it now resolves the
owning table through the directory index before deleting.
- Manifest lists and manifests were decoded with the embedded Avro
writer schema and no recursion depth limit; a self-referencing record
schema overflowed the stack (backlog#3080). Schemas are validated for
depth before decoding.
- Compaction decoded every input Parquet file into Arrow batches held in
memory; a decompression bomb OOM-killed the server (backlog#3085). Row
groups are streamed through a byte budget and oversized inputs are
refused.
- An S3 request for a key outside an indexed table prefix listed the
bucket's whole namespaces/ catalog prefix and failed once it held more
than 4096 objects (backlog#3086); a per-bucket directory index answers
index misses without the listing.
- The maintenance planner read whole orphan files to obtain their
modification time (backlog#3087); it reads metadata only.
- A client disconnect between persisting active_rename_id and clearing it
left the bucket's catalog entry locked at 503 (backlog#3088); the rename
runs detached from the request future and clears the flag on every exit.
- Table commits held the bucket publication fence until publication,
starving every object write (backlog#3111); commits now hold it under a
deadline and object writes are admitted between commits.
The directory index is managed through three new admin routes under
{warehouse}/catalog/directory, registered in the admin route policy.
The durable-strong store is left as is (its snapshot partitioning for
backlog#3109 and the strong-path deadline are not ported): that backing
is being retired separately, so the new tests cover the object backing.
* fix(table-catalog): close the review gaps in the scan-fix batch
Adversarial review of the table-catalog batch found the fixes narrower
than the issues they close and two cost regressions:
- backlog#3085: the compaction footer check bounded nothing a page header
could not override; parquet sizes its page buffers from the header.
Walk every column chunk's page headers (Thrift compact, no decompression)
and hold each page and the chunk total to the budget before decoding.
The stored-input budget is per rewrite group, so a multi-group job no
longer fails after writing earlier outputs.
- backlog#3086: the default index-miss scan still counted maintenance
reports toward its 4096-object limit; only table entries count now, and
the listing itself is bounded to 64 pages. The directory index remains
an optional accelerator.
- backlog#3111: a create wrote its metadata file under the fence as the
first durable mutation, which disarmed the deadline before the unbounded
registration inventory. The write precedes publication now. The deadline
gets its own knob (RUSTFS_TABLE_CATALOG_PUBLICATION_PREPARATION_TIMEOUT_SECS,
default 30 s, never below the lock-acquire timeout) instead of reusing
the 5 s lock timeout, and the ownership inventory reads entries
concurrently.
- Directory candidates are written only while a directory exists; the
backfill covers earlier entries. A directory on the migration-fence
path is reported as a conflict instead of a parse error.
- Dead test hooks, a stale comment and the support-matrix text that did
not match the code are removed or corrected.
* fix(select): preserve complete samples and Parquet batch limits
* fix(s3,ecstore): close the round-2 adversarial review findings
DeleteObject resolved x-rustfs-source-version-id for authorization on
every request while storage consumes it only for a REPLICA request, so a
non-replica caller with s3:DeleteObjectVersion could trigger a plain
delete of the current version. The hook now resolves the header under
the same predicate as storage, and the regression test covers the
non-replica case.
HEAD no longer skips its permission probes for the bucket owner; an
explicit bucket-policy Deny on the metadata actions binds the owner too.
Multipart index pages verify each root against the staging area and
prune markers whose staging root is gone, under the marker's own lease
with a re-check, so the per-drive index stays proportional to live
uploads instead of every key ever multipart-initiated on the drive.
SigV2 configuration PUTs that declare an empty body need no Content-MD5:
there is no body to replay. The minimum-progress floor stays independent
of RUSTFS_HTTP_REQUEST_BODY_READ_TIMEOUT; the knob's documentation now
says so instead of claiming 0 disables it. The quota bootstrap window
(503 until the background scan publishes the ledger) is kept, because
ledger admission must not come from scanner reporting; the compatibility
matrix now documents it.
The S3 compatibility matrix gains a section for the client-visible
validation bounds, and the drive-timeout tuning guide tabulates
RUSTFS_GET_DISK_READ_IDLE_TIMEOUT_SECS.
* fix(lock): queue table-bucket publication writers fairly
Table commits take the bucket publication fence as a write lock; under a
stream of commits the lock's try-and-retry acquisition let a new commit
jump ahead of object writers that had been waiting longer, which is part
of what starved them (backlog#3111). Requests for the publication lock
path now take a FIFO ticket per shard: a waiter is admitted only when it
is at the head of the queue, cancelled waiters release their turn, and
batch acquisitions join the same queue. Other lock paths are unchanged.
This moved here from the request-layer batch, where no cited issue
covered it; it serves the table-catalog issue this PR closes.
* fix(table-catalog): close the round-2 adversarial review findings
Parquet page-header walker: read the size fields by field id as parquet
does, accepting I16/I32/I64 wire kinds and rejecting any other kind, so
a header encoding uncompressed_page_size as I64 can no longer bypass the
budget while parquet allocates it. Bound every list, set and map count by
the bytes remaining in the chunk, and reject bool collection elements
(parquet skips them with zero width, so either width would desynchronize
the two parsers); this removes the 2^63-iteration spin on LIST<BOOL>.
Start each column-chunk walk at the dictionary page offset whenever the
footer declares one, exactly as parquet's byte_range does, instead of
min(dictionary, data).
Footer pre-walk: before parquet decodes a footer, walk FileMetaData with
the hardened cursor and charge the declared row_groups count times the
RowGroupMetaData size to the decode budget, because parquet 59 reserves
that Vec before reading one element.
Preparation deadline: clamp the
RUSTFS_TABLE_CATALOG_PUBLICATION_PREPARATION_TIMEOUT_SECS knob to 86,400
seconds so Instant + Duration cannot overflow.
Index-miss scan: after each listing page, jump the cursor past the
maintenance subtree the page ended in, and allow one extra page per table
entry already listed, so job-report history scales with the catalog while
a report flood under one table stays bounded.
Migration fence: read the fence-path object once and decide from its
stored kind whether it is directory state or a fence.
Expired or failed create: remove the initial metadata file when the
registration fails before any catalog write may have begun; nothing else
references it. Correct the comment and support matrix that claimed
maintenance reclaims it.
Tests cover each walker divergence, the footer row-group reservation, the
deadline ceiling, the paging budget for honest catalogs and for a report
flood, and the removal of the unpublished metadata file.
* test(admin): register the catalog directory routes in the route matrix
The table-catalog batch added GET, POST and DELETE
{warehouse}/catalog/directory on both the Iceberg and the compat admin
prefixes without listing them in the admin route matrix, so
test_admin_route_matrix_matches_registered_routes failed on CI.
* fix(s3,ecstore): close the round-3 adversarial review findings
Multipart index pages now treat the cursor as the last examined marker
rather than the last root, so a page that pruned a stale marker still
validates and ListMultipartUploads keeps using the index. A marker is
written before its staging root is published, so pruning spares markers
younger than a five-minute grace period and an existing marker is
touched on every new upload; the prune re-checks both under the marker
lease. Bounded delete-recovery inspection rides the durable-mutation
lane like the unbounded path instead of the one-worker foreground lane.
The on-demand-migration inline body budget scales with the object size
(at least 64 KiB/s) instead of a flat 120 s. The control-body digest
service honors RUSTFS_HTTP_REQUEST_BODY_READ_TIMEOUT=0 as "no
inactivity timeout" like every other reader of that knob.
Quota reconciliation throttle objects are deleted together with a
retired generation's ledger, so bucket turnover leaves no residue. The PUT guard
test shim is gone; tests call the shared guard. Bounded DiskAPI defaults
answer MethodNotAllowed like their siblings. Stale
budget docs, an unreachable CSV arm, an always-Some limit parameter,
commented-out constants and a misplaced doc comment are cleaned up;
weak test assertions name the expected error; lifecycle rule-ID control
characters gain a test. The compatibility matrix records the CopyObject
tag-copy permission, the exact SigV2 empty-body predicate, replica and
migration writes of control-character keys, and the SigV2 DeleteObjects
digest rule.
* fix(table-catalog,heal): close the round-3 review findings
A. Pack compaction rewrite groups to min(max-rewrite-bytes-per-job,
TABLE_COMPACTION_MAX_STORED_BYTES), reject an over-ceiling group before
any output is written, and name the ceiling in the commit-path errors;
document the effective cap in the support matrix and scheduler runbook.
B. Keep TaskCancelled/TaskTimeout classification on replacement passes:
the namespace verification runs only for completed or storage-failed
passes, in the erasure healer and the task bucket prepass.
C. Compare the replacement bucket pass per bucket: a recorded identity
that is gone or replaced still errors or restarts the pass, buckets
created mid-pass are appended to the identities, pending_buckets and a
persisted positional order without rewinding progress or the
checkpoint; verification ignores additions.
D. Read the migration-fence object once in the object-backed write permit
and dispatch on its kind.
E. Drop the strong store's copy of maintenance_cleanup_roots.
F. Remove the unused test hooks fail_put_prefix, corrupt_put_prefix and
pause_after_metadata; make compact_parquet_data_files and
PublicationTicket.state private.
G. Document the body-revision identity, the legacy revalidation budget
and its HTTP 503 in the support matrix.
H. Define HEAL_REPLACEMENT_RECOVERY_DIR once in ecstore and consume it
from heal through the storage_api facade.
* fix(ecstore): cap the index pages one disk may answer
Pruned multipart index pages may carry few roots, so the page cursor no
longer bounds how many pages a peer can make a listing fetch. Cut a disk
off after 4096 pages (about one million examined markers) so it
contributes no coverage and the listing falls back to the staging scan.
* fix(ecstore,table-catalog): close the round-4 adversarial review findings
Multipart index markers dated in the future (a backwards clock step
between recording a marker and publishing its staging root) were judged
old and could be pruned while their upload was in flight; a future-dated
marker is now kept. The marker touch on EEXIST is best effort, so a
filesystem that refuses to set times no longer fails the upload commit.
Pruning runs on the durable-mutation lane instead of the one-worker
foreground lane.
Quota reconciliation keeps one throttle object per bucket incarnation (a
quota revision change replaces it instead of orphaning the previous
revision's object), the retirement cleanup deletes it ahead of the ledger
and also when the ledger is already gone, and a scan the retirement
already refused writes nothing back. Bounded DiskAPI defaults answer
UnsupportedDisk, the verdict the aggregate gives a peer without the
protocol. The compaction ceiling pre-check runs over every rewrite group
before any output is written, and the worker's ceiling error no longer
names a placeholder group.
Docs: the inline on-demand-migration client budget, the protocol-2
downgrade conflict of replacement passes, the exact footprint numbers.
* fix(ecstore): retry metadata writes rejected before staging
* fix(ecstore): reuse registered disks during startup
* fix(ecstore): retain remote health monitors during reconnect
* test(heal): release the commit barrier during graceful shutdown
* test(e2e): register graceful shutdown barrier regressions
* fix(ecstore): roll back interrupted pool metadata commits
* test(upgrade): honor the 1.0.4 delete recovery protocol
* fix(heal): bound admission during object rate pressure
* test(e2e): handle rebalance completion before stop
* test(e2e): observe rebalance activity before the start reply
* test(rebalance): hold the first run until stop cancels admission
Use explicit buckets that preserve millisecond lock acquisitions while retaining visibility into long-held delete and recovery locks. Add an exporter-backed regression test for both histogram instruments.
Co-authored-by: zhi22915 <qiuzgang@gmail.com>
* fix(s3): treat duplicate CreateBucket as owned by the requester
The duplicate-bucket response was decided from the requester's root-owner
flag, so every IAM user recreating a bucket received 409
BucketAlreadyExists, including a user recreating a bucket it created.
BucketAlreadyExists means the bucket belongs to another account; a
deployment owns a single namespace, so the answer is purely region-based:
the default region returns the legacy 200 OK, any other region returns
BucketAlreadyOwnedByYou.
Both the existing-bucket fast path and the BucketExists path now share one
helper. Authorization still runs first, so a requester without
s3:CreateBucket gets AccessDenied and cannot probe bucket existence.
Refs #8493
* test(ci): refresh e2e selections for duplicate CreateBucket
The two new create_bucket_region cases join the e2e-smoke and e2e-full
selections, which main also extended with multipart_upload_id tests.
smoke is measured from a local nextest listing: removing the two new cases
reproduces main's committed digest exactly. e2e-full is derived the same way
from main's listing artifact (run 41933823399, babfeaed0) plus the two
multipart_upload_id cases, which reproduces main's committed linux digest
before the new cases are added; CI re-verifies both platforms.
test_bucket_create_exists_nonowner asserts BucketAlreadyExists for a second
identity, which a single-owner deployment cannot produce. Move it to
excluded_tests.txt with the deployment-ownership rationale; the IAM-user
duplicate paths stay covered by crates/e2e_test/src/create_bucket_region_test.rs.
UploadPart, UploadPartCopy, ListParts and CompleteMultipartUpload returned 400 InvalidArgument for a malformed upload ID, while AbortMultipartUpload returned 404 NoSuchUpload. AWS S3 documents NoSuchUpload for an upload ID that might be invalid.
Report MalformedUploadID as NoSuchUpload through one helper used by all operations that address an existing upload, replacing the Abort-only special case. ListMultipartUploads keeps InvalidArgument for a malformed upload-id-marker.
Add the multipart_upload_id_test e2e module to the e2e-smoke profile. It asserts 404 NoSuchUpload from every operation that addresses an existing upload, for a malformed upload ID and for an aborted one, and updates the e2e-smoke and Linux e2e-full selection digests.
Co-authored-by: Karl Bastarache <bastarachek@google.com>
Co-authored-by: Hauser Zhi <housemecn@gmail.com>
* perf(ecstore): probe bulk-delete write locks in parallel waves
Bulk delete acquired one distributed write lock per key in serialized
order, paying a full quorum round-trip per key. Probe the unique keys in
bounded parallel waves with a short timeout instead; any contention drops
the whole wave and the canonical serial order resolves it as before, so
two overlapping batches can never deadlock.
* perf(ecstore): probe bulk-delete transaction locks in parallel
The set-layer delete-transaction locks were also acquired one quorum
round-trip per key after the raw write locks. Apply the same optimistic
all-or-nothing probe wave with a canonical serial fallback, and keep the
per-key error reporting the fallback already provided.
* perf(ecstore): pre-fetch bulk-delete source metadata in parallel
The admission loop ran one metadata quorum fan-out per key in sequence
on top of the per-key lock round-trips. Pre-fetch the source reads the
loop needs with bounded parallelism under the locks already held, and
derive the per-key read inputs from one shared helper so the pre-fetch
and the loop cannot disagree.
* test(ecstore): drop unused import in batch-delete probe test
* refactor(ecstore,lock): await wave releases and unify probe waves
Adversarial review of the bulk-delete lock waves found three real
issues. Distributed guard release is fire-and-forget on drop, so the
serial fallback could re-acquire keys while unlock RPCs were still in
flight and block on locks it already owned: guards are now released
with an awaited unlock before the fallback runs. The wave also probed
every key after the first failure, adding up to one probe timeout per
chunk of pure overhead under contention: probing now stops at the
first failing chunk. Finally the two copy-pasted wave implementations
are replaced by one shared probe_lock_wave, and the per-key delete
read inputs are resolved once instead of being recomputed.
Tests now assert the wave/fallback counters so they prove which path
ran, and an overlapping-batch test pins the no-deadlock property.
* refactor(ecstore,lock): batch bulk-delete lock rounds per key
Replace the optimistic probe waves with the bulk lock primitive the lock
layer already supports on the wire: one lock_batch RPC per lock client
for a whole DeleteObjects key set, per-key quorum guards with the usual
TTL heartbeat, and per-key outcomes. The batched round is deadlock-free
by construction (every lock server orders concurrent batches by the
same global (shard, key) order), so the probe timeout, the wave
release-and-wait dance, and the canonical serial fallback all go away.
Per-key lock outcomes are now reported per key end to end: a locked-out
key fails alone in the response while the rest of the batch proceeds,
matching the S3 DeleteObjects contract and the set layer's existing
behavior. The set layer's raw-write-lock and delete-transaction-lock
waves fold into the same primitive via one shared resource-folding
helper, and the fast-lock batch fast path now counts acquisitions so
shard metrics keep their meaning.
* perf(ecstore): move journal publish off the one-worker namespace lane
A bulk delete publishes one recovery transaction per key, and every
journal inspection and settle-plan closure ran on the per-disk
foreground namespace lane (one worker), serializing the whole publish
phase per drive behind every other mutation. Route those closures onto
the durable-mutation lane the fused delete prepare already uses (eight
workers per disk, fsyncs coalesce in the filesystem journal), and keep
32 journal publications in flight so a 16-drive set can feed those
lanes.
* test(ecstore): flatten lock-error iteration in overlap test
* fix(ecstore,lock): refresh duplicate keys after recovery, bound unlock
Review findings on the batched lock rounds. A batch entry whose recovery
rolls back an interrupted prepared delete can bring a retained version
back; a later duplicate of the same key still held its pre-fetched
source-missing read, skipped the object-lock check, and deleted the
restored version. Recovery now evicts the pre-fetched reads of every
other entry naming the same object, so those entries take the loop's
live re-read and see the restored version.
The batch rollback release awaited the full unlock retry schedule,
including the deferred background pass, parking the request behind an
unreachable peer. The synchronous phase is now a single release pass
with the deferred retries spawned in the background.
* fix(ecstore): release local batch locks before deferred recovery
A set-layer batch delete that runs deferred key recovery re-acquires a
write lock on the same keys through recover_delete_object, but the raw
local batch guards were only ever released at function exit — the
distributed path releases its batch ids before the deferred loop and the
local path missed the mirror release, so the recovery deadlocked against
the batch's own locks and surfaced as a spurious Lock(Timeout) for every
deferred key.
A regression test plants the interrupted prepared delete this needs and
pins the end-to-end contract: a batch naming a GOVERNANCE-retained
version twice must deny both entries after the first entry's recovery
rolls the prepared delete back, and the restored version must survive.
* test(lock): cover batch acquire partial-grant rollback
The distributed acquire_guard_batch rollback path — a key granted by
fewer than quorum clients must fail alone and release its partial grants
before the call returns — was only reachable in distributed mode and had
no coverage. A scripted per-object LockClient pins both halves: the
below-quorum key reports QuorumNotReached with its partial vote, and the
orphaned grant is released synchronously while the quorate key's guard
stays held.
Track active descriptor volumes, exact paths, and component prefixes in FdCacheTracker so unmatched invalidate_under, invalidate_volume, and invalidate_exact calls return without scanning Moka or waking the background maintenance task.
* fix(s3): honor presigned x-amz-meta-* query parameters
The AWS SDK presigners hoist signable x-amz-* headers into the query
string (smithy moveHeadersToQuery), and AWS S3 accepts the hoisted form
as the header it came from. RustFS read user metadata from headers only,
so a presigned PutObject carrying ?x-amz-meta-user-id=u1 returned 200 and
stored an object with empty metadata (rustfs/rustfs#9142).
Add one shared extractor for user-metadata query parameters and wire it
into every metadata-accepting write path (PutObject including snowball
members, CreateMultipartUpload, CopyObject REPLACE) plus the snowball
extract trigger, instead of normalizing per operation the way the
presigned UploadPart checksum fix did. The stored key form is produced by
the same helper as the header path, so the hoisted and header forms land
on disk identically.
Only a signature-covered query is honored: SigV4 signs the canonical
query string, while SigV2 signs headers plus a fixed subresource list
only, so honoring its query would let a presigned-URL holder add object
properties the presigner never authorized. Keys and values that could not
have arrived as a header are rejected rather than stored, since the read
path turns them into header fields and a control character would fail
every Head/Get of that object.
* refactor(s3): hoist signed x-amz query params at the request boundary
Replace the per-write-path query metadata extractor with one boundary
mechanism: restore the `x-amz-*` request properties the SigV4 presigners
hoist into the query (`@smithy/signature-v4` `moveHeadersToQuery`) as
request headers before s3s routes and deserializes the request.
One hook now covers every reader at once — s3s input parsing,
`extract_metadata_from_mime`, the snowball extract trigger and its
transport-key filter, tagging, storage class, object lock and copy
source — so PutObject, CreateMultipartUpload and CopyObject need no
changes at all. It also replaces repeated query parsing along the write
path with a single pass at the boundary.
An existing header always wins, so the hoist is purely additive and the
unsigned-header guard keeps rejecting anything the signer did not sign;
that guard now also accepts a header whose single value equals a
same-named query parameter, which is what the hoist produces and what
the signature covers.
`query_string_is_signed` rules out SigV2 before it trusts any SigV4
marker. SigV2's string-to-sign covers headers plus a fixed subresource
list rather than arbitrary query parameters, so a SigV2 presigned URL
stays valid after `X-Amz-Algorithm`/`X-Amz-Signature` markers are
appended to it; without this rule a URL holder could turn on
`x-amz-meta-snowball-auto-extract` and have a plain upload untarred
(review of #9146).
* fix(s3): hoist signed query params after s3s signature verification
The request-boundary hoist ran before s3s, so the restored `x-amz-*`
headers reached s3s's own verifier, which rejects any `x-amz-*` header
outside `X-Amz-SignedHeaders` (s3s `ops/signature.rs`,
`reject_unsigned_amz_headers`). A valid presigned PUT with
`SignedHeaders=host` and `?x-amz-meta-user-id=u1` therefore came back
with 403 AccessDenied before any handler ran (review of #9146).
Move the hoist to `authorize_request`, RustFS's shared authorization
entry: s3s has already verified the signature there, and policy
evaluation plus the handler read request properties after it, so the
restored headers reach every operation without ever appearing unsigned
to the verifier. The mechanism is unchanged — one gated, validated pass
over the signature-covered query.
CopyObject's replacement metadata now also reads the headers, because
s3s parsed `input.metadata` before the hoist ran.
The hoist itself and its tests stay in `server::layer`; the tower layer
is gone.
* perf(s3): single-pass signature-aware query hoist in auth
Move the query hoist into `auth.rs`, next to the unsigned-header rules it
is part of, so `storage` no longer reaches into `server` for it. Its
documented contract — which request properties the signature covers, and
why SigV2 must never be widened — now lives with the rule it protects.
Make the hot path allocation-free and single-pass. The caller no longer
clones the query just to satisfy the borrow checker, the function no
longer clones it again, and one `form_urlencoded::parse` pass both
recognizes the presigned-SigV4 marker set and stages the hoistable
parameters. Nothing is validated or applied until the gate passes, so a
request whose query is not signature-covered is never rejected here. A
single ASCII scan skips the whole thing when the query names no `x-amz-`
parameter at all, which is every ordinary `?versionId=`/`?uploadId=`
request and every queryless one.
The duplicate-parameter and header-value checks stay strict; only when
they moved, not how they behave.
* fix(s3): keep verifying a query-carried payload hash after the hoist
The hoist restores `?x-amz-content-sha256=<digest>` as a header, but it
runs after s3s has already built the request body. s3s reads its
payload-hash input from the headers only, so it installs no hash
validator for a digest that arrived in the query, and the restored header
then made `get_content_sha256_with_query` conclude the auth layer was
enforcing it. Neither layer verified the body, so a presigned PUT whose
digest exists only in the query accepted a modified payload (review of
#9146).
The question is not whether a digest header is present but whether the
digest reached the request as a query parameter: that one is always the
storage layer's to verify. A digest sent as a header is still left to the
auth layer, which wraps the body in a hash-validating reader for it, so
no payload is hashed twice.
* fix(auth): limit late query hoisting to user metadata
* refactor(s3): drop the stale payload-hash compensation, scope the pre-filter
Limiting the hoist to `x-amz-meta-*` user metadata means it never
manufactures an `x-amz-content-sha256` header, so the payload-hash
predicate's original signal — a digest header present at handler time is
one s3s is enforcing — is exact again. Restore it and drop the
query-aware form: that change existed only to undo a header the hoist is
no longer allowed to create, and it would have hashed a payload twice
for a presigned request that declared a real digest both in the query and
as a header.
Match the pre-filter to the same scope so a query carrying only a typed
property (`x-amz-tagging`, `x-amz-storage-class`, `versionId`) skips the
parse entirely instead of scanning past it.
* test(access): use explicit imports in the signed-query lock regression
The architecture migration rule requires RustFS storage-owner modules to
name their imports instead of pulling the parent module in wholesale, and
`signed_query_lock_permission_regression` was still on `use super::*`.
Import the two symbols it actually uses and drop the unused one.
* test(access): reference FS through the parent import
The layer dependency guard tracks `use crate::...` edges, and naming
`crate::storage::ecfs::FS` from the signed-query regression looked like a
new `infra -> interface` reverse dependency. `access.rs` already reaches
the same type through `use super::ecfs::FS`, so reuse that import instead
of introducing a second, tracked spelling of an edge the module has
always had. No baseline entry, no new edge.
---------
Co-authored-by: overtrue <anzhengchao@gmail.com>
Scan findings in the alternate-protocol front ends and the KMS and
Keystone integrations:
- WebDAV and FTPS read RUSTFS_WEBDAV_CA_FILE / RUSTFS_FTPS_CA_FILE,
checked that the file existed, and then built their TLS listeners
with with_no_client_auth(), so mTLS was never enforced. A shared
protocols::common::tls::server_config builds the rustls config with a
client-certificate verifier from the configured CA, startup fails on
an invalid CA, and a CA cannot be combined with cleartext
(backlog#3069).
- The KMS readiness probe used a fixed key id that the key-lifecycle
admin routes accepted like any customer key, so kms:DisableKey on it
failed readiness on every node. The probe key is reserved from key
management and a disabled probe key no longer counts as a backend
failure (backlog#3115).
- The Keystone middleware answered anonymous requests carrying a junk
X-Auth-Token with the configured Keystone URL in the body while
Keystone was unreachable; failures no longer disclose upstream
locations, and the login cache is bypassed when a subscription token
refresh is required (backlog#3116).
Scan findings in crates whose behavior is driven by untrusted input and
that stand alone from the storage core:
- trusted-proxies: the simple implementation takes the client address
from the right-most untrusted X-Forwarded-For hop instead of the
left-most (backlog#3082); the legacy validator never falls back to the
proxy identity when the chain is rejected and reads X-Forwarded-For
unless RFC 7239 is explicitly selected (backlog#3051).
- targets: webhook and similar outbound clients drain responses up to a
byte limit instead of buffering arbitrary bodies (backlog#3062); the
on-disk notification queue has a shared byte budget in addition to the
entry count, configurable via RUSTFS_TARGET_STORE_MAX_BYTES
(backlog#3122); AMQP and Redis manifests declare the broker url as a
secret and bind credentials to their connection fields, and the
plugin-instance PUT refuses to restore a redacted secret when the
connection fields changed (backlog#3065, backlog#3066).
- obs: the debug-level s3s response dump is filtered so STS and
service-account credentials never reach the log (backlog#3029).
Scan findings in the IAM crate and the admin identity handlers, grouped
because they share the IamCache mutation lock, the STS session sweep and
the admin client-address plumbing:
- Admin and STS authorization used the socket peer as aws:SourceIp
behind a trusted proxy; they now read the verified client address the
S3 path already uses (backlog#3025), and the admin usage route passes
the bucket named in the path to the s3:ListBucket check (backlog#3033).
- AssumeRoleWithWebIdentity accepted HS256 ID tokens MACed with the OIDC
client secret; only provider-signed tokens verify (backlog#3028), and
JWKS refresh is single-flight with a failure cool-down (backlog#3103).
- A disabled group emptied the member's whole policy list, dropping
every Deny; direct and other-group policies are preserved (backlog#3042).
- Identity rewrites (secret change, add-user, status) ran without a lock
and without re-reading the record, so a user's password change could
undo an admin disable or reset (backlog#3048). add-user on an existing
user and in-flight AssumeRole both left sessions minted under the old
secret valid (backlog#3071, backlog#3079); rotation now revokes under
the per-key mutation lock and derived-credential publication is ordered
with it.
- aws:groups never carried a built-in user's IAM group membership and
jwt:iss was always the local marker, so group- and issuer-scoped
Denies never matched; both now come from live IAM state
(backlog#3095, backlog#3096).
- OPA mode skipped the account-status gates for service accounts and STS
sessions of a disabled user (backlog#3057).
- The periodic IAM reload published nothing when any cache write raced
it and kept stale partitions when a prefix listed empty, so a missed
revocation never applied (backlog#3064, backlog#3114).
- update-group-members never evaluated admin:RemoveUserFromGroup
(backlog#3098); ExportIAM compared the site-replicator account against
a name it never has (backlog#3045); add-user accepted the access key of
a live STS session or service account (backlog#3134); a service account
could create or extend a lifetime beyond its own (backlog#3112);
sts:DurationSeconds and svc:DurationSeconds were never populated
(backlog#3136); temporary-account-info returned service accounts
(backlog#3137); an access key the object layer refuses no longer holds
the service-account lock through the write retries (backlog#3133).
A put-like write without a version id to a key that is not
versioning-Enabled replaces the key's null version in place. The
commit-time Object Lock gate in put_object_with_old_current_size_inner
ran only when the write carried an explicit version id, and the
request-layer pre-check reads the key's latest version. When the
retained null version was not the latest entry (a UUID version or a
delete marker sat above it), a user holding only s3:PutObject could
overwrite it together with its COMPLIANCE retention or legal hold, and
CompleteMultipartUpload had no commit-time gate at all.
Name the version a commit replaces (the explicit id, or the nil id when
the write is not versioned and the bucket's Object Lock snapshot is not
ConfirmedAbsent) and run the existing WORM gate against it, tolerating
a null delete marker. complete_multipart_upload gets the same gate at
its commit point, and ECStore::handle_complete_multipart_upload attaches
the bucket's Object Lock snapshot the way a single PUT already does.
Refs rustfs/backlog#3019 (ANT-2026-EH17ZP59).
Eight scan findings in the shared policy engine, each a case where a
Deny did not apply or an Allow was broader than written:
- A bare `*` Action was parsed as `s3:*`, so `Deny *` never matched
admin:, kms: or sts: actions. `Action::All` now denies every family
while the historical S3-only Allow is preserved (backlog#3041).
- A bare IPv6 address in aws:SourceIp got `/32` appended, matching a
2^96-address network; a bare address now means one host, and IPv4
clients on a dual-stack listener are matched through their mapped
form (backlog#3030).
- Date conditions on aws:EpochTime parsed the epoch seconds as RFC 3339
and never matched, so a time-bounded Deny failed open (backlog#3036).
- NumericNotEquals and DateNotEquals evaluated to false when the key was
absent; negated operators now follow the AWS rule and match
(backlog#3123).
- Every trailing IfExists on a condition operator added a level of
recursion without bound; the chain is rejected before parsing
(backlog#3091).
- Policy variable substitution re-expanded its own output for up to ten
passes, so an access key holding k placeholders grew to k^10; values
are now substituted once as literal data (backlog#3067).
- A bucket-policy Service principal was matched against the signing
access key, so a service account named after the service received its
grants; Service principals round-trip but never match (backlog#3081).
- Resource matching cleaned the trailing slash off a directory object,
so a Deny on `prefix/*` missed `prefix/` itself (backlog#3125).
Also deduplicate large Resource sets through a HashSet instead of a
quadratic scan.
PutObjectLegalHold and PutObjectTagging apply the value in their XML
body, and a POST upload applies its form fields, while the
s3:object-lock-legal-hold, s3:RequestObjectTag/<key> and
s3:RequestObjectTagKeys condition keys (and every other header-derived
key for POST) were built from the HTTP headers. The policy engine
therefore judged a value the handler did not store: absent when the
client sent no header, and whatever the caller wrote when it did. A
user limited by a Deny on the requested legal hold or tag value could
release a hold and delete the held version, or retag an object so a
tag-based grant exposed it.
Record the body value in request extensions before authorization, the
way PutObjectRetention already does, and let authorization_conditions
replace the header-derived keys with it. For POST uploads project the
parsed form fields over the headers that feed the condition map, removing
a header whose field is absent.
Refs rustfs/backlog#3018 (ANT-2026-KONBE376).
DeleteObjects ran its Object Lock check on one version of a key and then
removed a different one. For an entry without a version id on a key that
is not versioning-Enabled, delete_objects_with_accounting judged the
key's latest entry and then had the disks remove the null version. When
a UUID version or a delete marker sat above a null version under
COMPLIANCE retention or legal hold, that retention was never read and a
user holding only s3:DeleteObject could remove the retained version. The
single-object DELETE on a Suspended key had the same gap.
Add SetDisks::check_null_version_object_lock_delete, which looks up the
null version under the lock the caller already holds and runs the
existing check_object_lock_delete against it when the latest entry
carries a UUID version id. The batch path is not gated on
source_missing because a delete marker on top makes the latest-version
lookup report the key as missing. Plain unversioned keys and
versioning-Enabled keys keep a single lookup and unchanged behavior.
Refs rustfs/backlog#3017 (ANT-2026-6ED2F7II).
Storage encodes a directory object `photos/` as `photos__XLDIR__/xl.meta`
and LocalDisk::scan_dir decodes every directory entry ending in the
suffix back into a marker. The S3 write path never refused the suffix in
client keys, so a 1-byte PutObject to the key `__XLDIR__` made the walk
schedule the directory being scanned as its own child and recurse until
the stack overflowed, aborting the server on every restart until the
directory was removed by hand. Keys below a `*__XLDIR__/` segment were
stored but hidden from every listing and from quota accounting.
validate_object_key now rejects any client key containing the suffix,
which covers PutObject, CopyObject, Snowball extraction and the admin
internal put; CreateMultipartUpload and CompleteMultipartUpload now run
the same validation so uploads created before the key was reserved
cannot publish it. scan_dir additionally ignores a marker whose decoded
name is empty, so malformed markers left by older writers or corrupt
disks can no longer recurse into their parent.
Refs rustfs/backlog#3020 (ANT-2026-SBJ1JLC2).
POST /rustfs/admin/v3/kms/reconfigure and configure accepted
server-local file paths in the Vault request body (Kubernetes jwt_path,
AppRole secret_id_file, TokenFile path, TLS ca/client cert and key
paths), read the named file with the server's own authority and sent
its content to the Vault address given in the same body. A user holding
only kms:Configure could read any UTF-8 file the process can read,
including /proc/self/environ with the root credentials, and exfiltrate
it to an address of their choosing.
normalize_configure_request_secrets now runs
validate_vault_local_file_access for both handlers: a Vault request
that names a server-local file, or inherits the stored credentials by
sending a blank token, must keep the operator-configured address and
namespace, the same file-based auth method (role, path, mount) and the
same TLS paths and verification setting. Requests with explicit inline
credentials and no file paths are unchanged.
Refs rustfs/backlog#3021 (ANT-2026-F8MRFKCC).
The object handlers accept several spellings of one version id (upper
case, unhyphenated, braced, urn:uuid:) and the storage layer resolves
all of them to the same version, but the s3:VersionId condition value
was the client string verbatim. A Deny with StringEquals on one version
could therefore be bypassed by re-spelling the id in ?versionId=.
Canonicalize the id once where the condition value is produced so every
hook and policy type (IAM, bucket, session) sees the spelling the server
emits: hyphenated lower-case UUID, with null in any case and the nil UUID
mapped to null. Non-UUID strings pass through unchanged and are rejected
by the handlers as before.
Refs rustfs/backlog#3022 (ANT-2026-O5MHY8VJ).
* fix(replication): replay the MRF journal periodically
The persisted MRF journal was replayed only at startup. Delete-marker
purges whose watch window drained with targets unreachable, and
replications that overflowed the worker queues, reach the journal while
the server runs, so they waited for a restart to converge.
Move the startup pass into replay_mrf_journal and rerun it every five
minutes from a background loop. The existing recovery leader lock keeps
one pass at a time across the cluster, including the startup pass.
* fix(replication): space periodic MRF replay across nodes
Every node runs the MRF replay loop and the recovery leader lock only
serializes passes, so an N-node cluster replayed the same retained
backlog N times per interval while a target stayed down.
Record the start time of a pass that finds entries in a shared replay
stamp and skip a periodic pass when any node started one within the
interval. The startup pass still always runs.
* fix(heal): treat SlowDown as a rate signal and bound page churn
* refactor(heal): unify page rate control and drop checkpoint clones
* refactor(heal): one pass-wide rate controller, single SlowDown observe
* fix(heal): address SlowDown backoff and checkpoint baseline review
Replace the single-mutex PREPARED_DELETE_RECEIPTS and ACTIVE_MULTIPART_SOURCE_HEADS maps in rustfs-ecstore with a 64-shard ShardedPathCache backed by SWAR canonical byte keys and xxh3_64 hashing. Replace PUT_FILE_TARGET_LOCKS in rustfs internode RPC with a 64-shard ShardedPutFileTargetLockRegistry using stack-buffered binary keys, in-place dead-Weak slot reuse, and amortized garbage collection.
* fix(replication): skip site-replication rules in replica delete fan-out
A site that receives a replicated delete forwards it to its own targets
so chained replication keeps propagating deletes. The fan-out used the
whole bucket config, including derived site-replication rules, so in a
site-replication mesh a replicated purge was echoed back to its origin
and duplicated to every other peer. The echo could remove the origin's
pending-purge version before its own delivery state was written back.
Ignore derived site-replication rules when computing the replica delete
fan-out. The origin already delivers and retries every delete to each
peer itself. Operator rules still fan out, so chained replication is
unchanged.
* fix(replication): keep operator rules matching the site-repl shape
Rule ids and target ARN shapes are not reserved for site replication:
an ordinary bucket target ARN carries its target id, so an operator rule
named site-repl-<target id> matched the derived shape and was dropped
from replica delete fan-out, breaking an A -> B -> C chain.
Skip a rule only when its deployment id is a current remote
site-replication peer, reading the peer set only when a rule has the
derived shape. If the state cannot be read, skip nothing.
Large metacache elements (an xl.meta grown past 16 MiB with its version
history) were accumulated in fixed 64 KiB reads, paying hundreds of
read rounds per entry on the listing path. Double the per-round chunk up
to a 4 MiB bound so each reservation stays incremental and bounded
(rustfs/rustfs#2715 corrupt-length-prefix safety is unchanged) while
wide listings and oversized entries decode in far fewer rounds.
Add regression coverage for rustfs/rustfs#9120: a listing whose
aggregate stream exceeds the per-value 16 MiB bound, and an oversized
entry listed among ordinary neighbors with peek/skip interleaving.
The rustfs_cluster_servers_offline_total gauge was event-driven only:
record_peer_reachable fired solely after a fresh connector.connect()
succeeded. When post-recovery traffic was served by a cached channel
(tonic auto-reconnect, addr-form cache-key split, or delete-recovery
channel), no fresh dial occurred and the gauge stayed stale until the
process restarted.
Add record_peer_reachable_if_offline (fast-path read / slow-path write)
and wire it into every successful peer interaction:
- remote-disk recovery probe success (gRPC DiskInfo)
- periodic TCP connectivity check success (bounded-interval reconcile)
- data-path RPC success (execute_with_timeout_for_op_and_health_action)
- lock-path RPC success (record_rpc_success and detached completion)
Fixesrustfs/rustfs#9119
Tracking: rustfs/backlog#3138
After the nightly packages are on the auto-testing nightly release and
the schema 2 R2 candidate exists, post repository_dispatch to
rustfs/auto-testing. Vault lanes stay parallel, and the workflow_run
functional chain is left in place.
Co-authored-by: hector <majinghe@users.noreply.github.com>
Remove pull_request from the distributed and upgrade workflows. Distributed
stays on the nightly cron, and upgrade stays on the Monday cron plus
version tags. Both remain manual-dispatch lanes and stay outside the
required merge checks.
Co-authored-by: majinghe <majinghe@users.noreply.github.com>
Give each mixed-workload request a bounded policy for the server's
explicit namespace lock timeout response and Retry-After advice. Keep
unrelated SDK failures and data mismatches immediately visible.
Preserve 24 workers, four rounds, cross-node checks and all retained
objects. Cover exhausted SDK retries, bodyless HEAD, fatal responses,
persistent contention and incorrect acknowledged bytes, and include the
new cases in the protected distributed profile inventory.
Co-authored-by: heihutu <heihutu@gmail.com>
Co-authored-by: zhi22915 <qiuzgang@gmail.com>
* ci(validation): resolve the newest nightly build on scheduled sweeps
The first scheduled sweep failed at candidate resolution: a schedule
carries no workflow inputs and resolve_functional_candidate.py only
derives a build from a workflow_run payload, so the empty
BUILD_RUN_ID crashed int(). Look up the newest successful
nightly-gnu run explicitly when the inputs are empty and feed it to
the resolver. Also downgrade the evidence artifact to
if-no-files-found: warn so an early infra failure does not cascade
into a second red step.
* ci(validation): read the resolver chain envelope correctly
CHAIN_OUTPUT is the chain envelope {schema, run_id, attempt,
workflow_sha, testing_sha, candidate}; the manifest lives under
candidate.manifest and the testing sha at the top level.
* ci(validation): sweep harness fixes from the first live run
- ssh -n in the inter-case cleanup loop: plain ssh consumed the case
list from the loop's stdin, so the first dispatch executed only one
of the twelve selected cases.
- Provide the managed Python 3.12 runtime for pool/ilm/replication
cases, mirroring the chain validation job (pool.status reported
INFRA_ERROR after its assertions passed; the missing runtime is the
prime suspect).
- Record the auto-testing sha in the report step environment.
- Retain the last case's evidence tree in the sweep artifact so gate
verdicts like INFRA_ERROR can be diagnosed from the run itself.
* ci(validation): install sweep Python client dependencies
Scheduled sweeps failed pool.status before any assertion because
pool_behavior.py exits when boto3 is missing. Install the auto-testing
validation requirements into a per-run virtualenv and put it on PATH,
matching the functional chain, so case scripts can import the client.
Co-authored-by: hector <majinghe@users.noreply.github.com>
---------
Co-authored-by: hector <majinghe@users.noreply.github.com>
* chore(deps): refresh crates and adopt openidconnect 4.1
Use published OIDC/OAuth2 releases and reject malformed HMAC signature
lengths before upstream verification. Refresh workspace and fuzz locks,
and advance the local zstd compatibility patch to 0.13.4 while retaining
the fixed 0.14.3 codec backend.
Co-Authored-By: heihutu <heihutu@gmail.com>
Co-Authored-By: zhi22915 <qiuzgang@gmail.com>
* chore(deps): refresh compatible lockfile resolutions
Resolve Prost build and derive dependencies with itertools 0.14 and
prefer windows-sys 0.61.2 within the existing supported version ranges.
Retain all package versions, sources and checksums.
Co-Authored-By: heihutu <heihutu@gmail.com>
Co-Authored-By: zhi22915 <qiuzgang@gmail.com>
* test(connect): separate cleanup checks from short probe windows
Use the functional measurement budget for the cleanup HTTP fixture so
setup and measured I/O can finish before checking delete status handling.
Exercise both normal scheduling and an in-flight PUT clock advance, and
include target classification in unexpected-failure diagnostics.
Retain the cleanup deadline, outer timeout and delete-stage assertions.
Co-Authored-By: heihutu <heihutu@gmail.com>
Co-Authored-By: zhi22915 <qiuzgang@gmail.com>
* test(ecstore): isolate stalled multipart disk timeouts
Apply the short injected timeout only to the deliberately stalled disk.
Healthy disks retain their normal operation budgets, while logical lock,
physical owner, immediate CopyObject and final shard assertions remain.
Keep overrides scoped across cloned and reconnected handles, preserve
shorter caller deadlines and explicit zero timeouts, and report drive
health when the real CopyObject assertion fails.
Co-Authored-By: heihutu <heihutu@gmail.com>
Co-Authored-By: zhi22915 <qiuzgang@gmail.com>
* fix(rebalance): retry metadata write lock quorum admission
Recognize typed write-lock quorum failures for rebalance.bin within the
existing bounded metadata admission retry policy. Each attempt must
acquire quorum before loading or saving metadata; data writes are not
replayed by this retry.
Cover recovered and persistent quorum failures, cancellation after a
failed attempt, and rejection of unrelated resource or read-lock errors.
Co-Authored-By: heihutu <heihutu@gmail.com>
Co-Authored-By: zhi22915 <qiuzgang@gmail.com>
---------
Co-authored-by: heihutu <heihutu@gmail.com>
Co-authored-by: zhi22915 <qiuzgang@gmail.com>
Add an opt-in `fips` Cargo feature to `rustfs` and `rustfs-tls-runtime`
that enables `rustls/fips` and `tokio-rustls/fips` (activating
`aws-lc-rs/fips` / `aws-lc-fips-sys` and `rustls-webpki/aws-lc-rs-fips`).
When compiled with `--features fips`, `startup_runtime_hooks` installs
`rustls::crypto::default_fips_provider()` as the process-wide default
crypto provider and asserts `provider.fips()` at startup.
Also lock `aws-lc-fips-sys v0.14.4` in `Cargo.lock`, allow the `OpenSSL`
license on `aws-lc-fips-sys` in `deny.toml`, and update
`docs/operations/kms-cryptographic-compliance.md` to document the scope
of `-p rustfs --features fips` vs. `rustfs-crypto`.
Co-authored-by: hector <42570491+majinghe@users.noreply.github.com>
Characterize how scanner metadata decoding and object listing handle a leading free version and a later delete marker.
Co-authored-by: heihutu <heihutu@gmail.com>
Co-authored-by: zhi22915 <qiuzgang@gmail.com>
Define the scope, authority, coverage, persistence, and resource guarantees needed before scanner-derived current-object statistics can be enabled.
Co-authored-by: heihutu <heihutu@gmail.com>
Co-authored-by: zhi22915 <qiuzgang@gmail.com>
* ci(validation): sweep the migrated canonical validation cases nightly
The canonical release-validation cases migrated into auto-testing by
rustfs/auto-testing#113 (66 implemented, all non-heal) currently run
only through the manual validation flow; the last fleet executions
stopped on 2026-10-27 and nothing schedules them. Add a scheduled
validation sweep that closes the gap:
- nightly at 19:30 UTC (after the functional chain settles), rotating
windows of the implemented catalog cases so the whole catalog is
covered roughly weekly, with a bounded batch per night
- workflow_dispatch for a single case (--case), one module (--module),
or an explicit batch (--limit), mirroring how the lanes are testable
standalone
- fleet admission through functional_environment enter/finish with the
same quarantine discipline as the lanes, inter-case node cleanup,
per-case logs + reports retained as a sweep artifact
- a markdown sweep report uploaded to rustfs/dashboard under
functional-reports/validation/ so the cases appear next to the lane
reports, plus an alert-on-failure job for scheduled runs
- heal.* cases stay on the interactive validation flow: many need the
reviewed native/tool collectors built by separate validation-tools
runs before fleet execution
The registry gains the workflow responsibility (monitor: workflow,
max-age 60h so the watchdog notices a dead sweep) and the wiring test
baseline count moves with it.
Verified locally: check_wiring, check_test_wiring, test_validation_registry
(9/9), test_functional_chain + test_functional_chain_health +
test_security_workflow (99/99).
* ci(validation): group the sweep report by catalog module
The validation cases cover distinct areas (pool lifecycle, ILM
expiration, bucket/site replication, kms, data services, identity,
s3 compatibility), so a flat case table hides the category breakdown.
Render one section per catalog module with its own case table and a
Subtotal line, and keep the single grand - Total:/- PASS:/- FAIL:
block for the dashboard overview parser. Subtotals deliberately use
a different bullet prefix so the overview regexes cannot mistake a
module subtotal for the grand total.
* ci(validation): sweep the eight planned cases whose executors exist
The planned catalog entries are not all unimplemented: all eight
(kms.sse-kms-round-trip, tiering.transition-and-restore,
events.delivery, audit.request-correlation, iam.policy-enforcement,
oidc.login-and-session, s3.basic-object-round-trip and
scanner.usage-convergence) have dedicated executors routed in
run_validation_case.sh and live PASS evidence recorded in the
migration map. Widen the sweep filter to include them: the rotating
nightly batch now covers 74 cases (66 implemented + these 8), and the
fresh evidence each sweep produces feeds their acceptance review.
heal.* procedure-ready cases remain excluded - their executors need
the reviewed native/tool collectors built by separate
validation-tools runs before fleet execution.
---------
Co-authored-by: Hauser Zhi <housemecn@gmail.com>
* perf(ecstore): fuse multipart commit, cache active upload heads, and scale fsync workers (#9077)
- Cache active multipart upload heads, committed part records, and per-part
transaction state in ACTIVE_MULTIPART_SOURCE_HEADS with invalidation on
mutation, retirement, abort, and delete paths.
- Fuse destination metadata preparation into MultipartFusedCommitOutcome,
coalesce prepared metadata writes before sync_data(), and gate direct
local part transaction settlement on disk.is_local().
- Retain quorum commit receipt publication before physical delete cleanup
using PREPARED_DELETE_RECEIPTS, batch TARGET and PREPARED syncs, and skip
redundant pre-reservation directory syncs for inline deletes.
- Scale DiskBlockingLane::Fsync to 8 workers per disk with idle_workers
accounting and bounded per-disk batch fairness so concurrent fsyncs
coalesce in the filesystem journal.
- Cache verified small stripe proof records in VERIFIED_STRIPE_RECORDS while
recomputing payload digests on every read.
Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
Copilot-Session: d85aaf47-2a4b-4c2c-97a3-dbe8c7717e2a
* fix(ecstore): align test/prod commit and cache paths and hold source lock across cache publication
* fix(ecstore): preserve fused commit cancellation/panic semantics and retire excess idle fsync workers
Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
Copilot-Session: d85aaf47-2a4b-4c2c-97a3-dbe8c7717e2a
---------
Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
Copilot-Session: d85aaf47-2a4b-4c2c-97a3-dbe8c7717e2a
Replace the immediate post-replacement leader identity comparison with a
polling stabilization loop (up to 20 x 500ms). A leader-lock refresh gap
during old-binary replacement can make leader() briefly observe a different
node; the loop waits for the measured new coordinator to be confirmed. A
persistent change to an old-binary node still fails with the observed
identity in the error message.
Removes the redundant immediate assertion that the stabilization loop
supersedes. Also consolidates the non-mixed leader/pid resolution to avoid
a double leader query.
Extend test_buffer_brotli_responses to verify body bytes are preserved
and Content-Length matches the exact body length. Add two new cases:
non-brotli responses pass through unmodified, and a body stream error
strips Content-Encoding so clients do not attempt to decode an empty
body as brotli (covers the fix from #9085).
backlog#2918 retires the usage floor as a quota proof. After dual-corruption
recovery has been attempted, an unreadable floor is reported and rebuilt
through a CAS overwrite with a bootstrap marker rather than stopping the
leader iteration for a supervisor retry. Transient I/O errors that prevent
reading the floor revision fall through to the existing stop-and-retry path.
The identity-less floor regression from rustfs/rustfs#8670 now verifies
automatic rebuild and recovery instead of bounded backoff.
When sequential test fixtures share the bootstrap InstanceContext, the
BucketMetadataSys stores a Weak<ECStore> to avoid a reference cycle. The
ExpiryState holds an Arc<ECStore> that keeps the store alive past the
test's own Arc drop; the next fixture's resize_workers releases it and
the weak goes dangling before init_bucket_metadata_sys installs a fresh
BucketMetadataSys. Any object_store() call in that window panicked.
Add Drop for ECStore that clears the context's BucketMetadataSys slot
when its weak handle is stale, so the next fixture always finds a clean
slot. Fixes the heal absence_receipt_regressions and dangling_grace
test failures under sequential execution.
When sequential test fixtures share the bootstrap InstanceContext, the
BucketMetadataSys stores a Weak<ECStore> to avoid a reference cycle. The
ExpiryState holds an Arc<ECStore> that keeps the store alive past the
test's own Arc drop; the next fixture's resize_workers releases it and
the weak goes dangling before init_bucket_metadata_sys installs a fresh
BucketMetadataSys. Any object_store() call in that window panicked.
Add Drop for ECStore that clears the context's BucketMetadataSys slot
when its weak handle is stale, so the next fixture always finds a clean
slot. Fixes the heal absence_receipt_regressions and dangling_grace
test failures under sequential execution.
* ci: gate the plan on a contract marker
A base planner predating the NIGHTLY_JOBS split still emits the old required
set, and the strict verifier would reject it. Select now stamps plan_contract=2
and the workflow refuses any plan without the current marker, falling back to
the full matrix, so a stale base planner can neither authorize the new narrower
selection nor force a job the new workflow no longer schedules.
* ci: rename the workspace lint-and-build job key and evidence path
The job key test-and-lint named a job that runs no tests: it does Clippy, the
test-binary archive, doctests and the log-rule anchors, while the tests live in
workspace-tests. Anyone reading needs: [test-and-lint] would conclude tests had
already gated that dependency. The key becomes workspace-lint-and-build to
match its display name, and the shared evidence directory becomes
artifacts/workspace so the path stops asserting the same thing. The aggregate's
display name stays Test and Lint because that is the live ruleset check
context. ci-gates.md also stops attributing the ECStore test run to the wrong
job.
Add a fail-closed release certification path so incomplete or substituted
evidence blocks every official publication exit. scripts/release/release_gate.py
validates the versioned release-validation plan, the candidate identity and the
G01 evidence-envelope set, then re-checks promoted bytes against the passed
list. build/docker/package/helm publication jobs require that certification and
stage artifacts before promoting identical bytes.
Connect Feature Boundaries, real io_uring and the rio-v2, Swift, SFTP and
HTTP/3 feature lanes no longer gate a merge. They run only in the scheduled
and manual full-validation runs, which report a regression within a day; on
the pull-request path they bought no earlier signal worth the runner
occupancy (Connect alone measured 53.5 median minutes and occasionally set
the wall). Workspace lint and tests, the short core contract, the debug
build, E2E smoke and the S3 lanes stay required on PRs and merge groups.
* ci: retain CI01 timing sample and fix the OIDC live gate tool
Record the measured PR feedback and runner occupancy for the CI plan's CI01
criterion, including the honest gap between the experimental targets and the
sampled medians, and replace the awscurl dependency in the live Keycloak gate
with the AWS CLI so the security suite runs on PEP 668 managed fleets.
* docs(testing): attach live run evidence to the event-drill index
Run cargo update --verbose and cargo upgrade --verbose twice from the
current main baseline. Upgrade the Tonic family to 0.14.9, Redis to 1.8.1,
sysinfo to 0.40.1, and hotpath to 0.30.2 with compatible transitive updates.
Synchronize the standalone fuzz lockfile and retain its PKCS#1 rc.4
resolution for compatibility with pinned RSA rc.18.
Co-authored-by: heihutu <heihutu@gmail.com>
Co-authored-by: zhi22915 <qiuzgang@gmail.com>
Co-authored-by: Chris An <anzhengchao@gmail.com>
When to_bytes fails in buffer_brotli_responses the error path replaced
the body with empty while keeping Content-Encoding: br, producing a
malformed response (empty body claiming brotli encoding). Strip the
header and log a warning so the degradation is visible and the client
does not attempt to decode a missing body.
tower_http's CompressionLayer streams brotli via Transfer-Encoding: chunked
without a Content-Length header. Firefox's incremental brotli decoder aborts
on this combination (NS_BINDING_ERROR), making the web console unusable in
Firefox. Chrome and Safari buffer the full body before decoding so they are
unaffected.
Add a thin middleware (buffer_brotli_responses) that runs after CompressionLayer,
collects the complete body for any brotli response that lacks Content-Length, and
re-emits it with the correct length header.
* feat(scanner): bound first-build enumeration with the cycle budget
Charge raw directory entries and their name bytes, and let one bucket scan hold a child budget that stops its own token while charging the cycle. A count budget also bounds one directory listing, so a wide first build can no longer finish a full read_dir before the first object is charged.
The new stop reason maps onto the persisted Directories variant so older readers keep decoding checkpoints. A directory with queued children keeps its index until they end, and every index on the active walk path is saved as an optional map field: losing either one would recharge its replay and stop a fixed budget from advancing.
* feat(scanner): keep bucket service order across restarts
Save each set's first-in-first-out bucket order as an optional map field of its usage cache, including after an incomplete cycle, which otherwise writes nothing. Durable age now outranks the dirty-first and leader-local cohort orders, which only break its ties, so a restart cannot hand the budget back to the same first buckets. Failed turns add visible debt without moving a bucket forward; a budget stop or a resource veto adds none. A bucket that spends its quantum and saves forward motion returns to the queue tail within the cycle.
The order authorizes nothing: another movement epoch or a corrupt value starts a new one, and a recreated bucket joins as a new arrival.
* test(scanner): cover quantum handoff and saved service order
Run the real worker queue with injected bucket shares and verify a spent bucket yields to waiters, a share without durable motion ends its turn, and the saved order keeps certified coverage while rejecting older owners.
* docs(scanner): describe budgeted first build and service order
* test(scanner): pin index retention and requeue motion
Bound the nested restart fixture tightly enough that dropping a directory index fails it, and cover a repeated share whose frontier never moves.
* fix(scanner): keep service generations and owner stops intact
Queue a new bucket behind waiters but ahead of buckets that already had a turn, matching the leader-local cohort, and keep the set cache untouched when the owner was cancelled so an interrupted cycle cannot advance its revision. The order holds one record per inventory bucket, like the set cache's entries, so a truncating bound that could starve the long-waiting buckets is removed. Publish the incomplete scope before saving so a contended lock cannot delay it.
* ci(e2e): refresh the full selection digests
* fix(scanner): correct a comment spelling
* fix(scanner): separate service debt from vetoes and stale owners
Keep a resource veto and a remote count-budget stop out of failure debt,
settle the claim before a requeue exposes the bucket to another worker, and
stamp the service order with its leader so an older owner cannot replace it
after an incomplete save. The checkpoint fixture's budget now covers both
service orders.
* fix(ecstore): create parents before resource-scoped writes
#9064 moved parent creation into the lazy writer, so a write that holds a
resource unit reached the native open without its parent directory and
failed with FileNotFound on nested paths.
* fix(e2e): retry SlowDown codes during upgrade write readiness
The rolling-upgrade readiness probe only retried 503 ServiceUnavailable,
so a transient 503 SlowDown from a still-recovering node failed the
four_node_rolling_upgrade test immediately. Accept SlowDown and
SlowDownRead alongside ServiceUnavailable, matching the existing chaos
test transient-error set.
Keep native publication recording in main's native_io helpers and resource
writers, and adapt merged tests to the owned Body, Bytes write input, and
fallible find_disk.
Record native publications on the paths main's commit-fusion work opened
directly: the lazy truncate writer and the fused delete-recovery activation
and receipt renames. Keep extract_params_header in the shared audit builder.
* feat(scanner): share node admission with background heal
* fix(scanner): drain candidates after listing completion
* fix(scanner): yield cache leases for shared node admission
* test(scanner): attest adaptive admission in process receipts
* fix(concurrency): declare Tokio time dependency
* feat(concurrency): retain bounded physical resource subsets
* fix(scanner): retain resource ownership in native filesystem jobs
* feat(concurrency): acquire physical groups before node work
* feat(scanner): gate background reads by physical device
* feat(concurrency): combine preallocated resource selections
* feat(scanner): share physical topology across disk plans
* fix(storage): retain only native disk resource subsets
* feat(storage): admit auxiliary physical resources without waiting
* feat(heal): admit physical resources before object repair
* fix(storage): retain resources during native format probes
* fix(storage): retain resources through direct writer I/O
* fix(storage): retain resources through buffered writer I/O
* fix(storage): retain physical resources through streamed reads
* fix(storage): retain resources through native filesystem jobs
* fix(storage): retain physical ownership during bitrot reads
* fix(ecstore): retain resources during positioned reads
* fix(ecstore): retain staged metadata write ownership
* fix(ecstore): retain native io_uring read ownership
* fix(ecstore): enforce receiver-side background admission
* fix(ecstore): retain resources through whole-file native I/O
* fix(ecstore): retain physical ownership during durable rename sync
* fix(storage): account for retained background stream buffers
* fix(storage): retain memory charges for background read results
* fix(storage): account for native io_uring read allocations
* fix(storage): retain memory charges for direct-write buffers
* fix(ecstore): account for heal and RPC reader buffers
* test(scanner): verify samples and clean concurrent collectors
* fix(storage): retain inline and HTTP writer memory reservations
* perf(rio): reuse reserved HTTP writer storage
* fix(rpc): reserve background scanner pipe storage
* perf(rpc): reuse reserved response reader storage
* feat(scanner): measure cache copy allocation requests
* fix(rpc): budget namespace scanner responses
* fix(scanner): retain the enabled supervisor task handle
* fix(scanner): budget namespace frame read staging
* fix(scanner): account namespace envelope decode storage
* fix(heal): retain and join the MRF consumer on shutdown
* test(heal): isolate process-global MRF consumer fixture
* fix(heal): join manager workers during shutdown
* test(heal): verify shutdown at native read boundaries
* fix(heal): drain channel commands before stopping MRF
* fix(heal): admit resume scans and retain replacement workers
* build: pin profiling sources and trust fixed ownership dependencies
* fix(heal): yield node admission while accepted native work drains
* fix(startup): retain shutdown work and preserve deadline failures
* fix(heal): admit inventory probes and retain replacement work
* fix(shutdown): preserve maintenance and frontend join failures
* fix(shutdown): retain HTTP child tasks through real retirement
* fix(shutdown): join actual TLS reload work before retirement
* test(multipart): preserve failures before fault barriers
* fix: keep physical admission inside repository guards
Route the multipart lock diagnostics filter through the e2e storage
compatibility module, reuse the parent module's s3s body aliases in the
namespace memory test, and share one Reed-Solomon verify error mapping
across the reserved-buffer and default paths.
* fix: reconcile physical admission with main's new call sites
Main added a source-admission metadata write layer, a CAS write fence,
fixture writers and scanner handoff tests against the old borrowed-slice
and node-permit shapes. Pass owned Bytes through the new metadata layer,
answer the scanner handoff with ScannerNodePermit, gate the test-only
disk lookup re-export, and clear the lint findings that stable 1.99
reports for the shutdown and admission code without suppressions.
* test(concurrency): name reversed attachment groups clearly
* fix(ecstore): box disk operation futures below admission layers
Every LocalDiskWrapper operation reaches the LocalDisk future through the
health, admission and timeout layers (track_cleanup_mutation, the
track_disk_health family, the io-permit and native-resource admission
split, and track_admitted_disk_operation). Each layer embedded that future,
and unoptimized builds copy it into every awaiting frame, so one local
delete cost about 1.3 MiB of stack and 16 tests overflowed libtest's
default 2 MiB stack.
Box the operation future once where track_admitted_disk_operation creates
it, through a non-async helper so the temporary never stays in a polled
frame. The layers above now hold only a pointer; the cost is one
allocation per local disk operation.
* fix(heal): box object heal futures below admission
The admission wrappers took the object heal future by value, so unoptimized
builds copied it into three awaiting frames, about 650 KiB per heal. Create
it through a non-async helper instead, leaving only a pointer in those frames.
* test(ecstore): count the I/O backend's root FD owner
* test(heal): isolate durable repair during root recovery
* test(heal): isolate the direct repair fallback
* fix(lock): use remaining acquire budget for retries
* test(lock): use virtual time for retry budget
* fix(lock): retain quorum failures across deadline retries
* test(heal): isolate multi-pool crash recovery
* fix(lock): retain quorum error on final timeout
* fix(heal): allow reads during read repair reconstruction
* fix(heal): retain deep scan retry for read repair
* test(heal): wait for metadata write tail
* perf(ecstore): fuse local PUT/DELETE commit I/O and eliminate redundant metadata disk reads (#9038)
- Fuse authoritative bucket metadata existence and parse reads via tokio::join!, add a LOCAL_META_MUTATION_GEN-validated local authority cache, and piggyback one-shot quota snapshot reads onto single-PUT Object Lock snapshot loads.
- Reuse prepared ObjectOptions in post-write lifecycle transition checks instead of re-reading bucket versioning and Object Lock config after commit.
- Defer temporary shard file open/mkdir/write into a single spawn_blocking call via LazyLocalFileWriter, pre-sync single-block shards before page-cache reclaim, and fuse rename_data preparation + commit into a single Fsync-lane dispatch on Linux.
- Enable rename_data old_current_source reuse by default across all-local sets, guard early-ack commit on quorum source stability, and skip redundant post-commit tmp_delete when all online local disks succeeded.
- Enable data-first batch_shard_pread when deferred parity handles are available and replace zero-filled allocation with Vec::with_capacity + read_to_end for offset-0 shard reads.
- Defer pre-delete get_object_info_for_delete to PrefixAccessDenied error handling, skip redundant single-pool unversioned delete pre-stats, and add a fused local delete preparation and 2-barrier settlement fast path.
Fixes#9038
Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
Copilot-Session: d85aaf47-2a4b-4c2c-97a3-dbe8c7717e2a
* fix(ecstore): preserve delete identity and durability policy
Co-Authored-By: heihutu <heihutu@gmail.com>
Co-Authored-By: zhi22915 <qiuzgang@gmail.com>
* test(ecstore): cover fused delete recovery stages
Co-Authored-By: heihutu <heihutu@gmail.com>
Co-Authored-By: zhi22915 <qiuzgang@gmail.com>
* fix(ecstore): preserve inline versions after commit sync failure
* perf(ecstore,s3): accelerate PutObjectLegalHold, CopyObject, and large PUT fdatasync (#9038)
- Fuse LocalDisk::update_metadata (read xl.meta + FileMeta::load + update_object_version_with_opts + marshal_msg + staged tmp write + rename_all_with_precondition_and_parent_sync_std) into a single Fsync-lane blocking operation, and skip redundant get_bucket_info / get_object_info calls in put_object_legal_hold when object lock is enabled and replication is not configured (PutObjectLegalHold: 314.8 -> 988.3 ops/s, 3.14x faster, 0.95x of MinIO).
- Pass destination bucket as orig_volume through create_bitrot_writer_with_orig_volume into LocalDisk::create_file so single-shot non-inline part.1 writes under syncs_data_shards() durability run fdatasync before fadvise(POSIX_FADV_DONTNEED) and skip the duplicate sync_dir_files_std(src_data_path) in FusedCommitAttempt without fsyncing the temporary parent directory (warp PUT 1MiB: 658.3 vs MinIO 656.8 obj/s, 1.00x parity, p50 26.10ms vs MinIO 46.60ms).
- Add single_pool() fast paths to ECStore::handle_copy_object and ECStore::handle_new_multipart_upload, skip no-op destination prelookup get_object_info when put_like_write_creates_new_version is true, and backfill previous_current_size from OldCurrentSize in copy_object (CopyObject 1KB: 398.4 -> 1,086.1 ops/s, 2.73x faster, 1.01x faster than MinIO).
Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
Copilot-Session: d85aaf47-2a4b-4c2c-97a3-dbe8c7717e2a
* fix(ecstore): provide non-Linux has_rename_data_test_failpoints stub for Windows CI
Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
Copilot-Session: d85aaf47-2a4b-4c2c-97a3-dbe8c7717e2a
* fix(ecstore): preserve presynced shard directory durability
Sync the shard directory before publishing presynced payloads and keep the file sync test probe available on non-Linux targets.
* fix(ecstore,s3): fix non-consuming crash_inject check, test hooks, and rubber-duck findings (#9038)
- Add non-consuming crash_inject::is_armed(point, key) so FusedCommitAttempt and LocalDisk::update_metadata do not consume one-shot crash arms before the fallback path runs (fixes crash_consistency::* and meta_write_crash_consistency::* tests).
- Check INLINE_PREPARATION_BEFORE_BACKUP and RENAME_DATA_AFTER_FIRST_PUBLICATION in has_rename_data_test_failpoints so inline preparation admission and observed-owner panic tests run on their expected path.
- Resolve staged_dir via get_object_path_for_io in test_rename_data_presynced_shard_keeps_directory_sync and avoid premature orig_volume access check during tmp staging in put_object (fixes put_object_failure_cleans_tmp_workspace_inline).
- Address rubber-duck review findings: fall back from FusedCommitAttempt on any existing xl.meta when sync_commit_metadata is true, fsync object dir after activating delete rollback dir before unlinking xl.meta, classify data-dir entries before removal in fused delete finalization, include self_copy_lock in copy_prelookup_required, and avoid .expect() on non-test mutexes.
Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
Copilot-Session: d85aaf47-2a4b-4c2c-97a3-dbe8c7717e2a
* fix(ecstore): defer shard sync until destination is known
* test(rustfs): cover legal hold replication fast path
Co-Authored-By: heihutu <heihutu@gmail.com>
Co-Authored-By: zhi22915 <qiuzgang@gmail.com>
---------
Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
Co-authored-by: Hauser <housemecn@gmail.com>
Co-authored-by: zhi22915 <qiuzgang@gmail.com>
Co-authored-by: Chris An <anzhengchao@gmail.com>
Co-authored-by: heihutu <heihutu@gmail.com>
Copilot-Session: d85aaf47-2a4b-4c2c-97a3-dbe8c7717e2a
* fix(s3select): match request enum values case-insensitively
ExpressionType, CompressionType, CSV FileHeaderInfo, JSON Type and CSV
QuoteFields were compared exactly, so lowercase values such as
`lines`, `document` or `gzip` were rejected. MinIO accepts any case.
Match them ASCII case-insensitively in request validation and rewrite
them to the canonical uppercase spelling, so every later comparison
keeps working unchanged. Unknown values keep their existing errors.
* fix(s3select): keep source key order for JSON LINES input
Arrow's JSON schema inference walks each record as a serde_json::Map,
which sorts keys unless serde_json's process-wide preserve_order
feature is on. JSON LINES `SELECT *` therefore returned keys sorted,
and CSV output columns were permuted.
Infer the LINES schema from the same object bytes and record limit as
DataFusion's JsonFormat, keep Arrow's types, and reorder fields
(nested structs included) by first-seen key order. Enabling
preserve_order globally was rejected: it changes map ordering for the
whole binary.
Drop the test-only preserve_order dependency so the crate's tests run
with production features; the DOCUMENT inference reference now
compares against Arrow's types in source order.
* fix(s3select): name nested path projections after their last key
`SELECT s.obj0.id` returned the key `s.obj0[id]`, DataFusion's display
name for the field access. AWS names an unaliased path projection after
its last key, e.g. `{"id": ...}`.
Alias such projections to their last key when that name is unique among
the output and ORDER BY column names. Queries with a wildcard keep
today's names, since the expanded columns could collide.
* fix(s3select): preserve names beside computed projections
* fix(s3select): preserve names for ambiguous path projections
* fix(s3select): name path projections from the planned schema
Path projections that the SQL rewrite leaves unaliased are now named
after their last key once DataFusion has planned the query. A path takes
its key only if no other output column keeps or receives that name. A
colliding rename is dropped and the check repeats, so the final names
are unique whenever DataFusion's own names are.
The SQL rewrite is unchanged, so GROUP BY can still refer to the names
it assigns and every query that planned before still plans.
Review cases:
- s.obj0.id, s.obj1.id, s.obj2."s.obj0[id]" keeps the three generated
names: the third key would duplicate the first retained name.
- s.obj."lower(s.id)", LOWER(s.id), in either order, keeps the generated
path name: its key is the computed column's name.
Paths beside computed columns, beside duplicated keys, and in
parentheses now take their key, as AWS S3 Select names them. Paths stay
unnamed when an ORDER BY expression is not projected, as before.
---------
Co-authored-by: Hauser <housemecn@gmail.com>
Co-authored-by: overtrue <anzhengchao@gmail.com>
Co-authored-by: GatewayJ <835269233@qq.com>
A request refused by the authorization layer was logged as
s3_authorization_denied but never reached the audit stream. Emit an audit
entry from DenialContext::deny with the account, bucket, object, denied
action and denial reason. Entries are bounded by a per-class token bucket
(lower for anonymous callers) with a suppressed count carried on the next
entry, quiet probe denials are skipped, DeleteObjects audits its refused
keys as one aggregate entry, and the S3 response is unchanged.
A replicated force-delete that a target rejected (for example a peer whose
credentials lack s3:ForceDeleteObject) was only logged; the per-target
replication counters never saw it, so `replication status` reported zero
failures while the sites had already diverged.
Build a ReplicatedTargetInfo for every target outcome of
replicate_force_delete_to_targets (missing client, offline target,
rejected remove, success) with a Pending previous status, and feed each
through ReplicationStats::update so rejections count as failures and
accepted force-deletes count as replicated.
Document that the managed site-replicator-0 policy must name
s3:ForceDeleteObject explicitly and that a bucket-replication target with
its own credentials needs the same grant.
* fix: preserve PUT lock failure diagnostics
Propagate the request operation into spawned PUT commit tasks so lock failures retain their metric attribution. Record bounded lock error classifications without exposing lock resources.
Co-Authored-By: heihutu <heihutu@gmail.com>
Co-Authored-By: zhi22915 <qiuzgang@gmail.com>
* test: remove redundant clone in PUT lock test
Co-Authored-By: heihutu <heihutu@gmail.com>
Co-Authored-By: zhi22915 <qiuzgang@gmail.com>
* fix: bound PUT commit context stack usage
Pin the commit future before wrapping it with request context to reduce stack pressure in PUT callers.
Co-Authored-By: heihutu <heihutu@gmail.com>
Co-Authored-By: zhi22915 <qiuzgang@gmail.com>
---------
Co-authored-by: heihutu <heihutu@gmail.com>
Co-authored-by: zhi22915 <qiuzgang@gmail.com>
* ci(ft): update the lane gate to the 54-case manifest with scenario F
auto-testing #118/#119 added FT scenario F and case B5, so the lane now
runs 54 cases over scenarios A,B,C,C2,D,E,F and passes 54/54 — but the
workflow gate still pinned the 44-case A-E manifest and failed the
report step. Align the gate and the contract-test fixture with the
54-case manifest.
* ci(heal): give heal a 90-minute run budget inside a 120-minute job
The heal task was still progressing (scanned 494 -> 571+, failed=0)
when the 45-minute step cap killed it, so the verification steps never
ran and the lane produced invalid evidence (chain 40796474356). The
step budget is now 90 minutes, and the job-level budget rises from 60
to 120 minutes so the enclosing job cannot cancel the raised step
budget. The timeout contract now records heal as the exception to the
shared 60/45 baseline.
* ci(chain): force the performance probe offline while pf-testing is broken
The pf-testing environment is temporarily broken, so every nightly
chain would end red in the performance lane. performance is
required=false in scheduled-validations.json, so forcing the probe's
performance_ready output to offline takes the chain through its
designed skip path: the lane job is skipped, aggregate receives
--allow-skipped performance, and the report records the optional-lane
skip with its reason. Marked TEMPORARY; restore the probe once the
environment is healthy again.
* test(table): drop a stale workflow-comment pin left by #9059#9059 rewrote the table suite step comments but kept the assertion for
the old '# Red only for harness/environment breakdowns' line, so
test_security_workflow.py fails against current main. The semantics
stay pinned by the continue-on-error and HARNESS_OK assertions.
* fix(audit): replay queued payloads for internal event names
Audit entries are persisted by the store targets and read back through
serde. Three internal events (CreateMultipartUpload, AbortMultipartUpload,
DeleteObjects) serialized to a string that EventName::deserialize rejected,
so every queued entry for them was dropped on replay. Read the persisted
event name through a lenient deserialize_with function; EventName's own
Deserialize and parse stay strict because they also back notification
configuration.
* fix(audit): emit entries for multipart parts and bucket config writes
UploadPart, UploadPartCopy, AbortMultipartUpload and the mutating bucket
configuration operations never produced an audit entry: the audit path
required a bucket notification event and these operations have none, or
(AbortMultipartUpload) never built a helper. Add an audit-only
OperationHelper constructor that builds no notification event and wrap
the S3 handlers for these operations with it. Operations without a
notification event use one coarse audit-only event name, with the
operation in api.name.
---------
Co-authored-by: Hauser Zhi <housemecn@gmail.com>
Since #9059 the shared test archives and debug binaries are named
with the run attempt, and the archive identity binds the attempt. A
"Re-run failed jobs" attempt does not rerun the successful producer,
so its consumers look for <name>-2, fail with "Artifact not found",
and every flaky test forces a full rerun.
Give those artifacts one name per run, overwrite them when a producer
does rerun, and bind the archive identity to the run instead of the
attempt. Source, tree, lock, toolchain and environment stay bound, and
an archive from another run is still rejected.
Sync main 84e24e767 into gateway/integration. Ports main-side additions to
the integration boundaries: the tier deletion diagnostics admin handlers move
to the owned admin request/response types, and the new lifecycle rule checks
in ecstore use the persisted lifecycle types instead of s3s DTOs. Keeps the
gateway git source in deny.toml while dropping the smithy-rs pin main retired,
and refreshes the test-count baseline for the merged tree.
Sync main 9601a7c4b into gateway/integration. Ports main-side additions to the
owned admin boundary (plugin delegation/attempt handlers, notification target
edit tests), drops a new ecstore s3s::dto reference in favour of the persisted
lifecycle filter, and refreshes the e2e selection digests and test-count
baseline for the merged tree.
main moved the local write path onto the disk blocking queue, which skips an
operation once its waiter is dropped, so a borrowed write cancelled while it is
queued no longer creates its file. The integration-only native publication
test still expected the queued open to run. Assert the new execution model
instead (the open never runs) while keeping its safety claims: the Started
record is taken before queueing and the cancelled publication never regrants
native history.
Source: main 4e48c390e7
Target: gateway/integration 1000860cb8
Conflicts resolved, keeping both sides:
- crates/ecstore/src/disk/{fs,local}.rs: main's batched rollback writes,
unified blocking write path and relocated open_file, with integration's
native publication records restored at each publication point and the
publication_history field kept beside main's multipart proof authority.
- crates/ecstore/src/set_disk/ops/multipart.rs: union of the imports.
- rustfs/src/storage/rpc/node_service.rs: main's MRF lifecycle reader
capability branch; the heal capability keeps integration's restore
copy-back probe.
- rustfs/src/admin/handlers/{audit,event,plugins_instances}.rs: main's
target_read inventory snapshot on integration's owned request boundary;
the new event.rs inventory test now builds owned requests.
- scripts/check_s3s_footprint.sh: integration's baseline-file version.
- .config/e2e-{full,smoke}-selection.txt: digests recomputed over the
union of both heads' CI listings (full 700 darwin / 708 linux, smoke 232).
Also migrated main's new rebalance delete-all expiry test to the persisted
lifecycle types integration uses in ecstore, and raised the test-count
baseline to the merged counts (every decrease comes from main).
`target_descriptor::build_json_response` returned the s3s response and
had no production caller left once the event, audit and plugin instance
handlers moved to `owned_build_json_response`; it survived only for the
legacy/owned equivalence test. Remove it, give the owned helper its own
documentation, and make the test assert the owned response directly:
status, `application/json`, the echoed `x-request-id` when present and
only then, and the body bytes. The file no longer imports s3s, so the
footprint baseline drops from 66 to 65 files.
* refactor(admin): migrate identity and cluster handlers to owned requests
Move the T1.12 batch 4 admin handlers (user, service_account, oidc, tier,
system, pools, quota, kms_dynamic, kms_backup, rebalance, module_switch,
bucket_meta, cluster_snapshot, realtime, bucket_recovery, profile) from the
s3s request/response types to the owned AdminRequest/AdminResponse
boundary. All 90 impl Operation blocks now take AdminRequest and return
AdminResponse; handler logic, authorization order, error codes and response
bytes are unchanged. The handlers use the shared owned helpers
(owned_json_response, owned_admin_json_response, authorize_profile_envelope,
the envelope-form runtime sources and audit constructors) instead of local
copies.
profile.rs drops the s3s forms of authorize_profile_request and
profile_not_implemented_response now that profile_admin.rs is on the
envelope gate too; profile_not_implemented_response answers with an
AdminResponse. The realtime metrics stream moves from the s3s ByteStream
adapter to Body::from_stream, and the rebalance idle-404 document still
renders through s3s::S3Error::to_http_response with its body converted at
the edge.
The idp_compat delegators ImportIamV2 and AddServiceAccountLdap move to the
owned trait as well: with their targets on both traits, the method-call
delegation became ambiguous between the owned impl and the router blanket
impl, so they now take the owned request directly.
KmsAdminAudit::from_request loses its last production caller with this batch
and is deleted; its comparison test now asserts the envelope facts directly.
Lower the s3s footprint baseline to the measured 68 files.
Part of rustfs/backlog#2788 (rustfs/backlog#2734 T1.12).
* refactor(admin): remove the legacy admin helpers batch 4 superseded
Batch 4 moved the last production callers of the s3s-typed admin helpers
to their owned forms, which left `admin_json_response`, `json_response`,
`authenticate_request`, `authorize_admin_request`, `object_store_from_req`
and `object_store_from_extensions` unused, along with the `Body`,
`S3Request` and `S3Response` re-exports of the admin `s3` facade. Remove
them. The owned forms keep their documentation; the behavior tests of the
removed `json_response` now exercise `owned_json_response`, and the
compat-prefix test asserts the owned response directly.
Two tests existed only to compare a removed legacy helper with its owned
twin (`owned_json_response_matches_the_legacy_shape`) or to cover the
removed legacy gate (`authorize_admin_request_without_credentials_is_rejected`);
they go with the helpers, and the owned equivalents stay. The rustfs
test-count row drops from 5957 to 5955 and the s3s footprint from 68 to 66
files.
* refactor(admin): complete the owned admin request boundary
Every type implementing the owned admin `Operation` now serves the legacy
router through one blanket impl that converts the s3s request and response
at that single edge, so handlers register with `AdminOperation(&Handler {})`
and need no adapter statics; `LegacyAdminAdapter` is removed.
`AdminRequest` gains the accessors handlers read today (`remote_addr`,
`client_info`, `request_context`, `req_info`, `req_info_mut`,
`install_req_info`, `envelope_mut`, `take_body`), the request helpers gain
envelope forms (`authenticate_envelope_credentials`,
`object_store_from_envelope`, the table-catalog `*_from_envelope` helpers,
`KmsAdminAudit::from_envelope`, `AccountAuditContext::from_envelope`,
`CallerIdentity::resolve_envelope`), `read_compatible_admin_body` accepts
either body type, and the owned `Body` gains `store_all_limited` and
`Default`. `admin::operation::test_support::TestRequest` builds owned
requests for handler tests. Unused accessors carry a per-method dead-code
expectation that each migrating handler batch removes.
No handler behavior changes; the remaining migrations follow in batches.
* chore(admin): add the T1.12 admin handler codemod
The script rewrites an admin handler file from the s3s request and response
types to the owned boundary (imports, signatures, request field and helper
calls, body access, response construction, and test request literals) and
leaves a TODO comment at every site it cannot rewrite mechanically. It is
idempotent and is removed once T1.12 completes.
* refactor(admin): migrate replication and KMS handlers to owned requests (#8951)
Move the T1.12 batch 2 admin handlers (site_replication, kms_keys, event,
replication, diagnostics, on_demand_migration, audit, account, kms_rekey,
kms_key_lifecycle, object_data_cache, tls_debug, integrity) from the s3s
request/response types to the owned AdminRequest/AdminResponse boundary.
Handler logic, authorization order, error codes and response bytes are
unchanged; the request/response plumbing now goes through the envelope
accessors and helpers introduced by the owned boundary.
The shared response helpers (utils::json_response, utils::empty_response,
handlers::admin_json_response, target_descriptor::build_json_response,
profile::authorize_profile_request) and the site replication repair
executor still answer with the s3s types, so the migrated files carry
owned-boundary forms of those helpers and convert the repair response at
its call edge. The streaming console-log body and the two devnull drains
move from the s3s byte stream to the owned Body stream/frame API.
Drop the dead_code expectations for the owned accessors and envelope
helpers this batch starts using, and lower the s3s footprint baseline to
the measured 99 files.
Part of rustfs/backlog#2788 (rustfs/backlog#2734 T1.12).
* refactor(admin): add owned forms of the shared admin response helpers
Handler batches were copying `json_response`, `empty_response`,
`admin_json_response`, and `build_json_response` into each migrated file
because the shared helpers still return the s3s response. Provide owned
forms next to each legacy helper (`owned_json_response`,
`owned_empty_response`, `owned_admin_json_response`,
`owned_build_json_response`, `authorize_profile_envelope`) with parity
tests against the legacy shape, derive `Debug` for `AdminResponse` so
tests can use `expect_err`, and fix three codemod rules: strip owned
string conversions in test credentials, annotate non-literal URI parses,
and leave a rebound request parameter immutable.
* refactor(admin): use the shared owned response helpers in batch 2 (#8954)
Replace the per-file owned-boundary copies of json_response,
empty_response, admin_json_response, build_json_response and
authorize_profile_request in the batch 2 admin handlers with the shared
owned_* helpers and authorize_profile_envelope that the boundary branch
now provides, and restore expect_err in the handler tests now that
AdminResponse implements Debug; the `.err().expect(..)` stand-ins were
failing the base branch's clippy step (clippy::err_expect under
-D warnings). The site replication repair call edge
keeps its conversion because the repair executor still answers with the
s3s response type.
Drop the dead_code expectations on the shared owned helpers this batch
now uses in non-test code.
Part of rustfs/backlog#2788 (rustfs/backlog#2734 T1.12).
* fix(admin): preserve legacy body limit error responses
* refactor(admin): move policy, ILM and batch 3 handlers to owned requests (#8956)
Batch 3 of T1.12 (rustfs/backlog#2788, parent rustfs/backlog#2734) moves
14 admin handler files to AdminRequest/AdminResponse and the owned
Operation trait: ilm_transition, policies, idp_compat, config_admin,
plugins_instances, profile_admin, group, heal, object_zip_download,
kms_key_metadata, durability, inspect_archive, sts and
gateway_key_inventory. Handler logic, authorization order, error codes
and response bytes are unchanged; test request literals use
test_support::TestRequest.
The handlers use the shared owned helpers (owned_json_response,
owned_admin_json_response, owned_build_json_response,
authorize_profile_envelope). The dead_code expectations on the
AdminRequest accessors and RequestEnvelope::uri_mut they now exercise
are removed. The legacy target_descriptor::build_json_response keeps
only its equivalence test caller and carries the matching expectation
until it retires.
object_zip_download builds the s3s request the S3 gate still reads from
the owned request facts; heal authenticates through
authenticate_envelope_credentials; the trace, inspect-archive and zip
bodies use Body::from_stream. extract_plugin_instance_filters takes the
URI so extensions.rs keeps calling it, and the two idp_compat
delegations stay on the legacy trait until their delegates migrate.
The s3s footprint baseline drops from 99 to 90 files.
* test(admin): anchor the replication diff body contract on the call
The source-contract test looked for `let body = read_compatible_admin_body(`
on one line; rustfmt wrapped that binding after the handler moved to the
owned request boundary, so the marker was no longer found. Anchor on the
call itself so formatting cannot change what the contract checks.
* refactor(admin): migrate table catalog and batch 1 to owned requests (#8955)
* fix(admin): preserve catalog body limit response bytes
Source: main 0f24511535
Target: gateway/integration 8cf6187ff4
Resolve the e2e-full selection digest conflict by recomputing both
platform digests over the merged nextest listing (685 darwin / 693
linux): main adds two rpc_fault_proxy unit tests, integration keeps its
dual-stack differential and anonymous-access additions. Raise the rustfs
test-count baseline row to the merged count (5932).
Brings in #8825, #8831, #8836 and #8839. Conflicts were import lists and test blocks (both sides kept), plus the new restore_copyback request flag next to main's data-movement field. Main replaced completed_multipart_object_part with prepare_complete_multipart_parts, so the stale import is dropped.
Main's xl.meta fast-path scanner (inspect_v2_object_slice) predates #8836 and classified a transitioned version with a pending v1 restore copyback as having no physical data dir, which made find_unshared_data_dir_for_version drop the copyback data dir. The scanner now reads the copyback marker with get_consistent_bytes semantics (both aliases, ASCII case-insensitive, empty or conflicting values count as absent), and uses_data_dir keeps main's allocation-free transition check plus #8836's copyback exclusion. find_unshared_data_dir_for_version_keeps_pending_copyback_data_dir failed on the plain merge and passes now.
Picks up main through 44a73b0be. Conflicts: rustfs/src/error.rs keeps the rustfs-s3-types ApiError conversion and drops the helper main's #8816 replaced; scripts/check_s3s_footprint.sh keeps the integration script, whose baseline lives in .config (main only retuned a constant the integration script does not have). Main's new tests use the rustfs-s3-types error code in admin replication and reach s3s::BodySizeLimitExceeded through the app's test-only s3 facade, so the s3s file count stays at 118; s3_error_lines tightens to 302. The test-count baseline is regenerated; the e2e selections are unchanged.
Brings in #8812 and #8818 from gateway/integration. #8812 predates main's heal error classifier (#8778); its two retryable variants (TooManyOpenFiles, OutdatedXLMeta) move into the classifier's transient arms for storage and disk errors, and its test is kept.
Since main's region-aware CreateBucket and GetBucketLocation (#8739, #8743), us-east-1 answers an empty LocationConstraint and 200 for an owned re-create, so the control no longer reported its own region. Also refresh the e2e-full linux selection from the CI listing (682 tests).
Picks up the 20 main commits since 62b6f5f3f. The one conflict is the tier free-version recovery test setup: it keeps the integration-only fixture isolation (background recovery disabled while the fixture starts) and main's mock tier registration (#8804 skips scans with no tier). The e2e selections are unchanged by these commits; the test-count baseline is regenerated.
The scheduled sync has stopped on conflicts since 2026-10-19. Resolved by keeping both sides: main's restructured delete-rollback path in disk/local.rs with the integration-only native-publication record, main's per-instance region with the rustfs-s3-types Region, main's new lifecycle, rebalance and admin code ported to the persisted lifecycle types and rustfs-s3-types errors, both sides' new scanner tests, and the union of the access_consumer imports. The e2e smoke, full (darwin) and repl-nightly selections and the test-count baseline are regenerated from the merged tree.
* refactor(s3select): run queries on an engine request built from gateway DTO shapes
Part of T1.5 (rustfs/backlog#2746), batch 2.
* fix(s3select): keep the select request conversion in the app layer
The layer guard forbids app -> storage; the conversion moves from storage/s3_compat/select.rs to app/select_request.rs unchanged.
* refactor(ecstore): move bucket configs onto gateway persisted types
Finish the ECStore batch of T1.4 (rustfs/backlog#2745). The remaining
eleven bucket configuration families (Website, CORS, Tagging, Versioning,
ObjectLock, Encryption, Notification, Logging, Accelerate, RequestPayment,
PublicAccessBlock) now read and write their stored XML through the
rustfs-gateway-types persistence and cors_tagging codecs instead of the
s3s codec, so the legacy s3s bucket-config codec and its
s3gate-metadata-xml compat marker are removed.
- ecstore: BucketConfigXml is implemented by the gateway codec for every
family; metadata, metadata_sys, versioning, object lock, replication and
lifecycle consumers take the persisted types. Region moves to
rustfs_s3_types::Region, and the ETag conversion to s3s moves to the
rustfs edge with its tests. Only warm_backend.rs and warm_backend_s3.rs
still reference s3s, held for T1.5 (rustfs/backlog#2746).
- scanner: object lock cache, item actions and data usage use the
persisted types.
- rustfs: the legacy S3 edge converts each family between s3s DTOs and the
persisted types in storage/storage_api.rs by writing one codec's XML and
reading it with the other; Tagging converts field by field.
- Bump the rustfs-gateway-types pin to 21283e87.
The 39 persisted-byte goldens are unchanged and pass, so a document the
old codec wrote reads back and re-encodes to the same bytes. s3s footprint:
files 181 -> 146, ecstore_files 28 -> 2, s3_error_lines unchanged.
* test(app): prepare the PutBucketEncryption config from the request
The uninitialized-store test passed an empty persisted configuration, which the use case now validates before it reaches the store check, so it answered MalformedXML instead of InternalError. Convert the request's configuration the way the S3 edge does.
* style(app): format the PutBucketEncryption test
* refactor(protocols): move error types to rustfs-s3-types by codemod
T1.2 batch 3 (rustfs/backlog#2743): s3select-api, s3select-query and
the error items of protocols move off s3s by scripts/codemods, run with
--hold crates/protocols/src/sftp/errors.rs. That file stays on s3s: it
downcasts the backend error, whose runtime type is still s3s::S3Error.
- rustfs/src/protocols/client.rs: the two webdav session methods now
return rustfs_s3_types::S3Result and convert the s3s error with From.
- codemod: a child module that calls s3_error! through `use super::*`
gets its own `use rustfs_s3_types::s3_error;` once its parent imports
that macro by name; otherwise it is listed as GLOB.
- s3-types: a table test pins code, status and the s3s error the legacy
edge renders for every variant a moved call site names.
- s3_error_lines 1585 -> 1584 (a lifecycle comment reworded).
* refactor(admin): move error types to rustfs-s3-types (WIP, no build)
T1.2 batch 4 (rustfs/backlog#2743), work in progress: rustfs check
still fails with 16 errors (listed in the PR body).
- codemod: --facade PATH treats a crate-local re-export of s3s's error
items like s3s for the four names (9 new self-tests, 9 mutants red).
- codemod output for rustfs/src/admin and rustfs/src/site_replication
(--facade crate::storage_api::site_replication::s3); 6 glob children
get an explicit rustfs_s3_types::s3_error import.
- admin/router.rs: the Operation trait returns rustfs_s3_types::S3Result.
- error.rs: From<ApiError> builds the rustfs-s3-types error; the s3s
conversion is derived from it.
* fix(admin): convert error types at the s3s boundary
Batch 4 of T1.2 left the admin and site_replication modules returning
rustfs_s3_types::S3Error while their callers and callees on the storage
and auth side still speak s3s::S3Error. Convert at each edge with the
existing From impls in rustfs-s3-types (compat_s3s.rs) instead of
changing any code, message or status:
- admin/auth.rs, admin/handlers/object_zip_download.rs,
admin/handlers/table_catalog/mod.rs: map the s3s result returned by
check_key_valid / authorize_request / authorize_internal_object_request
into the admin error type with .map_err(Into::into).
- storage/ecfs.rs: the s3s-facing side maps the admin site_replication
gate results the other way.
- admin/handlers/pools.rs: ApiError.code stays s3s::S3ErrorCode, so the
rustfs code is converted with .into() when the struct is built, in
production and in the two tests that build or compare it.
- admin/handlers/heal.rs, admin/runtime_sources.rs: the router returns
s3s errors (S3Route impl); the expected rustfs code is converted with
.into() before the comparison.
- admin/handlers/table_catalog/mod.rs: iceberg_rest_error takes
&'static str so the Cow<'static, str> custom code borrows it; every
caller already passes an ICEBERG_ERROR_* constant or a &'static str.
- admin/handlers/quota.rs: drop a now same-type .into() after s3_error!.
- storage_api.rs, site_replication/mod.rs: drop re-exports and an import
that no longer have users (clippy -D warnings).
cargo check -p rustfs --all-targets and cargo clippy -p rustfs
--all-targets -- -D warnings are clean; the s3s footprint baseline is
unchanged at files=172, s3_error_lines=303, ecstore_files=28.
Refs rustfs/backlog#2743, rustfs/backlog#2734.
Keep lifecycle rules, restore requests, Object Lock, and scanner cached tags in domain types. Convert at the existing HTTP boundary while preserving legacy XML, scanner cache bytes, date ordering, and tombstones.
Linux test validation before the final import-path-only adjustment:
cargo nextest run -p rustfs-ecstore bucket_config_goldens
39 passed; 6211 skipped by selector.
cargo nextest run --lib -p rustfs -E 'test(/test_zero_rule_lifecycle_tombstone_recognition/) | test(/test_local_lifecycle_staleness_axis_selection/) | test(/test_lifecycle_staleness_preserves_legacy_submillisecond_axis/) | test(/test_merge_incoming_lifecycle/) | test(/test_lifecycle_expiry_statement_matrix/) | test(/test_lifecycle_expiry_subset_xml/) | test(/put_bucket_lifecycle_validation_errors_keep_their_s3_code/) | test(/execute_put_bucket_lifecycle_configuration_rejects_missing_configuration/) | test(/execute_restore_object_rejects_missing_restore_request/) | test(/execute_restore_object_rejects_select_type/) | test(/execute_restore_object_maps_failures_to_typed_s3_errors/) | test(/execute_restore_object_returns_internal_error_when_store_uninitialized/) | test(/restore_request_conversion_preserves_legacy_output_values/) | test(/restore_request_conversion_preserves_presence_and_raw_enums/) | test(/restore_request_conversion_distinguishes_absent_and_empty_containers/)'
25 passed; 5318 skipped by selector.
cargo nextest run --profile ci -p rustfs-ecstore -p rustfs-lifecycle -p rustfs-replication -p rustfs-scanner
7872 passed; 21 skipped under the existing CI profile.
Previously verified mutation controls: 39 production faults, 41 RED/GREEN pairs.
cargo clippy --locked -p rustfs -p rustfs-ecstore -p rustfs-lifecycle -p rustfs-scanner --all-targets -- -D warnings
Passed in 40m 45s before the import-path adjustment, then in 1m 59s on
the final source with warnings denied. The final adjustment imports the
existing helper through the object domain facade; fmt and the architecture
check also passed.
Part of rustfs/backlog#2745. The remaining ecstore configuration families follow separately.
Keep the migrated compatibility adapters in their active modules while bringing in upstream fixes. Preserve native publication history on the new create, rename and borrowed-open paths.
Validation:
- `cargo nextest run --locked -p rustfs-ecstore --lib -E 'test(disk::os::native_creation_history_tests::) | test(disk::local::test::borrowed_open_cancellation_retains_started_history_before_execution) | test(disk::local::test::owned_direct_child_open_attempt_consumes_native_history)' --no-fail-fast`: 9/9 passed after restoring all 12 mutation cases at cea78f2590645eb0d303cd7cf56f4dd0d9c00e87; these source blobs are unchanged in this merge.
- `cargo nextest run --locked -p rustfs-s3select-query --lib -E 'test(dispatcher::json_document_schema::tests::)' --no-fail-fast`: 5/5 passed at 5421ba682d1329cc7ebeaac7c6a863da2cea1996, with the same tree.
- `cargo fmt --all --check`: passed.
- `./scripts/check_test_count_ratchet.sh --update`: 52 crates, 22007 tests, 165 ignored.
- Profile baselines use seven actual Darwin listings and seven actual Linux listings.
- Earlier conflict snapshot f5e0cdf0c58aed1ebe53b9243cf52f94832ab049: `cargo nextest run --locked -p rustfs-ecstore -p rustfs --lib -E 'test(disk::os::) | test(bucket::metadata_sys::) | test(server::legacy_compat::) | test(server::layer::tests::request_logging_watchdog) | test(server::http::tests::conditional_object_http::) | test(storage::helper::tests::operation_helper)' --no-fail-fast` passed 218/218 before the borrowed-open fix. This is historical evidence.
* wip: T1.4 replication family
* refactor(replication): move the replication config onto gateway types
The replication configuration family leaves s3s: rustfs-replication no longer depends on s3s, and ECStore bucket metadata, the scanner and the rustfs app/admin layers hold rustfs_gateway_types::persistence::PersistedReplicationConfiguration. The persisted XML goes through the gateway persistence codec behind a new BucketConfigXml trait (the 39 byte goldens pass with unchanged fixtures); the legacy s3s edge converts once in storage/ecfs.rs through storage/s3_compat/replication.rs; the scanner data usage cache keeps its MessagePack bytes through a serde mirror pinned by bytes recorded from s3s 5761ddfe. rustfs-gateway-types 0.32.0 is pinned once (rev 772d3385). Refs rustfs/backlog#2745.
* test: use S3 serializer for list response tests
Merge frozen main acd26afd8b and include
compatibility fixes and measured Darwin/Linux membership corrections
across six files.
Validation on candidate b4c3ab560a:
- All 20 local gates passed.
- Native smoke: 218 passed. Native full: 661 passed.
- Full passed on the same-head failed-only rerun; the initial failure
remains recorded.
- Gateway remains report-only: 38 passed, 180 failed, under the existing
exemption.
CI: https://github.com/rustfs/rustfs/actions/runs/38285788334
Run the three rollback metadata-cleanup cases in separate test processes so reused /proc/self/fd aliases from earlier cases cannot invalidate their creation witnesses. Keep the original fixture, guards, assertions and io_uring test selection, and verify that each child runs the intended case successfully.
Update the generated test-count baseline for the added macro arm.
Validation: cargo check -p rustfs-ecstore --tests; cargo fmt --all --check; all 19 quick-check commands, including make script-tests and the two Python checks. Linux io_uring acceptance and fault controls remain pending.
Keep the guarded rollback future behind a heap allocation so bucket creation does not store its nested state inline. The rollback body, all three fence references and pending-proof revocation order are unchanged.
The same three original Linux tests overflowed on both e611 CI attempts. Local format, ecstore test compilation and all 19 quick-check steps pass; the original Linux feature lanes still need to validate this candidate.
* test(rpc): pin the wire bytes of every internode rpc response body
rustfs/backlog#2747 batch 4, step 1 of 2, on the s3s bodies the rpc paths
serve today. Each response body builder in storage/rpc/http_service.rs is
served through hyper's HTTP/1.1 server and the raw bytes the peer reads are
compared with goldens captured from this commit: status, content-length or
chunked framing, chunk boundaries, the terminating chunk, HEAD, and the cut
connection a failure after data produces. The capability responses carry a
per-process proof, so their length header and payload are checked against
the body they encode instead.
The read_file_stream response builder moves into read_file_stream_response
so the test reaches the same code the handler runs; behaviour is unchanged.
* refactor(rpc): serve internode rpc bodies with the owned Body
rustfs/backlog#2747 batch 4, step 2 of 2. The internode rpc HTTP paths
build their responses with rustfs_s3_types::Body: Body::from_stream
replaces Body::from(StreamingBlob::wrap(..)) at the six stream sites, and
the in-memory and empty bodies keep their constructors. InternodeRpcService
converts its responses into the wrapped service's response body through
From<Body>, so the server keeps composing it with the s3s service as
before. The wire-byte goldens from the previous commit pass unchanged;
storage/rpc no longer names s3s.
* fix(rpc): simplify wire test response builder type
Name the boxed response builder used by the capability wire test so
all-target Clippy can check it without a type-complexity warning.
Validation (LANG=en_US.UTF-8, CARGO_INCREMENTAL=0):
- cargo clippy -p rustfs --all-targets -- -D warnings: exit 0.
- cargo nextest run --profile ci -p rustfs --lib -E 'test(/storage::rpc::/)': 279 tests run, 279 passed, 4842 skipped.
- All 19 quick-check commands, including script-tests, both Python scripts and cargo fmt --all --check: exit 0.
- Footprint baseline regenerated: files=194, s3_error_lines=1584, ecstore_files=36; test-count baseline regenerated: 52 crates, 21228 tests, 164 ignored.
- All 54 Cargo.toml files parse without duplicate keys.
No assertion was added or changed by this lint fix. R1-R8 are retained
in the existing PR verification record.
* fix(ecstore): resume fenced Object Lock creation intents
* fix(ecstore): bound creation retry futures on the stack
* fix(ecstore): require native proof for creation retries
The fixture constructor can take the production recovery lock and wait for 60 seconds before this test installs its queue and seeds metadata. Disable its background recovery only while creating this fixture, then restore the environment before calling the existing production recovery entrypoint.
Only the existing test setup changes. Its production entrypoint, default lock parameters, original assertions, 30-second timeout, cancellation and cleanup remain intact. No tests or assertions were added. Production and store/init.rs are byte-identical to the parent.
Independent acceptance ran on merge tree 8ec358cb42ba9ebb3bcfeafcef048a3f0366355b; the runner header reports parent HEAD, while before/after index and source hashes confirm this candidate. Worker task branch d8055766b8 has the same tree.
Every command used LANG=en_US.UTF-8. Task compilation also used CARGO_BUILD_JOBS=1, CARGO_INCREMENTAL=0, CARGO_NET_OFFLINE=true, an independent target, and empty RUSTC_WRAPPER/RUSTC_WORKSPACE_WRAPPER to keep compilers in its owned process group without changing repository configuration.
Command:
shellcheck --version && actionlint
Exit 0; observed output excerpt:
version: 0.11.0
Log SHA-256: fc9eec16b4191d058a295813ca9afbe9d2b37b59975da860170726be162a3924
Command:
cargo fmt --all --check
Exit 0; observed output excerpt:
(no stdout or stderr)
Log SHA-256: e3b0c44298fc1c149afbf4c8996fb92427ae41e4649b934ca495991b7852b855
Command:
./scripts/check_unsafe_code_allowances.sh
Exit 0; observed output excerpt:
(no stdout or stderr)
Log SHA-256: e3b0c44298fc1c149afbf4c8996fb92427ae41e4649b934ca495991b7852b855
Command:
./scripts/check_layer_dependencies.sh
Exit 0; observed output excerpt:
Layer dependency guard passed (no new reverse dependencies/cycles).
Log SHA-256: 69c6cb1896c0586b28d1167380d3260d99d8017a4265c8df5a2348f289e93a12
Command:
./scripts/check_architecture_migration_rules.sh
Exit 0; observed output excerpt:
Architecture migration rules passed.
Log SHA-256: 28790e70d070607ed4de0704f292a65030be92ec224ef106d4b72fd8296f896e
Command:
./scripts/check_logging_guardrails.sh
Exit 0; observed output excerpt:
✅ Logging guardrails check passed
Log SHA-256: 7adf41abff8ed8f7617decf86498888ddde78937099dd9ce47f6d359dfc2d317
Command:
./scripts/check_error_other_format_ratchet.sh
Exit 0; observed output excerpt:
error(format!) ratchet passed (total call sites: 579).
Log SHA-256: 66172aef18f150ba0605af122e26ffcf91f6dd86f43e38b05e2e982ce8d783ee
Command:
./scripts/check_no_tokio_io_uring.sh
Exit 0; observed output excerpt:
(no stdout or stderr)
Log SHA-256: e3b0c44298fc1c149afbf4c8996fb92427ae41e4649b934ca495991b7852b855
Command:
./scripts/check_extension_schema_boundaries.sh
Exit 0; observed output excerpt:
Extension schema boundaries passed.
Log SHA-256: f13d0acd0e2ce5d9815241b12b4dab7d01cc11b9b65bcf6dfb7812c878532a51
Command:
./scripts/check_body_cache_whitelist.sh
Exit 0; observed output excerpt:
Body-cache whitelist guard passed.
Log SHA-256: 6b828847b0383950ee6eef9cd846d0c322c945e47efd4c7d1ad881178810d6bb
Command:
./scripts/check_s3s_footprint.sh
Exit 0; observed output excerpt:
s3s footprint OK: files importing s3s is 195 (baseline files=195)
s3s footprint OK: s3s s3_error! lines is 1584 (baseline s3_error_lines=1584)
s3s footprint OK: ecstore files referencing s3s is 36 (baseline ecstore_files=36)
✅ s3s footprint baseline check passed
Log SHA-256: 47f53d04ef645618563fe1f22abc64667f19078f1974feedcca46e4f2c016844
Command:
./scripts/check_test_count_ratchet.sh
Exit 0; observed output excerpt:
test-count ratchet OK: 52 crates, 21340 tests, 164 ignored match .config/test-count-baseline.txt
Log SHA-256: 618a5e330e933de0c6ff484546df401457024a88bb5986d11d588794e33dab60
Command:
./scripts/check_fips_wording.sh
Exit 0; observed output excerpt:
FIPS wording guard passed (README.md and CHANGELOG.md contain no forbidden claims).
KMS wording guard passed (crates/kms claims no Vault KV2 Transit wrapping).
Log SHA-256: 99221cc9979c480ec4f46f50d76c52b18c72fe499f12e15e4f4a22a1b8bfb140
Command:
./scripts/check_embedded_secrets.sh
Exit 0; observed output excerpt:
Embedded secret guard passed (no private key material or provider credential literal in the tree).
Log SHA-256: 437a3ceba32094d2901889965ad191e411aad32cdb18bf0b8659e818011c406b
Command:
make script-tests
Exit 0; observed output excerpt:
test_cli_requires_the_repository_default_branch (__main__.SelfTests.test_cli_requires_the_repository_default_branch) ... ok
test_config_rejects_duplicate_and_invalid_entries (__main__.SelfTests.test_config_rejects_duplicate_and_invalid_entries) ... ok
test_freshness_boundaries (__main__.SelfTests.test_freshness_boundaries) ... ok
test_grace_requires_two_successful_queries_with_no_history (__main__.SelfTests.test_grace_requires_two_successful_queries_with_no_history) ... ok
test_invalid_api_evidence_fails_closed (__main__.SelfTests.test_invalid_api_evidence_fails_closed) ... ok
test_report_retains_every_workflow (__main__.SelfTests.test_report_retains_every_workflow) ... ok
test_requests_filter_schedule_default_branch_and_success_on_server (__main__.SelfTests.test_requests_filter_schedule_default_branch_and_success_on_server) ... ok
Ran 10 tests in 0.042s
OK
Ran 30 tests in 20.627s
OK
Ran 10 tests in 2.268s
OK
Ran 33 tests in 0.127s
OK
Ran 7 tests in 0.002s
OK
Ran 7 tests in 0.001s
OK
Ran 4 tests in 0.108s
OK
Ran 3 tests in 0.564s
OK
PASS: cold stampede script self-test
Log SHA-256: 75fda2d069a373c63667b908baaf99e6591aab9099a739b0c4c22ceb6da8d68c
Command:
python3 ./scripts/test_object_generation_protocol_model.py
Exit 0; observed output excerpt:
test_an_unavailable_decision_quorum_cannot_choose_a_candidate (__main__.ObjectGenerationProtocolModel.test_an_unavailable_decision_quorum_cannot_choose_a_candidate) ... ok
test_counter_allocation_requires_persistence_and_is_consumed_once (__main__.ObjectGenerationProtocolModel.test_counter_allocation_requires_persistence_and_is_consumed_once) ... ok
test_counter_restart_burns_persisted_allocations_before_or_after_issue (__main__.ObjectGenerationProtocolModel.test_counter_restart_burns_persisted_allocations_before_or_after_issue) ... ok
test_crash_after_promise_before_reply_does_not_restore_an_older_ballot (__main__.ObjectGenerationProtocolModel.test_crash_after_promise_before_reply_does_not_restore_an_older_ballot) ... ok
test_duplicate_or_unconfigured_voters_cannot_form_a_quorum (__main__.ObjectGenerationProtocolModel.test_duplicate_or_unconfigured_voters_cannot_form_a_quorum) ... ok
test_empty_or_unobserved_minority_slot_can_retry_without_allocating_another_revision (__main__.ObjectGenerationProtocolModel.test_empty_or_unobserved_minority_slot_can_retry_without_allocating_another_revision) ... ok
test_equal_ballot_cannot_accept_different_bytes_and_lower_ballot_is_rejected (__main__.ObjectGenerationProtocolModel.test_equal_ballot_cannot_accept_different_bytes_and_lower_ballot_is_rejected) ... ok
test_forgetting_tombstone_authority_allows_an_aba_create (__main__.ObjectGenerationProtocolModel.test_forgetting_tombstone_authority_allows_an_aba_create) ... ok
test_issuing_before_persistence_reuses_a_ballot_after_restart (__main__.ObjectGenerationProtocolModel.test_issuing_before_persistence_reuses_a_ballot_after_restart) ... ok
test_lost_ack_and_restart_preserve_the_chosen_operation (__main__.ObjectGenerationProtocolModel.test_lost_ack_and_restart_preserve_the_chosen_operation) ... ok
test_minority_accepted_work_is_adopted_and_finished (__main__.ObjectGenerationProtocolModel.test_minority_accepted_work_is_adopted_and_finished) ... ok
test_omitting_highest_accepted_adoption_has_a_double_choice_counterexample (__main__.ObjectGenerationProtocolModel.test_omitting_highest_accepted_adoption_has_a_double_choice_counterexample) ... ok
test_quorum_choice_does_not_require_an_uncontacted_disk_to_learn_it (__main__.ObjectGenerationProtocolModel.test_quorum_choice_does_not_require_an_uncontacted_disk_to_learn_it) ... ok
test_recovery_failure_or_stale_expected_head_does_not_allocate_a_successor (__main__.ObjectGenerationProtocolModel.test_recovery_failure_or_stale_expected_head_does_not_allocate_a_successor) ... ok
test_replacing_voters_with_empty_state_can_choose_a_conflicting_value (__main__.ObjectGenerationProtocolModel.test_replacing_voters_with_empty_state_can_choose_a_conflicting_value) ... ok
test_skipping_predecessor_recovery_chooses_a_broken_revision_chain (__main__.ObjectGenerationProtocolModel.test_skipping_predecessor_recovery_chooses_a_broken_revision_chain) ... ok
test_stale_publication_rollback_and_gc_cannot_change_the_winner (__main__.ObjectGenerationProtocolModel.test_stale_publication_rollback_and_gc_cannot_change_the_winner) ... ok
test_superseded_operation_identity_cannot_be_reused_for_different_input (__main__.ObjectGenerationProtocolModel.test_superseded_operation_identity_cannot_be_reused_for_different_input) ... ok
test_superseded_operation_retry_returns_its_original_successor (__main__.ObjectGenerationProtocolModel.test_superseded_operation_retry_returns_its_original_successor) ... ok
test_tombstone_and_null_marker_payloads_do_not_reset_authority_revision (__main__.ObjectGenerationProtocolModel.test_tombstone_and_null_marker_payloads_do_not_reset_authority_revision) ... ok
test_tombstone_rejects_a_delayed_never_present_create (__main__.ObjectGenerationProtocolModel.test_tombstone_rejects_a_delayed_never_present_create) ... ok
test_unlearned_predecessor_is_recovered_before_the_second_slot (__main__.ObjectGenerationProtocolModel.test_unlearned_predecessor_is_recovered_before_the_second_slot) ... ok
Ran 24 tests in 0.929s
OK
Log SHA-256: 0b4be926efb0609d497ae58930340153f01957b606206cbbdb3e0d47eb29d3a2
Command:
python3 ./scripts/check_test_wiring.py
Exit 0; observed output excerpt:
OK: e2e modules, runner selection, fuzz matrices, profiles, scheduled alerts, and Quick Checks are wired
Log SHA-256: 21f676f458451525535ab0ef73d6a9c0d9774e16d76a4d19e396ad34936dd39c
Command:
./scripts/check_no_planning_docs.sh
Exit 0; observed output excerpt:
(no stdout or stderr)
Log SHA-256: e3b0c44298fc1c149afbf4c8996fb92427ae41e4649b934ca495991b7852b855
Command:
./scripts/check_uring_lane_lib_only.sh
Exit 0; observed output excerpt:
OK: no 'uring_'-matching test functions under crates/ecstore/tests; --lib is safe
Log SHA-256: 43f3f851e27ec2ab9e34d6b05fefdede38af992699c473904bfe6f53a46d52e3
Command:
cargo nextest run --profile ci -p rustfs-ecstore --lib --features rio-v2 -E 'test(=bucket::lifecycle::bucket_lifecycle_ops::tests::tier_free_version_recovery_production_entrypoint_enqueues_seeded_item)' --no-fail-fast
Exit 0; wall 234.399s.
Observed output:
Finished `test` profile [unoptimized + debuginfo] target(s) in 3m 49s
Starting 1 test across 1 binary (5871 tests skipped)
SETUP PASS [ 0.006s] ecstore-base-stack: sh -c 'echo RUST_MIN_STACK=4194304 >> "$NEXTEST_ENV"'
PASS [ 0.440s] rustfs-ecstore bucket::lifecycle::bucket_lifecycle_ops::tests::tier_free_version_recovery_production_entrypoint_enqueues_seeded_item
Summary [ 0.586s] 1 test run: 1 passed, 5871 skipped
Log SHA-256: 34e973f86e8dc2102c1a0e483e0ac802ce7df3d1b40a603932f62c0805c4de27
Command:
cargo nextest run --profile ci -p rustfs-ecstore --lib --features rio-v2 -E 'test(=bucket::lifecycle::bucket_lifecycle_ops::tests::tier_free_version_recovery_eight_coordinators_hold_one_leader_and_take_over) | test(=store::bucket::tests::creation_commit_backfill_runs_under_the_config_write_fence) | test(=store::bucket::tests::orphaned_bucket_recovery_requires_every_volume_absent_and_is_retryable)' --no-fail-fast
Exit 0; wall 4.010s.
Observed output:
Finished `test` profile [unoptimized + debuginfo] target(s) in 0.43s
Starting 3 tests across 1 binary (5869 tests skipped)
SETUP PASS [ 0.005s] ecstore-base-stack: sh -c 'echo RUST_MIN_STACK=4194304 >> "$NEXTEST_ENV"'
PASS [ 0.068s] rustfs-ecstore bucket::lifecycle::bucket_lifecycle_ops::tests::tier_free_version_recovery_eight_coordinators_hold_one_leader_and_take_over
PASS [ 0.637s] rustfs-ecstore store::bucket::tests::creation_commit_backfill_runs_under_the_config_write_fence
PASS [ 0.846s] rustfs-ecstore store::bucket::tests::orphaned_bucket_recovery_requires_every_volume_absent_and_is_retryable
Summary [ 0.987s] 3 tests run: 3 passed, 5869 skipped
Log SHA-256: b5da2a927ff18ca9aa217762f7186a6d4371bd9e8feedd7b77940e10b64dffac
Command:
cargo clippy -p rustfs -p rustfs-ecstore --all-targets --features rio-v2 -- -D warnings
Exit 0; wall 1084.508s.
Observed output:
Finished `dev` profile [unoptimized + debuginfo] target(s) in 18m 03s
Log SHA-256: 28f98be7970faaa71cd118da1da4e6e2e3d23b9dda7e6449eaa7a37e741260b5
Compiled mutation command (unchanged for each red and restored run):
cargo nextest run --profile ci -p rustfs-ecstore --lib --features rio-v2 -E 'test(=bucket::lifecycle::bucket_lifecycle_ops::tests::tier_free_version_recovery_production_entrypoint_enqueues_seeded_item)' --no-fail-fast
M1: reverse only this test-local constructor flag from false to true. The mutant compiled, then failed the original recovered-item queue assertion after 30.481s. Restoring the entire observed candidate passed in 0.460s. Red log 55bf3b1535529ac0bcfb8071f9b56f9f40631007a85c315367e7e93786d65293; restored log 204d5fbe4574a7917f5e740a795d864fe28a7e7fb5e33d27b0a9cf68e2f17f52.
M2: replace only the actual awaited enqueue with false, consuming its original inputs. The mutant compiled, then failed the same original queue assertion after 30.516s. Restoring the entire observed candidate passed in 0.457s. Red log 89186330413215e231e2b06bf013cc31097ea8f4a106d91421a36d6c8986a7fd; restored log e848b42d2302401afa5dbcd9e37bf76d2e56a96ab6de5a488513b997f53c5e9b.
All temporary observation and mutation code was removed before final acceptance. Historical baseline and diagnostic failures remain recorded; they are not mutation credit. The separate Object Lock intent-retry diagnostic still fails, and these native macOS checks do not establish Linux CI recovery.
Pass the capture-free pending future factory as a function item in the existing slow-operation test. This fixes the redundant_closure error at disk_store.rs:2914 that stopped strict Clippy before the feature tests could run.
The single-line test edit preserves the factory's result, warning and timeout behavior. Reversing the replacement reproduces the parent file byte for byte. All 220 assertion lines and the production prefix are unchanged; no tests, assertions, guards, profiles or baselines were added or relaxed.
Original Linux CI evidence (run 37888552268 at 53245be9f):
- RIO job113685483992: strict Clippy exit101, redundant closure at disk_store.rs:2914; feature tests skipped.
- Workspace Test and Lint job113685484058: same strict Clippy exit101; nextest and doc tests skipped.
These are lint failures, not runtime mutation kills or known flakes.
Native macOS arm64 validation on tree7a5bf47e7533b29479bc85207416102f1337c566, LANG=en_US.UTF-8, one build job, CARGO_INCREMENTAL=0 and offline dependencies:
shellcheck --version && actionlint -> exit 0
cargo fmt --all --check -> exit 0
./scripts/check_unsafe_code_allowances.sh -> exit 0
./scripts/check_layer_dependencies.sh -> exit 0
./scripts/check_architecture_migration_rules.sh -> exit 0
./scripts/check_logging_guardrails.sh -> exit 0
./scripts/check_error_other_format_ratchet.sh -> exit 0
./scripts/check_no_tokio_io_uring.sh -> exit 0
./scripts/check_extension_schema_boundaries.sh -> exit 0
./scripts/check_body_cache_whitelist.sh -> exit 0
./scripts/check_s3s_footprint.sh -> exit 0
./scripts/check_test_count_ratchet.sh -> exit 0
./scripts/check_fips_wording.sh -> exit 0
./scripts/check_embedded_secrets.sh -> exit 0
make script-tests -> exit 0
python3 ./scripts/test_object_generation_protocol_model.py -> exit 0
python3 ./scripts/check_test_wiring.py -> exit 0
./scripts/check_no_planning_docs.sh -> exit 0
./scripts/check_uring_lane_lib_only.sh -> exit 0
All 19 checks: exit0, 312.161s.
$ cargo clippy -p rustfs -p rustfs-ecstore --all-targets --features rio-v2 -- -D warnings
Finished `dev` profile [unoptimized + debuginfo] target(s) in 20m 11s
exit0, 1213.210s total.
$ cargo nextest run --profile ci -p rustfs-ecstore --features rio-v2 slow_disk_operation_warns_once_while_pending_and_preserves_result --no-fail-fast
Finished `test` profile [unoptimized + debuginfo] target(s) in 4m 34s
PASS [0.026s] rustfs-ecstore disk::disk_store::tests::slow_disk_operation_warns_once_while_pending_and_preserves_result
Summary [ 0.170s] 1 test run: 1 passed, 5943 skipped
exit0, 281.920s total; 5943 existing filter skips.
warning: linker stderr: ld: __eh_frame section too large (max 16MB) to encode dwarf unwind offsets in compact unwind table, performance of exception handling might be affected
This linker warning is retained; flags, profiles and allowances are unchanged.
54 tracked Cargo.toml files parsed without duplicate keys. git diff --check passed. No new assertions were introduced, so no new runtime mutation is claimed. Native compilation and the existing helper test do not substitute for actual Linux RIO execution; the new integration CI run still needs to recover before the RPC merge.
Preservation and output hashes:
parent file SHA256: 8e59ee745e3aaaa22848b943d3b10f6353b515956282cde1f3af2c394200c9ac
final file SHA256: b69ba3237061b161448ae8b62b615885d5770b98a5b4a2479bcfbd70f1bd1f17
unchanged production prefix SHA256: 7903a6e288252c420231242d7b58a25fa2a7140ea3dd430cbe7f0dde11369dcb
RIO Linux failure log SHA256:421addb9868fd88e4109a922311e54f91f0dfd70862f8ba7fcaba193482ed90b
Workspace Linux failure log SHA256:e0aea8d0f014c022252a9534968e083c98cfc1fd760da445ecd04ec3f522e26a
19 output SHA256:35a54f28d707956ed1edb29d492c4dfe8a9fe753d103b111b5a2f1b5c41c2b98
RIO Clippy output SHA256:d6046e3868f2b3992fe11cdd04e9e98eb50a87d1234d9a70dbfeb154ff61638d
slow-operation output SHA256:605b7b258d2fd09c6ef2ab2ffc90d433d0509480f1e45125d20c823759e83608
Refs rustfs/backlog#2734.
Wait for the RPC/locks test sockets to pass the existing owner and mode
checks inside the original 5s readiness window and 10ms poll interval.
Use the existing admin S3 carrier for the new recovery and quota error
calls. Regenerate the test-count baseline for main's 118 existing tests
across seven crates. Pair the private pagination helper's raw-scan inputs
and remove the redundant Copy clone from the retirement event.
Keep the original assertions, responses, authorization, locks and storage
operations. The pagination body and raw-scan expressions at all five calls
are byte-identical; reinserting only the Copy clone restores the entire
base heal file. All ignored baseline values and s3s counts stay unchanged.
Validation base: c47432e353
Validation tree: f081084bc5
Patch SHA-256: 2dcf0028691bfb2772edfb38e262829b04f4151832dcce823c9dd9f7434b52de
Trace production-prefix SHA-256: fd994f218a21429d62866f4efb8eee1ee57b58f4a51865aa35d05a54ef1ccb45
All commands used LANG=en_US.UTF-8, CARGO_BUILD_JOBS=1,
CARGO_INCREMENTAL=0 and CARGO_NET_OFFLINE=true in one exclusive target.
Every gate retained the same tree and six source hashes before and after.
54 Cargo.toml files parse without duplicate keys.
cargo clippy -p rustfs --all-targets -- -D warnings
Finished `dev` profile [unoptimized + debuginfo] target(s) in 12m 50s
Exit: 0. Wall time: 770.855s.
The original 19 quick-check steps:
shellcheck --version && actionlint -> exit 0
cargo fmt --all --check -> exit 0
./scripts/check_unsafe_code_allowances.sh -> exit 0
./scripts/check_layer_dependencies.sh -> exit 0
./scripts/check_architecture_migration_rules.sh -> exit 0
./scripts/check_logging_guardrails.sh -> exit 0
./scripts/check_error_other_format_ratchet.sh -> exit 0
./scripts/check_no_tokio_io_uring.sh -> exit 0
./scripts/check_extension_schema_boundaries.sh -> exit 0
./scripts/check_body_cache_whitelist.sh -> exit 0
./scripts/check_s3s_footprint.sh -> exit 0
./scripts/check_test_count_ratchet.sh -> exit 0
./scripts/check_fips_wording.sh -> exit 0
./scripts/check_embedded_secrets.sh -> exit 0
make script-tests -> exit 0
python3 ./scripts/test_object_generation_protocol_model.py -> exit 0
python3 ./scripts/check_test_wiring.py -> exit 0
./scripts/check_no_planning_docs.sh -> exit 0
./scripts/check_uring_lane_lib_only.sh -> exit 0
Wall time: 286.337s.
cargo nextest run --profile ci -p rustfs --no-fail-fast
Summary [ 217.017s] 5382 tests run: 5382 passed (4 leaky), 40 skipped
Exit: 0. Wall time: 1324.414s.
cargo nextest run --profile ci -p rustfs-s3-types -p rustfs-protocols --features rustfs-protocols/swift --no-fail-fast
Summary [ 2.189s] 543 tests run: 543 passed, 0 skipped
Exit: 0. Wall time: 123.192s.
cargo nextest run --profile ci -p rustfs-ecstore list_objects_paginate --no-fail-fast
Summary [ 0.170s] 8 tests run: 8 passed, 5921 skipped
Exit: 0. Wall time: 261.324s.
cargo nextest run --profile ci -p rustfs-heal durable_mrf_anchor_retires_only_after_exact_bucket_generation_is_deleted --no-fail-fast
Summary [ 0.805s] 1 test run: 1 passed, 835 skipped
Exit: 0. Wall time: 199.801s.
Baseline regeneration:
./scripts/check_test_count_ratchet.sh --update
wrote .config/test-count-baseline.txt: 52 crates, 21340 tests, 164 ignored
Test-only increases: ecstore48, heal3, iam14, io-metrics2, obs3, policy14,
rustfs34. Existing ignored values and all other rows are byte-identical.
s3s footprint OK: files importing s3s is 195 (baseline files=195)
s3s footprint OK: s3s s3_error! lines is 1584 (baseline s3_error_lines=1584)
s3s footprint OK: ecstore files referencing s3s is 36 (baseline ecstore_files=36)
Historical runtime red controls on f622 (same retained two assertions):
cargo nextest run --profile ci -p rustfs --lib -E 'test(=connect::diagnostics::trace_runtime::tests::local_top_rpc_reads_service_http_completions_and_signs_offline) | test(=connect::diagnostics::trace_runtime::tests::local_top_locks_reads_service_manager_and_signs_offline)' --no-fail-fast
Original path readiness with a temporary 500ms bind/seal gap:
Summary [ 0.863s] 2 tests run: 0 passed, 2 failed, 5113 skipped
Exit: 100.
Sealed readiness in that same gap:
Summary [ 5.621s] 2 tests run: 2 passed, 5113 skipped
Exit: 0.
Compiled M1: replace the helper predicate with path.exists():
Summary [ 0.870s] 2 tests run: 0 passed, 2 failed, 5113 skipped
Exit: 100.
Restore the sealed predicate byte-for-byte:
Summary [ 5.640s] 2 tests run: 2 passed, 5113 skipped
Exit: 0.
Both reds observe mode755 with owner_matches=true and fail at the original
RPC/locks unwraps with StateSecurity. M1 failure locations: trace_runtime.rs
2547:29 and 2713:29. No new assertions were introduced.
cargo nextest run --profile ci -p rustfs --lib -E 'test(connect::diagnostics::trace_runtime::tests::)' --no-fail-fast
Summary [ 20.748s] 28 tests run: 28 passed (1 leaky), 5087 skipped
Exit: 0. This security sweep is historical; the full final unit gate above
covers the actual c474 candidate.
The first disk-stopped compile ran no tests and is excluded from red credit.
All temporary seal instrumentation is removed from the final source.
Final unit cleanup markers: 4 leaky, 40 original skips (exit0). The ld
__eh_frame linker warning also appears verbatim in the original f622 run
and the earlier c474 run; there are no new source warnings.
T1.2 batch 3 (rustfs/backlog#2743): s3select-api, s3select-query and
the error items of protocols move off s3s by scripts/codemods, run with
--hold crates/protocols/src/sftp/errors.rs. That file stays on s3s: it
downcasts the backend error, whose runtime type is still s3s::S3Error.
- rustfs/src/protocols/client.rs: the two webdav session methods now
return rustfs_s3_types::S3Result and convert the s3s error with From.
- codemod: a child module that calls s3_error! through `use super::*`
gets its own `use rustfs_s3_types::s3_error;` once its parent imports
that macro by name; otherwise it is listed as GLOB.
- s3-types: a table test pins code, status and the s3s error the legacy
edge renders for every variant a moved call site names.
- s3_error_lines 1585 -> 1584 (a lifecycle comment reworded).
rustfs/backlog#2747 batch 3. `Body::from_stream` wraps a chunk stream of
unknown length over `StreamBody`/`BoxBody`: one data frame per chunk, the
first error ends the body, the size hint stays unknown, and dropping the
body drops the stream. Parity tests pin it to the s3s `StreamingBlob::wrap`
it replaces, at the `http_body` level and byte for byte through hyper.
The Swift handlers in rustfs-protocols build their responses as the owned
`Body`. `SwiftService` converts them into the wrapped S3 service's response
body type through `From<Body>`, so rustfs keeps serving `s3s::Body` at the
legacy edge unchanged. Request bodies stay Send-only (`UnsyncBoxBody`); the
two storage writers and the DLO/SLO segment streams get `Sync` from
`sync_wrapper::SyncStream`, which is what s3s's `http_body_unsync` mutex
provided before. The DTO and error uses of s3s in protocols are untouched
(T1.5 and T1.2).
Add dual_stack_differential to e2e_test. It starts two rustfs processes
from the same binary, stack A on RUSTFS_S3_STACK=legacy and stack B on
RUSTFS_E2E_STACK_B (default legacy until rustfs/backlog#2734 T2.5),
sends both the same aws-sdk-s3 script covering the 73 methods of
impl S3 for FS (a normal request and a 4xx negative each), and compares
status, error code, an allowlist of response headers and a few typed
body fields. Error message text is never compared. Each run writes
<target>/dual-stack/<run>.json with compared_ops and the (op, field,
a, b) differences; any difference fails the test.
differential_detects_an_injected_difference starts B with
RUSTFS_REGION=eu-west-1 and requires GetBucketLocation to be reported.
The comparator is pure and unit-tested without a server.
Wire the module into the e2e-smoke filter and update the e2e-smoke and
e2e-full selection digests and the e2e_test test-count row.
Refs rustfs/backlog#2740
s3_error_lines counted every line holding the text `s3_error!`, which
also matches the RustFS-owned `rustfs_s3_types::s3_error!`. A codemod
that moves call sites onto rustfs-s3-types therefore could not lower it
(rustfs/backlog#2743, decision 3; rustfs/rustfs#8432, gotcha 2).
The counter now applies a textual, per-file and per-line rule that errs
on the side of counting (full rule in the script header):
- occurrences spelled `rustfs_s3_types::s3_error!` never count;
- a line holding `s3s::s3_error!` always counts;
- any other `s3_error!` line counts unless its file is cleared, that is,
it imports the macro from rustfs_s3_types by name on a line starting
with the use item (or defines it with `macro_rules! s3_error`) and does
not also bring s3s's macro in (`use s3s::s3_error`, a `use s3s::{..}`
tree naming it or a crate-root `*`, `use s3s::*`, or
`#[macro_use] extern crate s3s`);
- fail closed: a file matching neither form (`use super::*`, a
crate-local re-export, a renamed or glob rustfs_s3_types import, a
commented-out import, a comment-only mention) counts.
Census of the 1585 lines on 96fc06776: 1478 in 64 files that import the
macro from s3s; 107 in 15 files matching neither form (105 through
`use super::*`, 1 qualified `s3s::s3_error!`, 1 comment line); none in
files that import it from rustfs_s3_types or define it. Every line stays
counted, so the baseline remains s3_error_lines=1585 (files=202 and
ecstore_files=36 unchanged). Rewriting rustfs/src/auth.rs's import to
rustfs_s3_types lowers the new counter to 1567; the old one stayed 1585.
`--self-test` (wired into `make script-tests`) copies the script into
22 fixture trees: 6 positive cases and 16 negative ones (lines that must
not count, forms that must not clear a file, and a missing baseline,
missing key or failing classification search that must fail). Each tree
has a control line that must count, and each run gets s3s text on stdin.
Seven cases were red against the old counter. 19 mutants each turn their
case red: old definition, flipped import test, dropped fail-closed
branch, dropped e2e_test exclusion, dropped s3s veto, dropped qualified
s3s rule, dropped rustfs-qualified strip, unanchored rustfs import,
renamed import accepted, nested glob taken as root glob, single-line
s3s trees only, dropped s3s glob forms, dropped macro_use form, dropped
definition clearing, ignored classification rg errors, error-line search
without its path, classification searches without their path, missing
baseline skipped, missing key skipped.
* test(ecstore): pin the InvalidRange responses of new_getobjectreader
Both error branches of new_getobjectreader build an ErrorResponse with S3ErrorCode::InvalidRange, and no test observed either code. Pin the code, its wire name and the location fields of each branch before the code type moves to rustfs-s3-types (rustfs/backlog#2743), so a changed code on either branch fails a test. The ecstore test-count row rises by the two tests.
* chore(scripts): add the s3s-to-rustfs-s3-types error codemod
Task T1.2 of rustfs/backlog#2734 (rustfs/backlog#2743) moves S3Error, S3ErrorCode, S3Result and s3_error! from s3s to rustfs-s3-types by script, one crate closure per PR. The codemod rewrites use trees (splitting mixed s3s imports) and qualified code paths, never comments or literals, adds the rustfs-s3-types dependency, drops s3s only from a crate that no longer names it, refuses every file of the s3s edge allowlist, takes --hold for a file whose type is fixed outside the closure, and prints a per-crate census. A second run changes nothing. Its self-test runs under make script-tests.
* refactor(ecstore): take S3ErrorCode from rustfs-s3-types
Output of scripts/codemods/s3_error_to_s3_types.py over crates/ecstore, crates/lifecycle, crates/scanner and crates/replication (rustfs/backlog#2743 batches 1 and 2). The ErrorResponse code field, the two InvalidRange constructions in new_getobjectreader and an unused tier import now use rustfs_s3_types::S3ErrorCode; the variant names and wire names are unchanged. lifecycle, scanner and replication reference none of the four error names. warm_backend.rs and warm_backend_s3.rs are held: they compare against rustfs_s3_client::ErrorResponse.code, which stays the s3s type until s3-client moves.
Add rustfs_s3_types::Body, a BoxBody<Bytes, StdError> wrapper with
empty() and From<Bytes | Vec<u8> | String | &'static str>, and the
legacy From conversions to and from s3s::Body behind compat-s3s.
The signer is now generic over the request body: it only reads and
writes the request head, so it needs no body type and its callers keep
theirs. s3-client sends its requests with the owned Body, madmin signs a
bodiless head, and targets drops its two S3Request/S3Response header
wrappers in favour of extract_params_header.
rustfs-signer, rustfs-madmin and rustfs-targets no longer depend on
s3s; the s3s footprint baseline drops from 209 to 202 files.
Refs rustfs/backlog#2747, rustfs/backlog#2734
* ci: make the s3s footprint baseline exact and lower-only
The s3s footprint guard compared three counters against ceilings held in
the script itself, so a ceiling could sit above the tree and a drift was
fixable by raising it (rustfs/rustfs#5964 raised the s3_error! ceiling
to go green). The three values now live in
.config/s3s-footprint-baseline.txt as files, s3_error_lines and
ecstore_files, the script reads only that file, and a measurement that
differs from its baseline in either direction fails: growth prints
"s3s footprint grew: <measured> > <baseline>", shrinkage prints
"lower the baseline to <measured> in this PR". The counting method is
unchanged; the baseline is the current measurement of
gateway/integration (209 / 1585 / 36). The header records the end state:
once all three reach 0, T4.2 switches the script to failing on any
occurrence.
Refs rustfs/backlog#2739, rustfs/backlog#2734
* ci: add the s3s edge allowlist mode to the footprint guard (#8405)
`scripts/check_s3s_footprint.sh --mode allowlist` fails on every Rust
file that references s3s (`s3s::` or `s3s_sigv4`) and is not matched by
an entry of the new `.config/s3s-edge-allowlist.txt`; `--dry-run` lists
those files and exits 0. The allowlist carries the initial edge from the
spec, and the script rejects an entry that names or matches anything
under crates/ other than crates/e2e_test. The baseline mode and the
argument-less CI invocation are unchanged.
Refs rustfs/backlog#2741, rustfs/backlog#2734
Before any DTO or codec is replaced, pin the exact bytes that
deserialize followed by serialize produces for every persisted
bucket-configuration XML family in its three source shapes (AWS
documentation, MinIO-written, RustFS-written), and the bytes RustFS
writes today for a typed value. Byte equality only: a structural
comparison would let a codec replacement change the persisted encoding
unnoticed.
The generator script refuses to overwrite existing goldens without
--force, refuses on dirty codec paths, cannot report success from its
write mode, and records the generating commit in the fixtures README.
Refs rustfs/backlog#2744, rustfs/backlog#2734
Quick Checks now fails any PR that lowers the number of test attributes
in a workspace crate or adds #[ignore], measured per crate against the
exact rows of .config/test-count-baseline.txt (rustfs/backlog#2734
section 1 rule 3). The counter widens the spec's literal
#[(tokio::)?test] to the tokio attributes that carry arguments and both
spellings of #[test_case(...)], and anchors at line start so
commented-out attributes do not count. The initial baseline covers 52
crates, 21077 tests and 164 ignored.
Refs rustfs/backlog#2738
Refs rustfs/backlog#2734
`rustfs-s3-types` now provides `S3ErrorCode` (generated by build.rs from
src/codes.txt, the grep-derived call surface, plus `Custom`), `S3Error`,
`S3Result`, `StdError` and the `s3_error` macro with the same call syntax
as s3s, so the T1.2 codemod is a `use` swap. Statuses are hand-written and
pinned to the s3s behaviour RustFS ships today by an oracle test that uses
s3s as a dev-dependency only. The `From` conversions for the legacy edge
live behind the off-by-default `compat-s3s` feature (the orphan rule
forbids them in rustfs/src/error.rs); `rustfs` enables it.
Refs rustfs/backlog#2742, rustfs/backlog#2734
Move the ten tower layers that patch s3s behaviour out of
rustfs/src/server/layer.rs into rustfs/src/server/legacy_compat/, one file
per layer with its tests. Each file header names the s3s behaviour it
patches, the gateway mechanism that replaces it (Replaced by:), and the task
that removes it (Removed by: T2.12, rustfs/backlog#2771). The layers keep
their code, their order in external_service_stack!, and their behaviour; no
layer is deleted.
Add server::http::tests::stack_census, which renders the ordered service
chain of the real external_service_stack! expansion and compares it with a
committed constant, so a reorder, insertion or removal is a visible diff.
Route the two s3s error types the moved layers use through a
storage_api::server::legacy_compat shim so the split adds no s3s-importing
file, and lower the s3s footprint baselines to the measured values (files 209
to 208, s3_error! lines 1586 to 1585). Keep the server runtime-source bypass
guard covering the moved code.
Refs rustfs/backlog#2787, rustfs/backlog#2734
@@ -18,6 +18,7 @@ These implementation, style, and quality rules apply to all contributions, regar
- Do not include agent, tool, contributor, account, or organization names in branch names.
- Push to the user-requested remote or the repository's configured push remote. Do not hard-code or infer a remote from an account name.
- Check free space before artifact-heavy builds, tests, coverage, or downloads. Re-check before a broad gate when space is tight.
- Before handoff, remove the Cargo target directories and temporary worktrees the task created unless they stay intentionally or hold unmerged work. Identify build output by Cargo's markers (`.rustc_info.json`, `CACHEDIR.TAG`), not by directory name: `target` is also a source directory here.
- Remove only task-owned temporary/build artifacts. Never delete another task's worktree or uncommitted data.
- At handoff, mention disk or cleanup details only when they affected execution or artifacts/worktrees remain intentionally.
- For Docker verification, ensure the tested image contains the intended binary. `buildx build` without `--load` leaves the image in the build cache; use the Makefile's loaded image and rebuild without cache only when stale artifacts are evidenced.
description: Publish a RustFS version through Console, release-branch CI, preview acceptance, human confirmation, and final publication. Use only for an explicit release request.
description: Publish a RustFS version through frozen Linux candidate acceptance, desktop completion, manifest approval, byte-preserving publication and post-release follow-up. Use only for an explicit release request.
---
# RustFS Release
Require the exact final SemVer target before version edits or publication. If unspecified, inspect current releases/tags and ask with concrete candidates; do not guess the channel or bump. A named bump type resolves to an exact version that still needs confirmation. Report the confirmed target verbatim; resolve any later mismatch before proceeding. Independent read-only preflight may continue.
Require the exact final SemVer target before version edits or publication. If unspecified, inspect releases/tags and ask with concrete candidates; do not guess the channel or bump. Reuse the user's confirmed target and authorized delivery scope. Independent read-only preflight may continue.
## Invariants
- Cargo stores `<target>`, never `-preview.N`. Before publication, installation defaults retain the previous available deliverable. Final and preview binaries embed their own tag names, so version/asset names differ despite identical source.
-Use annotated tags without `v`. Preview names are exactly `<target>-preview.<digits>`; choose the next unused iteration after fetching tags. Preview Releases are prereleases, never Latest; publish only versioned assets/exact Docker tags, never latest assets, R2, Helm, moving Docker channels, or installation/banner updates.
-`release/<target>`owns the candidate. Retain its history, backport only reviewed fixes, and never merge main wholesale or force-push it. Pin `PREVIEW_HASH` to its exact commit with complete passing CI. Preview and final tags both point there; never tag a later main/branch HEAD or insert another bump.
-Phase failures block dependents. Keep preview Releases until final publication and notes generation finish; CI removes Releases while retaining tags. Never move an existing preview or final tag.
- Cargo and every candidate executable contain the final product version. `<target>-preview.<run_number>` is an immutable candidate identifier, not a public tag or displayed binary version.
-Freeze the exact source, reviewed workflow recipe, toolchain, Console, CLI, dependency lock, previous stable version and validation plan. Newer unrelated main or Console releases do not change those inputs.
- Retain`release/<target>`history when a release branch is used; backport only reviewed fixes. Pin `CANDIDATE_SHA` to its exact CI-validated commit. Never force-push a release branch or move a public tag.
-Build Linux first. Complete field acceptance before macOS/Windows and packaging. Full certification and final acceptance are required before publication; a partial Linux PASS grants no release authority.
- Promote the accepted bytes. Final publication never recompiles binaries or rebuilds packages/images. Infrastructure retries reuse successful artifacts; changed source or inputs require a new candidate and affected acceptance.
- Phase failures block dependents. No latest/default-channel update before versioned delivery. Use the [pipeline runbook](../../../docs/operations/release-pipeline.md) for executable entry points, locks and recovery.
## Read by Phase
Read only the current phase's procedure; retain completed evidence across turns. Follow root verification, Git/PR, and authorization rules.
## Read by phase
| Phase | Procedure |
|---|---|
| 0 | Clean tree, fetch main/tags, verify GitHub access, then complete the mandatory [Console release gate](references/console-gate.md). A green Console build alone is insufficient. |
| 1–2 | [Prepare/reuse the release candidate and run exact-SHA branch CI](references/release-candidate.md). Source version changes use `rustfs-release-version-bump` with authorized delivery scope. |
| 3 | [Verify the preview build, assets, channels, and notes](references/preview-release.md). |
| 4–5 | [Run the downloaded binary, Console CRUD, and latest-rc matrix](references/preview-acceptance.md). Every check must pass. |
| Confirmation | Complete the gate below and end the turn. |
| 6 | After fresh confirmation, [publish and verify the final tag](references/final-release.md) at `PREVIEW_HASH`. |
| 7 | [Reconcile Security Advisories, milestones, installation references, and the website; reconcile main and remove the completed release branch](references/post-release-updates.md) only after complete publication. |
| 0 | Clean tree, fetch main/tags, verify access, then complete the mandatory [Console gate](references/console-gate.md). |
| 1–2 | [Prepare the exact source and run branch CI](references/release-candidate.md). Version edits use `rustfs-release-version-bump`. |
| 3 | [Build and verify the Linux candidate](references/preview-release.md). |
| 4–5 | [Run artifact, field, Console and rc acceptance](references/preview-acceptance.md), then complete desktop/package/image/chart validation. |
| Confirmation | Report the complete manifest and obtain the confirmation below. |
| 6 | [Publish the accepted manifest](references/final-release.md). |
After Phases 3–5 pass, recheck that the release branch still equals `PREVIEW_HASH` and that the current Console asset tag/digest match the preview's recorded build inputs. Report the target, branch, preview tag/hash, branch CI run, preview Release URL, accepted Console asset identity/result, and rc matrix. Explicitly ask whether to publish the final tag and end the turn without creating/pushing it.
After all acceptance passes, report the target, source SHA, source CI run, Linux and completion run URLs, final manifest SHA-256, accepted Console/CLI identities, compatibility/performance results and backlog report. Explicitly ask whether to publish this exact manifest and end the turn without dispatchingpublication.
Only a new affirmative user reply to that exact report authorizes Phase 6. The original release request, earlier confirmation, silence, or automated follow-up does not count. Confirmation binds the target, preview tag, hash, and accepted build inputs; a changed value or repeated/failed acceptance cycle invalidates it. Revalidate and obtain fresh confirmation before continuing.
Only a new affirmative reply to that exact report authorizes Phase 6. An earlier general release request, silence or automatic continuation does not count. Reuse confirmation for retries of that same manifest; do not ask again merely because a destination failed. A changed source, input, manifest or acceptance result invalidates it and requires renewed acceptance/confirmation.
## Recovery
## Recovery and reporting
- Before the final tag exists, land fixes on main, backport to the candidate, verify target versions, and restart branch CI/acceptance. Use a new preview iteration once a preview was tagged; main may advance independently.
- If only an external build input changed, record it, create a new preview iteration even when the source SHA is unchanged, and repeat the affected build and artifact acceptance before fresh confirmation. Reuse source checks whose inputs remain valid.
- Once the final tag exists, retry publication jobs for that exact tag. A source fix requires a newly confirmed target, not retagging or a newer preview. Phase 7 retries recheck artifact availability and superseding releases.
- If abandoned after the bump merged, report the unpublished version on main and record whether to revert or let the next release overwrite it. Deleting unpublished candidate branches or preview tags requires an explicit decision. Completed release branches follow Phase 7 cleanup; keep release tags.
Report completed phase results and incomplete work with evidence: Console previous/latest tags, release need, hash and run/Release URLs; target, branch CI/SHA, previews and hash, confirmation state; preview/final assets, latest-channel state, Console/rc results and cleanup; advisory reconciliation results and pending disclosure decisions; both milestone states, installation PR checks/merge state, and website text/link/PR/deployment state. Keep artifact publication separate from follow-up delivery; do not call an open PR a live update. Report deviations and their authorization.
Resume failed infrastructure jobs without rebuilding successful units. Fix product defects on main, backport when needed, and start a new candidate. After a final tag exists, retry the same bytes or use a new patch version; never move the tag. Do not conflate implemented, CI-passed, field-accepted, approved, published and post-release-complete states. Report incomplete work and deviations with evidence; a tag or green build alone is not a completed release.
@@ -4,7 +4,7 @@ Read during Phase 0, before changing RustFS version files or tags. Follow the pa
### Console release gate
Complete this gate before changing any RustFS version file or creating any RustFS tag. RustFS `build.yml` downloads the asset returned by `repos/rustfs/console/releases/latest`, so a successful Console build alone is insufficient.
Complete this gate before changing any RustFS version file or creating any RustFS tag. Candidate creation resolves `repos/rustfs/console/releases/latest` once and freezes the exact asset for every build, so a successful Console build alone is insufficient. After freezing, later Console releases do not silently change or invalidate the accepted inputs.
1. Read the latest published Console tag and compare it with Console `main`:
Read only after fresh user confirmation of the preview acceptance report. The [parent confirmation gate](../SKILL.md#human-confirmation) applies.
Read only after fresh confirmation of the complete candidate report. The [parent confirmation gate](../SKILL.md#human-confirmation) applies.
## Phase 6 — Publish the final tag on the validated commit
Dispatch `release-publish.yml` on main with the successful completion run ID and approved `release-manifest.json` SHA-256. Do not create the tag manually. The workflow revalidates source, bytes and trusted certification, then creates the annotated final tag at `CANDIDATE_SHA`. A recipe change fails closed and requires a new candidate; unrelated main changes do not require a rebuild.
No second source version bump. Before creating the final tag, recheck the remote `release/<target>` HEAD equals `PREVIEW_HASH` and the current Console release tag/asset digest still match the accepted preview inputs. If either changed after confirmation, the approval no longer applies; validate the changed inputs with a new preview and obtain fresh confirmation. The final tag goes on the exact commit the preview validated:
Require both versioned publication and channel publication jobs to succeed. Verify the final tag's commit and manifest identity; all six platform ZIPs and compatibility aliases; DEB/RPM, checksums, SBOM, provenance and evidence bundle; both multi-architecture image variants in Docker Hub, GHCR and Quay; and the accepted Helm chart in the chart repository. Download a host-compatible final artifact and confirm its digest is exactly the accepted candidate digest and its version/source output agrees.
```bash
git fetch origin --tags
git rev-parse "<preview-tag>^{commit}"# must equal PREVIEW_HASH — abort if not
git tag -a "<target>" -m "Release <target>""$PREVIEW_HASH"
git push origin "<target>"
```
For stable releases verify GitHub Latest, R2 download aliases and `latest.json` unless a newer stable release already owns those channels. Prereleases must remain GitHub prereleases and leave stable pointers unchanged. Verify generated release notes compare against the frozen previous stable release, excluding internal previews.
- CI rebuilds from the same source with a different tag name, so the binary self-reports `<target>`. Other inputs can change: `build.yml` reads Console `releases/latest` during each build. Compare the actual Console tag, URL, and verified digest in every final build job with the accepted preview record, and inspect available build provenance for other relevant changes. A pre-tag check cannot prevent a later input change. On a mismatch or missing evidence, repeat affected acceptance against the final artifacts and report the limitation; failed acceptance leaves release validation incomplete even if publication already succeeded. Never move the final tag to conceal a failure.
- Verify the final tag's complete publication path: all matrix and release jobs green; `gh release view "<target>"` shows the full versioned and `-latest` asset set plus checksums, SBOM, and provenance; Docker and Helm workflows succeed; `latest.json` points to `<target>`. A stable target must have `isPrerelease=false` and `isLatest=true`. An alpha/beta/rc target must have `isPrerelease=true`; GitHub does not permit prereleases to be Latest, but the project `latest.json` still advances to the final non-preview target.
- Verify the final Release body contains `## What's Changed` and a Full Changelog link. When `PREVIOUS_DELIVERABLE` exists, the link MUST be `https://github.com/rustfs/rustfs/compare/<PREVIOUS_DELIVERABLE>...<target>` and the baseline MUST equal the preview Release baseline; for example, both `1.0.0-beta.12-preview.1` and `1.0.0-beta.12` compare from `1.0.0-beta.11`.
- Verify the preview cleanup: `cleanup-preview-releases` must succeed and `gh release view "<preview-tag>"` must then report `release not found` for every preview iteration of this target. Each surviving preview tag must still resolve to its own recorded commit; the accepted preview resolves to `PREVIEW_HASH`. If the job failed, delete the leftover Releases manually with `gh release delete "<preview-tag>" --yes` and report it.
- Verify a host-compatible final artifact's checksum and `./rustfs --version`; it must report `<target>` and the expected source SHA. This identity check does not replace affected acceptance when build inputs changed.
On infrastructure failure retry the same workflow/seal; matching destinations are reused. Never clobber an asset, force a tag, rebuild the image or silently accept a changed digest. A source fix after the final tag requires a new patch version. Cross-registry writes are not atomic: report the unfinished destination rather than claiming publication complete.
Never pass `--cleanup-tag` when removing preview Releases. Remove them manually only if `cleanup-preview-releases` failed, after final publication and notes generation. Record deleted Releases and surviving tags.
New candidates create no preview Releases to clean up. For a legacy previewrelease, preserve its tag and remove its Release only after final delivery and notes are verified. Do not pass `--cleanup-tag`. Complete the [post-release follow-up](post-release-updates.md) separately.
@@ -24,7 +24,7 @@ Reconcile advisories after every completed final release, including Console-only
1. Re-read the published deliverable state before edits or a retry. If a newer release has superseded this target, do not downgrade installation references or the website announcement. Report the superseding release and leave those defaults intact.
2. Invoke `rustfs-release-version-bump` with stage `post-release`, the exact target, and the authorized delivery scope. Prepare the follow-up on current main; do not edit or move either validated release tag. Cargo versions may already be preparing the next target and must remain untouched.
3. Verify the exact Docker image manifest, Release download links, RPM source archive, and Helm repository entry before publishing the installation PR. The chart workflow overrides chart/app versions from the final tag, pins the source checkout to that release, and waits for its default Docker image before packaging.
3. Verify the exact Docker image manifest, Release download links, RPM source archive, and Helm repository entry before publishing the installation PR. The chart is packaged and tested before sealing; publication uploads those same bytes only after versioned Docker images are available.
4. Report the installation PR and its merge state separately from the completed release. Reuse existing merge authorization and repository checks; do not infer new merge authority from this reference.
## rustfs.com top banner
@@ -54,4 +54,4 @@ Use the existing announcement configuration from [website PR #112](https://githu
1. Compare the exact published tag with current main. Audit every release-side commit since their merge base, including inherited backports. An ahead count is a history difference, not proof of missing changes. Use ancestry and patch equivalence, then inspect current code and regression evidence for squashed, adapted, superseded, or reverted changes. Record each commit as present, superseded, release-only, or missing; a positive `git cherry` result alone does not prove a missing fix.
2. Bring missing shared changes to main through a reviewed PR and the required checks before closing the follow-up. Preserve main's later changes; decide version bumps and pinned Console configuration separately. If a history merge is needed, use a merge commit and verify the release commit is an ancestor of main; squash does not establish that ancestry. Do not merge main into the published release branch or move its tags. Report unresolved changes instead of claiming reconciliation is complete.
3. After publication and authorized follow-ups are complete, delete the remote `release/<target>` branch. Immediately before deletion, verify its HEAD still equals the published tag's commit and no open PR targets or originates from it. Preserve a branch with newer work or open PRs and report the blocker. Use a deletion with an expected-old-SHA check to avoid removing a concurrently advanced branch.
4. Verify the remote branch is absent and the final and preview tags still resolve to their recorded commits. Remove a local release branch only if it is task-owned, unchecked out, and has no unpreserved work. Record cleanup separately from artifact publication; an explicitly authorized deletion before reconciliation does not make unresolved main changes complete.
4. Verify the remote branch is absent and the final tag and any legacy preview tags still resolve to their recorded commits. Remove a local release branch only if it is task-owned, unchecked out, and has no unpreserved work. Record cleanup separately from artifact publication; an explicitly authorized deletion before reconciliation does not make unresolved main changes complete.
Read after Phase 3 succeeds. Complete every check below before the parent skill's manual confirmation gate. These checks cover the downloaded artifact, embedded Console, and latest rc client.
Read after Phase 3 succeeds. Complete every check below before the parent skill's manual confirmation gate. These checks cover the downloaded artifact, embedded Console, and the rc release frozen in the candidate inputs.
## Phase 4 — Run the artifact locally, verify the console
Use a unique task-owned scratch directory. Select the versioned archive matching the host OS, architecture, and runtime, download its checksum file, and verify the archive before extraction. Do not substitute a different build when a matching artifact is unavailable; report that acceptance gap. The shell example below assumes the verified executable is at `RUSTFS_BINARY` (`rustfs.exe` on Windows), `SCRATCH` is the task directory, and `S3_PORT` and `CONSOLE_PORT` are distinct available ports. Use equivalent native commands when the shell differs.
Use a unique task-owned scratch directory. Select the versioned archive matching the acceptance host, and verify it against the sealed manifest before extraction. Before desktop completion, use the Linux acceptance environment rather than rebuilding for the local desktop. Do not substitute a different build when a matching artifact is unavailable; report that acceptance gap. The shell example below assumes the verified executable is at `RUSTFS_BINARY` (`rustfs.exe` on Windows), `SCRATCH` is the task directory, and `S3_PORT` and `CONSOLE_PORT` are distinct available ports. Use equivalent native commands when the shell differs.
Bind both listeners to loopback. Choose task-only credentials, set loopback proxy exclusions for the server/client processes, and record the server PID. A port probe does not reserve a port; retry startup with fresh ports on a bind conflict rather than connecting to an existing service.
@@ -24,16 +24,16 @@ PREVIEW_SERVER_PID=$!
Checks (all must pass):
-`"$RUSTFS_BINARY" --version` reports the preview tag name and the short SHA of `PREVIEW_HASH`. Reporting `<target>` without the `-preview.N` suffix means the build did not embed the tag — treat as FAIL and investigate before proceeding.
-`"$RUSTFS_BINARY" --version` reports the exact final `<target>` and full `CANDIDATE_SHA`. A preview suffix, another version or another source is a failure.
- Wait with a bounded readiness deadline while checking the recorded server process remains alive; `curl --noproxy '*' -fsS "$S3_ENDPOINT/health/ready"` must return ready from this process's listener.
- Startup log shows the embedded console being served (this was the regression that `fix(release): require embedded console assets` guards).
- Open `CONSOLE_ENDPOINT` in the browser and log in with the task credentials. The dashboard must render without JS console errors; create a bucket, upload a file, download it back (byte-identical), and delete the object and bucket. Keep the server running for Phase 5.
## Phase 5 — Validate with the latest rc client
## Phase 5 — Validate with the pinned rc client
`rc` is the RustFS CLI client from <https://github.com/rustfs/cli>.
- Resolve the latest release with `gh api repos/rustfs/cli/releases/latest --jq .tag_name`; verify the local client's version or download and verify a host-compatible client into the task directory. Set `RC_BIN` to that executable and record its version/digest. Avoid changing the user's installed client for this check.
- Read the CLI release and asset digests from `inputs.json`; verify the local client's version or download a host-compatible asset from that exact release into the task directory. If the host is a desktop, resolve its asset within the same pinned CLI release and record the digest; do not independently follow `latest`. Set `RC_BIN` to that executable and record its version/digest. Avoid changing the user's installed client for this check.
- Inspect the client's current help for an isolated configuration option and use it when supported. Otherwise choose a unique `PREVIEW_ALIAS` and confirm it is absent before creating it; never overwrite an existing alias. Use task-owned bucket/user names and input/output files. The names below are safe only inside the newly started isolated server.
- Point the client at `S3_ENDPOINT` with loopback proxy exclusions and run the command matrix, recording PASS/FAIL per command:
@@ -60,3 +60,9 @@ cmp <local-file> "$SCRATCH/out"
```
- Any FAIL blocks acceptance. On success or failure, stop only the recorded task process and remove only task-created aliases/data after retaining the needed failure evidence. Do not erase an existing client configuration or unrelated server data.
## Field acceptance and desktop completion
Before accepting the Linux manifest, record Scanner/Heal, multi-pool, memory, major features, high-concurrency 503, performance and previous-version integrity/upgrade results in backlog. Preserve the release-specific agreed limits and baseline; a PASS label without artifact identities or test results does not satisfy acceptance.
Dispatch `release-complete.yml` only after that report passes, using the exact Linux manifest digest and backlog URL. Require native macOS/Windows checks, package extraction, image startup/scans, chart tests and complete certification to pass. Complete any desktop Console/client checks applicable to the release against these same archives. Record the completion run and final manifest SHA-256 before entering the parent confirmation gate. Partial Linux acceptance does not authorize final publication.
Read for Phase 3 after the candidate passed exact-SHA CI. The [parent release invariants](../SKILL.md) apply.
Read for Phase 3 after the source passed exact-SHA CI. The [parent invariants](../SKILL.md) apply; commands and recovery live in the [pipeline runbook](../../../../docs/operations/release-pipeline.md).
## Preview channels
Dispatch `release-candidate.yml` on main with the confirmed final version and `CANDIDATE_SHA`. Record the run ID, frozen `inputs.json`, every Linux archive digest and `linux-manifest.json` SHA-256. Require the four Linux builds, native startup/S3 checks, previous-release upgrade matrix and Linux seal to succeed. Missing, skipped, cancelled or failed required work is incomplete acceptance.
- Use `<target>-preview.N` for every target, e.g. `1.0.0-beta.10-preview.3` or `1.1.0-preview.1`.
- The canonical suffix is exactly `-preview.<digits>`. `build.yml` recognizes it before alpha/beta/rc classification and routes it to the preview-only path; any other tag containing `-preview` fails closed instead of being treated as a release.
- A preview Release MUST be published with `isPrerelease=true` and `isLatest=false`. Any `*-latest` preview asset or preview-triggered `latest.json`, R2, Helm, or moving Docker channel tag is a pipeline failure. Docker images may use the exact preview version tag, including its existing variant suffixes.
- Preview Releases are cleaned up by the `cleanup-preview-releases` job after `publish-release` succeeds for the deliverable tag. It deletes every Release whose tag is exactly `<target>-preview.<digits>` and never passes `--cleanup-tag`, so the tags survive.
Verify the frozen Console identity equals the one accepted in Phase 0. Every matrix job consumes that same digest and exact toolchain. The binary must report the final product version and source commit; its candidate ID is separate. No public tag, Release, registry tag, R2 release path, Helm repository or latest pointer should change during this stage.
## Phase 3 — CI and preview Release verification
- Find and watch the tag build: `gh run list --workflow build.yml --branch "<preview-tag>" --limit 1` then `gh run watch <run-id>`. Every build matrix target must succeed (linux x86_64/aarch64 × musl/gnu, macos-aarch64, windows-x86_64).
- Confirm the preview tag resolves to the Phase 2 CI-validated `PREVIEW_HASH`. If the release branch advances during acceptance, repeat Phases 2–5 with a new preview tag.
- Confirm the Release publication jobs (`create-release`, `upload-release-assets`, and `publish-release`) succeed while `update-latest-version` is skipped.
- Record the inputs actually consumed by each build job: source SHA and the embedded Console release tag, asset URL, and verified SHA256 from the download logs. The workflow reads Console `releases/latest` separately for each build; confirm the matrix used the Console asset accepted in Phase 0. A missing identity or differing asset blocks acceptance until resolved. Retain available toolchain and build provenance with the artifact checksums; the same source SHA alone does not establish equivalent builds.
- Verify `gh release view "<preview-tag>" --json isPrerelease,assets,url`: `isPrerelease` must be `true`, and the Release must contain all 6 versioned platform zips, checksums, SBOM, and provenance with no `-latest` assets. Confirm `gh api repos/{owner}/{repo}/releases/latest --jq .tag_name` does not return `<preview-tag>`.
- Record `PREVIOUS_DELIVERABLE`, selected from published Releases by `publishedAt` after excluding the current tag and every `-preview.N` tag. Verify `gh release view "<preview-tag>" --json body --jq .body` contains `## What's Changed` and, when `PREVIOUS_DELIVERABLE` exists, `**Full Changelog**: https://github.com/rustfs/rustfs/compare/<PREVIOUS_DELIVERABLE>...<preview-tag>`. For a repository with no previous deliverable, verify a Full Changelog link exists and record the GitHub baseline fallback.
- Confirm Helm is skipped. If preview Docker images are published, confirm they use only exact preview tags (and variant suffixes), with no `latest`, `alpha`, `beta`, or `rc` channel updates. Preview validation covers the built RustFS binaries, embedded console, and rc compatibility; it does not authorize advancing any installation default.
## Release notes
- When a previous deliverable exists, GitHub Release notes for the preview and final tags MUST use it as their shared comparison baseline: the most recently published non-preview Release before the target. Internal `-preview.N` Releases are explicitly excluded from that selection, even when they point at the same commit as the final tag — cleanup runs after the notes are generated, so the preview Release is still present and would otherwise be picked as the baseline. If no previous deliverable exists, omit `previous_tag_name` and record that GitHub's default baseline fallback was used.
- Generated Release notes carry a workflow-management marker so retries can repair them. Before manually curating a generated body, remove that marker; unmarked non-placeholder notes are preserved by later workflow runs.
A green Linux run is ready for field acceptance, not publication. The partial certification must have `publish_allowed: false`. Retain the exact Linux manifest for the next stage; a new source/build-input candidate must not inherit the old approval. A newer unrelated main commit does not invalidate this evidence.
@@ -16,22 +16,13 @@ Read for Phases 1–2. The [parent release invariants](../SKILL.md) and Console
- Only when neither the release branch, an unfinished preview tag, nor current main already has `<target>`: if main is at a later version, select and verify a historical commit containing `<target>` rather than downgrading main; block if none exists. Otherwise invoke the `rustfs-release-version-bump` skill with stage `prepare`, the final `<target>` (NOT a preview version), and full GitHub flow (commit/push/PR).
- If a bump PR was needed, get it merged into main and record its merge commit. For a new release branch when main already has `<target>`, identify the main commit that introduced those versions. Phase 2 chooses a release candidate containing that commit; an existing release branch keeps its own history and a newer main HEAD is not required.
Do not wait for the moving latest main HEAD to become green. The release-branch CI in Phase 2 validates the chosen commit before a preview tag is created.
Do not wait for the moving latest main HEAD to become green. The release-branch CI in Phase 2 validates the chosen commit before a candidate build is dispatched.
## Phase 2 — Pin and validate the release candidate, then publish the preview tag
## Phase 2 — Pin and validate the release source
- Use `release/<target>`. On the first attempt, create it from a selected main commit containing the version bump and intended changes; it need not be the latest main HEAD. On a restart, retain the existing branch and backport only reviewed fixes from main. Verify the branch's Cargo versions still equal `<target>`. If adopting an existing preview, run this branch CI before continuing its acceptance; reuse the tag only when it points to the validated SHA.
- The current `ci.yml` runs automatically on main pushes, not release-branch pushes. Dispatch it explicitly with `gh workflow run ci.yml --ref "release/<target>"`. Locate that dispatch run, verify its `head_sha` equals the release branch's remote HEAD, and require the full expected CI job set to pass, including `Quick Checks`, `Test and Lint`, and `End-to-End Tests (full merge gate)`. Treat a missing or skipped required lane as incomplete. A successful PR check or CI run for a different SHA does not qualify. If the branch advances or a run is cancelled, validate the new SHA again. Do not tag a candidate with failing or incomplete CI.
- Use `release/<target>`. On the first attempt, create it from a selected main commit containing the version bump and intended changes; it need not be the latest main HEAD. On a restart, retain the existing branch and backport only reviewed fixes from main. Verify the branch's Cargo versions still equal `<target>`. A legacy preview can anchor source selection, but its tag and old artifacts do not constitute a candidate in the new protocol; freeze and verify a new candidate after the release recipe is aligned.
- The current `ci.yml` runs automatically on main pushes, not release-branch pushes. Dispatch it explicitly with `gh workflow run ci.yml --ref "release/<target>"`. Locate that dispatch run, verify its `head_sha` equals the release branch's remote HEAD, and require the full expected CI job set to pass, including `Quick Checks`, `Test and Lint`, and `End-to-End Tests (full merge gate)`. Treat a missing or skipped required lane as incomplete. A successful PR check or CI run for a different SHA does not qualify. If the branch advances or a run is cancelled, validate the new SHA again. Do not dispatch a candidate with failing or incomplete CI.
- Prefer protecting `release/*` against force pushes and deletion and requiring reviewed backport PRs. Check live rulesets rather than assuming the main ruleset covers release branches; report a missing release-branch rule as a gap.
- Set `PREVIEW_HASH` to the CI-validated release-branch commit and recheck the remote branch HEAD immediately before tagging. Report the branch, commit, and CI run URL; do not derive `PREVIEW_HASH` from a later `origin/main` fetch. When adopting an existing preview tag at this SHA, skip tag creation and continue Phase 3.
```bash
git tag -a "<preview-tag>" -m "Release <preview-tag>""$PREVIEW_HASH"
git push origin "<preview-tag>"
```
Pushing the tag triggers `.github/workflows/build.yml` ("Build and Release"); `docker.yml` chains off it via `workflow_run`.
The preview run builds versioned artifacts and publishes them in a GitHub prerelease. Docker may publish exact preview image tags. Latest-channel, R2, and Helm publication must be skipped; installation references and the website announcement stay unchanged.
On a restart after a backport, re-run release-branch CI before setting the new `PREVIEW_HASH`. Use the next unused preview iteration when the validated commit differs from an existing preview tag; never move an existing tag.
- Set `CANDIDATE_SHA` to the CI-validated release-branch commit. Record the branch, exact commit and CI run URL. The new candidate protocol starts from a commit whose release recipe matches the reviewed main dispatcher; backport the recipe before freezing an older release branch if needed.
- Dispatch the Linux candidate as described in [candidate verification](preview-release.md). Do not create a preview or final tag here. The final executable already contains `<target>`; publication later promotes those same bytes.
- After a source or build-input change, run the affected source checks and create a new candidate. Reuse the frozen candidate after an infrastructure failure. Do not restart acceptance for unrelated main advances.
@@ -4,18 +4,18 @@ description: Prepare an exact RustFS Cargo version or align installation referen
---
# Release Version Files
Require an exact target; ask if missing or ambiguous. Reject targets containing `-preview`: those are tag-only. Infer `post-release` only from an explicit request or parent workflow; otherwise use `prepare`. Delivery defaults to local edit/verify; commit, push, PR, and merge follow existing authorization and [Git rules](../../references/pull-requests.md).
Require an exact target; ask if missing or ambiguous. Reject targets containing `-preview`: those identify candidates, not product versions or new public tags. Infer `post-release` only from an explicit request or parent workflow; otherwise use `prepare`. Delivery defaults to local edit/verify; commit, push, PR, and merge follow existing authorization and [Git rules](../../references/pull-requests.md).
| Stage | Files and constraints |
|---|---|
| `prepare` | Only `Cargo.toml` and `Cargo.lock`: workspace package, internal dependency, and workspace member versions become `<target>`. External dependencies stay unchanged. |
| `post-release` | Only `README.md`, `README_ZH.md`, `flake.nix`, `helm/rustfs/Chart.yaml`, and `rustfs.spec`. Cargo may already target the next release; leave it untouched. |
Before installation edits, verify the exact non-preview Release, downloadable assets/source archive, pullable `rustfs/rustfs:<target>` image manifest, and published Helm chart. Block on missing artifacts. Do not downgrade defaults if a newer deliverable superseded the target. Preview preparation retains the previous published installation references; follow-up commits never replace either validated tag.
Before installation edits, verify the exact non-preview Release, downloadable assets/source archive, pullable `rustfs/rustfs:<target>` image manifest, and published Helm chart. Block on missing artifacts. Do not downgrade defaults if a newer deliverable superseded the target. Candidate preparation retains the previous published installation references; follow-up commits never replace validated tags or sealed artifacts.
Packaging policy:
- Docker image tags have no `v` prefix. Derive chart/app versions with `scripts/helm_chart_version.sh <target>` (`beta.N -> 0.N.0`; other versions unchanged). Helm CI derives versions from the final tag, so no early chart bump is required.
- Docker image tags have no `v` prefix. Derive chart/app versions with `scripts/helm_chart_version.sh <target>`; both equal the final product SemVer, including prerelease suffixes. Helm CI uses the frozen candidate version before sealing, so no early installation chart bump is required. Previously published chart versions remain unchanged.
- RPM `Version` is numeric; `Release` is the prerelease suffix or `1` for stable. Expanded `Source0` and the unpack directory must resolve to the exact target including its suffix. Add the changelog entry using current git identity, date, and target; never copy an example identity/date.
- Changing these mappings requires explicit confirmation.
@@ -20,11 +20,11 @@ planning-docs-check: ## Check that no planning-type documents are committed
./scripts/check_no_planning_docs.sh
.PHONY:pre-commit
pre-commit:fmt-checkunsafe-code-checkarchitecture-migration-checklogging-guardrails-checkerror-other-ratchet-checktokio-io-uring-checkextension-schema-checkbody-cache-whitelist-checks3s-footprint-checkfips-wording-checkembedded-secrets-checktest-wiring-checkdoc-paths-checkplanning-docs-checkquick-check## Run fast pre-commit checks without clippy/full tests
pre-commit:fmt-checkunsafe-code-checkarchitecture-migration-checklogging-guardrails-checkerror-other-ratchet-checktokio-io-uring-checkextension-schema-checkbody-cache-whitelist-checks3s-footprint-checktest-count-ratchet-checkfips-wording-checkembedded-secrets-checktest-wiring-checkdoc-paths-checkplanning-docs-checkquick-check## Run fast pre-commit checks without clippy/full tests
@echo "✅ All pre-commit checks passed!"
.PHONY:pre-pr
pre-pr:fmt-checkunsafe-code-checkarchitecture-migration-checklogging-guardrails-checkerror-other-ratchet-checktokio-io-uring-checkextension-schema-checkbody-cache-whitelist-checks3s-footprint-checkfips-wording-checkembedded-secrets-checktest-wiring-checkdoc-paths-checkplanning-docs-checklog-analyzer-rules-checkoffline-enrollment-e2e-checkclippy-checktest## Run full pre-PR checks with clippy and tests
pre-pr:fmt-checkunsafe-code-checkarchitecture-migration-checklogging-guardrails-checkerror-other-ratchet-checktokio-io-uring-checkextension-schema-checkbody-cache-whitelist-checks3s-footprint-checktest-count-ratchet-checkfips-wording-checkembedded-secrets-checktest-wiring-checkdoc-paths-checkplanning-docs-checklog-analyzer-rules-checkoffline-enrollment-e2e-checkclippy-checktest## Run full pre-PR checks with clippy and tests
"description":"Certification for immutable candidates. All shipped platforms require native identity and startup; stable and prerelease require the pinned upgrade matrix. Linux-only acceptance never authorizes publication.",
"release_types":[
"preview",
"prerelease",
"stable"
],
"targets":[
{
"os":"linux",
"arch":"x86_64",
"abi":"musl"
},
{
"os":"linux",
"arch":"aarch64",
"abi":"musl"
},
{
"os":"linux",
"arch":"x86_64",
"abi":"gnu"
},
{
"os":"linux",
"arch":"aarch64",
"abi":"gnu"
},
{
"os":"macos",
"arch":"aarch64",
"abi":"none"
},
{
"os":"windows",
"arch":"x86_64",
"abi":"msvc"
}
],
"targets_basis":"Platform matrix of .github/workflows/build.yml (prepare-platform-matrix). gnu and macOS targets build with the pyroscope feature; musl and Windows targets build with no extra features.",
"features":[
"pyroscope"
],
"trusted_workflows":[
{
"repository":"rustfs/rustfs",
"workflow_id":"release-candidate.yml"
},
{
"repository":"rustfs/rustfs",
"workflow_id":"release-complete.yml"
}
],
"trusted_workflows_basis":"Receipts are produced by the reviewed main workflows and fetched by exact run ID with GitHub artifact digest verification. Candidate inputs freeze the harness commit separately from product source.",
"lanes":[
{
"lane":"identity",
"harness_repository":"rustfs/rustfs",
"harness_sha":"candidate.workflow_sha",
"harness_basis":"The exact workflow commit is frozen in candidate inputs; no mutable branch is accepted as a harness identity.",
"timeout_seconds":1800,
"environment_capabilities":[
"process-exec"
],
"cases":[
"artifact-identity",
"startup"
],
"applies_to":{
"release_types":[
"preview",
"prerelease",
"stable"
],
"platforms":"all"
},
"applies_basis":"Artifact identity and startup checks are mandatory for every release type, preview included, on every shipped target (rustfs/backlog#2662 frozen design item 5)."
},
{
"lane":"upgrade",
"harness_repository":"rustfs/rustfs",
"harness_sha":"candidate.workflow_sha",
"harness_basis":"The exact workflow commit is frozen in candidate inputs; no mutable branch is accepted as a harness identity.",
"timeout_seconds":5400,
"environment_capabilities":[
"process-exec",
"previous-release-artifact"
],
"cases":[
"direct-upgrade",
"mixed-version-upgrade",
"bucket-config-upgrade",
"bucket-config-rollback",
"multipart-layout-upgrade",
"multipart-layout-baseline"
],
"cases_basis":"Case IDs of the pinned previous-release matrix in .github/workflows/e2e-upgrade.yml (source.version/asset/sha256 and the case list mapped there).",
"applies_to":{
"release_types":[
"prerelease",
"stable"
],
"platforms":[
{
"os":"linux",
"arch":"x86_64",
"abi":"gnu"
}
]
},
"applies_basis":"preview releases use the pre-declared smaller set (identity lane only). The upgrade lane is only exercised on linux-x86_64-gnu, the platform of the pinned source assets."
}
],
"upgrade_sources":{
"basis":"Exactly the two previous-release sources pinned in .github/workflows/e2e-upgrade.yml are exercised today (source.version / source.asset / source.sha256). No other N-1/N-2 compatibility is claimed, and these digests are not inferred from tags.",
"rollback_basis":"Only the bucket-config-rollback case (rollback_to_previous_release_reads_current_bucket_metadata) is exercised. Full data-plane rollback is not certified and is never a default claim."
@@ -22,9 +22,9 @@ The following files in the project root are essential for Docker operations:
| File | Description | Usage |
| :--- | :--- | :--- |
| **`docker-buildx.sh`** | **Multi-Arch Build Script**<br>Automates building and pushing Docker images for `amd64` and `arm64`. Supports release and dev channels. | `./docker-buildx.sh --push` |
| **`Dockerfile.glibc`** | **Production Image (Ubuntu)**<br>Standard image using glibc. Useful if you need specific dynamic libraries. | `docker build -f Dockerfile.glibc .` |
| **`Dockerfile.source`** | **Development Image**<br>Builds RustFS from source code. Includes build tools. Ideal for local development and CI. | `docker build -f Dockerfile.source .` |
| **`Dockerfile`** | **Production Image (Alpine)**<br>Lightweight image using musl libc. Downloads verified RustFS and `rc` binaries from GitHub Releases and bundles `/etc/rustfs/bootstrap`. | `docker build -t rustfs:latest .` |
| **`Dockerfile.glibc`** | **Production Image (Ubuntu)**<br>Standard image using glibc. Includes the bundled bootstrap configuration. | `docker build -f Dockerfile.glibc .` |
| **`Dockerfile.source`** | **Development Image**<br>Builds RustFS from source code and bundles `rc` and the default bootstrap configuration. | `docker build -f Dockerfile.source .` |
### Docker Compose Configurations
@@ -110,6 +110,36 @@ docker compose up -d
---
## Docker Compose environment values
Use map syntax when quoting environment values in YAML:
Create the host directories and grant the container user write access before starting this example; official images run as UID/GID `10001:10001`. Set both credential variables to your own values, then run `docker compose config` to inspect the resolved configuration locally before `docker compose up -d`. The resolved output includes credentials, so do not share it publicly.
With list syntax, use `- RUSTFS_OBS_LOG_DIRECTORY=/var/log/rustfs/` or quote the whole item as `- "RUSTFS_OBS_LOG_DIRECTORY=/var/log/rustfs/"`. Writing `- RUSTFS_OBS_LOG_DIRECTORY="/var/log/rustfs/"` passes the inner quotes literally, making this a relative path beginning with `"`, which can fail with `mkdir: cannot create directory '"': Permission denied`. This also applies to addresses and credentials: fix every affected list item, rather than only the log directory. The entrypoint preserves configured values, including literal quotes in valid paths and secrets.
# Licensed under the Apache License, Version 2.0 (the "License");
# you may not use this file except in compliance with the License.
# You may obtain a copy of the License at
#
# http://www.apache.org/licenses/LICENSE-2.0
#
# Unless required by applicable law or agreed to in writing, software
# distributed under the License is distributed on an "AS IS" BASIS,
# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
# See the License for the specific language governing permissions and
# limitations under the License.
name:"Reference RustFS binary"
description:"Download the pinned reference RustFS release, verify its sha256 and expose the executable path. A missing pin, missing asset or digest mismatch fails the step."
inputs:
pin-file:
description:"Pin file naming the reference release (repository-relative)"
required:false
default:".config/reference-binary.toml"
platform:
description:"Platform table in the pin file"
required:false
default:"linux-x86_64-gnu"
destination:
description:"Directory that receives the executable; defaults to $RUNNER_TEMP/reference-binary"
required:false
default:""
outputs:
path:
description:"Absolute path of the verified executable"
This repository is public. For a public repository, GitHub's standard hosted runners (`ubuntu-latest`, `windows-latest`, `macos-latest`) cost nothing and have no minute cap, whatever plan the organisation is on. GitHub bills only two things: larger hosted runners, which have custom labels and are not used here, and standard runners in private repositories.
This repository is public. For a public repository, GitHub's standard hosted runners (`ubuntu-latest`, `windows-latest`, `macos-latest`) cost nothing and have no minute cap, whatever plan the organisation is on. GitHub bills only two things: larger hosted runners, including the `macos-latest-xlarge` release lane used here, and standard runners in private repositories.
Every other label in this repository is capacity RustFS pays for. ARC pods and lab machines accrue cost while the machine exists. A Ubicloud label is metered for the minutes a job holds its VM:
@@ -13,6 +13,7 @@ Every other label in this repository is capacity RustFS pays for. ARC pods and l
|---|---|---|---|
| `ubuntu-latest` | nobody | GitHub-hosted, Ubuntu 24.04, 4 vCPU, 16 GB RAM, about 14 GB of free disk. Ships a C toolchain, `docker`, `gh`, `curl`, `wget`, `jq`, Python. Can `mount` tmpfs and loop devices; io_uring works | the default for every job |
| `windows-latest`, `macos-latest` | nobody | GitHub-hosted standard runners | Windows and macOS targets |
| `macos-latest-xlarge` | RustFS | GitHub-hosted larger Apple Silicon runner | macOS release builds, deferred until Linux acceptance for candidates |
| `sm-standard-2` | RustFS | ARC pod, Ubuntu 24.04, minimal image: no `gh`, no C toolchain; Docker only on the `dind-` label | only a job that a hosted runner cannot run (see the placement rule) |
| `sm-standard-4` | RustFS | ARC pod, Ubuntu 24.04, 14 CPUs and 28 GB (the label is not the core count), large disk | workspace-scale Cargo builds |
| `ubicloud-standard-16-arm` | RustFS | Ubicloud VM, alias of `ubicloud-standard-16-arm-ubuntu-2404`: Ubuntu 24.04 arm64, 16 vCPU, 48 GB, 300 GB disk. Reached through the `build.yml` matrix, so `actionlint.yaml` does not list it | native aarch64 release builds (`linux-aarch64-musl`, `linux-aarch64-gnu`) |
@@ -32,7 +33,7 @@ A job runs on `ubuntu-latest` unless it has one of these reasons, and the reason
Jobs that do only the following have no reason to be self-hosted: shell or Python scripts, `cargo fmt`, `cargo deny`, typos, actionlint, API calls (`gh`, dependency review, issue and stale bots, CLA checks), downloading an artifact and running a prebuilt binary, or a `cargo build -p rustfs` with its own cache key. `e2e-distributed.yml` and `e2e-upgrade.yml` build the server and the `e2e_test` harness on `ubuntu-latest` inside their budgets, so "it compiles something" alone is not a reason.
The functional chain's `prepare` and `complete-chain` jobs, the performance runner availability check, and runner hygiene use `ubuntu-latest`: they inspect APIs or artifacts, not the lab hosts. The actual functional and performance suites remain on their lab runners. Functional-chain health collection uses `ubuntu-latest` so a lab outage cannot also stop its independent monitor. Docker image packaging also uses `ubuntu-latest` for both musl and glibc variants; it downloads release binaries rather than compiling the workspace, and still sets up QEMU and Buildx explicitly.
The functional chain's `prepare` and `complete-chain` jobs, the performance runner availability check, and runner hygiene use `ubuntu-latest`: they inspect APIs or artifacts, not the lab hosts. The actual functional and performance suites remain on their lab runners. Functional-chain health collection uses `ubuntu-latest` so a lab outage cannot also stop its independent monitor. Docker image packaging also uses `ubuntu-latest` for both musl and glibc variants; it wraps verified candidate binaries rather than compiling the workspace, and still sets up QEMU and Buildx explicitly.
Moving these jobs off AKS reduces capacity demand, not necessarily the Azure bill immediately. VM costs fall only when nodes scale down or smaller capacity replaces them.
@@ -52,7 +53,7 @@ A hosted runner is also the only choice for a few things the pods cannot do: `mo
These are the patterns that have cost real money here without validating anything more:
- An `if: always()` summary or alert job on a self-hosted label inside a `workflow_run`-triggered workflow. The upstream workflow completes many times a day, the real jobs skip, and the summary still starts a pod each time. Gate it on the upstream job having run, and keep it on `ubuntu-latest`.
- Per-push workflows on `main`. A merge triggers the full`ci.yml` matrix, `cache-warm.yml` and the`build.yml` development build; with tens of merges a day that is as much self-hosted time as all pull requests together. Anything that does not verify the merged tree itself belongs on a schedule or behind a `paths` filter.
- Per-push workflows on `main`. A merge triggers the applicable`ci.yml` matrix and cachewarming;`build.yml` development builds are scheduled or manual; with tens of merges a day that is as much self-hosted time as all pull requests together. Anything that does not verify the merged tree itself belongs on a schedule or behind a `paths` filter.
-`apt-get install` of tools the hosted image already ships. It is only needed on the minimal pod images.
- A throttled `CARGO_BUILD_JOBS` on a 14-CPU pod. The limit exists for link-storm reasons documented in `ci.yml`; measure before raising it, and measure before lowering it further.
- Tiny jobs on ARC pods. Each one pays roughly a minute of pod start-up before its first step; a compile-free job that takes ten seconds costs more to schedule than to run.
@@ -66,7 +67,7 @@ gh api "repos/rustfs/rustfs/actions/runs/<run_id>/jobs?per_page=100" \
Only rows whose label starts with `sm-`, `dind-`, `ubicloud-`, `smoke-` or `pf-`cost anything. At the October 2026 review, a full pull-request run of `ci.yml` spent about 150 self-hosted minutes, a third of it in jobs that never compiled; treat that as the scale of what a placement mistake costs, not as a current figure.
Rows using `macos-latest-xlarge` or a label starting with `sm-`, `dind-`, `ubicloud-`, `smoke-` or `pf-`use paid capacity. At the October 2026 review, a full pull-request run of `ci.yml` spent about 150 self-hosted minutes, a third of it in jobs that never compiled; treat that as the scale of what a placement mistake costs, not as a current figure.
## Checklist for a new or moved job
@@ -76,3 +77,9 @@ Only rows whose label starts with `sm-`, `dind-`, `ubicloud-`, `smoke-` or `pf-`
- A self-hosted `runs-on` carries a comment naming its reason from the placement rule.
- Every `./.github/actions/setup` call states `cache-save-if`, and a key read from one pool is not written from the other.
- Moving a job does not change its check name, so the `main` ruleset and `docs/testing/ci-gates.md` stay as they are; renaming it does, and then that document's change checklist applies.
## Release stages
The [candidate release pipeline](../../docs/operations/release-pipeline.md) builds Linux first and defers paid desktop capacity until acceptance. Daily development builds also select Linux; use manual platform selection for desktop development artifacts. All six platforms remain required before publication. The pipeline records and promotes the exact tested bytes. Builder workflows have no public-release credentials. Keep periodic development builds, PR verification and field-test chains separate when their contracts differ; do not re-enable disabled workflows as part of a source-file migration.
Larger runners remain billed for public repositories; see [GitHub runner billing](https://docs.github.com/en/actions/concepts/runners/larger-runners#billing). Actual spending comes from organization billing, not runner labels or an estimated build duration.
@@ -26,6 +26,7 @@ and this project adheres to [Semantic Versioning](https://semver.org/spec/v2.0.0
- IAM snapshot/deletion replay, target-assigned delete-marker purges, timestamp ordering, and best-effort peer broadcast now close the control-plane gaps found by the R6 review ([#7195](https://github.com/rustfs/rustfs/pull/7195)).
- A `set-remote-target?update=true&creds=true` request on a site-replication peer's bucket target is now rejected with `400 InvalidRequest`, naming the target and why it counts as peer-owned. It used to return success while keeping the stored endpoint, credentials, and TLS settings, so an endpoint edit from the Console's bucket replication form appeared to apply but did not. Change a peer's address with a site-replication edit instead; see "Changing a site's address" in `docs/operations/site-replication-operations.md`. The `sync`, `bandwidth`, `proxy`, and `path` update groups are unchanged.
- Site-replication edits that omit the site name, such as `mc admin replicate update --deployment-id <ID> --endpoint <URL>`, now keep the stored name on the editing site and on every receiver instead of renaming the site after the new endpoint host. No persisted format changes.
- The persisted replication MRF journal is now replayed every 5 minutes while the server runs, not only at startup. Delete-marker purges whose targets were unreachable, and replications that overflowed the worker queues, now converge within one replay interval after the target returns instead of waiting for a restart. A target that stays down is re-attempted once per retained entry per interval across the cluster: the start time of the last pass is shared through a small `config/replication/mrf.bin.replay` object, so nodes do not each replay once per interval. Older versions ignore that object; existing persisted formats are unchanged.
- Upgrade and rollback: upgrade every node in one site consecutively and verify reconciliation before moving to the next site; do not intentionally run a site mixed-version. Target-version ledger keys are harmless on rollback, although old code cannot use their routing. Before rolling back past [#7307](https://github.com/rustfs/rustfs/pull/7307), drain or repair every pending version purge: older code can free a retained version's data directory before its remote purge is acknowledged. See `docs/operations/site-replication-operations.md`.
# Supports cross-compilation for amd64 and arm64 via TARGETPLATFORM.
ARG TARGETPLATFORM
ARG BUILDPLATFORM
ARG TARGETARCH
# Do not declare TARGETPLATFORM, BUILDPLATFORM or TARGETARCH here: a global
# declaration without a value shadows BuildKit's automatic values, so the
# stage-level re-declarations below see empty strings. They are only
# declared inside the stage that uses them.
ARGRUSTFS_BUILD_FEATURES=""
ARGRC_RELEASE=latest
# -----------------------------
# Build stage
# -----------------------------
FROMrust:1.97.1-trixieASbuilder
FROMrust:1.99.1-trixieASbuilder
# Re-declare args after FROM
ARG TARGETPLATFORM
ARG BUILDPLATFORM
ARG TARGETARCH
ARG RUSTFS_BUILD_FEATURES
ARG RC_RELEASE
# Debug: print platforms
RUNecho"Build info -> BUILDPLATFORM=${BUILDPLATFORM}, TARGETPLATFORM=${TARGETPLATFORM}"
@@ -52,6 +55,7 @@ RUN set -eux; \
build-essential \
ca-certificates \
curl \
jq \
git \
pkg-config \
libssl-dev \
@@ -166,6 +170,25 @@ RUN --mount=type=cache,target=/usr/local/cargo/registry \
;;\
esac
# TARGETARCH is only provided by BuildKit. Fall back to TARGETPLATFORM (for
# example set by docker-compose) and then to the build host, like the cargo
# step above, so every path that builds this file passes a real architecture.
RUNset -eu;\
rc_arch="${TARGETARCH:-}";\
if[ -z "${rc_arch}"];then\
case"${TARGETPLATFORM:-}" in \
linux/amd64)rc_arch=amd64 ;;\
linux/arm64)rc_arch=arm64 ;;\
*)\
case"$(uname -m)" in \
x86_64)rc_arch=amd64 ;;\
aarch64|arm64)rc_arch=arm64 ;;\
*)echo"TARGETARCH is empty and TARGETPLATFORM='${TARGETPLATFORM:-}' is not linux/amd64 or linux/arm64; cannot choose an rc architecture on $(uname -m)" >&2;exit1;;\
esac;;\
esac;\
fi;\
sh /usr/src/rustfs/scripts/download_rc.sh "${rc_arch}" musl "${RC_RELEASE}" /usr/local/bin/rc
The official image bundles this directory at `/etc/rustfs/bootstrap`. The default `buckets.json` is an empty JSON array, so the image starts RustFS directly and does not require a bootstrap endpoint or bootstrap credentials.
To create buckets, replace `buckets.json` in a derived image with a JSON array of bucket names. The entrypoint creates each bucket if it does not exist and applies `public-access-block.json`, whose default blocks all four public access paths. Edit that file in the derived image only when public access is an explicit requirement.
Optional per-bucket files use the bucket name as the filename without an extension:
| `cors/` | JSON or XML accepted by `rc bucket cors set` | Replace CORS rules |
| `notifications/` | JSON array of `{ "arn": "...", "events": ["..."] }` | Add or replace event rules by ARN (prefix/suffix filters are not currently accepted) |
Every per-bucket file must correspond to a bucket listed in `buckets.json`. Applying these files is repeatable: bucket creation ignores existing buckets, while each configuration operation replaces or reconciles that bucket's requested state.
When `buckets.json` is non-empty, set `RUSTFS_BOOTSTRAP_ENDPOINT` to the RustFS S3 endpoint reachable from the container, for example `http://rustfs:9000` for a Compose service or the internal service DNS name in Kubernetes. The entrypoint waits on that endpoint's `/health/ready`; it never assumes `localhost`. `RUSTFS_BOOTSTRAP_TIMEOUT` controls the readiness wait in seconds (default `120`, maximum `3600`). `RUSTFS_BOOTSTRAP_DIR` can point to a different in-image configuration directory; it defaults to `/etc/rustfs/bootstrap`. Set `RUSTFS_ACCESS_KEY` and `RUSTFS_SECRET_KEY`, or their `_FILE` variants, for bootstrap authentication. The endpoint and credentials are read at runtime and are not baked into the image.
RustFS built-in IAM policies remain unchanged; the default bootstrap installs no custom bucket policies, lifecycle rules, versioning, CORS rules, or notification rules. The bundled RustFS CLI is resolved from a published release and its release checksum is verified. Set the Docker build argument `RC_RELEASE` to a version to pin it. Build the RustFS image after a CLI release containing the `bucket public-access-block` commands is available. For Kubernetes, the configured service must route traffic to the starting pod before its readiness probe succeeds (for example, with `publishNotReadyAddresses: true`); otherwise the service endpoint cannot reach the pod while the entrypoint waits for readiness.
| **reliant** | [`src/reliant/`](src/reliant) | Tests that reuse an **externally started** server (SQL/select, conditional writes, lifecycle, deleted-object reads, node-interact). Run via [`scripts/run_e2e_tests.sh`](../../scripts/run_e2e_tests.sh); see [`src/reliant/README.md`](src/reliant/README.md) |
| **upgrade compatibility** | `upgrade_compatibility_test` | Pinned previous-release writes followed by current-build reads on the same data directory |
| **dual-stack differential** | `dual_stack_differential` | Two single-node servers answer one SDK script for all 73 S3 operations and every differing `(op, field, a, b)` is reported. Side A is the binary under test on `RUSTFS_S3_STACK=legacy`; side B runs `RUSTFS_E2E_STACK_B`. Set `RUSTFS_E2E_REFERENCE_BINARY` to a released `rustfs` binary to make side A that binary instead (started without `RUSTFS_S3_STACK`, which older releases do not know). A missing, non-file or non-executable path fails the run; it never skips or falls back to the same-binary mode. The report in `<target>/dual-stack/<run>.json` records `reference_side` and each side's `--version` output |
The external-tool `storage_metric_ownership_test` validates the OTLP/Collector/Prometheus path, including a rolling upgrade and node failures. See the [storage metrics guide](../../docs/operations/storage-metrics.md) for its required binaries and focused command.
@@ -64,6 +65,25 @@ Each build/run holds an exclusive `rustfs.e2e.lock` marker beside the binary; co
The protocols suite has its own fixed-port and single-worker contract in [`src/protocols/README.md`](src/protocols/README.md). Use its command under [Troubleshooting](#troubleshooting).
### Scanner mixed-version gate
The Linux N/O scanner gate needs a separately qualified `33482bb` server, which supports scoped ACKs. Set `RUSTFS_SCANNER_COMPAT_OLD_BINARY` to that executable and `RUSTFS_SCANNER_COMPAT_OLD_SHA256` to its independently verified digest. The release binary used by ordinary upgrade CI is a different baseline. Missing or mismatched inputs fail the test.
Enable the harness-only `scanner-peer-mixed-version` feature explicitly; keep the server wrapper features identical between build and run:
The separate `scanner_peer_bucket_new_coordinator_certifies_old_worker_peers_and_scopes_next_cycle` case uses the same feature and qualified old binary inputs. Select its exact name in the command above. It keeps the original N process enabled and starts the three official O replacements with `RUSTFS_SCANNER_ENABLED=false`: their autonomous scanner loops are disabled, while real S3 dirty tracking, authenticated snapshots, publication leases, scoped ACKs, and N-dispatched RemoteScanner workers remain active. It records each node's actual scanner status and retains the same identity, durability, root, ACK, data-conservation, and physical-walk requirements. This fixed-role case does not validate autonomous O scanning or all-enabled election behavior. The original all-enabled N/O test remains separate and has not passed acceptance; a fixed-role result cannot satisfy that gate.
For this fixed-role case, the `e2e-test-hooks` server also records the N client's complete bounded worker inventory. Requests are observed before opening the normal stream and paired only after authenticated source/plan validation and terminal EOF. Closed manifests and final health receipts must bind both cycles to their actual candidate/root; the next cycle needs exactly four sets of hot-only execution and at least one positive-progress worker mapped to a measured official O process. A zero-progress Current response, missing/extra request, unhealthy recorder, or missing physical coverage fails the gate. The recorder does not add O hooks or change any protocol or timeout. Final health reads check immutable closed-cycle evidence, not current N liveness.
### `#[ignore]` semantics
Ignored tests are excluded from the default `cargo nextest run` pass because
@@ -175,9 +195,9 @@ the same profile for membership and execution with one nightly worker.
Some files were not shown because too many files have changed in this diff
Show More
Reference in New Issue
Block a user
Blocking a user prevents them from interacting with repositories, such as opening or commenting on pull requests or issues. Learn more about blocking a user.
overtrue