ci(parity): run dual-stack differential against pinned release (#9315 )

* feat(server): add the RUSTFS_S3_STACK selector and dual-stack e2e lane (#8411) * refactor(server): group the s3s-compat tower layers under legacy_compat (#8410) Move the ten tower layers that patch s3s behaviour out of rustfs/src/server/layer.rs into rustfs/src/server/legacy_compat/, one file per layer with its tests. Each file header names the s3s behaviour it patches, the gateway mechanism that replaces it (Replaced by:), and the task that removes it (Removed by: T2.12, rustfs/backlog#2771). The layers keep their code, their order in external_service_stack!, and their behaviour; no layer is deleted. Add server::http::tests::stack_census, which renders the ordered service chain of the real external_service_stack! expansion and compares it with a committed constant, so a reorder, insertion or removal is a visible diff. Route the two s3s error types the moved layers use through a storage_api::server::legacy_compat shim so the split adds no s3s-importing file, and lower the s3s footprint baselines to the measured values (files 209 to 208, s3_error! lines 1586 to 1585). Keep the server runtime-source bypass guard covering the moved code. Refs rustfs/backlog#2787, rustfs/backlog#2734 * feat(s3-types): add the RustFS-owned S3 error type and s3_error! macro (#8406) `rustfs-s3-types` now provides `S3ErrorCode` (generated by build.rs from src/codes.txt, the grep-derived call surface, plus `Custom`), `S3Error`, `S3Result`, `StdError` and the `s3_error` macro with the same call syntax as s3s, so the T1.2 codemod is a `use` swap. Statuses are hand-written and pinned to the s3s behaviour RustFS ships today by an oracle test that uses s3s as a dev-dependency only. The `From` conversions for the legacy edge live behind the off-by-default `compat-s3s` feature (the orphan rule forbids them in rustfs/src/error.rs); `rustfs` enables it. Refs rustfs/backlog#2742, rustfs/backlog#2734 * ci: add the per-crate test-count ratchet (#8404) Quick Checks now fails any PR that lowers the number of test attributes in a workspace crate or adds #[ignore], measured per crate against the exact rows of .config/test-count-baseline.txt (rustfs/backlog#2734 section 1 rule 3). The counter widens the spec's literal #[(tokio::)?test] to the tokio attributes that carry arguments and both spellings of #[test_case(...)], and anchors at line start so commented-out attributes do not count. The initial baseline covers 52 crates, 21077 tests and 164 ignored. Refs rustfs/backlog#2738 Refs rustfs/backlog#2734 * test(ecstore): pin the persisted bucket-config XML bytes for 13 families (#8408) Before any DTO or codec is replaced, pin the exact bytes that deserialize followed by serialize produces for every persisted bucket-configuration XML family in its three source shapes (AWS documentation, MinIO-written, RustFS-written), and the bytes RustFS writes today for a typed value. Byte equality only: a structural comparison would let a codec replacement change the persisted encoding unnoticed. The generator script refuses to overwrite existing goldens without --force, refuses on dirty codec paths, cannot report success from its write mode, and records the generating commit in the fixtures README. Refs rustfs/backlog#2744, rustfs/backlog#2734 * ci: raise the test-count baseline to the integration tree (#8426) * ci: make the s3s footprint baseline exact and lower-only (#8403) * ci: make the s3s footprint baseline exact and lower-only The s3s footprint guard compared three counters against ceilings held in the script itself, so a ceiling could sit above the tree and a drift was fixable by raising it (rustfs/rustfs#5964 raised the s3_error! ceiling to go green). The three values now live in .config/s3s-footprint-baseline.txt as files, s3_error_lines and ecstore_files, the script reads only that file, and a measurement that differs from its baseline in either direction fails: growth prints "s3s footprint grew: <measured> > <baseline>", shrinkage prints "lower the baseline to <measured> in this PR". The counting method is unchanged; the baseline is the current measurement of gateway/integration (209 / 1585 / 36). The header records the end state: once all three reach 0, T4.2 switches the script to failing on any occurrence. Refs rustfs/backlog#2739, rustfs/backlog#2734 * ci: add the s3s edge allowlist mode to the footprint guard (#8405) `scripts/check_s3s_footprint.sh --mode allowlist` fails on every Rust file that references s3s (`s3s::` or `s3s_sigv4`) and is not matched by an entry of the new `.config/s3s-edge-allowlist.txt`; `--dry-run` lists those files and exits 0. The allowlist carries the initial edge from the spec, and the script rejects an entry that names or matches anything under crates/ other than crates/e2e_test. The baseline mode and the argument-less CI invocation are unchanged. Refs rustfs/backlog#2741, rustfs/backlog#2734 * test(ecstore): reach the bucket-config codec through the test facade (#8428) * refactor(s3-types): own the HTTP Body and drop s3s from three crates (#8431) Add rustfs_s3_types::Body, a BoxBody<Bytes, StdError> wrapper with empty() and From<Bytes | Vec<u8> | String | &'static str>, and the legacy From conversions to and from s3s::Body behind compat-s3s. The signer is now generic over the request body: it only reads and writes the request head, so it needs no body type and its callers keep theirs. s3-client sends its requests with the owned Body, madmin signs a bodiless head, and targets drops its two S3Request/S3Response header wrappers in favour of extract_params_header. rustfs-signer, rustfs-madmin and rustfs-targets no longer depend on s3s; the s3s footprint baseline drops from 209 to 202 files. Refs rustfs/backlog#2747, rustfs/backlog#2734 * refactor(ecstore): move S3ErrorCode to rustfs-s3-types by codemod (#8432) * test(ecstore): pin the InvalidRange responses of new_getobjectreader Both error branches of new_getobjectreader build an ErrorResponse with S3ErrorCode::InvalidRange, and no test observed either code. Pin the code, its wire name and the location fields of each branch before the code type moves to rustfs-s3-types (rustfs/backlog#2743), so a changed code on either branch fails a test. The ecstore test-count row rises by the two tests. * chore(scripts): add the s3s-to-rustfs-s3-types error codemod Task T1.2 of rustfs/backlog#2734 (rustfs/backlog#2743) moves S3Error, S3ErrorCode, S3Result and s3_error! from s3s to rustfs-s3-types by script, one crate closure per PR. The codemod rewrites use trees (splitting mixed s3s imports) and qualified code paths, never comments or literals, adds the rustfs-s3-types dependency, drops s3s only from a crate that no longer names it, refuses every file of the s3s edge allowlist, takes --hold for a file whose type is fixed outside the closure, and prints a per-crate census. A second run changes nothing. Its self-test runs under make script-tests. * refactor(ecstore): take S3ErrorCode from rustfs-s3-types Output of scripts/codemods/s3_error_to_s3_types.py over crates/ecstore, crates/lifecycle, crates/scanner and crates/replication (rustfs/backlog#2743 batches 1 and 2). The ErrorResponse code field, the two InvalidRange constructions in new_getobjectreader and an unused tier import now use rustfs_s3_types::S3ErrorCode; the variant names and wire names are unchanged. lifecycle, scanner and replication reference none of the four error names. warm_backend.rs and warm_backend_s3.rs are held: they compare against rustfs_s3_client::ErrorResponse.code, which stays the s3s type until s3-client moves. * ci(footprint): count only s3_error! lines still on s3s's macro s3_error_lines counted every line holding the text `s3_error!`, which also matches the RustFS-owned `rustfs_s3_types::s3_error!`. A codemod that moves call sites onto rustfs-s3-types therefore could not lower it (rustfs/backlog#2743, decision 3; rustfs/rustfs#8432, gotcha 2). The counter now applies a textual, per-file and per-line rule that errs on the side of counting (full rule in the script header): - occurrences spelled `rustfs_s3_types::s3_error!` never count; - a line holding `s3s::s3_error!` always counts; - any other `s3_error!` line counts unless its file is cleared, that is, it imports the macro from rustfs_s3_types by name on a line starting with the use item (or defines it with `macro_rules! s3_error`) and does not also bring s3s's macro in (`use s3s::s3_error`, a `use s3s::{..}` tree naming it or a crate-root `*`, `use s3s::*`, or `#[macro_use] extern crate s3s`); - fail closed: a file matching neither form (`use super::*`, a crate-local re-export, a renamed or glob rustfs_s3_types import, a commented-out import, a comment-only mention) counts. Census of the 1585 lines on 96fc06776: 1478 in 64 files that import the macro from s3s; 107 in 15 files matching neither form (105 through `use super::*`, 1 qualified `s3s::s3_error!`, 1 comment line); none in files that import it from rustfs_s3_types or define it. Every line stays counted, so the baseline remains s3_error_lines=1585 (files=202 and ecstore_files=36 unchanged). Rewriting rustfs/src/auth.rs's import to rustfs_s3_types lowers the new counter to 1567; the old one stayed 1585. `--self-test` (wired into `make script-tests`) copies the script into 22 fixture trees: 6 positive cases and 16 negative ones (lines that must not count, forms that must not clear a file, and a missing baseline, missing key or failing classification search that must fail). Each tree has a control line that must count, and each run gets s3s text on stdin. Seven cases were red against the old counter. 19 mutants each turn their case red: old definition, flipped import test, dropped fail-closed branch, dropped e2e_test exclusion, dropped s3s veto, dropped qualified s3s rule, dropped rustfs-qualified strip, unanchored rustfs import, renamed import accepted, nested glob taken as root glob, single-line s3s trees only, dropped s3s glob forms, dropped macro_use form, dropped definition clearing, ignored classification rg errors, error-line search without its path, classification searches without their path, missing baseline skipped, missing key skipped. * test(e2e): compare two RUSTFS_S3_STACK servers over every S3 operation (#8435) Add dual_stack_differential to e2e_test. It starts two rustfs processes from the same binary, stack A on RUSTFS_S3_STACK=legacy and stack B on RUSTFS_E2E_STACK_B (default legacy until rustfs/backlog#2734 T2.5), sends both the same aws-sdk-s3 script covering the 73 methods of impl S3 for FS (a normal request and a 4xx negative each), and compares status, error code, an allowlist of response headers and a few typed body fields. Error message text is never compared. Each run writes <target>/dual-stack/<run>.json with compared_ops and the (op, field, a, b) differences; any difference fails the test. differential_detects_an_injected_difference starts B with RUSTFS_REGION=eu-west-1 and requires GetBucketLocation to be reported. The comparator is pure and unit-tested without a server. Wire the module into the e2e-smoke filter and update the e2e-smoke and e2e-full selection digests and the e2e_test test-count row. Refs rustfs/backlog#2740 * feat(app): add the RustFS-owned RequestEnvelope and its s3s constructor (#8437) * feat(app): add the RustFS-owned RequestEnvelope and its s3s constructor * ci: classify the s3s edge adapters in storage/s3_compat as interface * feat(s3-types): add Body::from_stream and move Swift off s3s::Body (#8438) rustfs/backlog#2747 batch 3. `Body::from_stream` wraps a chunk stream of unknown length over `StreamBody`/`BoxBody`: one data frame per chunk, the first error ends the body, the size hint stays unknown, and dropping the body drops the stream. Parity tests pin it to the s3s `StreamingBlob::wrap` it replaces, at the `http_body` level and byte for byte through hyper. The Swift handlers in rustfs-protocols build their responses as the owned `Body`. `SwiftService` converts them into the wrapped S3 service's response body type through `From<Body>`, so rustfs keeps serving `s3s::Body` at the legacy edge unchanged. Request bodies stay Send-only (`UnsyncBoxBody`); the two storage writers and the DLO/SLO segment streams get `Sync` from `sync_wrapper::SyncStream`, which is what s3s's `http_body_unsync` mutex provided before. The DTO and error uses of s3s in protocols are untouched (T1.5 and T1.2). * refactor(protocols): move error types to rustfs-s3-types by codemod (#8439) T1.2 batch 3 (rustfs/backlog#2743): s3select-api, s3select-query and the error items of protocols move off s3s by scripts/codemods, run with --hold crates/protocols/src/sftp/errors.rs. That file stays on s3s: it downcasts the backend error, whose runtime type is still s3s::S3Error. - rustfs/src/protocols/client.rs: the two webdav session methods now return rustfs_s3_types::S3Result and convert the s3s error with From. - codemod: a child module that calls s3_error! through `use super::*` gets its own `use rustfs_s3_types::s3_error;` once its parent imports that macro by name; otherwise it is listed as GLOB. - s3-types: a table test pins code, status and the s3s error the legacy edge renders for every variant a moved call site names. - s3_error_lines 1585 -> 1584 (a lifecycle comment reworded). * test(connect): restore one-second drive request budgets Apply the exact test-only repair already landed on main in 5972a09cc3. The filesystem probe and production CLI success test request one second and keep their existing one-second duration and parameter checks. All outcome, I/O, error, cleanup and resource-limit assertions remain. Scope: rustfs/tests/connect_perf_drive.rs only. Patch-id: cb0a1dfc8f68a67727525adea7458e2cee7d09f8. Validation environment: LANG=en_US.UTF-8, CARGO_BUILD_JOBS=1, CARGO_INCREMENTAL=0, exclusive APFS-cloned CARGO_TARGET_DIR. Quick Checks: every command from .github/actions/quick-checks/action.yml, with captured output and exit status below. $ shellcheck --version && actionlint ShellCheck - shell script analysis tool version: 0.11.0 license: GNU General Public License, version 3 website: https://www.shellcheck.net exit_code=0 $ cargo fmt --all --check exit_code=0 $ ./scripts/check_unsafe_code_allowances.sh exit_code=0 $ ./scripts/check_layer_dependencies.sh Layer dependency guard passed (no new reverse dependencies/cycles). exit_code=0 $ ./scripts/check_architecture_migration_rules.sh Architecture migration rules passed. exit_code=0 $ ./scripts/check_logging_guardrails.sh ✅ Logging guardrails check passed exit_code=0 $ ./scripts/check_error_other_format_ratchet.sh error(format!) ratchet passed (total call sites: 582). exit_code=0 $ ./scripts/check_no_tokio_io_uring.sh exit_code=0 $ ./scripts/check_extension_schema_boundaries.sh Extension schema boundaries passed. exit_code=0 $ ./scripts/check_body_cache_whitelist.sh Body-cache whitelist guard passed. exit_code=0 $ ./scripts/check_s3s_footprint.sh s3s footprint OK: files importing s3s is 195 (baseline files=195) s3s footprint OK: s3s s3_error! lines is 1584 (baseline s3_error_lines=1584) s3s footprint OK: ecstore files referencing s3s is 36 (baseline ecstore_files=36) ✅ s3s footprint baseline check passed exit_code=0 $ ./scripts/check_test_count_ratchet.sh test-count ratchet OK: 52 crates, 21222 tests, 164 ignored match .config/test-count-baseline.txt exit_code=0 $ ./scripts/check_fips_wording.sh FIPS wording guard passed (README.md and CHANGELOG.md contain no forbidden claims). KMS wording guard passed (crates/kms claims no Vault KV2 Transit wrapping). exit_code=0 $ ./scripts/check_embedded_secrets.sh Embedded secret guard passed (no private key material or provider credential literal in the tree). exit_code=0 $ make script-tests Running script tests... entrypoint credential policy tests passed internode grpc A/B bench env tests passed object batch benchmark enhanced tests passed PASS package service script contract hotpath warp A/B gate tests passed hotpath warp ABBA tests passed Scanner validation artifacts written to /var/folders/0w/s4wxsn2n4w9b6f3cd4xrz53w0000gn/T/tmp.1pFe6Kl1Yu/out Scanner validation artifacts written to /var/folders/0w/s4wxsn2n4w9b6f3cd4xrz53w0000gn/T/tmp.1pFe6Kl1Yu/out-missing-pid ............................. ---------------------------------------------------------------------- Ran 29 tests in 181.286s OK {"decision":"verified","pending_gates":["G01","G03","G04","G05","G06","G07","G08","G09","G10","G11","G12","G13","G14","P1","P2","P3","P4","R-D","R-E","R-L"],"pending_lanes":["authority-coverage","checkpoint-and-crash","ec8-4-multiset","maintenance-producers","mixed-version-rollback","mrf-responsibility","scheduler-pressure","status-and-outcome"],"rejected_errors":[],"rejected_gate":null,"release_approved":false,"release_schema_capable":true,"schema":1,"verified_gate":"G02"} {"decision":"verified","pending_gates":["G01","G02","G03","G04","G05","G06","G07","G08","G09","G10","G11","G12","G13","G14","P1","P2","P3","P4","R-D","R-L"],"pending_lanes":["authority-coverage","checkpoint-and-crash","ec8-4-multiset","maintenance-producers","mixed-version-rollback","mrf-responsibility","scheduler-pressure","status-and-outcome"],"rejected_errors":[],"rejected_gate":null,"release_approved":false,"release_schema_capable":true,"schema":1,"verified_gate":"R-E"} {"decision":"verified","pending_gates":["G01","G03","G04","G05","G06","G07","G08","G09","G10","G11","G12","G13","G14","P1","P2","P3","P4","R-D","R-E","R-L"],"pending_lanes":["authority-coverage","checkpoint-and-crash","ec8-4-multiset","maintenance-producers","mixed-version-rollback","mrf-responsibility","scheduler-pressure","status-and-outcome"],"rejected_errors":[],"rejected_gate":null,"release_approved":false,"release_schema_capable":true,"schema":1,"verified_gate":"G02"} {"decision":"verified","pending_gates":["G01","G02","G03","G04","G05","G06","G07","G08","G09","G10","G11","G12","G13","G14","P1","P2","P3","P4","R-D","R-L"],"pending_lanes":["authority-coverage","checkpoint-and-crash","ec8-4-multiset","maintenance-producers","mixed-version-rollback","mrf-responsibility","scheduler-pressure","status-and-outcome"],"rejected_errors":[],"rejected_gate":null,"release_approved":false,"release_schema_capable":true,"schema":1,"verified_gate":"R-E"} {"decision":"verified","pending_gates":["G02","G03","G04","G05","G06","G07","G08","G09","G10","G11","G12","G13","G14","P1","P2","P3","P4","R-D","R-E","R-L"],"pending_lanes":["authority-coverage","checkpoint-and-crash","ec8-4-multiset","maintenance-producers","mixed-version-rollback","mrf-responsibility","scheduler-pressure","status-and-outcome"],"rejected_errors":[],"rejected_gate":null,"release_approved":false,"release_schema_capable":true,"schema":1,"verified_gate":"G01"} {"decision":"verified","pending_gates":["G01","G02","G04","G05","G06","G07","G08","G09","G10","G11","G12","G13","G14","P1","P2","P3","P4","R-D","R-E","R-L"],"pending_lanes":["authority-coverage","checkpoint-and-crash","ec8-4-multiset","maintenance-producers","mixed-version-rollback","mrf-responsibility","scheduler-pressure","status-and-outcome"],"rejected_errors":[],"rejected_gate":null,"release_approved":false,"release_schema_capable":true,"schema":1,"verified_gate":"G03"} {"decision":"verified","pending_gates":["G01","G02","G03","G04","G05","G06","G07","G08","G09","G10","G11","G12","G13","G14","P1","P2","P3","P4","R-D","R-E"],"pending_lanes":["authority-coverage","checkpoint-and-crash","ec8-4-multiset","maintenance-producers","mixed-version-rollback","mrf-responsibility","scheduler-pressure","status-and-outcome"],"rejected_errors":[],"rejected_gate":null,"release_approved":false,"release_schema_capable":true,"schema":1,"verified_gate":"R-L"} {"decision":"verified","pending_gates":["G01","G02","G03","G04","G05","G06","G07","G08","G09","G10","G11","G12","G13","P1","P2","P3","P4","R-D","R-E","R-L"],"pending_lanes":["authority-coverage","checkpoint-and-crash","ec8-4-multiset","maintenance-producers","mixed-version-rollback","mrf-responsibility","scheduler-pressure","status-and-outcome"],"rejected_errors":[],"rejected_gate":null,"release_approved":false,"release_schema_capable":true,"schema":1,"verified_gate":"G14"} {"decision":"verified","pending_gates":["G01","G02","G03","G04","G05","G06","G07","G08","G09","G10","G11","G12","G13","P1","P2","P3","P4","R-D","R-E","R-L"],"pending_lanes":["authority-coverage","checkpoint-and-crash","ec8-4-multiset","maintenance-producers","mixed-version-rollback","mrf-responsibility","scheduler-pressure","status-and-outcome"],"rejected_errors":[],"rejected_gate":null,"release_approved":false,"release_schema_capable":true,"schema":1,"verified_gate":"G14"} {"decision":"verified","pending_gates":["G01","G02","G03","G04","G05","G06","G07","G08","G09","G11","G12","G13","G14","P1","P2","P3","P4","R-D","R-E","R-L"],"pending_lanes":["authority-coverage","checkpoint-and-crash","ec8-4-multiset","maintenance-producers","mixed-version-rollback","mrf-responsibility","scheduler-pressure","status-and-outcome"],"rejected_errors":[],"rejected_gate":null,"release_approved":false,"release_schema_capable":true,"schema":1,"verified_gate":"G10"} {"decision":"verified","pending_gates":["G01","G02","G03","G04","G05","G06","G07","G08","G09","G10","G11","G12","G13","G14","P2","P3","P4","R-D","R-E","R-L"],"pending_lanes":["authority-coverage","checkpoint-and-crash","ec8-4-multiset","maintenance-producers","mixed-version-rollback","mrf-responsibility","scheduler-pressure","status-and-outcome"],"rejected_errors":[],"rejected_gate":null,"release_approved":false,"release_schema_capable":true,"schema":1,"verified_gate":"P1"} {"decision":"verified","pending_gates":["G01","G02","G03","G04","G05","G06","G07","G08","G09","G10","G11","G12","G13","G14","P1","P3","P4","R-D","R-E","R-L"],"pending_lanes":["authority-coverage","checkpoint-and-crash","ec8-4-multiset","maintenance-producers","mixed-version-rollback","mrf-responsibility","scheduler-pressure","status-and-outcome"],"rejected_errors":[],"rejected_gate":null,"release_approved":false,"release_schema_capable":true,"schema":1,"verified_gate":"P2"} {"decision":"verified","pending_gates":["G01","G02","G03","G04","G05","G06","G07","G08","G09","G10","G11","G12","G13","G14","P1","P2","P4","R-D","R-E","R-L"],"pending_lanes":["authority-coverage","checkpoint-and-crash","ec8-4-multiset","maintenance-producers","mixed-version-rollback","mrf-responsibility","scheduler-pressure","status-and-outcome"],"rejected_errors":[],"rejected_gate":null,"release_approved":false,"release_schema_capable":true,"schema":1,"verified_gate":"P3"} {"decision":"verified","pending_gates":["G01","G02","G03","G04","G06","G07","G08","G09","G10","G11","G12","G13","G14","P1","P2","P3","P4","R-D","R-E","R-L"],"pending_lanes":["authority-coverage","checkpoint-and-crash","ec8-4-multiset","maintenance-producers","mixed-version-rollback","mrf-responsibility","scheduler-pressure","status-and-outcome"],"rejected_errors":[],"rejected_gate":null,"release_approved":false,"release_schema_capable":true,"schema":1,"verified_gate":"G05"} {"decision":"verified","pending_gates":["G01","G02","G03","G04","G05","G07","G08","G09","G10","G11","G12","G13","G14","P1","P2","P3","P4","R-D","R-E","R-L"],"pending_lanes":["authority-coverage","checkpoint-and-crash","ec8-4-multiset","maintenance-producers","mixed-version-rollback","mrf-responsibility","scheduler-pressure","status-and-outcome"],"rejected_errors":[],"rejected_gate":null,"release_approved":false,"release_schema_capable":true,"schema":1,"verified_gate":"G06"} {"decision":"verified","pending_gates":["G01","G02","G03","G04","G05","G06","G07","G08","G09","G10","G11","G12","G13","G14","P1","P2","P3","P4","R-E","R-L"],"pending_lanes":["authority-coverage","checkpoint-and-crash","ec8-4-multiset","maintenance-producers","mixed-version-rollback","mrf-responsibility","scheduler-pressure","status-and-outcome"],"rejected_errors":[],"rejected_gate":null,"release_approved":false,"release_schema_capable":true,"schema":1,"verified_gate":"R-D"} Status-and-outcome release descriptor verified: /private/var/folders/0w/s4wxsn2n4w9b6f3cd4xrz53w0000gn/T/tmp8rwn1pe0/descriptor/release-bundle-status-outcome.json {"decision":"verified","pending_gates":["G01","G02","G03","G04","G06","G07","G08","G09","G10","G11","G12","G13","G14","P1","P2","P3","P4","R-D","R-E","R-L"],"pending_lanes":["authority-coverage","checkpoint-and-crash","ec8-4-multiset","maintenance-producers","mixed-version-rollback","mrf-responsibility","scheduler-pressure","status-and-outcome"],"rejected_errors":[],"rejected_gate":null,"release_approved":false,"release_schema_capable":true,"schema":1,"verified_gate":"G05"} {"decision":"verified","pending_gates":["G01","G02","G03","G04","G05","G07","G08","G09","G10","G11","G12","G13","G14","P1","P2","P3","P4","R-D","R-E","R-L"],"pending_lanes":["authority-coverage","checkpoint-and-crash","ec8-4-multiset","maintenance-producers","mixed-version-rollback","mrf-responsibility","scheduler-pressure","status-and-outcome"],"rejected_errors":[],"rejected_gate":null,"release_approved":false,"release_schema_capable":true,"schema":1,"verified_gate":"G06"} {"decision":"verified","pending_gates":["G01","G02","G03","G04","G05","G06","G07","G08","G09","G10","G11","G12","G13","G14","P1","P2","P3","P4","R-E","R-L"],"pending_lanes":["authority-coverage","checkpoint-and-crash","ec8-4-multiset","maintenance-producers","mixed-version-rollback","mrf-responsibility","scheduler-pressure","status-and-outcome"],"rejected_errors":[],"rejected_gate":null,"release_approved":false,"release_schema_capable":true,"schema":1,"verified_gate":"R-D"} {"decision":"verified","pending_gates":["G01","G02","G03","G04","G05","G06","G08","G09","G10","G11","G12","G13","G14","P1","P2","P3","P4","R-D","R-E","R-L"],"pending_lanes":["authority-coverage","checkpoint-and-crash","ec8-4-multiset","maintenance-producers","mixed-version-rollback","mrf-responsibility","scheduler-pressure","status-and-outcome"],"rejected_errors":[],"rejected_gate":null,"release_approved":false,"release_schema_capable":true,"schema":1,"verified_gate":"G07"} {"decision":"verified","pending_gates":["G01","G02","G03","G04","G05","G06","G07","G09","G10","G11","G12","G13","G14","P1","P2","P3","P4","R-D","R-E","R-L"],"pending_lanes":["authority-coverage","checkpoint-and-crash","ec8-4-multiset","maintenance-producers","mixed-version-rollback","mrf-responsibility","scheduler-pressure","status-and-outcome"],"rejected_errors":[],"rejected_gate":null,"release_approved":false,"release_schema_capable":true,"schema":1,"verified_gate":"G08"} {"decision":"verified","pending_gates":["G01","G02","G03","G04","G05","G06","G07","G08","G09","G10","G11","G12","G13","G14","P1","P2","P3","R-D","R-E","R-L"],"pending_lanes":["authority-coverage","checkpoint-and-crash","ec8-4-multiset","maintenance-producers","mixed-version-rollback","mrf-responsibility","scheduler-pressure","status-and-outcome"],"rejected_errors":[],"rejected_gate":null,"release_approved":false,"release_schema_capable":true,"schema":1,"verified_gate":"P4"} exact 1MiB handoff ABBA tests passed pinned paired ABBA bench tests passed ==> /var/folders/0w/s4wxsn2n4w9b6f3cd4xrz53w0000gn/T/tmp.mYmuI01hNq/bin/path_containment (-max_total_time=7, -seed=123456789, -artifact_prefix=/var/folders/0w/s4wxsn2n4w9b6f3cd4xrz53w0000gn/T/tmp.mYmuI01hNq/explicit-artifacts/path_containment/, corpus=/private/tmp/gw-orchestrator-2734/ci-drive/fuzz/corpus/path_containment) -seed=123456789 target=path_containment seed=123456789 max_total_time=7 git_revision=4d15b816236630b2c42f321ffc9ed3654edc94e3 git_dirty=true runner_mode=prebuilt ==> /var/folders/0w/s4wxsn2n4w9b6f3cd4xrz53w0000gn/T/tmp.mYmuI01hNq/bin/path_containment (-max_total_time=1, -seed=2322617706, -artifact_prefix=/var/folders/0w/s4wxsn2n4w9b6f3cd4xrz53w0000gn/T/tmp.mYmuI01hNq/auto-artifacts/path_containment/, corpus=/private/tmp/gw-orchestrator-2734/ci-drive/fuzz/corpus/path_containment) -seed=2322617706 ==> cargo +nightly fuzz run path_containment (-max_total_time=9, -seed=987654321, -artifact_prefix=/var/folders/0w/s4wxsn2n4w9b6f3cd4xrz53w0000gn/T/tmp.mYmuI01hNq/cargo-artifacts/path_containment/) +nightly fuzz run path_containment -seed=987654321 seed=987654321 runner_mode=cargo-fuzz fuzz runner seed manifest tests passed ✅ scripts/python_bin.sh resolver checks passed self-test ok: clean fixture with every exemption present passes self-test ok: all 11 pattern families fire self-test ok: moving a pinned source is reported Embedded secret guard self-test passed. self-test ok: p1-s3s-import: 'use s3s::s3_error;' counts every line, the rustfs_s3_types module's too (s3_error_lines=3) self-test ok: p2-s3s-mixed-tree: a multi-line s3s use tree naming s3_error counts (s3_error_lines=3) self-test ok: p3-s3s-glob: a crate-root glob of s3s, bare or inside a tree, counts (s3_error_lines=5) self-test ok: p4-macro-use: '#[macro_use] extern crate s3s' counts (s3_error_lines=3) self-test ok: p5-qualified-s3s: a qualified s3s::s3_error! line counts in a cleared file (s3_error_lines=2) self-test ok: p6-exact-pass: a baseline equal to the tree passes (exit 0) self-test ok: n1-rustfs-import: 'use rustfs_s3_types::s3_error;' alone counts nothing (s3_error_lines=1) self-test ok: n2-rustfs-mixed-tree: a rustfs_s3_types tree plus an s3s tree without the macro counts nothing (s3_error_lines=1) self-test ok: n3-defines: the file defining the macro counts nothing (s3_error_lines=1) self-test ok: n4-qualified-rustfs: a qualified rustfs_s3_types::s3_error! line counts nothing (s3_error_lines=1) self-test ok: n5-renamed-call: a call through a renamed macro is not text-matched (out of scope) (s3_error_lines=1) self-test ok: n6-comment-string: a comment or string mentioning s3_error! in a cleared file counts nothing (s3_error_lines=1) self-test ok: n7-e2e-excluded: crates/e2e_test is excluded (s3_error_lines=1) self-test ok: n8-super-glob: an unrecognised import form ('use super::*;') counts (s3_error_lines=3) self-test ok: n9-facade: a crate-local re-export counts (s3_error_lines=2) self-test ok: n10-renamed-import: a renamed rustfs_s3_types import does not clear s3_error! lines (s3_error_lines=2) self-test ok: n11-commented-import: a commented-out rustfs_s3_types import does not clear (s3_error_lines=2) self-test ok: n12-rustfs-glob: a glob import of rustfs_s3_types does not clear (s3_error_lines=2) self-test ok: n13-comment-only: a comment mentioning s3_error! in a file with no import counts (s3_error_lines=2) self-test ok: f1-baseline-missing: a missing baseline file fails (exit 1) self-test ok: f2-key-missing: a baseline without s3_error_lines fails (exit 1) self-test ok: f3-rg-fails: a failing classification search fails the guard (exit 1) ✅ s3s footprint self-test passed test_core_gate_rejects_missing_ignored_filtered_and_corrupt_inputs (__main__.SelfTests.test_core_gate_rejects_missing_ignored_filtered_and_corrupt_inputs) ... ok test_e2e_requires_registration (__main__.SelfTests.test_e2e_requires_registration) ... ok test_fuzz_runtime_uses_matrix_target (__main__.SelfTests.test_fuzz_runtime_uses_matrix_target) ... ok test_ilm_lane_keeps_the_measured_cargo_build_budget (__main__.SelfTests.test_ilm_lane_keeps_the_measured_cargo_build_budget) ... ok test_profile_listing_binds_platform_digest (__main__.SelfTests.test_profile_listing_binds_platform_digest) ... ok test_profile_listing_enforces_selection (__main__.SelfTests.test_profile_listing_enforces_selection) ... ok test_quick_checks_commands_propagate_failure (__main__.SelfTests.test_quick_checks_commands_propagate_failure) ... ok test_quick_checks_rejects_caller_and_execution_bypasses (__main__.SelfTests.test_quick_checks_rejects_caller_and_execution_bypasses) ... ok test_runner_readiness_fails_closed (__main__.SelfTests.test_runner_readiness_fails_closed) ... ok test_s3_runner_rejects_unbounded_failure_locals (__main__.SelfTests.test_s3_runner_rejects_unbounded_failure_locals) ... ok test_scanner_heal_begin_requires_embedded_source_provenance (__main__.SelfTests.test_scanner_heal_begin_requires_embedded_source_provenance) ... ok test_scanner_heal_case_does_not_approve_pending_release (__main__.SelfTests.test_scanner_heal_case_does_not_approve_pending_release) ... ok test_scanner_heal_case_lane_is_required_for_runner_profile (__main__.SelfTests.test_scanner_heal_case_lane_is_required_for_runner_profile) ... ok test_scanner_heal_case_only_schema_cannot_approve_release (__main__.SelfTests.test_scanner_heal_case_only_schema_cannot_approve_release) ... ok test_scanner_heal_crash_case_rejects_restart_oracle (__main__.SelfTests.test_scanner_heal_crash_case_rejects_restart_oracle) ... ok test_scanner_heal_ec84_case_rejects_wrong_erasure_geometry (__main__.SelfTests.test_scanner_heal_ec84_case_rejects_wrong_erasure_geometry) ... ok test_scanner_heal_finish_collects_oracles_from_registry (__main__.SelfTests.test_scanner_heal_finish_collects_oracles_from_registry) ... ok test_scanner_heal_multipool_case_accepts_deferred_optional_outage_write (__main__.SelfTests.test_scanner_heal_multipool_case_accepts_deferred_optional_outage_write) ... ok test_scanner_heal_pending_gate_cannot_map_to_implemented_lane (__main__.SelfTests.test_scanner_heal_pending_gate_cannot_map_to_implemented_lane) ... ok test_scanner_heal_receipts_reject_reuse_and_missing_builds (__main__.SelfTests.test_scanner_heal_receipts_reject_reuse_and_missing_builds) ... ok test_scanner_heal_rejects_boolean_fractional_and_out_of_geometry_integers (__main__.SelfTests.test_scanner_heal_rejects_boolean_fractional_and_out_of_geometry_integers) ... ok test_scanner_heal_rejects_broken_execution_and_artifacts (__main__.SelfTests.test_scanner_heal_rejects_broken_execution_and_artifacts) ... ok test_scanner_heal_rejects_copied_junit_and_wrong_suite_build (__main__.SelfTests.test_scanner_heal_rejects_copied_junit_and_wrong_suite_build) ... ok test_scanner_heal_release_bundle_accepts_complete_measured_evidence (__main__.SelfTests.test_scanner_heal_release_bundle_accepts_complete_measured_evidence) ... ok test_scanner_heal_release_bundle_assembler_copies_measured_artifacts (__main__.SelfTests.test_scanner_heal_release_bundle_assembler_copies_measured_artifacts) ... ok test_scanner_heal_release_bundle_assembler_merges_measured_descriptors (__main__.SelfTests.test_scanner_heal_release_bundle_assembler_merges_measured_descriptors) ... ok test_scanner_heal_release_bundle_assembler_rejects_duplicate_descriptor_gate (__main__.SelfTests.test_scanner_heal_release_bundle_assembler_rejects_duplicate_descriptor_gate) ... ok test_scanner_heal_release_bundle_assembler_rejects_fixture_descriptor (__main__.SelfTests.test_scanner_heal_release_bundle_assembler_rejects_fixture_descriptor) ... ok test_scanner_heal_release_bundle_assembler_rejects_hash_mismatch (__main__.SelfTests.test_scanner_heal_release_bundle_assembler_rejects_hash_mismatch) ... ok test_scanner_heal_release_bundle_assembler_rejects_mixed_descriptor_revisions (__main__.SelfTests.test_scanner_heal_release_bundle_assembler_rejects_mixed_descriptor_revisions) ... ok test_scanner_heal_release_bundle_enforces_topology_duration_profile_and_versions (__main__.SelfTests.test_scanner_heal_release_bundle_enforces_topology_duration_profile_and_versions) ... ok test_scanner_heal_release_bundle_fixture_writer_is_not_release_evidence (__main__.SelfTests.test_scanner_heal_release_bundle_fixture_writer_is_not_release_evidence) ... ok test_scanner_heal_release_bundle_gate_accepts_single_measured_gate (__main__.SelfTests.test_scanner_heal_release_bundle_gate_accepts_single_measured_gate) ... ok test_scanner_heal_release_bundle_gate_rejects_missing_field (__main__.SelfTests.test_scanner_heal_release_bundle_gate_rejects_missing_field) ... ok test_scanner_heal_release_bundle_gate_rejects_unknown_gate (__main__.SelfTests.test_scanner_heal_release_bundle_gate_rejects_unknown_gate) ... ok test_scanner_heal_release_bundle_rejects_synthetic_or_missing_fields (__main__.SelfTests.test_scanner_heal_release_bundle_rejects_synthetic_or_missing_fields) ... ok test_scanner_heal_release_bundle_requires_domain_evidence (__main__.SelfTests.test_scanner_heal_release_bundle_requires_domain_evidence) ... ok test_scanner_heal_release_bundle_requires_field_provenance (__main__.SelfTests.test_scanner_heal_release_bundle_requires_field_provenance) ... ok test_scanner_heal_release_bundle_requires_json_artifact_provenance (__main__.SelfTests.test_scanner_heal_release_bundle_requires_json_artifact_provenance) ... ok test_scanner_heal_release_bundle_requires_mixed_version_field_roles (__main__.SelfTests.test_scanner_heal_release_bundle_requires_mixed_version_field_roles) ... ok test_scanner_heal_release_bundle_requires_same_gate_measurement_window (__main__.SelfTests.test_scanner_heal_release_bundle_requires_same_gate_measurement_window) ... ok test_scanner_heal_release_consumes_multiple_registry_oracles (__main__.SelfTests.test_scanner_heal_release_consumes_multiple_registry_oracles) ... ok test_scanner_heal_release_matrix_lanes_remain_pending_without_real_evidence (__main__.SelfTests.test_scanner_heal_release_matrix_lanes_remain_pending_without_real_evidence) ... ok test_scanner_heal_release_requirements_cannot_be_cleared (__main__.SelfTests.test_scanner_heal_release_requirements_cannot_be_cleared) ... ok test_scanner_heal_release_status_rejects_focused_case_run (__main__.SelfTests.test_scanner_heal_release_status_rejects_focused_case_run) ... ok test_scanner_heal_release_status_rejects_synthetic_case (__main__.SelfTests.test_scanner_heal_release_status_rejects_synthetic_case) ... ok test_scanner_heal_required_evidence_fields_cannot_be_removed (__main__.SelfTests.test_scanner_heal_required_evidence_fields_cannot_be_removed) ... ok test_scheduled_alerts_require_completion_watchdog (__main__.SelfTests.test_scheduled_alerts_require_completion_watchdog) ... ok test_update_profile_selection_changes_only_requested_platform (__main__.SelfTests.test_update_profile_selection_changes_only_requested_platform) ... ok test_validate_includes_quick_checks (__main__.SelfTests.test_validate_includes_quick_checks) ... ok test_vault_tests_require_cross_process_serialization (__main__.SelfTests.test_vault_tests_require_cross_process_serialization) ... ok test_workflow_readiness_requires_dependency_probe (__main__.SelfTests.test_workflow_readiness_requires_dependency_probe) ... ok ---------------------------------------------------------------------- Ran 52 tests in 14.301s OK e2e-full selection OK: 1 tests, sha256=29e30e6ed2ed38d9e5b0a7cdd9536abf5c719f7021a02b578ae1381563a3c273 ................................... ---------------------------------------------------------------------- Ran 35 tests in 1.161s OK ................... ---------------------------------------------------------------------- Ran 19 tests in 17.397s OK .......... ---------------------------------------------------------------------- Ran 10 tests in 0.183s OK test_actual_selector_bootstrap_uses_base_policy_and_fails_closed (__main__.SelfTests.test_actual_selector_bootstrap_uses_base_policy_and_fails_closed) ... ok test_documentation_paths_do_not_hide_build_or_fixture_changes (__main__.SelfTests.test_documentation_paths_do_not_hide_build_or_fixture_changes) ... ok test_e2e_scope_keeps_production_and_shared_configuration_full (__main__.SelfTests.test_e2e_scope_keeps_production_and_shared_configuration_full) ... ok test_e2e_shortcut_rejects_new_direct_renamed_and_target_dependencies (__main__.SelfTests.test_e2e_shortcut_rejects_new_direct_renamed_and_target_dependencies) ... ok test_e2e_shortcut_traverses_implicit_path_dependencies (__main__.SelfTests.test_e2e_shortcut_traverses_implicit_path_dependencies) ... ok test_event_contract_requires_complete_candidate_and_optional_lanes (__main__.SelfTests.test_event_contract_requires_complete_candidate_and_optional_lanes) ... ok test_every_wrong_result_missing_job_or_selection_fails_closed (__main__.SelfTests.test_every_wrong_result_missing_job_or_selection_fails_closed) ... ok test_full_e2e_gate_preserves_workflow_branch_and_event_scope (__main__.SelfTests.test_full_e2e_gate_preserves_workflow_branch_and_event_scope) ... ok test_git_range_includes_deleted_source_and_rename_origins (__main__.SelfTests.test_git_range_includes_deleted_source_and_rename_origins) ... ok test_job_ids_and_display_names_cannot_hide_validation (__main__.SelfTests.test_job_ids_and_display_names_cannot_hide_validation) ... ok test_report_only_stack_lane_mirrors_e2e_smoke_outside_the_aggregate (__main__.SelfTests.test_report_only_stack_lane_mirrors_e2e_smoke_outside_the_aggregate) ... ok test_repository_wiring_and_missing_dependency_regression (__main__.SelfTests.test_repository_wiring_and_missing_dependency_regression) ... ok test_verify_command_preserves_failures (__main__.SelfTests.test_verify_command_preserves_failures) ... ok ---------------------------------------------------------------------- Ran 13 tests in 3.280s OK security coverage self-test passed test_api_failures_preserve_other_evidence_and_never_enter_grace (__main__.SelfTests.test_api_failures_preserve_other_evidence_and_never_enter_grace) ... ok test_attempt_outcome_does_not_replace_recent_success (__main__.SelfTests.test_attempt_outcome_does_not_replace_recent_success) ... ok test_cancelled_attempt_cannot_refresh_expired_success (__main__.SelfTests.test_cancelled_attempt_cannot_refresh_expired_success) ... ok test_cli_requires_the_repository_default_branch (__main__.SelfTests.test_cli_requires_the_repository_default_branch) ... ok test_config_rejects_duplicate_and_invalid_entries (__main__.SelfTests.test_config_rejects_duplicate_and_invalid_entries) ... ok test_freshness_boundaries (__main__.SelfTests.test_freshness_boundaries) ... ok test_grace_requires_two_successful_queries_with_no_history (__main__.SelfTests.test_grace_requires_two_successful_queries_with_no_history) ... ok test_invalid_api_evidence_fails_closed (__main__.SelfTests.test_invalid_api_evidence_fails_closed) ... ok test_report_retains_every_workflow (__main__.SelfTests.test_report_retains_every_workflow) ... ok test_requests_filter_schedule_default_branch_and_success_on_server (__main__.SelfTests.test_requests_filter_schedule_default_branch_and_success_on_server) ... ok ---------------------------------------------------------------------- Ran 10 tests in 0.051s OK .............................. ---------------------------------------------------------------------- Ran 30 tests in 22.354s OK .......... ---------------------------------------------------------------------- Ran 10 tests in 2.969s OK ................................. ---------------------------------------------------------------------- Ran 33 tests in 0.160s OK Online functional runner labels: pf-testing ....... ---------------------------------------------------------------------- Ran 7 tests in 0.002s OK ....... ---------------------------------------------------------------------- Ran 7 tests in 0.001s OK .... ---------------------------------------------------------------------- Ran 4 tests in 0.128s OK ... ---------------------------------------------------------------------- Ran 3 tests in 0.547s OK PASS: cold stampede script self-test exit_code=0 $ python3 ./scripts/test_object_generation_protocol_model.py test_a_recovery_barrier_cannot_choose_an_empty_value (__main__.ObjectGenerationProtocolModel.test_a_recovery_barrier_cannot_choose_an_empty_value) ... ok test_all_bounded_two_proposer_interleavings_choose_at_most_one_value (__main__.ObjectGenerationProtocolModel.test_all_bounded_two_proposer_interleavings_choose_at_most_one_value) ... ok test_an_unavailable_decision_quorum_cannot_choose_a_candidate (__main__.ObjectGenerationProtocolModel.test_an_unavailable_decision_quorum_cannot_choose_a_candidate) ... ok test_counter_allocation_requires_persistence_and_is_consumed_once (__main__.ObjectGenerationProtocolModel.test_counter_allocation_requires_persistence_and_is_consumed_once) ... ok test_counter_restart_burns_persisted_allocations_before_or_after_issue (__main__.ObjectGenerationProtocolModel.test_counter_restart_burns_persisted_allocations_before_or_after_issue) ... ok test_crash_after_promise_before_reply_does_not_restore_an_older_ballot (__main__.ObjectGenerationProtocolModel.test_crash_after_promise_before_reply_does_not_restore_an_older_ballot) ... ok test_duplicate_or_unconfigured_voters_cannot_form_a_quorum (__main__.ObjectGenerationProtocolModel.test_duplicate_or_unconfigured_voters_cannot_form_a_quorum) ... ok test_empty_or_unobserved_minority_slot_can_retry_without_allocating_another_revision (__main__.ObjectGenerationProtocolModel.test_empty_or_unobserved_minority_slot_can_retry_without_allocating_another_revision) ... ok test_equal_ballot_cannot_accept_different_bytes_and_lower_ballot_is_rejected (__main__.ObjectGenerationProtocolModel.test_equal_ballot_cannot_accept_different_bytes_and_lower_ballot_is_rejected) ... ok test_forgetting_tombstone_authority_allows_an_aba_create (__main__.ObjectGenerationProtocolModel.test_forgetting_tombstone_authority_allows_an_aba_create) ... ok test_issuing_before_persistence_reuses_a_ballot_after_restart (__main__.ObjectGenerationProtocolModel.test_issuing_before_persistence_reuses_a_ballot_after_restart) ... ok test_lost_ack_and_restart_preserve_the_chosen_operation (__main__.ObjectGenerationProtocolModel.test_lost_ack_and_restart_preserve_the_chosen_operation) ... ok test_minority_accepted_work_is_adopted_and_finished (__main__.ObjectGenerationProtocolModel.test_minority_accepted_work_is_adopted_and_finished) ... ok test_omitting_highest_accepted_adoption_has_a_double_choice_counterexample (__main__.ObjectGenerationProtocolModel.test_omitting_highest_accepted_adoption_has_a_double_choice_counterexample) ... ok test_quorum_choice_does_not_require_an_uncontacted_disk_to_learn_it (__main__.ObjectGenerationProtocolModel.test_quorum_choice_does_not_require_an_uncontacted_disk_to_learn_it) ... ok test_recovery_failure_or_stale_expected_head_does_not_allocate_a_successor (__main__.ObjectGenerationProtocolModel.test_recovery_failure_or_stale_expected_head_does_not_allocate_a_successor) ... ok test_replacing_voters_with_empty_state_can_choose_a_conflicting_value (__main__.ObjectGenerationProtocolModel.test_replacing_voters_with_empty_state_can_choose_a_conflicting_value) ... ok test_skipping_predecessor_recovery_chooses_a_broken_revision_chain (__main__.ObjectGenerationProtocolModel.test_skipping_predecessor_recovery_chooses_a_broken_revision_chain) ... ok test_stale_publication_rollback_and_gc_cannot_change_the_winner (__main__.ObjectGenerationProtocolModel.test_stale_publication_rollback_and_gc_cannot_change_the_winner) ... ok test_superseded_operation_identity_cannot_be_reused_for_different_input (__main__.ObjectGenerationProtocolModel.test_superseded_operation_identity_cannot_be_reused_for_different_input) ... ok test_superseded_operation_retry_returns_its_original_successor (__main__.ObjectGenerationProtocolModel.test_superseded_operation_retry_returns_its_original_successor) ... ok test_tombstone_and_null_marker_payloads_do_not_reset_authority_revision (__main__.ObjectGenerationProtocolModel.test_tombstone_and_null_marker_payloads_do_not_reset_authority_revision) ... ok test_tombstone_rejects_a_delayed_never_present_create (__main__.ObjectGenerationProtocolModel.test_tombstone_rejects_a_delayed_never_present_create) ... ok test_unlearned_predecessor_is_recovered_before_the_second_slot (__main__.ObjectGenerationProtocolModel.test_unlearned_predecessor_is_recovered_before_the_second_slot) ... ok ---------------------------------------------------------------------- Ran 24 tests in 0.958s OK model: states=51023, transitions=167520, terminal=732, chosen_candidates=2 negative control: double-choice trace=((0, 'prepare', 0), (0, 'prepare', 1), (0, 'prepare', 2), (0, 'accept', 0), (0, 'accept', 1), (0, 'accept', 2), (1, 'prepare', 0), (1, 'prepare', 1), (1, 'prepare', 2), (1, 'accept', 0), (1, 'accept', 1), (1, 'accept', 2)) multi-slot round schedules: 640 exit_code=0 $ python3 ./scripts/check_test_wiring.py OK: e2e modules, runner selection, fuzz matrices, profiles, scheduled alerts, and Quick Checks are wired exit_code=0 $ ./scripts/check_no_planning_docs.sh exit_code=0 $ ./scripts/check_uring_lane_lib_only.sh OK: no 'uring_'-matching test functions under crates/ecstore/tests; --lib is safe exit_code=0 Initial complete gate $ cargo nextest run --profile ci -p rustfs --test connect_perf_drive --no-fail-fast Nextest run ID e07fc5b9-c6c8-4cdd-ae61-54839bf24437 with nextest profile: ci Starting 10 tests across 1 binary PASS [ 0.016s] rustfs::connect_perf_drive cancellation_and_single_collector_leave_no_scratch PASS [ 0.011s] rustfs::connect_perf_drive cleanup_failure_overrides_cancellation_and_remains_attributed PASS [ 0.009s] rustfs::connect_perf_drive consent_and_resource_budgets_fail_before_scratch_io PASS [ 0.009s] rustfs::connect_perf_drive invalid_roots_and_existing_scratch_fail_without_clobbering PASS [ 16.950s] rustfs::connect_perf_drive production_cli_measures_and_signs_exact_binary_provenance PASS [ 4.025s] rustfs::connect_perf_drive production_cli_rejects_missing_l1_consent_and_limits_before_artifacts PASS [ 0.008s] rustfs::connect_perf_drive read_only_root_and_full_drive_errors_are_classified PASS [ 0.015s] rustfs::connect_perf_drive real_local_write_read_is_measured_and_scratch_is_removed PASS [ 0.007s] rustfs::connect_perf_drive runtime_timeout_and_full_drive_use_schema_valid_failed_reason PASS [ 0.035s] rustfs::connect_perf_drive successful_result_has_signed_bounded_private_offline_export ──────────── Summary [ 21.087s] 10 tests run: 10 passed, 0 skipped exit_code=0 Restored complete gate $ cargo nextest run --profile ci -p rustfs --test connect_perf_drive --no-fail-fast Nextest run ID 233b7307-5361-4eb7-94d5-b384f17d2d6a with nextest profile: ci Starting 10 tests across 1 binary PASS [ 0.009s] rustfs::connect_perf_drive cancellation_and_single_collector_leave_no_scratch PASS [ 0.008s] rustfs::connect_perf_drive cleanup_failure_overrides_cancellation_and_remains_attributed PASS [ 0.008s] rustfs::connect_perf_drive consent_and_resource_budgets_fail_before_scratch_io PASS [ 0.008s] rustfs::connect_perf_drive invalid_roots_and_existing_scratch_fail_without_clobbering PASS [ 19.518s] rustfs::connect_perf_drive production_cli_measures_and_signs_exact_binary_provenance PASS [ 4.146s] rustfs::connect_perf_drive production_cli_rejects_missing_l1_consent_and_limits_before_artifacts PASS [ 0.009s] rustfs::connect_perf_drive read_only_root_and_full_drive_errors_are_classified PASS [ 0.018s] rustfs::connect_perf_drive real_local_write_read_is_measured_and_scratch_is_removed PASS [ 0.009s] rustfs::connect_perf_drive runtime_timeout_and_full_drive_use_schema_valid_failed_reason PASS [ 0.054s] rustfs::connect_perf_drive successful_result_has_signed_bounded_private_offline_export ──────────── Summary [ 23.787s] 10 tests run: 10 passed, 0 skipped exit_code=0 Mutation M1: success_measurement reports max(actual elapsed, requested budget + 1), forcing 1001 ms for the one-second request. The retained duration assertion at rustfs/tests/connect_perf_drive.rs:105 fails. $ cargo nextest run --profile ci -p rustfs --test connect_perf_drive --no-fail-fast -E 'test(real_local_write_read_is_measured_and_scratch_is_removed)' info: experimental features enabled: setup-scripts Compiling rustfs v1.0.1 (/private/tmp/gw-orchestrator-2734/ci-drive/rustfs) warning: linker stderr: ld: __eh_frame section too large (max 16MB) to encode dwarf unwind offsets in compact unwind table, performance of exception handling might be affected | = note: `#[warn(linker_messages)]` on by default warning: `rustfs` (bin "rustfs") generated 1 warning warning: `rustfs` (bin "rustfs-cli") generated 1 warning (1 duplicate) Finished `test` profile [unoptimized + debuginfo] target(s) in 7m 06s ──────────── Nextest run ID 9f618777-7bf6-4cbc-8803-dd7f50f449ed with nextest profile: ci Starting 1 test across 1 binary (9 tests skipped) FAIL [ 0.016s] rustfs::connect_perf_drive real_local_write_read_is_measured_and_scratch_is_removed stdout ─── running 1 test test real_local_write_read_is_measured_and_scratch_is_removed ... FAILED failures: failures: real_local_write_read_is_measured_and_scratch_is_removed test result: FAILED. 0 passed; 1 failed; 0 ignored; 0 measured; 9 filtered out; finished in 0.01s stderr ─── thread 'real_local_write_read_is_measured_and_scratch_is_removed' (106986846) panicked at rustfs/tests/connect_perf_drive.rs:105:5: assertion failed: (1..=1_000).contains(&data.duration_millis) note: run with `RUST_BACKTRACE=1` environment variable to display a backtrace ──────────── Summary [ 0.017s] 1 test run: 0 passed, 1 failed, 9 skipped FAIL [ 0.016s] rustfs::connect_perf_drive real_local_write_read_is_measured_and_scratch_is_removed error: test run failed exit_code=100 Mutation M2: target_parameters reports requested duration + 1. The in-process assertion at rustfs/tests/connect_perf_drive.rs:111 and CLI JSON assertion at rustfs/tests/connect_perf_drive.rs:527 both reject 1001 against 1000. $ cargo nextest run --profile ci -p rustfs --test connect_perf_drive --no-fail-fast -E 'test(real_local_write_read_is_measured_and_scratch_is_removed) | test(production_cli_measures_and_signs_exact_binary_provenance)' info: experimental features enabled: setup-scripts Compiling rustfs v1.0.1 (/private/tmp/gw-orchestrator-2734/ci-drive/rustfs) warning: linker stderr: ld: __eh_frame section too large (max 16MB) to encode dwarf unwind offsets in compact unwind table, performance of exception handling might be affected | = note: `#[warn(linker_messages)]` on by default warning: `rustfs` (bin "rustfs") generated 1 warning warning: `rustfs` (bin "rustfs-cli") generated 1 warning (1 duplicate) Finished `test` profile [unoptimized + debuginfo] target(s) in 7m 20s ──────────── Nextest run ID b73d46ab-c910-4e1e-8063-7fc7e332ec15 with nextest profile: ci Starting 2 tests across 1 binary (8 tests skipped) FAIL [ 13.968s] rustfs::connect_perf_drive production_cli_measures_and_signs_exact_binary_provenance stdout ─── running 1 test test production_cli_measures_and_signs_exact_binary_provenance ... FAILED failures: failures: production_cli_measures_and_signs_exact_binary_provenance test result: FAILED. 0 passed; 1 failed; 0 ignored; 0 measured; 9 filtered out; finished in 13.96s stderr ─── thread 'production_cli_measures_and_signs_exact_binary_provenance' (107015462) panicked at rustfs/tests/connect_perf_drive.rs:527:5: assertion `left == right` failed left: Number(1001) right: 1000 note: run with `RUST_BACKTRACE=1` environment variable to display a backtrace FAIL [ 0.015s] rustfs::connect_perf_drive real_local_write_read_is_measured_and_scratch_is_removed stdout ─── running 1 test test real_local_write_read_is_measured_and_scratch_is_removed ... FAILED failures: failures: real_local_write_read_is_measured_and_scratch_is_removed test result: FAILED. 0 passed; 1 failed; 0 ignored; 0 measured; 9 filtered out; finished in 0.01s stderr ─── thread 'real_local_write_read_is_measured_and_scratch_is_removed' (107016667) panicked at rustfs/tests/connect_perf_drive.rs:111:5: assertion `left == right` failed left: 1001 right: 1000 note: run with `RUST_BACKTRACE=1` environment variable to display a backtrace ──────────── Summary [ 13.985s] 2 tests run: 0 passed, 2 failed, 8 skipped FAIL [ 13.968s] rustfs::connect_perf_drive production_cli_measures_and_signs_exact_binary_provenance FAIL [ 0.015s] rustfs::connect_perf_drive real_local_write_read_is_measured_and_scratch_is_removed error: test run failed exit_code=100 Transient production mutations were restored byte-for-byte before the complete green rerun. Original/restored SHA256: 8d389e6d8836c041aab8c6a93e56addd65ba3f70b33264e1c74f639d0572a2da. $ git diff --check exit_code=0 $ git diff --cached --check exit_code=0 Independent orchestrator acceptance Tested code tree: e67c72fb01 Tested tree: 16fbbd2ea4 shellcheck --version && actionlint -> exit 0 cargo fmt --all --check -> exit 0 ./scripts/check_unsafe_code_allowances.sh -> exit 0 ./scripts/check_layer_dependencies.sh -> exit 0 ./scripts/check_architecture_migration_rules.sh -> exit 0 ./scripts/check_logging_guardrails.sh -> exit 0 ./scripts/check_error_other_format_ratchet.sh -> exit 0 ./scripts/check_no_tokio_io_uring.sh -> exit 0 ./scripts/check_extension_schema_boundaries.sh -> exit 0 ./scripts/check_body_cache_whitelist.sh -> exit 0 ./scripts/check_s3s_footprint.sh -> exit 0 ./scripts/check_test_count_ratchet.sh -> exit 0 ./scripts/check_fips_wording.sh -> exit 0 ./scripts/check_embedded_secrets.sh -> exit 0 make script-tests -> exit 0 python3 ./scripts/test_object_generation_protocol_model.py -> exit 0 python3 ./scripts/check_test_wiring.py -> exit 0 ./scripts/check_no_planning_docs.sh -> exit 0 ./scripts/check_uring_lane_lib_only.sh -> exit 0 $ LANG=en_US.UTF-8 CARGO_BUILD_JOBS=1 CARGO_INCREMENTAL=0 cargo nextest run --profile ci -p rustfs --test connect_perf_drive --no-fail-fast info: experimental features enabled: setup-scripts warning: linker stderr: ld: __eh_frame section too large (max 16MB) to encode dwarf unwind offsets in compact unwind table, performance of exception handling might be affected | = note: `#[warn(linker_messages)]` on by default warning: `rustfs` (bin "rustfs-cli") generated 1 warning warning: `rustfs` (bin "rustfs") generated 1 warning (1 duplicate) Finished `test` profile [unoptimized + debuginfo] target(s) in 0.69s ──────────── Nextest run ID cd1edf4c-f1c0-456d-ad1a-76cc00ce05c9 with nextest profile: ci Starting 10 tests across 1 binary PASS [ 0.012s] rustfs::connect_perf_drive cancellation_and_single_collector_leave_no_scratch PASS [ 0.010s] rustfs::connect_perf_drive cleanup_failure_overrides_cancellation_and_remains_attributed PASS [ 0.010s] rustfs::connect_perf_drive consent_and_resource_budgets_fail_before_scratch_io PASS [ 0.009s] rustfs::connect_perf_drive invalid_roots_and_existing_scratch_fail_without_clobbering PASS [ 20.328s] rustfs::connect_perf_drive production_cli_measures_and_signs_exact_binary_provenance PASS [ 5.527s] rustfs::connect_perf_drive production_cli_rejects_missing_l1_consent_and_limits_before_artifacts PASS [ 0.013s] rustfs::connect_perf_drive read_only_root_and_full_drive_errors_are_classified PASS [ 0.019s] rustfs::connect_perf_drive real_local_write_read_is_measured_and_scratch_is_removed PASS [ 0.011s] rustfs::connect_perf_drive runtime_timeout_and_full_drive_use_schema_valid_failed_reason PASS [ 0.049s] rustfs::connect_perf_drive successful_result_has_signed_bounded_private_offline_export ──────────── Summary [ 25.991s] 10 tests run: 10 passed, 0 skipped exit_code=0 All 54 tracked Cargo.toml files parse without duplicate keys. git diff --check passed. The production source remains unchanged; the two deliberate implementation mutations above were restored before this complete green run. * fix(test): compare legacy body wire parity as raw bytes Lossy UTF-8 comparison concealed a changed binary response byte. Compare the already captured Vec<u8> values directly. Validation: - Binary mutant: migrated payload 0x84 vs legacy 0x83; old comparator passed 1/1, raw comparator failed 0/1 (exit 100); restored green. - Non-Swift body-drain mutant: untouched-body delegation test failed with empty vs payload (exit 100); restored handler without a diff. - cargo nextest run --profile ci -p rustfs-s3-types -p rustfs-protocols --features rustfs-protocols/swift --no-fail-fast: 543 tests run, 543 passed, 0 skipped after both mutants restored. - cargo fmt --all --check and git diff --check passed. Orchestrator acceptance (LANG=en_US.UTF-8): cargo nextest run --profile ci -p rustfs-s3-types -p rustfs-protocols --features rustfs-protocols/swift --no-fail-fast Summary [2.445s] 543 tests run: 543 passed, 0 skipped. All shared quick checks on the final code tree: shellcheck --version && actionlint -> exit 0 cargo fmt --all --check -> exit 0 ./scripts/check_unsafe_code_allowances.sh -> exit 0 ./scripts/check_layer_dependencies.sh -> exit 0 ./scripts/check_architecture_migration_rules.sh -> exit 0 ./scripts/check_logging_guardrails.sh -> exit 0 ./scripts/check_error_other_format_ratchet.sh -> exit 0 ./scripts/check_no_tokio_io_uring.sh -> exit 0 ./scripts/check_extension_schema_boundaries.sh -> exit 0 ./scripts/check_body_cache_whitelist.sh -> exit 0 ./scripts/check_s3s_footprint.sh -> exit 0 ./scripts/check_test_count_ratchet.sh -> exit 0 ./scripts/check_fips_wording.sh -> exit 0 ./scripts/check_embedded_secrets.sh -> exit 0 make script-tests -> exit 0 python3 ./scripts/test_object_generation_protocol_model.py -> exit 0 python3 ./scripts/check_test_wiring.py -> exit 0 ./scripts/check_no_planning_docs.sh -> exit 0 ./scripts/check_uring_lane_lib_only.sh -> exit 0 Final independent acceptance after the drive-test CI repair Parent: a99d8f4f89 Tested code tree: 93cbd96bf0 Tested tree: 075ae5161c2b62a96812bc1e3c904a7663b4f2f8 shellcheck --version && actionlint -> exit 0 cargo fmt --all --check -> exit 0 ./scripts/check_unsafe_code_allowances.sh -> exit 0 ./scripts/check_layer_dependencies.sh -> exit 0 ./scripts/check_architecture_migration_rules.sh -> exit 0 ./scripts/check_logging_guardrails.sh -> exit 0 ./scripts/check_error_other_format_ratchet.sh -> exit 0 ./scripts/check_no_tokio_io_uring.sh -> exit 0 ./scripts/check_extension_schema_boundaries.sh -> exit 0 ./scripts/check_body_cache_whitelist.sh -> exit 0 ./scripts/check_s3s_footprint.sh -> exit 0 ./scripts/check_test_count_ratchet.sh -> exit 0 ./scripts/check_fips_wording.sh -> exit 0 ./scripts/check_embedded_secrets.sh -> exit 0 make script-tests -> exit 0 python3 ./scripts/test_object_generation_protocol_model.py -> exit 0 python3 ./scripts/check_test_wiring.py -> exit 0 ./scripts/check_no_planning_docs.sh -> exit 0 ./scripts/check_uring_lane_lib_only.sh -> exit 0 $ LANG=en_US.UTF-8 CARGO_BUILD_JOBS=1 CARGO_INCREMENTAL=0 cargo nextest run --profile ci -p rustfs-s3-types -p rustfs-protocols --features rustfs-protocols/swift --no-fail-fast Finished `test` profile [unoptimized + debuginfo] target(s) in 10m 08s Nextest run ID 5c7ff102-ec6d-479a-86e0-810859b28bbf with nextest profile: ci Starting 543 tests across 8 binaries Summary [ 6.729s] 543 tests run: 543 passed, 0 skipped exit_code=0 The only source change is the raw-byte wire comparator. Cargo manifests and the ratchet baselines remain unchanged from the independently validated parent. The invalid-byte and consumed-request mutations remain recorded above; their implementation changes were restored. * fix: repair integration checks and IPC test readiness Wait for the RPC/locks test sockets to pass the existing owner and mode checks inside the original 5s readiness window and 10ms poll interval. Use the existing admin S3 carrier for the new recovery and quota error calls. Regenerate the test-count baseline for main's 118 existing tests across seven crates. Pair the private pagination helper's raw-scan inputs and remove the redundant Copy clone from the retirement event. Keep the original assertions, responses, authorization, locks and storage operations. The pagination body and raw-scan expressions at all five calls are byte-identical; reinserting only the Copy clone restores the entire base heal file. All ignored baseline values and s3s counts stay unchanged. Validation base: c47432e353 Validation tree: f081084bc5 Patch SHA-256: 2dcf0028691bfb2772edfb38e262829b04f4151832dcce823c9dd9f7434b52de Trace production-prefix SHA-256: fd994f218a21429d62866f4efb8eee1ee57b58f4a51865aa35d05a54ef1ccb45 All commands used LANG=en_US.UTF-8, CARGO_BUILD_JOBS=1, CARGO_INCREMENTAL=0 and CARGO_NET_OFFLINE=true in one exclusive target. Every gate retained the same tree and six source hashes before and after. 54 Cargo.toml files parse without duplicate keys. cargo clippy -p rustfs --all-targets -- -D warnings Finished `dev` profile [unoptimized + debuginfo] target(s) in 12m 50s Exit: 0. Wall time: 770.855s. The original 19 quick-check steps: shellcheck --version && actionlint -> exit 0 cargo fmt --all --check -> exit 0 ./scripts/check_unsafe_code_allowances.sh -> exit 0 ./scripts/check_layer_dependencies.sh -> exit 0 ./scripts/check_architecture_migration_rules.sh -> exit 0 ./scripts/check_logging_guardrails.sh -> exit 0 ./scripts/check_error_other_format_ratchet.sh -> exit 0 ./scripts/check_no_tokio_io_uring.sh -> exit 0 ./scripts/check_extension_schema_boundaries.sh -> exit 0 ./scripts/check_body_cache_whitelist.sh -> exit 0 ./scripts/check_s3s_footprint.sh -> exit 0 ./scripts/check_test_count_ratchet.sh -> exit 0 ./scripts/check_fips_wording.sh -> exit 0 ./scripts/check_embedded_secrets.sh -> exit 0 make script-tests -> exit 0 python3 ./scripts/test_object_generation_protocol_model.py -> exit 0 python3 ./scripts/check_test_wiring.py -> exit 0 ./scripts/check_no_planning_docs.sh -> exit 0 ./scripts/check_uring_lane_lib_only.sh -> exit 0 Wall time: 286.337s. cargo nextest run --profile ci -p rustfs --no-fail-fast Summary [ 217.017s] 5382 tests run: 5382 passed (4 leaky), 40 skipped Exit: 0. Wall time: 1324.414s. cargo nextest run --profile ci -p rustfs-s3-types -p rustfs-protocols --features rustfs-protocols/swift --no-fail-fast Summary [ 2.189s] 543 tests run: 543 passed, 0 skipped Exit: 0. Wall time: 123.192s. cargo nextest run --profile ci -p rustfs-ecstore list_objects_paginate --no-fail-fast Summary [ 0.170s] 8 tests run: 8 passed, 5921 skipped Exit: 0. Wall time: 261.324s. cargo nextest run --profile ci -p rustfs-heal durable_mrf_anchor_retires_only_after_exact_bucket_generation_is_deleted --no-fail-fast Summary [ 0.805s] 1 test run: 1 passed, 835 skipped Exit: 0. Wall time: 199.801s. Baseline regeneration: ./scripts/check_test_count_ratchet.sh --update wrote .config/test-count-baseline.txt: 52 crates, 21340 tests, 164 ignored Test-only increases: ecstore48, heal3, iam14, io-metrics2, obs3, policy14, rustfs34. Existing ignored values and all other rows are byte-identical. s3s footprint OK: files importing s3s is 195 (baseline files=195) s3s footprint OK: s3s s3_error! lines is 1584 (baseline s3_error_lines=1584) s3s footprint OK: ecstore files referencing s3s is 36 (baseline ecstore_files=36) Historical runtime red controls on f622 (same retained two assertions): cargo nextest run --profile ci -p rustfs --lib -E 'test(=connect::diagnostics::trace_runtime::tests::local_top_rpc_reads_service_http_completions_and_signs_offline) | test(=connect::diagnostics::trace_runtime::tests::local_top_locks_reads_service_manager_and_signs_offline)' --no-fail-fast Original path readiness with a temporary 500ms bind/seal gap: Summary [ 0.863s] 2 tests run: 0 passed, 2 failed, 5113 skipped Exit: 100. Sealed readiness in that same gap: Summary [ 5.621s] 2 tests run: 2 passed, 5113 skipped Exit: 0. Compiled M1: replace the helper predicate with path.exists(): Summary [ 0.870s] 2 tests run: 0 passed, 2 failed, 5113 skipped Exit: 100. Restore the sealed predicate byte-for-byte: Summary [ 5.640s] 2 tests run: 2 passed, 5113 skipped Exit: 0. Both reds observe mode755 with owner_matches=true and fail at the original RPC/locks unwraps with StateSecurity. M1 failure locations: trace_runtime.rs 2547:29 and 2713:29. No new assertions were introduced. cargo nextest run --profile ci -p rustfs --lib -E 'test(connect::diagnostics::trace_runtime::tests::)' --no-fail-fast Summary [ 20.748s] 28 tests run: 28 passed (1 leaky), 5087 skipped Exit: 0. This security sweep is historical; the full final unit gate above covers the actual c474 candidate. The first disk-stopped compile ran no tests and is excluded from red credit. All temporary seal instrumentation is removed from the final source. Final unit cleanup markers: 4 leaky, 40 original skips (exit0). The ld __eh_frame linker warning also appears verbatim in the original f622 run and the earlier c474 run; there are no new source warnings. * fix: remove redundant pending closure in disk test Pass the capture-free pending future factory as a function item in the existing slow-operation test. This fixes the redundant_closure error at disk_store.rs:2914 that stopped strict Clippy before the feature tests could run. The single-line test edit preserves the factory's result, warning and timeout behavior. Reversing the replacement reproduces the parent file byte for byte. All 220 assertion lines and the production prefix are unchanged; no tests, assertions, guards, profiles or baselines were added or relaxed. Original Linux CI evidence (run 37888552268 at 53245be9f): - RIO job113685483992: strict Clippy exit101, redundant closure at disk_store.rs:2914; feature tests skipped. - Workspace Test and Lint job113685484058: same strict Clippy exit101; nextest and doc tests skipped. These are lint failures, not runtime mutation kills or known flakes. Native macOS arm64 validation on tree7a5bf47e7533b29479bc85207416102f1337c566, LANG=en_US.UTF-8, one build job, CARGO_INCREMENTAL=0 and offline dependencies: shellcheck --version && actionlint -> exit 0 cargo fmt --all --check -> exit 0 ./scripts/check_unsafe_code_allowances.sh -> exit 0 ./scripts/check_layer_dependencies.sh -> exit 0 ./scripts/check_architecture_migration_rules.sh -> exit 0 ./scripts/check_logging_guardrails.sh -> exit 0 ./scripts/check_error_other_format_ratchet.sh -> exit 0 ./scripts/check_no_tokio_io_uring.sh -> exit 0 ./scripts/check_extension_schema_boundaries.sh -> exit 0 ./scripts/check_body_cache_whitelist.sh -> exit 0 ./scripts/check_s3s_footprint.sh -> exit 0 ./scripts/check_test_count_ratchet.sh -> exit 0 ./scripts/check_fips_wording.sh -> exit 0 ./scripts/check_embedded_secrets.sh -> exit 0 make script-tests -> exit 0 python3 ./scripts/test_object_generation_protocol_model.py -> exit 0 python3 ./scripts/check_test_wiring.py -> exit 0 ./scripts/check_no_planning_docs.sh -> exit 0 ./scripts/check_uring_lane_lib_only.sh -> exit 0 All 19 checks: exit0, 312.161s. $ cargo clippy -p rustfs -p rustfs-ecstore --all-targets --features rio-v2 -- -D warnings Finished `dev` profile [unoptimized + debuginfo] target(s) in 20m 11s exit0, 1213.210s total. $ cargo nextest run --profile ci -p rustfs-ecstore --features rio-v2 slow_disk_operation_warns_once_while_pending_and_preserves_result --no-fail-fast Finished `test` profile [unoptimized + debuginfo] target(s) in 4m 34s PASS [0.026s] rustfs-ecstore disk::disk_store::tests::slow_disk_operation_warns_once_while_pending_and_preserves_result Summary [ 0.170s] 1 test run: 1 passed, 5943 skipped exit0, 281.920s total; 5943 existing filter skips. warning: linker stderr: ld: __eh_frame section too large (max 16MB) to encode dwarf unwind offsets in compact unwind table, performance of exception handling might be affected This linker warning is retained; flags, profiles and allowances are unchanged. 54 tracked Cargo.toml files parsed without duplicate keys. git diff --check passed. No new assertions were introduced, so no new runtime mutation is claimed. Native compilation and the existing helper test do not substitute for actual Linux RIO execution; the new integration CI run still needs to recover before the RPC merge. Preservation and output hashes: parent file SHA256: 8e59ee745e3aaaa22848b943d3b10f6353b515956282cde1f3af2c394200c9ac final file SHA256: b69ba3237061b161448ae8b62b615885d5770b98a5b4a2479bcfbd70f1bd1f17 unchanged production prefix SHA256: 7903a6e288252c420231242d7b58a25fa2a7140ea3dd430cbe7f0dde11369dcb RIO Linux failure log SHA256:421addb9868fd88e4109a922311e54f91f0dfd70862f8ba7fcaba193482ed90b Workspace Linux failure log SHA256:e0aea8d0f014c022252a9534968e083c98cfc1fd760da445ecd04ec3f522e26a 19 output SHA256:35a54f28d707956ed1edb29d492c4dfe8a9fe753d103b111b5a2f1b5c41c2b98 RIO Clippy output SHA256:d6046e3868f2b3992fe11cdd04e9e98eb50a87d1234d9a70dbfeb154ff61638d slow-operation output SHA256:605b7b258d2fd09c6ef2ab2ffc90d433d0509480f1e45125d20c823759e83608 Refs rustfs/backlog#2734. * test(ecstore): isolate tier-free recovery fixture startup The fixture constructor can take the production recovery lock and wait for 60 seconds before this test installs its queue and seeds metadata. Disable its background recovery only while creating this fixture, then restore the environment before calling the existing production recovery entrypoint. Only the existing test setup changes. Its production entrypoint, default lock parameters, original assertions, 30-second timeout, cancellation and cleanup remain intact. No tests or assertions were added. Production and store/init.rs are byte-identical to the parent. Independent acceptance ran on merge tree 8ec358cb42ba9ebb3bcfeafcef048a3f0366355b; the runner header reports parent HEAD, while before/after index and source hashes confirm this candidate. Worker task branch d8055766b8 has the same tree. Every command used LANG=en_US.UTF-8. Task compilation also used CARGO_BUILD_JOBS=1, CARGO_INCREMENTAL=0, CARGO_NET_OFFLINE=true, an independent target, and empty RUSTC_WRAPPER/RUSTC_WORKSPACE_WRAPPER to keep compilers in its owned process group without changing repository configuration. Command: shellcheck --version && actionlint Exit 0; observed output excerpt: version: 0.11.0 Log SHA-256: fc9eec16b4191d058a295813ca9afbe9d2b37b59975da860170726be162a3924 Command: cargo fmt --all --check Exit 0; observed output excerpt: (no stdout or stderr) Log SHA-256: e3b0c44298fc1c149afbf4c8996fb92427ae41e4649b934ca495991b7852b855 Command: ./scripts/check_unsafe_code_allowances.sh Exit 0; observed output excerpt: (no stdout or stderr) Log SHA-256: e3b0c44298fc1c149afbf4c8996fb92427ae41e4649b934ca495991b7852b855 Command: ./scripts/check_layer_dependencies.sh Exit 0; observed output excerpt: Layer dependency guard passed (no new reverse dependencies/cycles). Log SHA-256: 69c6cb1896c0586b28d1167380d3260d99d8017a4265c8df5a2348f289e93a12 Command: ./scripts/check_architecture_migration_rules.sh Exit 0; observed output excerpt: Architecture migration rules passed. Log SHA-256: 28790e70d070607ed4de0704f292a65030be92ec224ef106d4b72fd8296f896e Command: ./scripts/check_logging_guardrails.sh Exit 0; observed output excerpt: ✅ Logging guardrails check passed Log SHA-256: 7adf41abff8ed8f7617decf86498888ddde78937099dd9ce47f6d359dfc2d317 Command: ./scripts/check_error_other_format_ratchet.sh Exit 0; observed output excerpt: error(format!) ratchet passed (total call sites: 579). Log SHA-256: 66172aef18f150ba0605af122e26ffcf91f6dd86f43e38b05e2e982ce8d783ee Command: ./scripts/check_no_tokio_io_uring.sh Exit 0; observed output excerpt: (no stdout or stderr) Log SHA-256: e3b0c44298fc1c149afbf4c8996fb92427ae41e4649b934ca495991b7852b855 Command: ./scripts/check_extension_schema_boundaries.sh Exit 0; observed output excerpt: Extension schema boundaries passed. Log SHA-256: f13d0acd0e2ce5d9815241b12b4dab7d01cc11b9b65bcf6dfb7812c878532a51 Command: ./scripts/check_body_cache_whitelist.sh Exit 0; observed output excerpt: Body-cache whitelist guard passed. Log SHA-256: 6b828847b0383950ee6eef9cd846d0c322c945e47efd4c7d1ad881178810d6bb Command: ./scripts/check_s3s_footprint.sh Exit 0; observed output excerpt: s3s footprint OK: files importing s3s is 195 (baseline files=195) s3s footprint OK: s3s s3_error! lines is 1584 (baseline s3_error_lines=1584) s3s footprint OK: ecstore files referencing s3s is 36 (baseline ecstore_files=36) ✅ s3s footprint baseline check passed Log SHA-256: 47f53d04ef645618563fe1f22abc64667f19078f1974feedcca46e4f2c016844 Command: ./scripts/check_test_count_ratchet.sh Exit 0; observed output excerpt: test-count ratchet OK: 52 crates, 21340 tests, 164 ignored match .config/test-count-baseline.txt Log SHA-256: 618a5e330e933de0c6ff484546df401457024a88bb5986d11d588794e33dab60 Command: ./scripts/check_fips_wording.sh Exit 0; observed output excerpt: FIPS wording guard passed (README.md and CHANGELOG.md contain no forbidden claims). KMS wording guard passed (crates/kms claims no Vault KV2 Transit wrapping). Log SHA-256: 99221cc9979c480ec4f46f50d76c52b18c72fe499f12e15e4f4a22a1b8bfb140 Command: ./scripts/check_embedded_secrets.sh Exit 0; observed output excerpt: Embedded secret guard passed (no private key material or provider credential literal in the tree). Log SHA-256: 437a3ceba32094d2901889965ad191e411aad32cdb18bf0b8659e818011c406b Command: make script-tests Exit 0; observed output excerpt: test_cli_requires_the_repository_default_branch (__main__.SelfTests.test_cli_requires_the_repository_default_branch) ... ok test_config_rejects_duplicate_and_invalid_entries (__main__.SelfTests.test_config_rejects_duplicate_and_invalid_entries) ... ok test_freshness_boundaries (__main__.SelfTests.test_freshness_boundaries) ... ok test_grace_requires_two_successful_queries_with_no_history (__main__.SelfTests.test_grace_requires_two_successful_queries_with_no_history) ... ok test_invalid_api_evidence_fails_closed (__main__.SelfTests.test_invalid_api_evidence_fails_closed) ... ok test_report_retains_every_workflow (__main__.SelfTests.test_report_retains_every_workflow) ... ok test_requests_filter_schedule_default_branch_and_success_on_server (__main__.SelfTests.test_requests_filter_schedule_default_branch_and_success_on_server) ... ok Ran 10 tests in 0.042s OK Ran 30 tests in 20.627s OK Ran 10 tests in 2.268s OK Ran 33 tests in 0.127s OK Ran 7 tests in 0.002s OK Ran 7 tests in 0.001s OK Ran 4 tests in 0.108s OK Ran 3 tests in 0.564s OK PASS: cold stampede script self-test Log SHA-256: 75fda2d069a373c63667b908baaf99e6591aab9099a739b0c4c22ceb6da8d68c Command: python3 ./scripts/test_object_generation_protocol_model.py Exit 0; observed output excerpt: test_an_unavailable_decision_quorum_cannot_choose_a_candidate (__main__.ObjectGenerationProtocolModel.test_an_unavailable_decision_quorum_cannot_choose_a_candidate) ... ok test_counter_allocation_requires_persistence_and_is_consumed_once (__main__.ObjectGenerationProtocolModel.test_counter_allocation_requires_persistence_and_is_consumed_once) ... ok test_counter_restart_burns_persisted_allocations_before_or_after_issue (__main__.ObjectGenerationProtocolModel.test_counter_restart_burns_persisted_allocations_before_or_after_issue) ... ok test_crash_after_promise_before_reply_does_not_restore_an_older_ballot (__main__.ObjectGenerationProtocolModel.test_crash_after_promise_before_reply_does_not_restore_an_older_ballot) ... ok test_duplicate_or_unconfigured_voters_cannot_form_a_quorum (__main__.ObjectGenerationProtocolModel.test_duplicate_or_unconfigured_voters_cannot_form_a_quorum) ... ok test_empty_or_unobserved_minority_slot_can_retry_without_allocating_another_revision (__main__.ObjectGenerationProtocolModel.test_empty_or_unobserved_minority_slot_can_retry_without_allocating_another_revision) ... ok test_equal_ballot_cannot_accept_different_bytes_and_lower_ballot_is_rejected (__main__.ObjectGenerationProtocolModel.test_equal_ballot_cannot_accept_different_bytes_and_lower_ballot_is_rejected) ... ok test_forgetting_tombstone_authority_allows_an_aba_create (__main__.ObjectGenerationProtocolModel.test_forgetting_tombstone_authority_allows_an_aba_create) ... ok test_issuing_before_persistence_reuses_a_ballot_after_restart (__main__.ObjectGenerationProtocolModel.test_issuing_before_persistence_reuses_a_ballot_after_restart) ... ok test_lost_ack_and_restart_preserve_the_chosen_operation (__main__.ObjectGenerationProtocolModel.test_lost_ack_and_restart_preserve_the_chosen_operation) ... ok test_minority_accepted_work_is_adopted_and_finished (__main__.ObjectGenerationProtocolModel.test_minority_accepted_work_is_adopted_and_finished) ... ok test_omitting_highest_accepted_adoption_has_a_double_choice_counterexample (__main__.ObjectGenerationProtocolModel.test_omitting_highest_accepted_adoption_has_a_double_choice_counterexample) ... ok test_quorum_choice_does_not_require_an_uncontacted_disk_to_learn_it (__main__.ObjectGenerationProtocolModel.test_quorum_choice_does_not_require_an_uncontacted_disk_to_learn_it) ... ok test_recovery_failure_or_stale_expected_head_does_not_allocate_a_successor (__main__.ObjectGenerationProtocolModel.test_recovery_failure_or_stale_expected_head_does_not_allocate_a_successor) ... ok test_replacing_voters_with_empty_state_can_choose_a_conflicting_value (__main__.ObjectGenerationProtocolModel.test_replacing_voters_with_empty_state_can_choose_a_conflicting_value) ... ok test_skipping_predecessor_recovery_chooses_a_broken_revision_chain (__main__.ObjectGenerationProtocolModel.test_skipping_predecessor_recovery_chooses_a_broken_revision_chain) ... ok test_stale_publication_rollback_and_gc_cannot_change_the_winner (__main__.ObjectGenerationProtocolModel.test_stale_publication_rollback_and_gc_cannot_change_the_winner) ... ok test_superseded_operation_identity_cannot_be_reused_for_different_input (__main__.ObjectGenerationProtocolModel.test_superseded_operation_identity_cannot_be_reused_for_different_input) ... ok test_superseded_operation_retry_returns_its_original_successor (__main__.ObjectGenerationProtocolModel.test_superseded_operation_retry_returns_its_original_successor) ... ok test_tombstone_and_null_marker_payloads_do_not_reset_authority_revision (__main__.ObjectGenerationProtocolModel.test_tombstone_and_null_marker_payloads_do_not_reset_authority_revision) ... ok test_tombstone_rejects_a_delayed_never_present_create (__main__.ObjectGenerationProtocolModel.test_tombstone_rejects_a_delayed_never_present_create) ... ok test_unlearned_predecessor_is_recovered_before_the_second_slot (__main__.ObjectGenerationProtocolModel.test_unlearned_predecessor_is_recovered_before_the_second_slot) ... ok Ran 24 tests in 0.929s OK Log SHA-256: 0b4be926efb0609d497ae58930340153f01957b606206cbbdb3e0d47eb29d3a2 Command: python3 ./scripts/check_test_wiring.py Exit 0; observed output excerpt: OK: e2e modules, runner selection, fuzz matrices, profiles, scheduled alerts, and Quick Checks are wired Log SHA-256: 21f676f458451525535ab0ef73d6a9c0d9774e16d76a4d19e396ad34936dd39c Command: ./scripts/check_no_planning_docs.sh Exit 0; observed output excerpt: (no stdout or stderr) Log SHA-256: e3b0c44298fc1c149afbf4c8996fb92427ae41e4649b934ca495991b7852b855 Command: ./scripts/check_uring_lane_lib_only.sh Exit 0; observed output excerpt: OK: no 'uring_'-matching test functions under crates/ecstore/tests; --lib is safe Log SHA-256: 43f3f851e27ec2ab9e34d6b05fefdede38af992699c473904bfe6f53a46d52e3 Command: cargo nextest run --profile ci -p rustfs-ecstore --lib --features rio-v2 -E 'test(=bucket::lifecycle::bucket_lifecycle_ops::tests::tier_free_version_recovery_production_entrypoint_enqueues_seeded_item)' --no-fail-fast Exit 0; wall 234.399s. Observed output: Finished `test` profile [unoptimized + debuginfo] target(s) in 3m 49s Starting 1 test across 1 binary (5871 tests skipped) SETUP PASS [ 0.006s] ecstore-base-stack: sh -c 'echo RUST_MIN_STACK=4194304 >> "$NEXTEST_ENV"' PASS [ 0.440s] rustfs-ecstore bucket::lifecycle::bucket_lifecycle_ops::tests::tier_free_version_recovery_production_entrypoint_enqueues_seeded_item Summary [ 0.586s] 1 test run: 1 passed, 5871 skipped Log SHA-256: 34e973f86e8dc2102c1a0e483e0ac802ce7df3d1b40a603932f62c0805c4de27 Command: cargo nextest run --profile ci -p rustfs-ecstore --lib --features rio-v2 -E 'test(=bucket::lifecycle::bucket_lifecycle_ops::tests::tier_free_version_recovery_eight_coordinators_hold_one_leader_and_take_over) | test(=store::bucket::tests::creation_commit_backfill_runs_under_the_config_write_fence) | test(=store::bucket::tests::orphaned_bucket_recovery_requires_every_volume_absent_and_is_retryable)' --no-fail-fast Exit 0; wall 4.010s. Observed output: Finished `test` profile [unoptimized + debuginfo] target(s) in 0.43s Starting 3 tests across 1 binary (5869 tests skipped) SETUP PASS [ 0.005s] ecstore-base-stack: sh -c 'echo RUST_MIN_STACK=4194304 >> "$NEXTEST_ENV"' PASS [ 0.068s] rustfs-ecstore bucket::lifecycle::bucket_lifecycle_ops::tests::tier_free_version_recovery_eight_coordinators_hold_one_leader_and_take_over PASS [ 0.637s] rustfs-ecstore store::bucket::tests::creation_commit_backfill_runs_under_the_config_write_fence PASS [ 0.846s] rustfs-ecstore store::bucket::tests::orphaned_bucket_recovery_requires_every_volume_absent_and_is_retryable Summary [ 0.987s] 3 tests run: 3 passed, 5869 skipped Log SHA-256: b5da2a927ff18ca9aa217762f7186a6d4371bd9e8feedd7b77940e10b64dffac Command: cargo clippy -p rustfs -p rustfs-ecstore --all-targets --features rio-v2 -- -D warnings Exit 0; wall 1084.508s. Observed output: Finished `dev` profile [unoptimized + debuginfo] target(s) in 18m 03s Log SHA-256: 28f98be7970faaa71cd118da1da4e6e2e3d23b9dda7e6449eaa7a37e741260b5 Compiled mutation command (unchanged for each red and restored run): cargo nextest run --profile ci -p rustfs-ecstore --lib --features rio-v2 -E 'test(=bucket::lifecycle::bucket_lifecycle_ops::tests::tier_free_version_recovery_production_entrypoint_enqueues_seeded_item)' --no-fail-fast M1: reverse only this test-local constructor flag from false to true. The mutant compiled, then failed the original recovered-item queue assertion after 30.481s. Restoring the entire observed candidate passed in 0.460s. Red log 55bf3b1535529ac0bcfb8071f9b56f9f40631007a85c315367e7e93786d65293; restored log 204d5fbe4574a7917f5e740a795d864fe28a7e7fb5e33d27b0a9cf68e2f17f52. M2: replace only the actual awaited enqueue with false, consuming its original inputs. The mutant compiled, then failed the same original queue assertion after 30.516s. Restoring the entire observed candidate passed in 0.457s. Red log 89186330413215e231e2b06bf013cc31097ea8f4a106d91421a36d6c8986a7fd; restored log e848b42d2302401afa5dbcd9e37bf76d2e56a96ab6de5a488513b997f53c5e9b. All temporary observation and mutation code was removed before final acceptance. Historical baseline and diagnostic failures remain recorded; they are not mutation credit. The separate Object Lock intent-retry diagnostic still fails, and these native macOS checks do not establish Linux CI recovery. * refactor(rpc): serve internode rpc bodies with the owned Body (#8441) * test(rpc): pin the wire bytes of every internode rpc response body rustfs/backlog#2747 batch 4, step 1 of 2, on the s3s bodies the rpc paths serve today. Each response body builder in storage/rpc/http_service.rs is served through hyper's HTTP/1.1 server and the raw bytes the peer reads are compared with goldens captured from this commit: status, content-length or chunked framing, chunk boundaries, the terminating chunk, HEAD, and the cut connection a failure after data produces. The capability responses carry a per-process proof, so their length header and payload are checked against the body they encode instead. The read_file_stream response builder moves into read_file_stream_response so the test reaches the same code the handler runs; behaviour is unchanged. * refactor(rpc): serve internode rpc bodies with the owned Body rustfs/backlog#2747 batch 4, step 2 of 2. The internode rpc HTTP paths build their responses with rustfs_s3_types::Body: Body::from_stream replaces Body::from(StreamingBlob::wrap(..)) at the six stream sites, and the in-memory and empty bodies keep their constructors. InternodeRpcService converts its responses into the wrapped service's response body through From<Body>, so the server keeps composing it with the s3s service as before. The wire-byte goldens from the previous commit pass unchanged; storage/rpc no longer names s3s. * fix(rpc): simplify wire test response builder type Name the boxed response builder used by the capability wire test so all-target Clippy can check it without a type-complexity warning. Validation (LANG=en_US.UTF-8, CARGO_INCREMENTAL=0): - cargo clippy -p rustfs --all-targets -- -D warnings: exit 0. - cargo nextest run --profile ci -p rustfs --lib -E 'test(/storage::rpc::/)': 279 tests run, 279 passed, 4842 skipped. - All 19 quick-check commands, including script-tests, both Python scripts and cargo fmt --all --check: exit 0. - Footprint baseline regenerated: files=194, s3_error_lines=1584, ecstore_files=36; test-count baseline regenerated: 52 crates, 21228 tests, 164 ignored. - All 54 Cargo.toml files parse without duplicate keys. No assertion was added or changed by this lint fix. R1-R8 are retained in the existing PR verification record. * fix(ecstore): resume fenced Object Lock creation intents * fix(ecstore): bound creation retry futures on the stack * fix(ecstore): require native proof for creation retries * fix(ecstore): bound bucket creation stack usage (#8466) * fix(ecstore): retain creation fences through rollback Keep lifecycle, metadata-transaction, and namespace guards around rollback and metadata cleanup. Preserve the original publication error when a guard is lost. Build the timestamp retry fixture through the real interrupted producer so it checks a valid pending creation. Validated source tree: 4e8e207999. Two independent source reviews found no blocking regression. Validation (LANG=en_US.UTF-8): - bash -c 'shellcheck --version && actionlint' — exit 0 - cargo fmt --all --check — exit 0 - ./scripts/check_unsafe_code_allowances.sh — exit 0 - ./scripts/check_layer_dependencies.sh — exit 0 - ./scripts/check_architecture_migration_rules.sh — exit 0 - ./scripts/check_logging_guardrails.sh — exit 0 - ./scripts/check_error_other_format_ratchet.sh — exit 0 - ./scripts/check_no_tokio_io_uring.sh — exit 0 - ./scripts/check_extension_schema_boundaries.sh — exit 0 - ./scripts/check_body_cache_whitelist.sh — exit 0 - ./scripts/check_s3s_footprint.sh — exit 0 - ./scripts/check_test_count_ratchet.sh — exit 0 - ./scripts/check_fips_wording.sh — exit 0 - ./scripts/check_embedded_secrets.sh — exit 0 - make script-tests — exit 0 - python3 ./scripts/test_object_generation_protocol_model.py — exit 0 - python3 ./scripts/check_test_wiring.py — exit 0 - ./scripts/check_no_planning_docs.sh — exit 0 - ./scripts/check_uring_lane_lib_only.sh — exit 0 - cargo nextest run --profile ci -p rustfs-ecstore --lib --locked --no-fail-fast — 5894 passed, 0 skipped - cargo check -p rustfs-ecstore --all-targets --locked — exit 0 - cargo nextest run --profile ci -p rustfs --lib --locked --no-fail-fast -E 'test(=admin::handlers::table_catalog::tests::cold_strong_registration_completes_while_other_bucket_backup_waits) | test(=admin::handlers::table_catalog::tests::commit_backend_borrowed_put_preserves_owner_authorization_and_precondition) | test(=admin::handlers::table_catalog::tests::disable_table_bucket_preserves_objects_and_can_be_reenabled) | test(=admin::handlers::table_catalog::tests::load_table_handler_negotiates_vended_credentials_without_breaking_metadata_only_callers) | test(=admin::handlers::table_catalog::tests::namespace_property_handler_updates_object_backed_catalog_and_maps_errors) | test(=app::object::get::tests::execute_get_object_resumes_from_relocated_pool_without_splicing_body) | test(=table_catalog::store::tests::catalog_writer_loss_prevents_recovery_storage_commit)' — 7 passed - cargo nextest run --profile ci -p rustfs --lib --locked --no-fail-fast -E 'test(unsupported_remote_disk_)' — 2 passed Additional fault-injection evidence remains incomplete. Timestamp M1 reached its original assertion; its restore run is being collected. M2–M5 and the three planned rollback faults have not run. An automated risk check interrupted the timestamp proof task; no assertions or validation settings were weakened. * fix(ecstore): box nested bucket rollback state Keep the guarded rollback future behind a heap allocation so bucket creation does not store its nested state inline. The rollback body, all three fence references and pending-proof revocation order are unchanged. The same three original Linux tests overflowed on both e611 CI attempts. Local format, ecstore test compilation and all 19 quick-check steps pass; the original Linux feature lanes still need to validate this candidate. * test(ecstore): isolate rollback guard loss fixtures Run the three rollback metadata-cleanup cases in separate test processes so reused /proc/self/fd aliases from earlier cases cannot invalidate their creation witnesses. Keep the original fixture, guards, assertions and io_uring test selection, and verify that each child runs the intended case successfully. Update the generated test-count baseline for the added macro arm. Validation: cargo check -p rustfs-ecstore --tests; cargo fmt --all --check; all 19 quick-check commands, including make script-tests and the two Python checks. Linux io_uring acceptance and fault controls remain pending. * refactor(replication): move the replication config onto gateway types (#8443) * wip: T1.4 replication family * refactor(replication): move the replication config onto gateway types The replication configuration family leaves s3s: rustfs-replication no longer depends on s3s, and ECStore bucket metadata, the scanner and the rustfs app/admin layers hold rustfs_gateway_types::persistence::PersistedReplicationConfiguration. The persisted XML goes through the gateway persistence codec behind a new BucketConfigXml trait (the 39 byte goldens pass with unchanged fixtures); the legacy s3s edge converts once in storage/ecfs.rs through storage/s3_compat/replication.rs; the scanner data usage cache keeps its MessagePack bytes through a serde mirror pinned by bytes recorded from s3s 5761ddfe. rustfs-gateway-types 0.32.0 is pinned once (rev 772d3385). Refs rustfs/backlog#2745. * test: use S3 serializer for list response tests * fix(ci): restore integration test compilation Gate the diagnostic test import with its feature, initialize the conditional MetadataWriteStore fields explicitly, and decode the scanner replication fixture through its existing cache boundary. Preserve the original fixture and assertions. Validation was run on tree 544acb80c0 at 6003d4b8. LANG=en_US.UTF-8; no source changes follow validation. Quick checks: 19/19 passed. - shellcheck --version && actionlint: exit 0 - cargo fmt --all --check: exit 0 - ./scripts/check_unsafe_code_allowances.sh: exit 0 - ./scripts/check_layer_dependencies.sh: exit 0 - ./scripts/check_architecture_migration_rules.sh: exit 0 - ./scripts/check_logging_guardrails.sh: exit 0 - ./scripts/check_error_other_format_ratchet.sh: exit 0 - ./scripts/check_no_tokio_io_uring.sh: exit 0 - ./scripts/check_extension_schema_boundaries.sh: exit 0 - ./scripts/check_body_cache_whitelist.sh: exit 0 - ./scripts/check_s3s_footprint.sh: exit 0 - ./scripts/check_test_count_ratchet.sh: exit 0 - ./scripts/check_fips_wording.sh: exit 0 - ./scripts/check_embedded_secrets.sh: exit 0 - make script-tests: exit 0 - python3 ./scripts/test_object_generation_protocol_model.py: exit 0 - python3 ./scripts/check_test_wiring.py: exit 0 - ./scripts/check_no_planning_docs.sh: exit 0 - ./scripts/check_uring_lane_lib_only.sh: exit 0 Darwin validation: - cargo clippy --locked -p rustfs-ecstore -p rustfs-scanner --all-targets -- -D warnings: exit 0 - cargo nextest run --locked -p rustfs-ecstore -p rustfs-scanner --lib -E 'test(config::com::tests::save_config_preserves_retryable_system_volume_errors) | test(scanner_io::tests::replication_maintenance_refreshes_compacted_stats_across_cycles_and_retries)' --test-threads 1 --no-fail-fast: exit 0, 2/2 passed - cargo clippy --locked -p rustfs-ecstore -p rustfs-scanner --all-targets --features rustfs-ecstore/diagnostic-read-version-caller,rustfs-scanner/diagnostic-read-version-caller -- -D warnings: exit 101 - cargo nextest run --locked -p rustfs-ecstore -p rustfs-scanner --lib --features rustfs-ecstore/diagnostic-read-version-caller,rustfs-scanner/diagnostic-read-version-caller -E 'test(config::com::tests::save_config_preserves_retryable_system_volume_errors) | test(scanner_io::tests::replication_maintenance_refreshes_compacted_stats_across_cycles_and_retries) | test(cluster::rpc::runtime_sources::tests::) | test(config::com::tests::diagnostic_config_write_retains_origin_and_api_for_results_and_cancellation)' --test-threads 1 --no-fail-fast: exit 0, 14/14 passed Diagnostic strict Clippy remains red on four unchanged production lints: runtime_sources.rs if_same_then_else and three redundant_locals reports in set_disk/core/metadata_read.rs and set_disk/read.rs. Their source ranges and feature wiring match the base; the base was not compiled separately. Linux aarch64 validation (Rust/Clippy 1.99.0, UID 501, jobs=1): - cargo clippy --all-targets -- -D warnings: exit 0; Finished dev profile in 68m 12s - cargo clippy -p rustfs -p rustfs-ecstore --all-targets --features rio-v2 -- -D warnings: exit 0; Finished dev profile in 49m 45s Source HEAD/tree and clean status matched before and after both commands. No compiler or stack overrides were used. protoc 35.1 was installed from its verified ARM64 release; flatc was absent and not invoked by the workspace build scripts. The local environment differs from hosted CI. * fix(ecstore): recover delete batches with missing versions Treat an explicit non-null version as a no-op when it is absent from both recovery sources. Keep rejecting missing null versions, delete-marker mismatches and conflicting local generations. Add returning-disk and partial-receipt regressions, extend journal boundary checks, and regenerate the ecstore test-count baseline. Production faults exercise the 24 added assertion sites and 7 semantic checks: 33 faults failed at the expected sites and passed after full restoration. Validation (LANG=en_US.UTF-8, one build job): cargo nextest run --locked -p rustfs-ecstore --lib -E 'test(delete_recovery) | test(quorum_delete_cleanup_coordinator_binds_single_and_batch) | test(crash_delete_retry_batch_reclaims_inline_and_reserved_external)' --test-threads 1 --no-fail-fast Summary [ 34.580s] 69 tests run: 69 passed, 6058 skipped; exit 0 cargo clippy --locked -p rustfs-ecstore --all-targets -- -D warnings Finished `dev` profile [unoptimized + debuginfo] target(s) in 5m 37s; exit 0 All 19 final Quick Checks completed with exit 0: - shellcheck --version && actionlint - cargo fmt --all --check - ./scripts/check_unsafe_code_allowances.sh - ./scripts/check_layer_dependencies.sh - ./scripts/check_architecture_migration_rules.sh - ./scripts/check_logging_guardrails.sh - ./scripts/check_error_other_format_ratchet.sh - ./scripts/check_no_tokio_io_uring.sh - ./scripts/check_extension_schema_boundaries.sh - ./scripts/check_body_cache_whitelist.sh - ./scripts/check_s3s_footprint.sh - ./scripts/check_test_count_ratchet.sh - ./scripts/check_fips_wording.sh - ./scripts/check_embedded_secrets.sh - make script-tests - python3 ./scripts/test_object_generation_protocol_model.py - python3 ./scripts/check_test_wiring.py - ./scripts/check_no_planning_docs.sh - ./scripts/check_uring_lane_lib_only.sh test-count ratchet OK: 52 crates, 22009 tests, 165 ignored match .config/test-count-baseline.txt PASS: cold stampede script self-test model: states=51023, transitions=167520, terminal=732, chosen_candidates=2 multi-slot round schedules: 640 OK: e2e modules, runner selection, fuzz matrices, profiles, scheduled alerts, and Quick Checks are wired * fix(ecstore): retry tier commit intent lock contention Retry transient namespace lock timeouts within the existing three-attempt CAS budget. Reload and verify the original mutation identity after contention, and preserve the final typed error when attempts are exhausted. Cover save/read contention, exhausted retries, replacement identities and CAS reconciliation without changing the existing timeout or lock policy. Validation: - cargo nextest run --locked -p rustfs-ecstore --features test-util --lib -E 'test(tier_mutation)' --test-threads 1 --no-fail-fast: 52 passed. - cargo nextest run --locked -p rustfs-ecstore --features test-util --lib -E 'test(intent_advance_) | test(=services::tier::tier::tests::caller_cancellation_after_durable_prepare_does_not_hide_the_mutation)' --test-threads 1 --no-fail-fast: 5 passed. - cargo clippy --locked -p rustfs-ecstore --all-targets -- -D warnings: passed. - cargo clippy --locked -p rustfs-ecstore --all-targets --features test-util -- -D warnings: passed. - 27 mutation pairs failed at the intended assertions and passed after restoring the production code. - bash scripts/check_test_count_ratchet.sh --update, then bash scripts/check_test_count_ratchet.sh: 52 crates, 22013 tests, 165 ignored; ecstore increases from 6581 to 6585. * refactor(lifecycle): replace s3s domain types with Gateway DTOs Keep lifecycle rules, restore requests, Object Lock, and scanner cached tags in domain types. Convert at the existing HTTP boundary while preserving legacy XML, scanner cache bytes, date ordering, and tombstones. Linux test validation before the final import-path-only adjustment: cargo nextest run -p rustfs-ecstore bucket_config_goldens 39 passed; 6211 skipped by selector. cargo nextest run --lib -p rustfs -E 'test(/test_zero_rule_lifecycle_tombstone_recognition/) | test(/test_local_lifecycle_staleness_axis_selection/) | test(/test_lifecycle_staleness_preserves_legacy_submillisecond_axis/) | test(/test_merge_incoming_lifecycle/) | test(/test_lifecycle_expiry_statement_matrix/) | test(/test_lifecycle_expiry_subset_xml/) | test(/put_bucket_lifecycle_validation_errors_keep_their_s3_code/) | test(/execute_put_bucket_lifecycle_configuration_rejects_missing_configuration/) | test(/execute_restore_object_rejects_missing_restore_request/) | test(/execute_restore_object_rejects_select_type/) | test(/execute_restore_object_maps_failures_to_typed_s3_errors/) | test(/execute_restore_object_returns_internal_error_when_store_uninitialized/) | test(/restore_request_conversion_preserves_legacy_output_values/) | test(/restore_request_conversion_preserves_presence_and_raw_enums/) | test(/restore_request_conversion_distinguishes_absent_and_empty_containers/)' 25 passed; 5318 skipped by selector. cargo nextest run --profile ci -p rustfs-ecstore -p rustfs-lifecycle -p rustfs-replication -p rustfs-scanner 7872 passed; 21 skipped under the existing CI profile. Previously verified mutation controls: 39 production faults, 41 RED/GREEN pairs. cargo clippy --locked -p rustfs -p rustfs-ecstore -p rustfs-lifecycle -p rustfs-scanner --all-targets -- -D warnings Passed in 40m 45s before the import-path adjustment, then in 1m 59s on the final source with warnings denied. The final adjustment imports the existing helper through the object domain facade; fmt and the architecture check also passed. Part of rustfs/backlog#2745. The remaining ecstore configuration families follow separately. * refactor(admin): move error types to rustfs-s3-types (#8440) * refactor(protocols): move error types to rustfs-s3-types by codemod T1.2 batch 3 (rustfs/backlog#2743): s3select-api, s3select-query and the error items of protocols move off s3s by scripts/codemods, run with --hold crates/protocols/src/sftp/errors.rs. That file stays on s3s: it downcasts the backend error, whose runtime type is still s3s::S3Error. - rustfs/src/protocols/client.rs: the two webdav session methods now return rustfs_s3_types::S3Result and convert the s3s error with From. - codemod: a child module that calls s3_error! through `use super::*` gets its own `use rustfs_s3_types::s3_error;` once its parent imports that macro by name; otherwise it is listed as GLOB. - s3-types: a table test pins code, status and the s3s error the legacy edge renders for every variant a moved call site names. - s3_error_lines 1585 -> 1584 (a lifecycle comment reworded). * refactor(admin): move error types to rustfs-s3-types (WIP, no build) T1.2 batch 4 (rustfs/backlog#2743), work in progress: rustfs check still fails with 16 errors (listed in the PR body). - codemod: --facade PATH treats a crate-local re-export of s3s's error items like s3s for the four names (9 new self-tests, 9 mutants red). - codemod output for rustfs/src/admin and rustfs/src/site_replication (--facade crate::storage_api::site_replication::s3); 6 glob children get an explicit rustfs_s3_types::s3_error import. - admin/router.rs: the Operation trait returns rustfs_s3_types::S3Result. - error.rs: From<ApiError> builds the rustfs-s3-types error; the s3s conversion is derived from it. * fix(admin): convert error types at the s3s boundary Batch 4 of T1.2 left the admin and site_replication modules returning rustfs_s3_types::S3Error while their callers and callees on the storage and auth side still speak s3s::S3Error. Convert at each edge with the existing From impls in rustfs-s3-types (compat_s3s.rs) instead of changing any code, message or status: - admin/auth.rs, admin/handlers/object_zip_download.rs, admin/handlers/table_catalog/mod.rs: map the s3s result returned by check_key_valid / authorize_request / authorize_internal_object_request into the admin error type with .map_err(Into::into). - storage/ecfs.rs: the s3s-facing side maps the admin site_replication gate results the other way. - admin/handlers/pools.rs: ApiError.code stays s3s::S3ErrorCode, so the rustfs code is converted with .into() when the struct is built, in production and in the two tests that build or compare it. - admin/handlers/heal.rs, admin/runtime_sources.rs: the router returns s3s errors (S3Route impl); the expected rustfs code is converted with .into() before the comparison. - admin/handlers/table_catalog/mod.rs: iceberg_rest_error takes &'static str so the Cow<'static, str> custom code borrows it; every caller already passes an ICEBERG_ERROR_* constant or a &'static str. - admin/handlers/quota.rs: drop a now same-type .into() after s3_error!. - storage_api.rs, site_replication/mod.rs: drop re-exports and an import that no longer have users (clippy -D warnings). cargo check -p rustfs --all-targets and cargo clippy -p rustfs --all-targets -- -D warnings are clean; the s3s footprint baseline is unchanged at files=172, s3_error_lines=303, ecstore_files=28. Refs rustfs/backlog#2743, rustfs/backlog#2734. * refactor(ecstore): move bucket configs onto gateway persisted types (#8709) * refactor(ecstore): move bucket configs onto gateway persisted types Finish the ECStore batch of T1.4 (rustfs/backlog#2745). The remaining eleven bucket configuration families (Website, CORS, Tagging, Versioning, ObjectLock, Encryption, Notification, Logging, Accelerate, RequestPayment, PublicAccessBlock) now read and write their stored XML through the rustfs-gateway-types persistence and cors_tagging codecs instead of the s3s codec, so the legacy s3s bucket-config codec and its s3gate-metadata-xml compat marker are removed. - ecstore: BucketConfigXml is implemented by the gateway codec for every family; metadata, metadata_sys, versioning, object lock, replication and lifecycle consumers take the persisted types. Region moves to rustfs_s3_types::Region, and the ETag conversion to s3s moves to the rustfs edge with its tests. Only warm_backend.rs and warm_backend_s3.rs still reference s3s, held for T1.5 (rustfs/backlog#2746). - scanner: object lock cache, item actions and data usage use the persisted types. - rustfs: the legacy S3 edge converts each family between s3s DTOs and the persisted types in storage/storage_api.rs by writing one codec's XML and reading it with the other; Tagging converts field by field. - Bump the rustfs-gateway-types pin to 21283e87. The 39 persisted-byte goldens are unchanged and pass, so a document the old codec wrote reads back and re-encodes to the same bytes. s3s footprint: files 181 -> 146, ecstore_files 28 -> 2, s3_error_lines unchanged. * test(app): prepare the PutBucketEncryption config from the request The uninitialized-store test passed an empty persisted configuration, which the use case now validates before it reaches the store check, so it answered MalformedXML instead of InternalError. Convert the request's configuration the way the S3 edge does. * style(app): format the PutBucketEncryption test * refactor(s3-client): move remote error codes and DTO enums off s3s (#8783) Part of T1.5 (rustfs/backlog#2746), batch 1. * refactor(s3select): run queries on gateway DTO request shapes (#8794) * refactor(s3select): run queries on an engine request built from gateway DTO shapes Part of T1.5 (rustfs/backlog#2746), batch 2. * fix(s3select): keep the select request conversion in the app layer The layer guard forbids app -> storage; the conversion moves from storage/s3_compat/select.rs to app/select_request.rs unchanged. * fix(heal): retry transient metadata conflicts (#8812) * fix(heal): retry transient metadata conflicts * chore: update heal test count baseline * test: reserve nextest capacity for progress probes (#8818) * test(e2e): use a non-default region for the differential negative control Since main's region-aware CreateBucket and GetBucketLocation (#8739, #8743), us-east-1 answers an empty LocationConstraint and 200 for an owned re-create, so the control no longer reported its own region. Also refresh the e2e-full linux selection from the CI listing (682 tests). * refactor(protocols): move StorageBackend to gateway types (#8825) * refactor(protocols): move storage backend to gateway types * test: update e2e test count baseline * fix: preserve copy source keys and unknown lengths * test: update protocols test count baseline * fix(protocols): preserve typed SFTP backend errors * test(protocols): use gateway list input in policy tests * fix(protocols): clear clippy failures after T1.5 merge (#8831) fix(protocols): clear clippy failures after merge * fix(ecstore): finalize restore metadata after local copy-back (#8836) * fix(ecstore): finalize restore metadata after local copy-back * test(ecstore): ratchet added restore finalization cases * fix(ecstore): keep restore copyback compatible with older peers * fix(ecstore): gate restore copyback on fleet compatibility * fix(ecstore): retain transition identity during restore copyback * fix(ecstore): route committed restore copyback locally * fix restore copyback cleanup * fix(ecstore): honor restore probe test fallback * refactor(storage): migrate option errors to domain types (#8839) * refactor(s3): migrate private multipart validation errors (#8844) * refactor(auth): derive policy conditions from AuthzFacts (#8442) * refactor(auth): extract authorization facts * test(e2e): update smoke selection baseline * test(e2e): update full selection baseline * refactor(admin): migrate health and prefix usage to owned requests (#8887) * refactor(admin): add owned health router boundary * refactor(admin): migrate prefix usage to owned request * refactor(admin): complete the owned admin request boundary (#8949) * refactor(admin): complete the owned admin request boundary Every type implementing the owned admin `Operation` now serves the legacy router through one blanket impl that converts the s3s request and response at that single edge, so handlers register with `AdminOperation(&Handler {})` and need no adapter statics; `LegacyAdminAdapter` is removed. `AdminRequest` gains the accessors handlers read today (`remote_addr`, `client_info`, `request_context`, `req_info`, `req_info_mut`, `install_req_info`, `envelope_mut`, `take_body`), the request helpers gain envelope forms (`authenticate_envelope_credentials`, `object_store_from_envelope`, the table-catalog `*_from_envelope` helpers, `KmsAdminAudit::from_envelope`, `AccountAuditContext::from_envelope`, `CallerIdentity::resolve_envelope`), `read_compatible_admin_body` accepts either body type, and the owned `Body` gains `store_all_limited` and `Default`. `admin::operation::test_support::TestRequest` builds owned requests for handler tests. Unused accessors carry a per-method dead-code expectation that each migrating handler batch removes. No handler behavior changes; the remaining migrations follow in batches. * chore(admin): add the T1.12 admin handler codemod The script rewrites an admin handler file from the s3s request and response types to the owned boundary (imports, signatures, request field and helper calls, body access, response construction, and test request literals) and leaves a TODO comment at every site it cannot rewrite mechanically. It is idempotent and is removed once T1.12 completes. * refactor(admin): migrate replication and KMS handlers to owned requests (#8951) Move the T1.12 batch 2 admin handlers (site_replication, kms_keys, event, replication, diagnostics, on_demand_migration, audit, account, kms_rekey, kms_key_lifecycle, object_data_cache, tls_debug, integrity) from the s3s request/response types to the owned AdminRequest/AdminResponse boundary. Handler logic, authorization order, error codes and response bytes are unchanged; the request/response plumbing now goes through the envelope accessors and helpers introduced by the owned boundary. The shared response helpers (utils::json_response, utils::empty_response, handlers::admin_json_response, target_descriptor::build_json_response, profile::authorize_profile_request) and the site replication repair executor still answer with the s3s types, so the migrated files carry owned-boundary forms of those helpers and convert the repair response at its call edge. The streaming console-log body and the two devnull drains move from the s3s byte stream to the owned Body stream/frame API. Drop the dead_code expectations for the owned accessors and envelope helpers this batch starts using, and lower the s3s footprint baseline to the measured 99 files. Part of rustfs/backlog#2788 (rustfs/backlog#2734 T1.12). * refactor(admin): add owned forms of the shared admin response helpers Handler batches were copying `json_response`, `empty_response`, `admin_json_response`, and `build_json_response` into each migrated file because the shared helpers still return the s3s response. Provide owned forms next to each legacy helper (`owned_json_response`, `owned_empty_response`, `owned_admin_json_response`, `owned_build_json_response`, `authorize_profile_envelope`) with parity tests against the legacy shape, derive `Debug` for `AdminResponse` so tests can use `expect_err`, and fix three codemod rules: strip owned string conversions in test credentials, annotate non-literal URI parses, and leave a rebound request parameter immutable. * refactor(admin): use the shared owned response helpers in batch 2 (#8954) Replace the per-file owned-boundary copies of json_response, empty_response, admin_json_response, build_json_response and authorize_profile_request in the batch 2 admin handlers with the shared owned_* helpers and authorize_profile_envelope that the boundary branch now provides, and restore expect_err in the handler tests now that AdminResponse implements Debug; the `.err().expect(..)` stand-ins were failing the base branch's clippy step (clippy::err_expect under -D warnings). The site replication repair call edge keeps its conversion because the repair executor still answers with the s3s response type. Drop the dead_code expectations on the shared owned helpers this batch now uses in non-test code. Part of rustfs/backlog#2788 (rustfs/backlog#2734 T1.12). * fix(admin): preserve legacy body limit error responses * refactor(admin): move policy, ILM and batch 3 handlers to owned requests (#8956) Batch 3 of T1.12 (rustfs/backlog#2788, parent rustfs/backlog#2734) moves 14 admin handler files to AdminRequest/AdminResponse and the owned Operation trait: ilm_transition, policies, idp_compat, config_admin, plugins_instances, profile_admin, group, heal, object_zip_download, kms_key_metadata, durability, inspect_archive, sts and gateway_key_inventory. Handler logic, authorization order, error codes and response bytes are unchanged; test request literals use test_support::TestRequest. The handlers use the shared owned helpers (owned_json_response, owned_admin_json_response, owned_build_json_response, authorize_profile_envelope). The dead_code expectations on the AdminRequest accessors and RequestEnvelope::uri_mut they now exercise are removed. The legacy target_descriptor::build_json_response keeps only its equivalence test caller and carries the matching expectation until it retires. object_zip_download builds the s3s request the S3 gate still reads from the owned request facts; heal authenticates through authenticate_envelope_credentials; the trace, inspect-archive and zip bodies use Body::from_stream. extract_plugin_instance_filters takes the URI so extensions.rs keeps calling it, and the two idp_compat delegations stay on the legacy trait until their delegates migrate. The s3s footprint baseline drops from 99 to 90 files. * test(admin): anchor the replication diff body contract on the call The source-contract test looked for `let body = read_compatible_admin_body(` on one line; rustfmt wrapped that binding after the handler moved to the owned request boundary, so the marker was no longer found. Anchor on the call itself so formatting cannot change what the contract checks. * refactor(admin): migrate table catalog and batch 1 to owned requests (#8955) * fix(admin): preserve catalog body limit response bytes * refactor(admin): migrate identity and cluster handlers to owned requests (#8966) * refactor(admin): migrate identity and cluster handlers to owned requests Move the T1.12 batch 4 admin handlers (user, service_account, oidc, tier, system, pools, quota, kms_dynamic, kms_backup, rebalance, module_switch, bucket_meta, cluster_snapshot, realtime, bucket_recovery, profile) from the s3s request/response types to the owned AdminRequest/AdminResponse boundary. All 90 impl Operation blocks now take AdminRequest and return AdminResponse; handler logic, authorization order, error codes and response bytes are unchanged. The handlers use the shared owned helpers (owned_json_response, owned_admin_json_response, authorize_profile_envelope, the envelope-form runtime sources and audit constructors) instead of local copies. profile.rs drops the s3s forms of authorize_profile_request and profile_not_implemented_response now that profile_admin.rs is on the envelope gate too; profile_not_implemented_response answers with an AdminResponse. The realtime metrics stream moves from the s3s ByteStream adapter to Body::from_stream, and the rebalance idle-404 document still renders through s3s::S3Error::to_http_response with its body converted at the edge. The idp_compat delegators ImportIamV2 and AddServiceAccountLdap move to the owned trait as well: with their targets on both traits, the method-call delegation became ambiguous between the owned impl and the router blanket impl, so they now take the owned request directly. KmsAdminAudit::from_request loses its last production caller with this batch and is deleted; its comparison test now asserts the envelope facts directly. Lower the s3s footprint baseline to the measured 68 files. Part of rustfs/backlog#2788 (rustfs/backlog#2734 T1.12). * refactor(admin): remove the legacy admin helpers batch 4 superseded Batch 4 moved the last production callers of the s3s-typed admin helpers to their owned forms, which left `admin_json_response`, `json_response`, `authenticate_request`, `authorize_admin_request`, `object_store_from_req` and `object_store_from_extensions` unused, along with the `Body`, `S3Request` and `S3Response` re-exports of the admin `s3` facade. Remove them. The owned forms keep their documentation; the behavior tests of the removed `json_response` now exercise `owned_json_response`, and the compat-prefix test asserts the owned response directly. Two tests existed only to compare a removed legacy helper with its owned twin (`owned_json_response_matches_the_legacy_shape`) or to cover the removed legacy gate (`authorize_admin_request_without_credentials_is_rejected`); they go with the helpers, and the owned equivalents stay. The rustfs test-count row drops from 5957 to 5955 and the s3s footprint from 68 to 66 files. * refactor(admin): drop the legacy target descriptor JSON response helper (#8968) `target_descriptor::build_json_response` returned the s3s response and had no production caller left once the event, audit and plugin instance handlers moved to `owned_build_json_response`; it survived only for the legacy/owned equivalence test. Remove it, give the owned helper its own documentation, and make the test assert the owned response directly: status, `application/json`, the echoed `x-request-id` when present and only then, and the body bytes. The file no longer imports s3s, so the footprint baseline drops from 66 to 65 files. * test(ecstore): pin native history on cancellation under the cancellable queue main moved the local write path onto the disk blocking queue, which skips an operation once its waiter is dropped, so a borrowed write cancelled while it is queued no longer creates its file. The integration-only native publication test still expected the queued open to run. Assert the new execution model instead (the open never runs) while keeping its safety claims: the Started record is taken before queueing and the cancelled publication never regrants native history. * chore(gateway): raise test-count rows for tests merged from main * test(s3select): use the owned JSON input types in merged planner tests * fix(ecstore): scope the merged direct-writer test hook to test builds * fix(ecstore): create parents before admitted truncate opens Align the heal tests with removal-disabled preservation and local-only outcome fields, and refresh the e2e-full selection for the synced tests. * chore(gateway): update test-count baseline after main sync Main #9241 removed the node pool-metadata readiness cache with 8 tests and added 6, so the rustfs row drops from 6440 to 6438. The ecstore and s3select rows rise for tests added by #9237, #9239 and #9241. * feat(scripts): add junit_diff.py for dual-stack JUnit comparison (#9275) * test(e2e): allow a pinned release binary as differential reference side RUSTFS_E2E_REFERENCE_BINARY makes side A of the dual-stack differential a released rustfs binary, started without RUSTFS_S3_STACK. A missing, non-file or non-executable path fails the run instead of falling back to the same-binary mode. The report records which side is the reference and each side's --version output. * ci(gateway): make the test-count ratchet a floor, not exact (#9278) ci(gateway): make the test-count ratchet a floor instead of an exact baseline Tests below a row still fail; tests above it, a member without a row and ignores below the row now pass with a note. Ignores above a row still fail. Adds contract tests wired into make script-tests. * chore(gateway): record ignored microbenchmarks merged from main (#9290) chore(gateway): record the ignored microbenchmarks merged from main main added three throughput microbenchmarks that are ignored by design (#9217): one each in rustfs-checksums, rustfs-rio and rustfs-utils. The test-count ratchet treats a higher ignored count as a failure until the row is raised, so raise those three rows to 1 and tighten the test rows to the merged tree with --update. * test(e2e): update smoke selection for reference-binary unit tests * ci(s3-tests): add reference parity lane against pinned 1.0.5 (#9288) * ci(parity): run the dual-stack differential against the pinned reference Adds a differential job to reference-parity.yml: side A is the pinned release, side B the candidate artifact built for the s3-tests comparison. The job prints reference_side, version_a and version_b from the report, fails when side A is not the reference or both versions are equal, and lists every difference in the job summary. * ci(parity): print the first version line in the differential summary * chore: restore main's Cargo and baseline files after merging main --------- Co-authored-by: rustfs-cla-assistant[bot] <273121464+rustfs-cla-assistant[bot]@users.noreply.github.com>
ci(e2e): exclude reference unit tests from e2e-full selection (#9342 )
2026-11-11 20:54:09 +00:00 · 2026-11-11 15:15:12 -05:00 · 2026-11-11 14:13:10 -05:00 · 2026-11-11 14:12:54 -05:00
21 changed files with 335 additions and 1736 deletions
@@ -946,6 +946,10 @@ path = "junit.xml"
 # parallel-safe — the same property e2e-smoke relies on. The exceptions are the
 # 4-disk reliability / degraded-read fault-injection tests, multi-node heal
 # clusters, and the fixed-port Vault tests, all serialized below.
+# `dual_stack_differential::reference::` holds hermetic unit tests of how the
+# differential picks side A (shell-script stand-ins, no server, no reference
+# binary). They already run on every pull request in e2e-smoke, so e2e-full
+# excludes them to keep its selection digest unchanged.
 [profile.e2e-full]
 default-filter = """
    package(e2e_test)
@@ -956,6 +960,7 @@ default-filter = """
    & !test(/^replication_target_matrix_test::/)
    & !test(/^on_demand_migration::(concurrency_test|fault_test|interop_test|real_source_test)::/)
    & !test(/^upgrade_compatibility_test::upgrade_write_readiness_tests::/)
+    & !test(/^dual_stack_differential::reference::/)
 """
 fail-fast = false

@@ -30,6 +30,16 @@
 # result and honours .config/reference-parity-tolerated.txt, whose entries must
 # each match a real difference.
 #
+# The `differential` job adds the 73-operation dual-stack differential
+# (crates/e2e_test/src/dual_stack_differential, rustfs/backlog#2740) against the
+# same pinned reference: side A is the released binary (RUSTFS_E2E_REFERENCE_BINARY,
+# started without RUSTFS_S3_STACK), side B is the candidate built above on its
+# legacy stack. It reuses the candidate build artifact, never builds the candidate
+# twice, and prints reference_side, version_a and version_b from the report the test
+# wrote; it fails when side A was not the reference or both sides report the same
+# version. Every (op, field, a, b) difference goes to the job summary. Report-only
+# until the first result has been reviewed.
+#
 # Report-only: not part of the Test and Lint aggregate. Scheduled failures are
 # owned by the scheduled-validation watchdog (.github/scheduled-validations.json).

@@ -45,6 +55,7 @@ on:
      - ".config/reference-parity-tolerated.txt"
      - "scripts/s3-tests/reference_parity.py"
      - "scripts/junit_diff.py"
+      - "crates/e2e_test/src/dual_stack_differential/**"
  schedule:
    - cron: "37 1 * * *"
  workflow_dispatch:
@@ -253,6 +264,131 @@ jobs:
          if-no-files-found: error
          retention-days: 14

+  differential:
+    name: Dual-stack differential (reference vs candidate)
+    needs: [ build-candidate ]
+    # Compiles the e2e_test harness against the ci-dev cache like the other
+    # sm-standard-4 jobs; the candidate server itself is the downloaded artifact.
+    runs-on: sm-standard-4
+    timeout-minutes: 90
+    env:
+      FORCE_JAVASCRIPT_ACTIONS_TO_NODE24: "true"
+      NO_PROXY: 127.0.0.1,localhost,::1
+    steps:
+      - name: Checkout repository
+        uses: actions/checkout@f548e57e544e1ff5a4c46bf1e1b8685f8e4a348a # v7
+        with:
+          persist-credentials: false
+
+      - name: Setup Rust environment
+        uses: ./.github/actions/setup
+        with:
+          rust-version: stable
+          cache-shared-key: ci-dev
+          cache-save-if: 'false'
+          install-build-packaging-tools: 'false'
+
+      - name: Fetch pinned reference binary
+        id: reference
+        uses: ./.github/actions/reference-binary
+
+      - name: Download candidate binary
+        uses: actions/download-artifact@3e5f45b2cfb9172054b4087a40e8e0b5a5461e7c # v8.0.1
+        with:
+          name: reference-parity-candidate-binary
+          path: target/debug
+
+      - name: Run dual-stack differential
+        id: differential
+        continue-on-error: true
+        env:
+          REFERENCE_PATH: ${{ steps.reference.outputs.path }}
+          RUSTFS_E2E_LOG_DIR: ${{ runner.temp }}/rustfs-differential-logs
+        run: |
+          set -euo pipefail
+          export no_proxy="$NO_PROXY"
+          chmod +x target/debug/rustfs
+          test -x "$REFERENCE_PATH"
+          mkdir -p "$RUSTFS_E2E_LOG_DIR"
+          # Side A: the released binary. Side B: the candidate on its legacy stack.
+          export RUSTFS_E2E_REFERENCE_BINARY="$REFERENCE_PATH"
+          test_name='dual_stack_differential::tests::both_stacks_answer_every_operation_identically'
+          python3 scripts/e2e_binary.py run --features e2e-test-hooks -- \
+            cargo test --locked -p e2e_test --lib "$test_name" -- --exact --nocapture \
+            2>&1 | tee "$RUNNER_TEMP/differential-test.log"
+
+      - name: Summarize report and verify the reference side
+        if: always()
+        run: |
+          set -euo pipefail
+          python3 - <<'PY'
+          import glob, json, os, sys
+
+          summary = os.environ["GITHUB_STEP_SUMMARY"]
+          reports = sorted(glob.glob("target/dual-stack/self-comparison-*.json"))
+          lines = ["## Dual-stack differential: reference vs candidate", ""]
+          failures = []
+          if len(reports) != 1:
+              failures.append(f"expected exactly one self-comparison report, found {len(reports)}: {reports}")
+          else:
+              report = json.load(open(reports[0]))
+              side = report.get("reference_side")
+              version_a = report.get("version_a")
+              version_b = report.get("version_b")
+              lines += [
+                  f"- report: `{reports[0]}`",
+                  f"- reference_side: `{side}`",
+                  f"- version_a: `{(version_a or '').splitlines()[0] if version_a else version_a}`",
+                  f"- version_b: `{(version_b or '').splitlines()[0] if version_b else version_b}`",
+                  f"- stack_a: `{report.get('stack_a')}`, stack_b: `{report.get('stack_b')}`",
+                  f"- compared_ops: {report.get('compared_ops')}, compared_cases: {report.get('compared_cases')}",
+                  "",
+              ]
+              print(f"reference_side={side}\nversion_a={version_a}\nversion_b={version_b}")
+              lines += ["```", "version_a:", version_a or "", "", "version_b:", version_b or "", "```", ""]
+              if side != "a":
+                  failures.append(f"reference_side is {side!r}, expected 'a'")
+              if not version_a or not version_b:
+                  failures.append("version_a or version_b is missing")
+              elif version_a == version_b:
+                  failures.append("version_a equals version_b: the reference was replaced by the candidate")
+              diffs = report.get("differences") or []
+              lines.append(f"### Differences: {len(diffs)}")
+              by_op = {}
+              for d in diffs:
+                  by_op.setdefault(d["op"], []).append(d)
+              for op in sorted(by_op):
+                  lines += ["", f"#### {op}", "", "| case | field | a (reference) | b (candidate) |", "| --- | --- | --- | --- |"]
+                  for d in by_op[op]:
+                      cell = lambda v: "`" + str(v).replace("|", "\\|").replace("`", "'") + "`"
+                      lines.append(f"| {d['case']} | {d['field']} | {cell(d['a'])} | {cell(d['b'])} |")
+          if failures:
+              lines += ["", "### Observation check failed", ""] + [f"- {f}" for f in failures]
+          open(summary, "a").write("\n".join(lines) + "\n")
+          print("\n".join(lines))
+          for f in failures:
+              print(f"::error::{f}")
+          sys.exit(1 if failures else 0)
+          PY
+
+      - name: Upload differential reports
+        if: always()
+        uses: actions/upload-artifact@043fb46d1a93c77aae656e7c1c64a875d1fc6a0a # v7.0.1
+        with:
+          name: reference-differential-report
+          overwrite: true
+          path: |
+            target/dual-stack/*.json
+            ${{ runner.temp }}/differential-test.log
+          if-no-files-found: warn
+          retention-days: 14
+
+      - name: Fail when the differential found differences
+        if: steps.differential.outcome == 'failure'
+        run: |
+          echo "::error::the dual-stack differential failed; see the job summary for every (op, field, a, b)"
+          exit 1
+
  compare:
    name: Compare targets
    needs: [ plan, s3tests ]
@@ -18,8 +18,6 @@ and this project adheres to [Semantic Versioning](https://semver.org/spec/v2.0.0

 ### Security

- **Key-scoped KMS policy statements now hold for every spelling of an AWS key**: a statement such as `Deny kms:Decrypt` on `arn:aws:kms:::key/<key-id>` was matched against the key identifier exactly as the request spelled it, so naming the same key by its AWS ARN (`arn:aws:kms:<region>:<account>:key/<key-id>`) was evaluated as a different key and the `Deny` did not apply. With `RUSTFS_KMS_ENFORCE_SSE_KEY_POLICY=true` and the AWS KMS backend, an identity allowed on all keys but denied one could still write under that key, and read objects stored under its ARN spelling; the per-key admin endpoints were affected the same way. A key ARN is now evaluated as its key id, on the SSE-KMS data path and the admin endpoints alike. A request that names a key by alias, which only the backend can resolve, now fails closed against key-scoped statements: every key-scoped `Deny` applies and only an `Allow` on every key covers it. The KMS deletion impact report recognizes the default key and bucket default-encryption references across the id and ARN spellings as well. Deployments without key-scoped KMS statements, and the other KMS backends, whose key ids have a single spelling, are unaffected.
-
 - **Header-signed SigV4 requests honour only signed headers** (GHSA-xm99-m3gq-83g8): a request authenticated with a SigV4 `Authorization` header that carries an `x-amz-*` request header not listed in its `SignedHeaders` is now rejected with `403 AccessDenied` ("There were headers present in the request which were not signed"), matching AWS S3 and the presigned rule from GHSA-g8w9-qw9q-fghr. Previously anyone holding one header-signed `PutObject` request could add an unsigned `x-amz-copy-source` and turn it into a `CopyObject` that ran with the signer's permissions, copying any object the signer could read into the target. An `Authorization` header whose algorithm token is not `AWS4-HMAC-SHA256` is now rejected instead of being verified as SigV4. The request-envelope headers `x-amz-content-sha256`, `x-amz-decoded-content-length`, `x-amz-trailer` and `x-amz-checksum-algorithm` (the same set the upstream `s3s` fix exempts) and `x-amz-cf-id` (CloudFront) remain tolerated unsigned; AWS SDKs and RustFS's own signers already sign every other `x-amz-*` header. SigV2, JWT and anonymous requests are unchanged.

 ### Replication
@@ -39,10 +37,6 @@ and this project adheres to [Semantic Versioning](https://semver.org/spec/v2.0.0
 - **Presigned URLs honour only signed headers** (GHSA-g8w9-qw9q-fghr): a SigV4 presigned request that carries an `x-amz-*` request header not listed in `X-Amz-SignedHeaders` is now rejected with `403 AccessDenied` ("There were headers present in the request which were not signed"), matching AWS S3. Previously the holder of a presigned `PutObject` URL could add unsigned `x-amz-tagging`, `x-amz-storage-class`, `x-amz-website-redirect-location`, ACL, metadata, Object Lock or SSE headers and have them applied. Presigners that intend a property must set it before signing so the SDK lists the header in `SignedHeaders`; `x-amz-cf-id` (CloudFront) remains tolerated unsigned. Header-signed SigV4 and SigV2 requests are unchanged.

 ### Fixed
-
- **KMS keys could not be described by ARN through the path route**: `GET /rustfs/admin/v3/kms/keys/{key_id}` read the path segment verbatim, so a key id that has to be percent-encoded to fit in one segment — an AWS key ARN, with its `:` and `/` — was looked up, authorized and audited under its encoded spelling and answered `404`. The segment is now percent-decoded first; a segment that is not valid UTF-8 after decoding is refused with `400 InvalidArgument`. The `describe-key?keyId=` form was unaffected.
-
- **SSE-KMS objects written through the AWS KMS backend could not be read**: `GetObject`, `CopyObject` and every other read of an `aws:kms` object failed with `400 InvalidRequest` ("SSE-S3 requires RUSTFS_SSE_S3_MASTER_KEY to be set ...") on a deployment using `RUSTFS_KMS_BACKEND=aws`, although the write and `HeadObject` succeeded. The AWS backend stored the `GenerateDataKey` blob as it came from AWS, the read path took those unrecognized bytes for a locally wrapped key, and AWS was never asked to decrypt. The backend now stores the blob inside the same data-key envelope the other backends use, and a blob stored without one is routed to AWS when the AWS backend is active. Objects written by 1.0.4 and 1.0.5 are intact and become readable after the upgrade, with no rewrite. Setting `RUSTFS_SSE_S3_MASTER_KEY` was never a workaround. An object written by this release is not readable by an older one; the same was already true of every object the older release wrote itself.
 - **Immediate KMS key deletion no longer reports success without destroying the key** (behavior change): with `RUSTFS_KMS_ALLOW_IMMEDIATE_DELETION=true`, a `DELETE /rustfs/admin/v3/kms/keys/delete` request with `force_immediate` against an `Enabled` or `Disabled` key on the Vault KV2 or Vault Transit backend returned success but only moved the key to `PendingDeletion`; the key material stayed in Vault and objects encrypted under it remained readable, while the Local backend destroyed the key in that one request. Key destruction is now the same two explicit steps on every backend: schedule the deletion, then either let the waiting window elapse or send `force_immediate` to destroy the pending key early. `force_immediate` against a key that is not `PendingDeletion` is refused with `400` and changes nothing, so a success response always means the key is gone and no single request can destroy a key in use. Automation that destroyed Local keys with one `force_immediate` request must schedule the deletion first. On Vault Transit, a read under a key that no longer exists in Vault now fails with `KMS.NotFoundException` instead of `503 ServiceUnavailable`. No persisted format changes. See `docs/operations/kms-key-destruction.md`.
 - **`mc admin info` and `mc admin replicate resync` work again**: since the cluster snapshot view (#3803), `GET /v3/info` and `GET /v3/storageinfo` returned the madmin `InfoMessage` / `StorageInfo` nested under an `info` key next to `admin_discovery` (and, for `/info`, `bitrotSelftest`). madmin decodes those bodies directly, so `mc admin info` printed empty values and `mc admin replicate resync start/status` rejected every alias with "alias provided is not part of cluster replication" because the local `deploymentID` read as empty. Requests on the MinIO-compatible `/minio/admin/v3` prefix now receive the bare madmin body again; `/rustfs/admin/v3`, which rc and the console use, keeps the envelope unchanged.
 - **Replication status writebacks no longer invalidate scanner cycles**: every replication status writeback (`persist_replication_state_if_current` in `Update` mode, including the heals the scanner queues during its own walk) marked the bucket dirty on the namespace path, advanced the dirty-usage generation and woke the scanner, so a site-replication resync or any steadily replicated bucket ended every cycle `superseded`, the authoritative usage baseline was never certified, and the console reported usage freshness as stale for as long as replication ran (#8375). Those writebacks are now status class: they do not touch the dirty-usage generation, the activity snapshot, the dirty-usage wake or the durable dirty journal, and only the process-local producer coverage is recorded. They need no rescan queue, because a deployment with an active replication rule already walks every bucket on every cycle and that walk refreshes `replication_stats`. Tier-expiration cleanup, which removes data, stays on the namespace path. No persisted format, protocol, admin API or configuration changes. Design: `docs/architecture/scanner-per-bucket-convergence.md`.
@@ -56,7 +56,6 @@
 //! schedule/cancel deletion, enable/disable) are executed at most once because
 //! a lost response could otherwise be replayed into a second side effect.

-use std::borrow::Cow;
 use std::collections::HashMap;
 use std::future::Future;

@@ -70,7 +69,6 @@ use tokio_util::sync::CancellationToken;

 use super::{BackendCapabilities, ExpiredKeyRemoval, KmsBackend, empty_key_page, list_keys_page_size};
 use crate::config::{BackendConfig, KmsConfig};
-use crate::encryption::{DataKeyEnvelope, is_data_key_envelope};
 use crate::error::{KmsError, Result};
 use crate::policy::{self, AttemptError, ErrorClass, OpClass, RetryPolicy, classify_status};
 use crate::types::*;
@@ -290,22 +288,6 @@ pub struct AwsKmsBackend {
    cancel: CancellationToken,
 }

-/// The blob to hand to AWS `Decrypt` for a stored ciphertext.
-///
-/// [`KmsBackend::generate_data_key`] wraps the AWS blob in a
-/// [`DataKeyEnvelope`]; [`KmsBackend::encrypt`] returns it bare, and so did
-/// `generate_data_key` through 1.0.5. Both shapes stay decryptable. The
-/// envelope's recorded key id and context are not consulted: AWS resolves the
-/// key from the blob and authenticates the caller's context against it.
-fn native_ciphertext(ciphertext: &[u8]) -> Result<Cow<'_, [u8]>> {
-    if !is_data_key_envelope(ciphertext) {
-        return Ok(Cow::Borrowed(ciphertext));
-    }
-    let envelope: DataKeyEnvelope = serde_json::from_slice(ciphertext)
-        .map_err(|e| KmsError::cryptographic_error("parse", format!("Failed to parse data key envelope: {e}")))?;
-    Ok(Cow::Owned(envelope.encrypted_key))
-}
-
 impl AwsKmsBackend {
    /// Build the backend from a KMS configuration.
    ///
@@ -501,8 +483,7 @@ impl KmsBackend for AwsKmsBackend {
        // AWS ciphertext blobs identify their own key, so no key id is needed
        // here — and AWS refuses to decrypt with a disabled or pending-deletion
        // key, unlike the RustFS-managed backends.
-        let ciphertext = native_ciphertext(&request.ciphertext)?;
-        let ciphertext = ciphertext.as_ref();
+        let ciphertext = request.ciphertext.as_slice();
        let context = encryption_context(&request.encryption_context);
        let tokens = grant_tokens(&request.grant_tokens);

@@ -582,29 +563,10 @@ impl KmsBackend for AwsKmsBackend {
            .ciphertext_blob
            .ok_or_else(|| KmsError::backend_error(format!("AWS KMS GenerateDataKey returned no ciphertext for key {key_id}")))?;

-        let master_key_id = output.key_id.unwrap_or_else(|| request.key_id.clone());
-        // The envelope is what lets a reader recognize these bytes as
-        // KMS-wrapped: the native blob identifies its key to AWS, but nothing
-        // in it identifies AWS to RustFS.
-        let envelope = DataKeyEnvelope {
-            key_id: uuid::Uuid::new_v4().to_string(),
-            master_key_id: master_key_id.clone(),
-            key_spec: request.key_spec.as_str().to_string(),
-            encrypted_key: ciphertext.into_inner(),
-            nonce: Vec::new(),
-            encryption_context: request.encryption_context,
-            created_at: Zoned::now(),
-            // AWS addresses backing keys internally and never exposes a version.
-            master_key_version: None,
-            // AWS binds the context into the blob itself; this field describes
-            // only the local DekCrypto binding, which this backend never uses.
-            context_binding: None,
-        };
-
        Ok(GenerateDataKeyResponse {
-            key_id: master_key_id,
+            key_id: output.key_id.unwrap_or_else(|| request.key_id.clone()),
            plaintext_key: plaintext.into_inner(),
-            ciphertext_blob: serde_json::to_vec(&envelope)?,
+            ciphertext_blob: ciphertext.into_inner(),
        })
    }

@@ -864,12 +826,6 @@ impl KmsBackend for AwsKmsBackend {
            .with_production_supported(true)
    }

-    /// Releases through 1.0.5 persisted the `GenerateDataKey` blob without an
-    /// envelope, so objects written by them carry it bare.
-    fn reads_bare_data_key_ciphertext(&self) -> bool {
-        true
-    }
-
    /// Observe, never destroy.
    ///
    /// AWS runs the deletion window itself, so the sweep's job here is only to
@@ -1016,49 +972,15 @@ mod tests {
        insta::assert_json_snapshot!("aws_backend_capabilities", capabilities_snapshot(backend.capabilities()));
    }

-    /// A data key blob shaped like AWS's: binary, not UTF-8, and nothing in it
-    /// that RustFS could recognize.
-    fn native_blob() -> Vec<u8> {
-        let mut blob = vec![0x01, 0x02, 0x02, 0x00, 0x78];
-        blob.extend((0u8..=178).map(|byte| byte.wrapping_mul(151).wrapping_add(0x80)));
-        blob
-    }
-
-    fn generate_data_key_event(plaintext: &[u8], blob: &[u8]) -> ReplayEvent {
-        ok_event(serde_json::json!({
-            "KeyId": "arn:aws:kms:us-east-1:111122223333:key/test-key",
-            "Plaintext": BASE64.encode_to_string(plaintext),
-            "CiphertextBlob": BASE64.encode_to_string(blob),
-        }))
-    }
-
-    fn decrypt_event(plaintext: &[u8]) -> ReplayEvent {
-        ok_event(serde_json::json!({
-            "KeyId": "arn:aws:kms:us-east-1:111122223333:key/test-key",
-            "Plaintext": BASE64.encode_to_string(plaintext),
-            "EncryptionAlgorithm": "SYMMETRIC_DEFAULT",
-        }))
-    }
-
-    /// The JSON body of the `index`-th request the backend actually sent.
-    fn sent_body(http_client: &StaticReplayClient, index: usize) -> serde_json::Value {
-        let request = http_client
-            .actual_requests()
-            .nth(index)
-            .expect("the backend should have sent this request");
-        serde_json::from_slice(request.body().bytes().expect("request body should be buffered"))
-            .expect("AWS KMS requests are JSON")
-    }
-
-    /// The data key has to come back in the envelope every reader classifies
-    /// KMS-wrapped keys by. Returning the bare blob made each object written
-    /// through this backend unreadable: the read path took the unrecognized
-    /// bytes for a locally wrapped key and never called AWS at all.
    #[tokio::test]
-    async fn generate_data_key_wraps_the_native_blob_in_an_envelope() {
+    async fn generate_data_key_maps_the_native_response() {
        let plaintext = vec![7u8; 32];
-        let blob = native_blob();
-        let (http_client, backend) = scripted_backend(vec![generate_data_key_event(&plaintext, &blob)]);
+        let ciphertext = b"encrypted-data-key".to_vec();
+        let (http_client, backend) = scripted_backend(vec![ok_event(serde_json::json!({
+            "KeyId": "arn:aws:kms:us-east-1:111122223333:key/test-key",
+            "Plaintext": BASE64.encode_to_string(&plaintext),
+            "CiphertextBlob": BASE64.encode_to_string(&ciphertext),
+        }))]);

        let response = backend
            .generate_data_key(generate_request())
@@ -1066,89 +988,23 @@ mod tests {
            .expect("generate_data_key should map the AWS response");

        assert_eq!(response.plaintext_key, plaintext);
+        assert_eq!(response.ciphertext_blob, ciphertext);
        assert_eq!(response.key_id, "arn:aws:kms:us-east-1:111122223333:key/test-key");
-        assert!(
-            is_data_key_envelope(&response.ciphertext_blob),
-            "the stored data key must be recognizable as KMS-wrapped from its bytes alone"
-        );
-        let envelope: DataKeyEnvelope = serde_json::from_slice(&response.ciphertext_blob).expect("envelope must parse");
-        assert_eq!(envelope.encrypted_key, blob, "the envelope must carry the AWS blob byte for byte");
-        assert_eq!(envelope.master_key_id, "arn:aws:kms:us-east-1:111122223333:key/test-key");
-        assert_eq!(envelope.key_spec, "AES_256");
-        assert_eq!(envelope.encryption_context, generate_request().encryption_context);
-        assert_eq!(envelope.master_key_version, None);
-        assert_eq!(envelope.context_binding, None);
        assert_eq!(http_client.actual_requests().count(), 1);
    }

-    /// What `generate_data_key` returns is what `decrypt` is later handed, so
-    /// AWS must receive the blob it issued — not the envelope around it — along
-    /// with the caller's context, which AWS authenticates.
    #[tokio::test]
-    async fn decrypt_opens_a_generated_data_key_with_the_native_blob() {
-        let plaintext = vec![9u8; 32];
-        let blob = native_blob();
-        let (http_client, backend) =
-            scripted_backend(vec![generate_data_key_event(&plaintext, &blob), decrypt_event(&plaintext)]);
-
-        let data_key = backend
-            .generate_data_key(generate_request())
-            .await
-            .expect("generate_data_key should succeed");
-        let response = backend
-            .decrypt(DecryptRequest {
-                ciphertext: data_key.ciphertext_blob,
-                encryption_context: generate_request().encryption_context,
-                grant_tokens: Vec::new(),
-            })
-            .await
-            .expect("a generated data key must decrypt");
-
-        assert_eq!(response.plaintext, plaintext);
-        let sent = sent_body(&http_client, 1);
-        assert_eq!(sent["CiphertextBlob"], BASE64.encode_to_string(&blob));
-        assert_eq!(sent["EncryptionContext"], serde_json::json!({ "bucket": "aws" }));
-    }
-
-    /// The caller's context reaches AWS even when it disagrees with the one
-    /// recorded in the envelope: the recorded copy is unauthenticated, so AWS,
-    /// which bound the real context into the blob, has to be the judge.
-    #[tokio::test]
-    async fn decrypt_sends_the_callers_context_not_the_envelopes() {
-        let plaintext = vec![9u8; 32];
-        let (http_client, backend) = scripted_backend(vec![
-            generate_data_key_event(&plaintext, &native_blob()),
-            error_event(400, "InvalidCiphertextException", "context does not match"),
-        ]);
-
-        let data_key = backend
-            .generate_data_key(generate_request())
-            .await
-            .expect("generate_data_key should succeed");
-        backend
-            .decrypt(DecryptRequest {
-                ciphertext: data_key.ciphertext_blob,
-                encryption_context: HashMap::from([("bucket".to_string(), "other".to_string())]),
-                grant_tokens: Vec::new(),
-            })
-            .await
-            .expect_err("AWS rejects a context that does not match the blob");
-
-        assert_eq!(sent_body(&http_client, 1)["EncryptionContext"], serde_json::json!({ "bucket": "other" }));
-    }
-
-    /// Objects written through 1.0.5 store the AWS blob bare, and `encrypt`
-    /// still returns it that way, so a ciphertext that is not an envelope goes
-    /// to AWS untouched.
-    #[tokio::test]
-    async fn decrypt_sends_a_bare_blob_unchanged() {
+    async fn decrypt_recovers_the_plaintext_without_a_key_id() {
        let plaintext = b"recovered-data-key".to_vec();
-        let blob = native_blob();
-        let (http_client, backend) = scripted_backend(vec![decrypt_event(&plaintext)]);
+        let (_http, backend) = scripted_backend(vec![ok_event(serde_json::json!({
+            "KeyId": "arn:aws:kms:us-east-1:111122223333:key/test-key",
+            "Plaintext": BASE64.encode_to_string(&plaintext),
+            "EncryptionAlgorithm": "SYMMETRIC_DEFAULT",
+        }))]);

        let response = backend
            .decrypt(DecryptRequest {
-                ciphertext: blob.clone(),
+                ciphertext: b"blob".to_vec(),
                encryption_context: HashMap::new(),
                grant_tokens: Vec::new(),
            })
@@ -1157,44 +1013,6 @@ mod tests {

        assert_eq!(response.plaintext, plaintext);
        assert_eq!(response.encryption_algorithm.as_deref(), Some("SYMMETRIC_DEFAULT"));
-        assert_eq!(sent_body(&http_client, 0)["CiphertextBlob"], BASE64.encode_to_string(&blob));
-        assert!(backend.reads_bare_data_key_ciphertext());
-    }
-
-    /// The object layer's write and read, through the same service entry
-    /// points the S3 handlers use: the key a PUT stores must be the key a GET
-    /// recovers, under the object's own context.
-    #[tokio::test]
-    async fn object_data_key_round_trips_through_the_encryption_service() {
-        use crate::manager::KmsManager;
-        use crate::service::ObjectEncryptionService;
-
-        let plaintext = vec![3u8; 32];
-        let blob = native_blob();
-        let (http_client, backend) =
-            scripted_backend(vec![generate_data_key_event(&plaintext, &blob), decrypt_event(&plaintext)]);
-        let manager = KmsManager::new(std::sync::Arc::new(backend), KmsConfig::aws(Some("us-east-1".to_string())));
-        let service = ObjectEncryptionService::new(manager);
-        let context = ObjectEncryptionContext::new("bucket".to_string(), "dir/object".to_string());
-
-        let (written, stored) = service
-            .create_data_key(&Some("test-key".to_string()), &context)
-            .await
-            .expect("the write path must obtain a data key");
-        assert!(is_data_key_envelope(&stored));
-        assert!(service.reads_bare_data_key_ciphertext());
-
-        let read = service
-            .decrypt_data_key(&stored, &context)
-            .await
-            .expect("the read path must recover the data key");
-
-        assert_eq!(read.plaintext_key, written.plaintext_key);
-        assert_eq!(sent_body(&http_client, 1)["CiphertextBlob"], BASE64.encode_to_string(&blob));
-        assert_eq!(
-            sent_body(&http_client, 1)["EncryptionContext"],
-            sent_body(&http_client, 0)["EncryptionContext"]
-        );
    }

    /// Asserts that a mapped failure landed on the intended `KmsError` variant.
@@ -584,18 +584,6 @@ pub trait KmsBackend: Send + Sync {
        BackendCapabilities::minimal()
    }

-    /// Whether objects on disk may carry this backend's data-key ciphertext
-    /// without a [`crate::encryption::DataKeyEnvelope`] around it.
-    ///
-    /// Every backend wraps the data keys it generates in an envelope, which is
-    /// what lets a reader recognize a KMS-wrapped key from its bytes alone. A
-    /// backend returns `true` only when an earlier release persisted its
-    /// provider-native ciphertext bare, so those bytes still have to be routed
-    /// to [`Self::decrypt`] even though nothing in them identifies the backend.
-    fn reads_bare_data_key_ciphertext(&self) -> bool {
-        false
-    }
-
    /// Remove a key whose scheduled deletion deadline has passed.
    ///
    /// Used by the background deletion worker. Implementations must re-check
@@ -670,11 +670,6 @@ impl KmsManager {
        self.backend.capabilities()
    }

-    /// See [`KmsBackend::reads_bare_data_key_ciphertext`].
-    pub fn reads_bare_data_key_ciphertext(&self) -> bool {
-        self.backend.reads_bare_data_key_ciphertext()
-    }
-
    /// Direct handle to the configured backend, bypassing the metadata cache.
    /// Used by background maintenance that must observe fresh state.
    pub(crate) fn backend(&self) -> Arc<dyn KmsBackend> {
@@ -304,15 +304,6 @@ impl ObjectEncryptionService {
        self.kms_manager.backend_capabilities()
    }

-    /// Whether a stored data key that is not a RustFS envelope may still belong
-    /// to the active backend, and so must be offered to
-    /// [`Self::decrypt_data_key`] instead of being treated as foreign.
-    ///
-    /// See [`crate::backends::KmsBackend::reads_bare_data_key_ciphertext`].
-    pub fn reads_bare_data_key_ciphertext(&self) -> bool {
-        self.kms_manager.reads_bare_data_key_ciphertext()
-    }
-
    /// Re-wrap an object's encrypted data key onto its master key's current
    /// version, without the plaintext data key ever reaching the caller.
    ///
@@ -2534,139 +2534,6 @@ mod test {
        Ok(())
    }

-    const AWS_KEY_ID: &str = "74e5921e-107a-4584-bfda-a5dadeb3e13e";
-    const AWS_KEY_ARN: &str = "arn:aws:kms:eu-north-1:111122223333:key/74e5921e-107a-4584-bfda-a5dadeb3e13e";
-    const OTHER_AWS_KEY_ARN: &str = "arn:aws:kms:eu-north-1:111122223333:key/00000000-0000-4000-8000-000000000000";
-
-    /// A key-scoped statement has to hold however the request spells the key.
-    /// Matched literally, a `Deny` written on the key id let the same key
-    /// through when the request named it by its ARN — on the write and on the
-    /// read of an SSE-KMS object alike.
-    #[tokio::test]
-    async fn test_kms_key_statement_covers_the_aws_arn_spelling_of_the_key() -> Result<()> {
-        use crate::policy::action::{Action, KmsAction};
-
-        let deny = Policy::parse_config(
-            format!(
-                r#"{{
-  "Version": "2012-10-17",
-  "Statement": [
-    {{ "Effect": "Allow", "Action": ["kms:*"], "Resource": ["arn:aws:kms:::*"] }},
-    {{ "Effect": "Deny", "Action": ["kms:Decrypt", "kms:GenerateDataKey"], "Resource": ["arn:aws:kms:::key/{AWS_KEY_ID}"] }}
-  ]
-}}"#
-            )
-            .as_bytes(),
-        )?;
-        let allow = Policy::parse_config(
-            format!(
-                r#"{{
-  "Version": "2012-10-17",
-  "Statement": [
-    {{ "Effect": "Allow", "Action": ["kms:Decrypt"], "Resource": ["arn:aws:kms:::key/{AWS_KEY_ID}"] }}
-  ]
-}}"#
-            )
-            .as_bytes(),
-        )?;
-        let conditions = HashMap::new();
-        let claims = HashMap::new();
-        let decrypt = Action::KmsAction(KmsAction::DecryptAction);
-
-        for spelling in [AWS_KEY_ID, AWS_KEY_ARN] {
-            assert!(
-                !deny.is_allowed(&kms_args(decrypt, spelling, &conditions, &claims)).await,
-                "a Deny on the key id must cover the key named as {spelling}"
-            );
-            assert!(
-                allow.is_allowed(&kms_args(decrypt, spelling, &conditions, &claims)).await,
-                "an Allow on the key id must cover the key named as {spelling}"
-            );
-        }
-        assert!(
-            deny.is_allowed(&kms_args(decrypt, OTHER_AWS_KEY_ARN, &conditions, &claims))
-                .await,
-            "a Deny on one key must not spread to another key's ARN"
-        );
-        assert!(
-            !allow
-                .is_allowed(&kms_args(decrypt, OTHER_AWS_KEY_ARN, &conditions, &claims))
-                .await,
-            "an Allow on one key must not spread to another key's ARN"
-        );
-
-        // Shapes the AWS KMS API would not accept as a key are not reinterpreted.
-        for not_a_key_arn in [
-            format!("arn:aws:s3:::key/{AWS_KEY_ID}"),
-            format!("arn:aws:kms:eu-north-1:111122223333:key/nested/{AWS_KEY_ID}"),
-            format!("prefix-{AWS_KEY_ARN}"),
-        ] {
-            assert!(
-                !allow
-                    .is_allowed(&kms_args(decrypt, &not_a_key_arn, &conditions, &claims))
-                    .await,
-                "{not_a_key_arn} must not be taken for the key"
-            );
-        }
-
-        Ok(())
-    }
-
-    /// An alias names a key only the backend can resolve, so it must not be a
-    /// way around a statement scoped to particular keys.
-    #[tokio::test]
-    async fn test_kms_alias_request_fails_closed_against_key_scoped_statements() -> Result<()> {
-        use crate::policy::action::{Action, KmsAction};
-
-        let conditions = HashMap::new();
-        let claims = HashMap::new();
-        let decrypt = Action::KmsAction(KmsAction::DecryptAction);
-        let aliases = ["alias/payroll", "arn:aws:kms:eu-north-1:111122223333:alias/payroll"];
-        let policy = |statements: &str| {
-            Policy::parse_config(format!(r#"{{ "Version": "2012-10-17", "Statement": [{statements}] }}"#).as_bytes())
-        };
-
-        let all_keys = policy(r#"{ "Effect": "Allow", "Action": ["kms:*"], "Resource": ["arn:aws:kms:::*"] }"#)?;
-        let unscoped = policy(r#"{ "Effect": "Allow", "Action": ["kms:*"] }"#)?;
-        let one_key = policy(&format!(
-            r#"{{ "Effect": "Allow", "Action": ["kms:*"], "Resource": ["arn:aws:kms:::key/{AWS_KEY_ID}"] }}"#
-        ))?;
-        let all_but_one = policy(&format!(
-            r#"{{ "Effect": "Allow", "Action": ["kms:*"], "NotResource": ["arn:aws:kms:::key/{AWS_KEY_ID}"] }}"#
-        ))?;
-        let all_with_key_deny = policy(&format!(
-            r#"{{ "Effect": "Allow", "Action": ["kms:*"], "Resource": ["arn:aws:kms:::*"] }},
-               {{ "Effect": "Deny", "Action": ["kms:Decrypt"], "Resource": ["arn:aws:kms:::key/{AWS_KEY_ID}"] }}"#
-        ))?;
-
-        for alias in aliases {
-            assert!(
-                all_keys.is_allowed(&kms_args(decrypt, alias, &conditions, &claims)).await,
-                "an Allow on every key covers whatever key {alias} names"
-            );
-            assert!(
-                unscoped.is_allowed(&kms_args(decrypt, alias, &conditions, &claims)).await,
-                "an unscoped Allow covers whatever key {alias} names"
-            );
-            assert!(
-                !one_key.is_allowed(&kms_args(decrypt, alias, &conditions, &claims)).await,
-                "an Allow on one key cannot be shown to cover {alias}"
-            );
-            assert!(
-                !all_but_one.is_allowed(&kms_args(decrypt, alias, &conditions, &claims)).await,
-                "an Allow excluding one key cannot be shown to exclude {alias}"
-            );
-            assert!(
-                !all_with_key_deny
-                    .is_allowed(&kms_args(decrypt, alias, &conditions, &claims))
-                    .await,
-                "a Deny on one key must apply to {alias}, which may name it"
-            );
-        }
-
-        Ok(())
-    }
-
    #[tokio::test]
    async fn test_kms_alias_resource_parses_but_matches_no_key() -> Result<()> {
        use crate::policy::action::{Action, KmsAction};
@@ -89,51 +89,6 @@ enum ActionFamily {
    Mixed,
 }

-/// A requested KMS key identifier, reduced to what `key/<key_id>` patterns are
-/// written against.
-///
-/// Policies name a key by its id, and [`Resource`] validation refuses a key
-/// pattern containing a separator. An identifier reaching the matcher in any
-/// other spelling of the same key was therefore compared as a different key: a
-/// `Deny` on `key/<id>` did not cover a request naming that key by its AWS ARN.
-/// Only the spellings the AWS KMS API itself accepts are recognized; anything
-/// else is matched literally.
-#[derive(Debug, PartialEq)]
-pub enum RequestedKmsKey<'a> {
-    /// A key id, taken from the identifier itself or from a key ARN
-    /// (`arn:<partition>:kms:<region>:<account>:key/<key_id>`).
-    Key(&'a str),
-    /// An alias name or alias ARN: it names a key only the backend can resolve.
-    Alias,
-}
-
-impl<'a> RequestedKmsKey<'a> {
-    /// Whether two identifiers provably name the same key. An alias is never
-    /// provably the same key as anything, itself included.
-    pub fn same_key(left: &str, right: &str) -> bool {
-        matches!(
-            (RequestedKmsKey::parse(left), RequestedKmsKey::parse(right)),
-            (RequestedKmsKey::Key(left), RequestedKmsKey::Key(right)) if left == right
-        )
-    }
-
-    pub fn parse(identifier: &'a str) -> Self {
-        let fields: Vec<&str> = identifier.splitn(6, ':').collect();
-        let resource = match fields.as_slice() {
-            ["arn", _partition, "kms", _region, _account, resource] => *resource,
-            _ if identifier.starts_with(Resource::KMS_ALIAS_SEGMENT) => return Self::Alias,
-            _ => return Self::Key(identifier),
-        };
-        if resource.starts_with(Resource::KMS_ALIAS_SEGMENT) {
-            return Self::Alias;
-        }
-        match resource.strip_prefix(Resource::KMS_KEY_SEGMENT) {
-            Some(key_id) if !key_id.is_empty() && !key_id.contains('/') => Self::Key(key_id),
-            _ => Self::Key(identifier),
-        }
-    }
-}
-
 impl Statement {
    fn skips_resource_match_for_args(&self, args: &Args<'_>) -> bool {
        if self.is_sts() {
@@ -223,9 +178,8 @@ impl Statement {
    /// patterns instead of the bucket/object path grammar used by S3 statements.
    ///
    /// Call-site contract (wired up by the admin/SSE authorization paths): the requested
-    /// key identifier travels in `args.object`, as the request spelled it, with `args.bucket`
-    /// left empty; [`RequestedKmsKey`] reduces it to the key id the patterns are written
-    /// against. An empty `args.object` means the caller did not scope the request to a
+    /// key identifier (pre-alias-resolution) travels in `args.object` with `args.bucket`
+    /// left empty. An empty `args.object` means the caller did not scope the request to a
    /// key, which preserves the legacy match-every-key behaviour.
    async fn kms_key_scope_matches(&self, args: &Args<'_>, resolver: &VariableResolver) -> bool {
        if self.not_resources.iter().any(|resource| matches!(resource, Resource::All)) {
@@ -269,27 +223,7 @@ impl Statement {
            return true;
        }

-        let key_id = match RequestedKmsKey::parse(args.object) {
-            RequestedKmsKey::Key(key_id) => key_id,
-            RequestedKmsKey::Alias => {
-                // Which key an alias names is known only to the backend, so a
-                // statement scoped to particular keys cannot be shown to cover
-                // or to exclude it. Fail closed in both directions: every
-                // key-scoped Deny applies, and only an Allow on all keys does.
-                return match self.effect {
-                    Effect::Deny => true,
-                    Effect::Allow => {
-                        kms_not_resources.is_empty()
-                            && kms_resources.iter().any(|resource| match resource {
-                                Resource::All => true,
-                                Resource::Kms(pattern) => pattern == "*",
-                                Resource::S3(_) => false,
-                            })
-                    }
-                };
-            }
-        };
-        let requested = format!("{}{}", Resource::KMS_KEY_SEGMENT, key_id);
+        let requested = format!("{}{}", Resource::KMS_KEY_SEGMENT, args.object);

        if !kms_resources.is_empty() {
            let mut matched = false;
@@ -60,14 +60,14 @@ There is no key-version metadata key. Object metadata carries the key id (`x-rus
 | Vault Transit (`crates/kms/src/backends/vault_transit.rs`) | Yes | `vault:vN:` prefix of the ciphertext in `encrypted_key`; the envelope's version field is deliberately `None` | Yes, by parsing that prefix |
 | Local (`crates/kms/src/backends/local.rs`) | No, rotation is rejected | Nowhere; hardcoded `None` | Moot while rotation is rejected |
 | Static (`crates/kms/src/backends/static_kms.rs`) | No | Nowhere; hardcoded `None` | Moot |
-| AWS (`crates/kms/src/backends/aws.rs`) | AWS-managed | Inside the opaque `CiphertextBlob`, which the envelope carries in `encrypted_key`; objects written through 1.0.5 store that blob with no `DataKeyEnvelope` at all | **No** |
+| AWS (`crates/kms/src/backends/aws.rs`) | AWS-managed | Inside the opaque `CiphertextBlob`; no `DataKeyEnvelope` | **No** |

 Contract rules that follow:

 - **`None` does not mean one thing.** KV2: pre-versioning envelope, resolved by `resolve_envelope_master_key_version` to the key's recorded baseline, never implicitly to "current". Transit: permanent and expected; read the ciphertext prefix. Local/Static: unconditional. Version extraction must be dispatched by backend, never inferred from the field alone.
 - **Local's `None` is coupled to the Local blocker.** If Local gains rotation history (`rustfs/backlog#1565`), envelope version recording must land in the same change, or Local becomes a second unreadable backend.
 - The primitive exposes the wrapping version through **one backend-dispatched accessor** (`KmsManager::describe_data_key_wrapping`) and reports **"already at target state" as an outcome distinct from "re-wrapped"**.
- **AWS is a scoping exception.** Its ciphertext is opaque whether or not an envelope surrounds it, so no scan can skip, report version composition, or self-evidence completion; a re-run would rewrap everything. AWS-backed keys are out of scope and must be refused at admission.
+- **AWS is a scoping exception.** Its ciphertext is opaque, so no scan can skip, report version composition, or self-evidence completion; a re-run would rewrap everything. AWS-backed keys are out of scope and must be refused at admission.
 - Skip detection costs a base64 decode plus JSON parse (plus a prefix parse on Transit) per work unit: CPU, not I/O, and part of the rate budget rather than free.

 ## Failure Semantics
@@ -1,281 +0,0 @@
-# AWS KMS backend: setup, verification, and operations
-
-**Use this when:** connecting RustFS to AWS KMS for the first time, verifying an existing AWS-backed deployment, upgrading one from 1.0.4 or 1.0.5, or deciding how to run and protect the AWS keys RustFS depends on.
-**Source of truth:** `crates/kms/src/backends/aws.rs` (backend behavior and its deviations), `crates/kms/src/config.rs` (`ENV_KMS_AWS_*`), `crates/kms/src/policy.rs` (retry, deadline and circuit breaker), `rustfs/src/storage/sse.rs` (SSE data path). 中文版: [kms-aws-backend-guide.zh-CN.md](kms-aws-backend-guide.zh-CN.md).
-
-The statements about behavior on this page were observed against the real AWS KMS service (region `eu-north-1`) and, for failures AWS cannot be asked to produce on demand, against a loopback stand-in that speaks the AWS KMS wire protocol. Where a statement is guidance that was not exercised, it says so. For the contract-level differences between AWS and the other backends, see [KMS backend security properties](kms-backend-security.md#aws-kms-deviations-from-the-shared-backend-contract).
-
-## What RustFS does with AWS KMS
-
-RustFS uses envelope encryption. For each encrypted object it asks AWS KMS for a data key (`GenerateDataKey`), encrypts the object with the plaintext data key, and stores only the wrapped data key next to the object. To read the object it sends the wrapped key back (`Decrypt`). The KMS key itself never leaves AWS, and RustFS never stores AWS credentials.
-
-Consequences that shape everything below:
-
- **AWS KMS is on the read path.** If the key is disabled, scheduled for deletion, deleted, or AWS is unreachable, encrypted objects cannot be read. Nothing stored in RustFS can substitute for the key.
- **Every encrypted write and every encrypted read is one AWS request.** Multipart uploads add one `Decrypt` per uploaded part. Plan request volume and cost accordingly.
- **Both `aws:kms` and `AES256` objects use the key.** With a KMS backend configured, SSE-S3 (`AES256`) wraps its data keys under the configured default key, so SSE-S3 objects depend on AWS exactly as SSE-KMS objects do.
-
-## Before you start
-
-| Item | Requirement |
-| --- | --- |
-| RustFS release | One that contains the AWS read-path fix. Releases 1.0.4 and 1.0.5 can write but not read AWS-backed objects; see [Upgrading from 1.0.4 or 1.0.5](#upgrading-from-104-or-105). |
-| KMS key | Symmetric, usage "Encrypt and decrypt", in the region RustFS will be configured with. |
-| Network | Every RustFS node must reach `kms.<region>.amazonaws.com` over HTTPS (or a VPC interface endpoint for KMS). The AWS SDK inside RustFS does not use `HTTP_PROXY` / `HTTPS_PROXY`. |
-| Identity | One IAM identity per deployment that every node can obtain credentials for. |
-| Clock | Node clocks in sync; AWS rejects requests whose signature time is off by more than a few minutes. |
-
-## Step 1: create the KMS key
-
-In the AWS console: **KMS → Customer managed keys → Create key**.
-
-1. Key type **Symmetric**, key usage **Encrypt and decrypt**. Leave "Advanced options" at their defaults (key material origin KMS, single-Region) unless you have decided on multi-Region keys for disaster recovery.
-2. Alias and description: for humans only. RustFS addresses the key by its **key ID or ARN**, never by alias.
-3. **Key administrators**: select only the human role that will manage this key (disable, rotate, schedule deletion). Do not select AWS service-linked roles (`AWSServiceRoleFor…`) or organization-managed roles. Keep "Allow key administrators to delete this key" only if those administrators should hold that power.
-4. **Key users**: select the IAM identity RustFS runs as (created in step 2; you can come back and add it). This grants `kms:Encrypt`, `kms:Decrypt`, `kms:ReEncrypt*`, `kms:GenerateDataKey*` and `kms:DescribeKey` on this key through the key policy.
-5. Finish, then record the **region** and the **key ARN**.
-
-Administrators and users should be different identities. The identity RustFS runs as needs to use the key, not to manage it.
-
-## Step 2: create the identity RustFS runs as
-
-Prefer short-lived role credentials over long-lived access keys:
-
-| Where RustFS runs | Recommended identity |
-| --- | --- |
-| EC2 | An instance profile role |
-| EKS | IAM Roles for Service Accounts (IRSA) or EKS Pod Identity |
-| ECS | A task role |
-| Outside AWS | A dedicated IAM user with an access key, rotated on a schedule, or IAM Roles Anywhere |
-
-RustFS resolves credentials through the standard AWS provider chain: environment variables, the shared profile (`AWS_PROFILE`, `~/.aws/credentials`), then container and instance metadata. It never persists them, and the KMS admin API rejects credential fields in a configuration body.
-
-Attach this identity policy. `kms:ListKeys` cannot be granted by a key policy because it is not scoped to one key, and RustFS calls it as its startup health check; without it the KMS backend does not start.
-
-```json
-{
-  "Version": "2012-10-17",
-  "Statement": [
-    {
-      "Sid": "RustfsKmsHealthCheck",
-      "Effect": "Allow",
-      "Action": "kms:ListKeys",
-      "Resource": "*"
-    }
-  ]
-}
-```
-
-If you would rather not rely on the console's "key users" list, grant the data-path actions in the same identity policy, scoped to the key:
-
-```json
-{
-  "Sid": "RustfsKmsDataPath",
-  "Effect": "Allow",
-  "Action": ["kms:GenerateDataKey", "kms:Decrypt", "kms:DescribeKey"],
-  "Resource": "arn:aws:kms:<region>:<account-id>:key/<key-id>"
-}
-```
-
-Only if operators will manage AWS keys **through the RustFS admin API** does the identity also need the management actions below. Most deployments should leave these out and manage keys in AWS directly, so that a compromised RustFS node cannot disable or delete keys.
-
-| RustFS admin operation | AWS action |
-| --- | --- |
-| Create key | `kms:CreateKey` |
-| Enable / disable | `kms:EnableKey`, `kms:DisableKey` |
-| Rotate | `kms:RotateKeyOnDemand` |
-| Schedule / cancel deletion | `kms:ScheduleKeyDeletion`, `kms:CancelKeyDeletion` |
-
-## Step 3: configure RustFS
-
-Set these on every node and restart:
-
-```bash
-RUSTFS_KMS_ENABLE=true
-RUSTFS_KMS_BACKEND=aws
-RUSTFS_KMS_AWS_REGION=eu-north-1
-RUSTFS_KMS_DEFAULT_KEY_ID=arn:aws:kms:eu-north-1:<account-id>:key/<key-id>
-```
-
-Notes:
-
- `RUSTFS_KMS_DEFAULT_KEY_ID` is the key used when a request asks for encryption without naming a key, and the key that wraps every SSE-S3 data key. The key ID and the ARN are both accepted.
- Do **not** set `RUSTFS_SSE_S3_MASTER_KEY`. It configures the no-KMS fallback and has no role when a KMS backend is active.
- `RUSTFS_KMS_AWS_ENDPOINT_URL` is only for emulators and non-default endpoints. A plain `http://` endpoint is refused unless `RUSTFS_KMS_ALLOW_INSECURE_DEV_DEFAULTS=true`, which is for development only. With a VPC interface endpoint and private DNS enabled, the default regional name already resolves to the endpoint and nothing needs to be set.
- The same configuration can be applied at runtime through the admin API (`POST /rustfs/admin/v3/kms/configure`, body `{"backend_type": "AWS", "region": "...", "default_key_id": "..."}`). Credentials still come from each node's own environment.
-
-Startup is fail-closed. If the health check cannot reach AWS, or the endpoint is plain HTTP without the development switch, the RustFS process exits during startup instead of serving without encryption.
-
-## Step 4: verify
-
-Write, inspect and read back one object. With any S3 client:
-
-```bash
-aws s3api put-object --endpoint-url https://rustfs.example.com --bucket demo --key kms-check \
-  --body ./sample.bin --server-side-encryption aws:kms
-aws s3api head-object --endpoint-url https://rustfs.example.com --bucket demo --key kms-check
-aws s3api get-object  --endpoint-url https://rustfs.example.com --bucket demo --key kms-check ./readback.bin
-cmp ./sample.bin ./readback.bin
-```
-
-Expected: all three calls succeed, `head-object` reports `ServerSideEncryption: aws:kms` and the key you configured, and the read-back bytes are identical.
-
-Then confirm the round trip in AWS, which proves RustFS really used the key rather than a local fallback: in **CloudTrail → Event history**, filter by event source `kms.amazonaws.com`. A write produces `GenerateDataKey`; a read produces `Decrypt`. Each event carries the encryption context RustFS sent, whose entry maps the bucket name to `bucket/object-key`, so CloudTrail shows which object each KMS call served. **A read that returns data without a matching `Decrypt` event, or an error with no `Decrypt` attempt, means the read never reached AWS.**
-
-Check the service state:
-
-```bash
-GET /rustfs/admin/v3/kms/status
-```
-
-`backend_status` should be `healthy`, and `capabilities` should report `rotate`, `enable_disable` and `schedule_deletion` true with `versioning`, `physical_delete` and `rewrap` false.
-
-## Behavior reference
-
-Observed on the AWS backend. Status codes are what an S3 client receives.
-
-### Key identifiers
-
-| Situation | Result |
-| --- | --- |
-| Key named by ID or by ARN in `x-amz-server-side-encryption-aws-kms-key-id` | Accepted. The response and later `HEAD`/`GET` echo the same spelling the write used. |
-| Key that does not exist, or an ARN in another region | `400 KMS.NotFoundException`; no object is created. |
-| Create a key through RustFS with a chosen name (including the `mc`-compatible `create-key?key-id=`) | `501 NotImplemented`. AWS assigns key IDs; create without a name and use the returned ID. |
-| Describe a key by ARN through `GET /rustfs/admin/v3/kms/keys/{key_id}` | Works when the ARN is percent-encoded into the path segment; the `describe-key?keyId=` form takes it as a query value. |
-
-### Pairing with a Vault site under site replication
-
-**Delivery requirement.** Site replication synchronizes key names verbatim and never synchronizes keys, so every site must hold a key of the same name ([details](site-replication-operations.md)). AWS assigns key IDs and this backend does not manage aliases, so in a Vault and AWS pair the Vault site must follow the AWS names:
-
-1. Create the key in AWS first and note its key ID.
-2. On each Vault site (`vault-transit` or `vault-kv2`), create a key whose name is that bare key ID, for example `74e5921e-107a-4584-bfda-a5dadeb3e13e`.
-3. Set `RUSTFS_KMS_DEFAULT_KEY_ID` to the same bare key ID on every site, the AWS site included. Do not configure the AWS site's default key as an ARN: the configured spelling is what gets synchronized, and only the bare-ID form was verified to be usable as a Vault key name.
-4. Repeat steps 1 and 2 for every additional key that a bucket default or an IAM policy will name.
-
-Observed with this layout on `vault-transit` + AWS and `vault-kv2` + AWS: bucket defaults naming a key, and `aws:kms` defaults naming none, work for writes on both sites and replicate in both directions, with each replica sealed by the receiving site's own KMS. Without the alignment the receiving site answers `400 KMS.NotFoundException` and replication of the affected objects stays `FAILED`.
-
-### Key state
-
-| AWS key state | `PUT` (encrypt) | `GET` / range / copy source (decrypt) | `HEAD` |
-| --- | --- | --- | --- |
-| Enabled | 200 | 200 | 200 |
-| Disabled | `400 InvalidRequest` (`DisabledException`) | `400 InvalidRequest` (`DisabledException`) | 200 |
-| Pending deletion | `400 InvalidRequest` (`KMSInvalidStateException`) | `400 InvalidRequest` (`KMSInvalidStateException`) | 200 |
-| Deleted | `400 KMS.NotFoundException` | fails; the data is unrecoverable | 200 |
-
-This is the sharpest difference from the RustFS-managed backends, which keep reads working under a disabled key. On AWS, **disabling a key takes every object under it offline until the key is enabled again**; re-enabling restores access with no change on the RustFS side. AWS key state changes can take a few seconds to be visible to all requests.
-
-Cancelling a scheduled deletion leaves the key **Disabled** in AWS. Enable it explicitly before expecting reads or writes to work.
-
-### Operations the AWS backend does not offer
-
-| Operation | Result | Why |
-| --- | --- | --- |
-| Immediate deletion (`force_immediate`) | `501` | AWS only offers a 7–30 day waiting period. |
-| Tag, untag, update description | `501` | Not implemented for this backend; manage tags in AWS. |
-| Bulk rekey / rewrap | refused | AWS key versions are opaque; AWS decrypts with the right backing key automatically. |
-| KMS backup export | not applicable | The key material lives in AWS and cannot be exported. |
-| SSE-S3 key auto-creation, synthetic KMS probe | unavailable | Both address a key by a name RustFS chooses. |
-
-### When AWS misbehaves
-
-| Condition | Client sees | Notes |
-| --- | --- | --- |
-| A single throttled call | 200 | Retried transparently. |
-| Sustained throttling, AWS `5xx`, or the endpoint unreachable | `503 ServiceUnavailable` within a fraction of a second | Retryable. No partial object is left behind by a refused write. |
-| AWS answers `AccessDeniedException` | `403 AccessDenied` | Usually a missing IAM or key-policy grant. |
-| During any of the above | Unencrypted objects and `HEAD` on encrypted objects keep working | Only operations that need the key fail. |
-| After AWS recovers | Encrypted reads and writes resume **without a restart**, about 30 seconds later | The backend circuit breaker stays open for 30 seconds before it probes again. |
-
-### Per-key authorization
-
-RustFS can check `kms:GenerateDataKey` / `kms:Decrypt` against RustFS identity policies per key (`RUSTFS_KMS_ENFORCE_SSE_KEY_POLICY=true`; see [Per-key KMS authorization](kms-per-key-authorization.md)). Write policy resources with the **key ID** (`arn:aws:kms:::key/<key-id>`). A request, or a stored object, may name the key by its ID or by its ARN; both are evaluated as the same key, so an `Allow` or `Deny` on the key ID holds for either spelling. A request that names a key by alias is treated as possibly any key: every key-scoped `Deny` applies to it and no key-scoped `Allow` does, so use the key ID or ARN in requests wherever key-scoped policies are in force. AWS-side controls (key policy, IAM, encryption-context conditions) remain the outer boundary that holds even if a RustFS node is compromised.
-
-## Upgrading from 1.0.4 or 1.0.5
-
-Releases 1.0.4 and 1.0.5 store the AWS data key in a form their own read path does not recognize. On those releases, with the AWS backend:
-
- `PUT` succeeds and `HEAD` reports `aws:kms`, but `GET` fails with `400 InvalidRequest` and a message asking for `RUSTFS_SSE_S3_MASTER_KEY`.
- `UploadPart` of an SSE-KMS multipart upload fails the same way.
- CloudTrail shows `GenerateDataKey` for the write and no `Decrypt` for the read.
-
-Setting `RUSTFS_SSE_S3_MASTER_KEY` does not help and should not be done.
-
-The objects those releases wrote are intact. After upgrading to a fixed release, on the same data directories and with the same key and credentials:
-
- Existing objects read back unchanged, including range reads. No migration, rewrite or re-upload is needed.
- A server-side copy of such an object onto itself rewrites it in the current format; this is optional.
- New objects are written in the current format. **A release older than the fix cannot read them**, so do not roll back after writing with the fixed release; the older releases could not read their own objects either.
-
-Verify an upgrade by reading one object that was written before it and checking for the `Decrypt` event in CloudTrail.
-
-## Operating AWS KMS safely
-
-The key is the data. Every recommendation here follows from that.
-
-### Protect the key from deletion and disablement
-
- **Deleting the KMS key destroys every object encrypted under it.** There is no recovery path in RustFS or in AWS once the waiting period ends. Treat `kms:ScheduleKeyDeletion` as you would treat permission to delete all data.
- Use the longest waiting period (30 days) and alert on the request, not the deletion: create an EventBridge rule for the CloudTrail events `ScheduleKeyDeletion` and `DisableKey` on your keys and route it to the on-call channel. The waiting period only helps if someone notices.
- Deny `kms:ScheduleKeyDeletion` and `kms:DisableKey` for everyone except a small break-glass role, with an organization SCP or an explicit `Deny` in the key policy. The identity RustFS runs as should never hold either.
- Before retiring a key, prove nothing depends on it: disable it first and watch for `400 InvalidRequest` (`DisabledException`) on reads for at least one full business cycle. Disabling is reversible; deletion is not.
- RustFS reports which bucket default-encryption configurations reference a key when you schedule its deletion through the RustFS admin API. That report covers configuration, **not objects**: an empty report does not mean no object uses the key.
-
-### Least privilege and separation of duties
-
- Three roles, three identities: key administrators (manage the key, cannot use it), the RustFS service identity (uses the key, cannot manage it), and auditors (read CloudTrail and key metadata).
- Scope the service identity to the specific key ARNs. Avoid `kms:*` and avoid `Resource: "*"` for anything except `kms:ListKeys`.
- Do not use the account root user or a human administrator's credentials for the service.
- Add an encryption-context condition where a key should serve only certain buckets. RustFS sends the bucket name as an encryption-context key, so `kms:EncryptionContextKeys` can restrict a key to named buckets even if the service identity is compromised. This was not exercised in the test run behind this page; validate it in a non-production account first.
- Keep AWS keys per environment, and consider per tenant or per data class. A separate key gives a separate blast radius and a separate off switch.
-
-### Credentials
-
- Prefer instance, pod or task roles. They rotate automatically and leave no secret on disk.
- If a static access key is unavoidable: one key per deployment, stored with file mode `0600` or in a secret manager, rotated on a schedule, and never placed in RustFS configuration, images or version control.
- Every node needs the same permissions and region. A node with missing permissions fails its own requests while the others succeed, which looks like intermittent errors.
-
-### Rotation
-
- AWS automatic key rotation is transparent to RustFS: AWS keeps every earlier backing key and decrypts old data keys with the right one. RustFS neither enables nor reports on it; turn it on in AWS.
- On-demand rotation through RustFS calls `RotateKeyOnDemand`. AWS limits how many on-demand rotations a key may have and bills beyond a free allowance; check the current AWS limits before automating it.
- Rotation protects future data keys. It does not re-encrypt existing data keys or objects, and RustFS bulk rekey is not available on this backend. If a key is suspected to be compromised, the response is a new key plus rewriting the affected objects, not rotation.
-
-### Availability and disaster recovery
-
- AWS KMS regional availability is now part of your storage availability. During an AWS KMS outage, encrypted reads and writes return `503` and resume about 30 seconds after AWS recovers, without a restart.
- Use a VPC interface endpoint for KMS when RustFS runs in a VPC: it removes the dependency on internet egress and lets an endpoint policy restrict which keys and principals may be used from the network.
- For a disaster-recovery site in another region, AWS multi-Region keys let the replica key decrypt data keys wrapped by the primary. This was not exercised in the test run behind this page; test a full restore in the recovery region before depending on it.
- RustFS replication re-encrypts at the target, so each site uses its own KMS configuration. Losing the source site's key does not affect objects already replicated.
-
-### Monitoring and audit
-
- Enable CloudTrail in every region you use and retain KMS events. Each `GenerateDataKey` and `Decrypt` records the caller and the encryption context, which names the bucket and object.
- Alert on a sustained rise of `AccessDeniedException`, `DisabledException`, `KMSInvalidStateException` and `ThrottlingException` for your keys.
- On the RustFS side, watch the KMS backend metrics and the circuit breaker gauge `rustfs_kms_backend_circuit_open`; see the [KMS observability runbook](kms-observability-runbook.md).
- Request an AWS KMS request-quota increase before a large migration. Symmetric cryptographic operations share one per-region rate quota across the account, and RustFS issues one request per encrypted object operation.
-
-### Cost
-
- Each customer managed key has a fixed monthly charge, and each API request is billed. Encrypted object traffic translates one-to-one into KMS requests: one per write, one per read, one more per multipart part.
- Keys pending deletion are not billed. Keys left behind by tests are.
-
-## Troubleshooting
-
-| Symptom | Likely cause | What to do |
-| --- | --- | --- |
-| `GET` returns `400 InvalidRequest` mentioning `RUSTFS_SSE_S3_MASTER_KEY`, with KMS configured | A release before the AWS read-path fix | Upgrade. Do not set the variable. |
-| RustFS exits at startup with the AWS backend | Health check failed: no credentials, `kms:ListKeys` not granted, wrong region, or no route to the endpoint | Check the startup log for the KMS error, then credentials, the identity policy, and network reachability from that node. |
-| `400 KMS.NotFoundException` on write | Key ID typo, key in another region or account, or the key was deleted | Compare the key ARN with `RUSTFS_KMS_AWS_REGION` and the account. |
-| `403 AccessDenied` on encrypted reads or writes only | The service identity lacks `kms:GenerateDataKey` or `kms:Decrypt` on that key | Fix the key policy or identity policy; no restart needed. |
-| `400 InvalidRequest` with `DisabledException` or `KMSInvalidStateException` | The key is disabled or pending deletion | Enable the key (after cancelling deletion if needed). |
-| `503 ServiceUnavailable` on encrypted operations | AWS KMS unreachable or throttling | Check AWS health and quotas. Recovery is automatic about 30 seconds after AWS responds again. |
-| Encrypted operations fail on one node only | That node's credentials, permissions or network differ | Compare the node's identity and reachability with a working node. |
-| `501 NotImplemented` creating a key by name | AWS assigns key IDs | Create without a name, or create the key in AWS and reference its ID. |
-
-## Related documents
-
- [KMS backend security properties](kms-backend-security.md)
- [KMS admin API contract](kms-admin-contract.md)
- [Per-key KMS authorization](kms-per-key-authorization.md)
- [KMS observability runbook](kms-observability-runbook.md)
@@ -1,281 +0,0 @@
-# AWS KMS 后端：接入、验证与运维
-
-**适用场景：** 首次把 RustFS 接入 AWS KMS、验证已有的 AWS 后端部署、从 1.0.4 或 1.0.5 升级，或者规划如何运维并保护 RustFS 所依赖的 AWS 密钥。
-**事实来源：** `crates/kms/src/backends/aws.rs`（后端行为及其差异）、`crates/kms/src/config.rs`（`ENV_KMS_AWS_*`）、`crates/kms/src/policy.rs`（重试、超时与熔断）、`rustfs/src/storage/sse.rs`（SSE 数据面）。English: [kms-aws-backend-guide.md](kms-aws-backend-guide.md)。
-
-本页关于行为的描述，均来自对真实 AWS KMS 服务（区域 `eu-north-1`）的实测；对于无法让 AWS 按需产生的故障，则使用一个实现了 AWS KMS 线上协议的本机替身来实测。属于未经实测的建议之处，文中会明确标注。AWS 与其它后端在契约层面的差异，见 [KMS backend security properties](kms-backend-security.md#aws-kms-deviations-from-the-shared-backend-contract)。
-
-## RustFS 如何使用 AWS KMS
-
-RustFS 采用信封加密。每个加密对象写入时，RustFS 向 AWS KMS 申请一把数据密钥（`GenerateDataKey`），用明文数据密钥加密对象，只把被包裹的数据密钥存在对象旁边。读取时再把被包裹的密钥发回 AWS（`Decrypt`）。KMS 密钥本身从不离开 AWS，RustFS 也从不保存 AWS 凭据。
-
-由此得出贯穿全文的三点：
-
- **AWS KMS 在读路径上。** 密钥被禁用、被计划删除、被删除，或 AWS 不可达时，加密对象无法读取。RustFS 中存储的任何内容都替代不了这把密钥。
- **每次加密写入、每次加密读取，都是一次 AWS 请求。** 多段上传每上传一段还会多一次 `Decrypt`。请据此估算请求量与费用。
- **`aws:kms` 与 `AES256` 对象都依赖这把密钥。** 配置了 KMS 后端后，SSE-S3（`AES256`）的数据密钥由默认密钥包裹，因此 SSE-S3 对象对 AWS 的依赖与 SSE-KMS 对象完全相同。
-
-## 开始之前
-
-| 项目 | 要求 |
-| --- | --- |
-| RustFS 版本 | 包含 AWS 读路径修复的版本。1.0.4 与 1.0.5 能写不能读 AWS 后端的对象，见[从 1.0.4 或 1.0.5 升级](#从-104-或-105-升级)。 |
-| KMS 密钥 | 对称密钥，用途为"加密和解密"，位于 RustFS 将要配置的区域。 |
-| 网络 | 每个 RustFS 节点都要能通过 HTTPS 访问 `kms.<region>.amazonaws.com`（或 KMS 的 VPC 接口终端节点）。RustFS 内置的 AWS SDK 不使用 `HTTP_PROXY` / `HTTPS_PROXY`。 |
-| 身份 | 每套部署一个 IAM 身份，且每个节点都能取得它的凭据。 |
-| 时钟 | 各节点时钟同步；签名时间偏差超过几分钟的请求会被 AWS 拒绝。 |
-
-## 第一步：创建 KMS 密钥
-
-在 AWS 控制台：**KMS → 客户管理的密钥 → 创建密钥**。
-
-1. 密钥类型选**对称**，密钥用途选**加密和解密**。"高级选项"保持默认（密钥材料来源 KMS、单区域），除非你已决定用多区域密钥做灾备。
-2. 别名与描述：只给人看。RustFS 通过**密钥 ID 或 ARN** 寻址，从不使用别名。
-3. **密钥管理员**：只勾选负责管理这把密钥（禁用、轮换、计划删除）的人员角色。不要勾选 AWS 服务关联角色（`AWSServiceRoleFor…`）和组织托管角色。只有当这些管理员确实应当拥有删除权时，才保留"允许密钥管理员删除此密钥"。
-4. **密钥用户**：勾选 RustFS 运行所用的 IAM 身份（第二步创建，可以回头再添加）。这会通过密钥策略授予该密钥上的 `kms:Encrypt`、`kms:Decrypt`、`kms:ReEncrypt*`、`kms:GenerateDataKey*` 和 `kms:DescribeKey`。
-5. 完成后记下**区域**和**密钥 ARN**。
-
-管理员与用户应当是不同的身份。RustFS 运行所用的身份只需要使用密钥，不需要管理密钥。
-
-## 第二步：创建 RustFS 的运行身份
-
-优先使用短期的角色凭据，而不是长期访问密钥：
-
-| RustFS 运行位置 | 推荐身份 |
-| --- | --- |
-| EC2 | 实例配置文件角色 |
-| EKS | IRSA（服务账号 IAM 角色）或 EKS Pod Identity |
-| ECS | 任务角色 |
-| AWS 之外 | 专用 IAM 用户的访问密钥（定期轮换），或 IAM Roles Anywhere |
-
-RustFS 通过标准的 AWS 凭据链解析凭据：环境变量、共享 profile（`AWS_PROFILE`、`~/.aws/credentials`），然后是容器与实例元数据。RustFS 不会持久化凭据，KMS admin API 也会拒绝配置体中的凭据字段。
-
-为该身份附加下面的身份策略。`kms:ListKeys` 不针对某一把密钥，密钥策略无法授予；而 RustFS 启动时用它做健康检查，缺少它 KMS 后端无法启动。
-
-```json
-{
-  "Version": "2012-10-17",
-  "Statement": [
-    {
-      "Sid": "RustfsKmsHealthCheck",
-      "Effect": "Allow",
-      "Action": "kms:ListKeys",
-      "Resource": "*"
-    }
-  ]
-}
-```
-
-如果不想依赖控制台的"密钥用户"列表，可以把数据面权限写进同一份身份策略，并限定到具体密钥：
-
-```json
-{
-  "Sid": "RustfsKmsDataPath",
-  "Effect": "Allow",
-  "Action": ["kms:GenerateDataKey", "kms:Decrypt", "kms:DescribeKey"],
-  "Resource": "arn:aws:kms:<region>:<account-id>:key/<key-id>"
-}
-```
-
-只有当运维人员要**通过 RustFS 的 admin API** 管理 AWS 密钥时，该身份才需要下面的管理权限。多数部署应当不授予这些权限，直接在 AWS 中管理密钥，这样即使某个 RustFS 节点被攻破，也无法禁用或删除密钥。
-
-| RustFS admin 操作 | AWS 权限 |
-| --- | --- |
-| 创建密钥 | `kms:CreateKey` |
-| 启用 / 禁用 | `kms:EnableKey`、`kms:DisableKey` |
-| 轮换 | `kms:RotateKeyOnDemand` |
-| 计划 / 取消删除 | `kms:ScheduleKeyDeletion`、`kms:CancelKeyDeletion` |
-
-## 第三步：配置 RustFS
-
-在每个节点上设置并重启：
-
-```bash
-RUSTFS_KMS_ENABLE=true
-RUSTFS_KMS_BACKEND=aws
-RUSTFS_KMS_AWS_REGION=eu-north-1
-RUSTFS_KMS_DEFAULT_KEY_ID=arn:aws:kms:eu-north-1:<account-id>:key/<key-id>
-```
-
-说明：
-
- `RUSTFS_KMS_DEFAULT_KEY_ID` 是请求要求加密但未指定密钥时使用的密钥，也是包裹所有 SSE-S3 数据密钥的密钥。密钥 ID 和 ARN 两种写法都可以。
- **不要**设置 `RUSTFS_SSE_S3_MASTER_KEY`。它配置的是无 KMS 时的回退方案，KMS 后端启用后不起任何作用。
- `RUSTFS_KMS_AWS_ENDPOINT_URL` 仅用于模拟器和非默认端点。明文 `http://` 端点会被拒绝，除非设置 `RUSTFS_KMS_ALLOW_INSECURE_DEV_DEFAULTS=true`，而该开关只用于开发环境。使用 VPC 接口终端节点并开启私有 DNS 时，默认的区域域名已经解析到终端节点，无需额外设置。
- 同样的配置也可以在运行时通过 admin API 下发（`POST /rustfs/admin/v3/kms/configure`，请求体 `{"backend_type": "AWS", "region": "...", "default_key_id": "..."}`）。凭据仍然来自各节点自己的环境。
-
-启动是失败关闭的。如果健康检查访问不到 AWS，或端点是明文 HTTP 又没有开发开关，RustFS 进程会在启动阶段直接退出，而不是在没有加密的状态下提供服务。
-
-## 第四步：验证
-
-写入、查看并读回一个对象。任意 S3 客户端均可：
-
-```bash
-aws s3api put-object --endpoint-url https://rustfs.example.com --bucket demo --key kms-check \
-  --body ./sample.bin --server-side-encryption aws:kms
-aws s3api head-object --endpoint-url https://rustfs.example.com --bucket demo --key kms-check
-aws s3api get-object  --endpoint-url https://rustfs.example.com --bucket demo --key kms-check ./readback.bin
-cmp ./sample.bin ./readback.bin
-```
-
-预期：三个调用全部成功，`head-object` 显示 `ServerSideEncryption: aws:kms` 以及你配置的密钥，读回的内容逐字节一致。
-
-然后在 AWS 侧确认这次往返，以证明 RustFS 确实使用了这把密钥，而不是某种本地回退：在 **CloudTrail → 事件历史记录**中按事件源 `kms.amazonaws.com` 筛选。写入会产生 `GenerateDataKey`，读取会产生 `Decrypt`。每个事件都带有 RustFS 发送的加密上下文，其中一项把桶名映射到 `桶名/对象键`，因此 CloudTrail 能看出每次 KMS 调用服务的是哪个对象。**如果读取返回了数据却没有对应的 `Decrypt` 事件，或者报错且没有任何 `Decrypt` 尝试，说明这次读取根本没有到达 AWS。**
-
-检查服务状态：
-
-```bash
-GET /rustfs/admin/v3/kms/status
-```
-
-`backend_status` 应为 `healthy`；`capabilities` 中 `rotate`、`enable_disable`、`schedule_deletion` 为 true，`versioning`、`physical_delete`、`rewrap` 为 false。
-
-## 行为参考
-
-以下为 AWS 后端的实测行为。状态码是 S3 客户端收到的结果。
-
-### 密钥标识
-
-| 情形 | 结果 |
-| --- | --- |
-| 在 `x-amz-server-side-encryption-aws-kms-key-id` 中用 ID 或 ARN 指定密钥 | 均可。响应以及之后的 `HEAD`/`GET` 会原样回显写入时使用的写法。 |
-| 不存在的密钥，或其它区域的 ARN | `400 KMS.NotFoundException`，不会创建对象。 |
-| 通过 RustFS 用自定义名称创建密钥（含兼容 `mc` 的 `create-key?key-id=`） | `501 NotImplemented`。密钥 ID 由 AWS 分配，创建时不要带名称，使用返回的 ID。 |
-| 通过 `GET /rustfs/admin/v3/kms/keys/{key_id}` 用 ARN 描述密钥 | 把 ARN 百分号编码后放入路径段即可；`describe-key?keyId=` 形式则把它作为查询参数。 |
-
-### 站点复制下与 Vault 站点配对
-
-**交付约束**：站点复制原样同步密钥名称，但从不同步密钥本身，所以每个站点都必须有同名密钥（[详见](site-replication-operations.md)）。密钥 ID 由 AWS 分配，本后端也不管理别名，因此 Vault 与 AWS 配对时，必须由 Vault 站点跟随 AWS 的命名：
-
-1. 先在 AWS 创建密钥，记下它的密钥 ID。
-2. 在每个 Vault 站点（`vault-transit` 或 `vault-kv2`）创建一个以该裸密钥 ID 命名的密钥，例如 `74e5921e-107a-4584-bfda-a5dadeb3e13e`。
-3. 所有站点（包括 AWS 站点）的 `RUSTFS_KMS_DEFAULT_KEY_ID` 都设为同一个裸密钥 ID。AWS 站点的默认密钥不要配成 ARN：同步出去的就是配置里的写法，而只有裸 ID 形式经过验证可以用作 Vault 密钥名。
-4. 桶默认加密或 IAM 策略还会点名其它密钥时，对每个密钥重复第 1、2 步。
-
-按此布局在 `vault-transit` + AWS 与 `vault-kv2` + AWS 上实测：点名密钥的桶默认加密，以及不点名密钥的 `aws:kms` 默认加密，在两个站点上都能写入并双向复制，每个副本由接收站点自己的 KMS 封装。未对齐时，接收站点返回 `400 KMS.NotFoundException`，相关对象的复制状态停在 `FAILED`。
-
-### 密钥状态
-
-| AWS 密钥状态 | `PUT`（加密） | `GET` / Range / 拷贝源（解密） | `HEAD` |
-| --- | --- | --- | --- |
-| 已启用 | 200 | 200 | 200 |
-| 已禁用 | `400 InvalidRequest`（`DisabledException`） | `400 InvalidRequest`（`DisabledException`） | 200 |
-| 待删除 | `400 InvalidRequest`（`KMSInvalidStateException`） | `400 InvalidRequest`（`KMSInvalidStateException`） | 200 |
-| 已删除 | `400 KMS.NotFoundException` | 失败，数据无法恢复 | 200 |
-
-这是与 RustFS 自管后端差别最大的一点：自管后端在密钥禁用后仍允许读取。而在 AWS 上，**禁用一把密钥会让它名下的所有对象下线，直到密钥重新启用**；重新启用后访问即恢复，RustFS 一侧不需要任何操作。AWS 的密钥状态变更可能需要几秒才对所有请求生效。
-
-取消计划删除后，密钥在 AWS 中处于**已禁用**状态。必须显式启用，读写才会恢复。
-
-### AWS 后端不提供的操作
-
-| 操作 | 结果 | 原因 |
-| --- | --- | --- |
-| 立即删除（`force_immediate`） | `501` | AWS 只提供 7–30 天的等待期。 |
-| 打标签、去标签、修改描述 | `501` | 该后端未实现，请在 AWS 中管理标签。 |
-| 批量 rekey / rewrap | 拒绝 | AWS 的密钥版本对外不可见，AWS 会自动用正确的底层密钥解密。 |
-| 导出 KMS 备份 | 不适用 | 密钥材料在 AWS 中，无法导出。 |
-| SSE-S3 密钥自动创建、KMS 合成探测 | 不可用 | 两者都要用 RustFS 自选的名称寻址密钥。 |
-
-### AWS 异常时
-
-| 情况 | 客户端看到 | 说明 |
-| --- | --- | --- |
-| 单次被限流 | 200 | 自动重试，对客户端透明。 |
-| 持续限流、AWS 返回 `5xx`，或端点不可达 | 不到一秒内返回 `503 ServiceUnavailable` | 可重试。被拒绝的写入不会留下残缺对象。 |
-| AWS 返回 `AccessDeniedException` | `403 AccessDenied` | 通常是 IAM 或密钥策略缺少授权。 |
-| 上述任一情况期间 | 未加密对象的读写、加密对象的 `HEAD` 不受影响 | 只有需要密钥的操作失败。 |
-| AWS 恢复后 | 加密读写**无需重启**即可恢复，约 30 秒后 | 后端熔断器会保持断开 30 秒，然后才重新探测。 |
-
-### 按密钥授权
-
-RustFS 可以按密钥对 `kms:GenerateDataKey` / `kms:Decrypt` 做身份策略校验（`RUSTFS_KMS_ENFORCE_SSE_KEY_POLICY=true`，见 [Per-key KMS authorization](kms-per-key-authorization.md)）。策略资源请用**密钥 ID** 书写（`arn:aws:kms:::key/<key-id>`）。请求或已存储的对象可以用密钥 ID，也可以用 ARN 来指定密钥，两者都会按同一把密钥来判定，因此针对密钥 ID 的 `Allow` 或 `Deny` 对两种写法都生效。用别名指定密钥的请求会被当作"可能是任何一把密钥"：所有限定到具体密钥的 `Deny` 都对它生效，而限定到具体密钥的 `Allow` 都不生效，所以在使用按密钥策略的环境里，请求中请使用密钥 ID 或 ARN。AWS 侧的控制（密钥策略、IAM、加密上下文条件）仍然是最外层的边界，即使某个 RustFS 节点被攻破也依然有效。
-
-## 从 1.0.4 或 1.0.5 升级
-
-1.0.4 与 1.0.5 存储 AWS 数据密钥的形式，它们自己的读路径无法识别。在这两个版本上使用 AWS 后端时：
-
- `PUT` 成功，`HEAD` 显示 `aws:kms`，但 `GET` 返回 `400 InvalidRequest`，错误信息要求设置 `RUSTFS_SSE_S3_MASTER_KEY`。
- SSE-KMS 多段上传的 `UploadPart` 以同样的方式失败。
- CloudTrail 中写入有 `GenerateDataKey`，读取没有 `Decrypt`。
-
-设置 `RUSTFS_SSE_S3_MASTER_KEY` 没有用，也不应该设置。
-
-这两个版本写入的对象是完好的。升级到已修复的版本后，沿用原来的数据目录、密钥和凭据：
-
- 已有对象可以原样读回，Range 读取同样正确。不需要迁移、重写或重新上传。
- 对这类对象做一次原地的服务端拷贝，会把它改写成当前格式；这一步是可选的。
- 新对象按当前格式写入。**早于修复的版本读不了这些新对象**，所以用修复版本写入之后不要回滚；不过旧版本本来也读不了它们自己写的对象。
-
-验证升级的方法：读取一个升级前写入的对象，并在 CloudTrail 中确认出现了 `Decrypt` 事件。
-
-## 安全地运维 AWS KMS
-
-密钥就是数据。下面每一条建议都由此而来。
-
-### 防止密钥被删除或禁用
-
- **删除 KMS 密钥，等于销毁它加密的所有对象。** 等待期结束后，RustFS 和 AWS 都没有任何恢复手段。对待 `kms:ScheduleKeyDeletion` 权限，应当与对待"删除全部数据"的权限同等严格。
- 使用最长的等待期（30 天），并对"发起请求"而不是"完成删除"告警：为你的密钥上的 CloudTrail 事件 `ScheduleKeyDeletion` 和 `DisableKey` 建立 EventBridge 规则，推送到值班渠道。等待期只有在有人注意到的情况下才有意义。
- 用组织 SCP 或密钥策略中的显式 `Deny`，对除少数应急角色以外的所有人拒绝 `kms:ScheduleKeyDeletion` 和 `kms:DisableKey`。RustFS 的运行身份永远不应持有这两个权限。
- 退役一把密钥之前，先证明没有任何东西依赖它：先禁用，并观察至少一个完整业务周期内是否出现读取报 `400 InvalidRequest`（`DisabledException`）。禁用可以撤销，删除不能。
- 通过 RustFS admin API 计划删除密钥时，RustFS 会报告哪些桶的默认加密配置引用了它。这份报告覆盖的是配置，**不是对象**：报告为空并不代表没有对象在使用这把密钥。
-
-### 最小权限与职责分离
-
- 三种角色，三个身份：密钥管理员（管理密钥，但不能使用）、RustFS 服务身份（使用密钥，但不能管理）、审计员（读取 CloudTrail 与密钥元数据）。
- 把服务身份限定到具体的密钥 ARN。不要使用 `kms:*`；除 `kms:ListKeys` 之外，不要使用 `Resource: "*"`。
- 不要把账户根用户或某位管理员的个人凭据用于服务。
- 如果一把密钥只应服务于某些桶，可以加上加密上下文条件。RustFS 把桶名作为加密上下文的键发送，因此可以用 `kms:EncryptionContextKeys` 把密钥限定到指定的桶，即使服务身份泄露也有效。这一项不在本页所依据的测试范围内，请先在非生产账户中验证。
- 按环境区分 AWS 密钥，并考虑按租户或数据等级区分。独立的密钥意味着独立的影响范围和独立的"总开关"。
-
-### 凭据
-
- 优先使用实例、Pod 或任务角色。它们自动轮换，不在磁盘上留下任何秘密。
- 如果只能使用静态访问密钥：每套部署一把，以 `0600` 权限的文件或密钥管理服务保存，定期轮换，绝不写进 RustFS 配置、镜像或版本库。
- 每个节点都需要相同的权限和区域。某个节点权限不全时，它自己的请求会失败而其它节点成功，表现为间歇性错误。
-
-### 轮换
-
- AWS 的自动密钥轮换对 RustFS 透明：AWS 保留所有历史底层密钥，并自动用正确的那一把解密旧的数据密钥。RustFS 既不会开启它，也不会报告它的状态，请在 AWS 中开启。
- 通过 RustFS 做按需轮换时调用的是 `RotateKeyOnDemand`。AWS 对每把密钥的按需轮换次数有上限，超出免费额度后还会计费；自动化之前请先核对 AWS 当前的限制。
- 轮换保护的是今后的数据密钥，不会重新加密已有的数据密钥或对象，而 RustFS 的批量 rekey 在该后端不可用。如果怀疑密钥泄露，应对方式是换一把新密钥并重写受影响的对象，而不是轮换。
-
-### 可用性与灾备
-
- AWS KMS 的区域可用性从此成为你的存储可用性的一部分。AWS KMS 故障期间，加密读写返回 `503`，AWS 恢复约 30 秒后自动恢复，无需重启。
- RustFS 运行在 VPC 中时，为 KMS 使用 VPC 接口终端节点：这样不再依赖公网出口，并且可以用终端节点策略限制从该网络能使用哪些密钥和主体。
- 对于另一个区域的灾备站点，AWS 多区域密钥允许副本密钥解密由主密钥包裹的数据密钥。这一项不在本页所依据的测试范围内，依赖它之前请在灾备区域完整演练一次恢复。
- RustFS 复制会在目标端重新加密，因此每个站点使用各自的 KMS 配置。源站点的密钥丢失不影响已经复制过去的对象。
-
-### 监控与审计
-
- 在所有用到的区域开启 CloudTrail 并保留 KMS 事件。每条 `GenerateDataKey` 和 `Decrypt` 都记录了调用者和加密上下文，后者指明了桶和对象。
- 对你的密钥上 `AccessDeniedException`、`DisabledException`、`KMSInvalidStateException`、`ThrottlingException` 的持续上升告警。
- RustFS 一侧关注 KMS 后端指标和熔断器指标 `rustfs_kms_backend_circuit_open`，见 [KMS observability runbook](kms-observability-runbook.md)。
- 大规模迁移之前，先申请提高 AWS KMS 的请求配额。对称加解密操作在每个区域共用账户级的速率配额，而 RustFS 每个加密对象操作都会发出一次请求。
-
-### 费用
-
- 每把客户管理的密钥有固定月费，每次 API 请求另外计费。加密对象的流量会一比一地变成 KMS 请求：每次写入一次、每次读取一次，多段上传每段再加一次。
- 处于待删除状态的密钥不计费。测试遗留的密钥会继续计费。
-
-## 排障
-
-| 现象 | 可能原因 | 处理 |
-| --- | --- | --- |
-| 已配置 KMS，`GET` 却返回 `400 InvalidRequest` 并提到 `RUSTFS_SSE_S3_MASTER_KEY` | 版本早于 AWS 读路径修复 | 升级。不要设置该变量。 |
-| 使用 AWS 后端时 RustFS 启动即退出 | 健康检查失败：没有凭据、未授予 `kms:ListKeys`、区域不对，或到端点的网络不通 | 查看启动日志中的 KMS 错误，再依次核对凭据、身份策略、该节点的网络可达性。 |
-| 写入返回 `400 KMS.NotFoundException` | 密钥 ID 写错、密钥在其它区域或账户，或密钥已被删除 | 把密钥 ARN 与 `RUSTFS_KMS_AWS_REGION` 及账户对照。 |
-| 只有加密读写返回 `403 AccessDenied` | 服务身份在该密钥上缺少 `kms:GenerateDataKey` 或 `kms:Decrypt` | 修正密钥策略或身份策略，无需重启。 |
-| `400 InvalidRequest` 并带有 `DisabledException` 或 `KMSInvalidStateException` | 密钥被禁用或处于待删除状态 | 启用密钥（如有需要先取消删除）。 |
-| 加密操作返回 `503 ServiceUnavailable` | AWS KMS 不可达或被限流 | 检查 AWS 健康状态与配额。AWS 恢复响应约 30 秒后自动恢复。 |
-| 只有某一个节点上的加密操作失败 | 该节点的凭据、权限或网络与其它节点不同 | 把该节点的身份和可达性与正常节点对比。 |
-| 按名称创建密钥返回 `501 NotImplemented` | 密钥 ID 由 AWS 分配 | 创建时不带名称，或在 AWS 中创建后引用其 ID。 |
-
-## 相关文档
-
- [KMS backend security properties](kms-backend-security.md)
- [KMS admin API contract](kms-admin-contract.md)
- [Per-key KMS authorization](kms-per-key-authorization.md)
- [KMS observability runbook](kms-observability-runbook.md)
@@ -262,10 +262,6 @@ The AWS backend is exempt from `backends::contract_tests::assert_state_machine_c

 Key versions are opaque: RustFS reports `key_version` as 1 and cannot enumerate versions. Rotation uses `RotateKeyOnDemand`, which retains prior backing keys for decryption; AWS's automatic yearly rotation is neither enabled nor reported on by RustFS.

-For the step-by-step AWS setup, verification, observed status codes and key-protection practices, see the [AWS KMS backend guide](kms-aws-backend-guide.md).
-
-Stored data keys: the `GenerateDataKey` blob is persisted inside a `DataKeyEnvelope`, in `encrypted_key`, like every other backend's wrapped key. Releases through 1.0.5 persisted the blob with no envelope and then could not read those objects back, because the read path recognizes a KMS-wrapped key by its envelope. Such objects are intact: while the AWS backend is active, a stored data key that is neither an envelope nor a local-master-key wrap is sent to AWS `Decrypt` as it is. No rewrite is needed. The encryption context AWS authenticates is always the one rebuilt for the request, never the unauthenticated copy recorded in the envelope.
-
 The KMS admin API accepts the backend as `"backend_type": "AWS"` (aliases `aws`, `aws-kms`, `aws_kms`, `AwsKms`) on `/v3/kms/configure` and `/v3/kms/reconfigure`. The body carries `region` (**required**), and optionally `endpoint_url`, `default_key_id`, and the shared timeout/retry/cache settings; credential fields are rejected as unknown, because every node resolves credentials through its own provider chain. `region` is mandatory even though `RUSTFS_KMS_AWS_REGION` is optional at startup: the admin configuration is replayed on every node, and a request that left the region to each node's ambient chain would let nodes address different regions — and therefore different keys — under an identical configuration. `default_key_id` must be an AWS key id or ARN that already exists.

 ## Vault TLS: custom CA and mutual TLS
@@ -14,7 +14,7 @@ KMS resources use the same empty-account ARN form as S3 resources:
 | `arn:aws:kms:::key/<key_id>` | exactly that key |
 | `arn:aws:kms:::key/app-*` | every key whose id starts with `app-` |
 | `arn:aws:kms:::*` | every key |
-| `arn:aws:kms:::alias/<name>` | reserved; matches nothing, including a request that names that alias |
+| `arn:aws:kms:::alias/<name>` | reserved; matches nothing until alias resolution ships |

 Rules worth knowing before writing a policy:

@@ -24,8 +24,7 @@ Rules worth knowing before writing a policy:
 - A KMS statement whose resources are S3 ARNs predates KMS resources. It is still loadable, but the S3 patterns never constrained key access, so evaluation ignores them, treats the statement as unscoped, and logs a warning. Rewrite these with real KMS resources.
 - `PutBucketPolicy` rejects any statement carrying a `kms:` action or a KMS resource. KMS grants belong to identities. Bucket policies stored before this check still load; evaluation skips their pure-KMS statements and warns.
 - `Deny` wins, as everywhere else: a `Deny` on `arn:aws:kms:::key/payroll-*` overrides an `Allow` on `arn:aws:kms:::*`.
- Write a key pattern with the **key id**. A request may name an AWS key by its id or by its ARN (`arn:<partition>:kms:<region>:<account>:key/<key_id>`); both are matched as that key id, so one statement covers either spelling. An ARN cannot be written as a pattern, because a key pattern may not contain `/`.
- A request that names a key by **alias** (`alias/<name>` or an alias ARN) names a key only the backend can resolve, so it fails closed against key-scoped statements: every `Deny` that carries a KMS resource applies to it, and only an `Allow` on every key (`arn:aws:kms:::*`, or a statement with no resource) covers it. Name keys by id or ARN wherever a key-scoped policy is in force.
+- The key identifier matched is the one the request names, before any alias resolution.

 ## Built-in role templates

@@ -166,7 +166,7 @@ impl Operation for ChangeOwnPasswordHandler {
            .credentials()
            .ok_or_else(|| s3::error(S3ErrorCode::AccessDenied, "authentication required"))?
            .require_secret_key()
-            .map_err(|err| s3::error(S3ErrorCode::InternalError, err.to_string()))?
+            .map_err(|_| s3::error(S3ErrorCode::AccessDenied, "the request credential carries no secret key"))?
            .to_string();
        let caller = CallerIdentity::resolve_envelope(req.envelope()).await?;
        let audit = AccountAuditContext::from_envelope(req.envelope());
@@ -541,6 +541,23 @@ fn validate_new_secret_key(request: &ChangePasswordRequest) -> S3Result<()> {
 mod tests {
    use super::*;
    use crate::server::ADMIN_PREFIX;
+
+    #[tokio::test]
+    async fn change_own_password_without_a_secret_key_is_access_denied() {
+        let req = crate::admin::operation::test_support::TestRequest::new(
+            Method::POST,
+            format!("{ADMIN_PREFIX}/v3/change-own-password")
+                .parse::<http::Uri>()
+                .expect("uri"),
+        )
+        .credentials_without_secret("account-user", None)
+        .build();
+        let err = ChangeOwnPasswordHandler {}
+            .call(req, Params::new())
+            .await
+            .expect_err("a credential without a secret key cannot rotate its password");
+        assert_eq!(err.code(), &S3ErrorCode::AccessDenied);
+    }
    use rustfs_madmin::account::{AccountMutability, CredentialsSource};

    #[test]
@@ -61,12 +61,11 @@ pub fn register_bucket_recovery_route(router: &mut S3Router<AdminOperation>) ->
 impl Operation for RecoverOrphanedBucketHandler {
    async fn call(&self, mut req: AdminRequest, params: Params<'_, '_>) -> S3Result<AdminResponse> {
        authorize_admin_envelope(req.envelope(), vec![Action::AdminAction(AdminAction::RecoverOrphanedBucketAction)]).await?;
-        let secret_key = req
+        let credentials = req
            .credentials()
-            .ok_or_else(|| S3Error::with_message(S3ErrorCode::InvalidRequest, "authentication required"))?
-            .require_secret_key()
-            .map(|secret| Zeroizing::new(secret.to_owned()))
-            .map_err(|err| S3Error::with_message(S3ErrorCode::InternalError, err.to_string()))?;
+            .ok_or_else(|| S3Error::with_message(S3ErrorCode::InvalidRequest, "authentication required"))?;
+        // Checked before the recovery runs, so a refusal leaves the bucket untouched.
+        let response_secret = compat_response_secret(req.uri().path(), credentials.secret_key())?;
        let bucket = params
            .get("bucket")
            .filter(|bucket| !bucket.is_empty())
@@ -125,22 +124,66 @@ impl Operation for RecoverOrphanedBucketHandler {
            "orphaned bucket metadata recovery completed"
        );

-        super::owned_admin_json_response(
-            req.uri().path(),
-            &secret_key,
-            StatusCode::OK,
-            &RecoverOrphanedBucketResponse {
-                bucket: bucket.to_owned(),
-                retired_incarnation_id: request.expected_incarnation_id,
-            },
-        )
+        let response = RecoverOrphanedBucketResponse {
+            bucket: bucket.to_owned(),
+            retired_incarnation_id: request.expected_incarnation_id,
+        };
+        match response_secret {
+            Some(secret_key) => super::owned_admin_json_response(req.uri().path(), &secret_key, StatusCode::OK, &response),
+            None => crate::admin::utils::owned_json_response(StatusCode::OK, &response),
+        }
    }
 }

+/// The key the response body is sealed with. Only the MinIO compat prefix
+/// encrypts admin responses, so only that path needs the caller's secret.
+///
+/// # Errors
+///
+/// `AccessDenied` on the compat prefix when the verified credential carries no
+/// secret key.
+fn compat_response_secret(path: &str, secret: Option<&str>) -> S3Result<Option<Zeroizing<String>>> {
+    if !crate::admin::utils::is_compat_admin_request(path) {
+        return Ok(None);
+    }
+    secret
+        .map(|secret| Some(Zeroizing::new(secret.to_owned())))
+        .ok_or_else(|| S3Error::with_message(S3ErrorCode::AccessDenied, "the request credential carries no secret key"))
+}
+
 #[cfg(test)]
 mod tests {
    use super::*;

+    const OWNED_PATH: &str = "/rustfs/admin/v3/recover-orphaned-bucket/orphan";
+    const COMPAT_PATH: &str = "/minio/admin/v3/recover-orphaned-bucket/orphan";
+
+    #[test]
+    fn compat_recovery_without_a_secret_key_is_refused_before_it_runs() {
+        let err = compat_response_secret(COMPAT_PATH, None).expect_err("compat response needs a sealing key");
+        assert_eq!(err.code(), &S3ErrorCode::AccessDenied);
+    }
+
+    #[test]
+    fn owned_recovery_without_a_secret_key_proceeds_unsealed() {
+        assert!(
+            compat_response_secret(OWNED_PATH, None)
+                .expect("owned path needs no key")
+                .is_none()
+        );
+        assert!(
+            compat_response_secret(OWNED_PATH, Some("secret"))
+                .expect("owned path")
+                .is_none()
+        );
+    }
+
+    #[test]
+    fn compat_recovery_with_a_secret_key_seals_with_it() {
+        let secret = compat_response_secret(COMPAT_PATH, Some("secret")).expect("compat path with key");
+        assert_eq!(secret.as_deref().map(String::as_str), Some("secret"));
+    }
+
    #[test]
    fn recovery_request_rejects_missing_or_nil_generation_identity() {
        assert!(serde_json::from_slice::<RecoverOrphanedBucketRequest>(b"{}").is_err());
@@ -27,7 +27,6 @@ use crate::server::ADMIN_PREFIX;
 use http::header::CONTENT_TYPE;
 use hyper::{HeaderMap, Method, StatusCode};
 use matchit::Params;
-use percent_encoding::percent_decode_str;
 use rustfs_config::MAX_ADMIN_REQUEST_BODY_SIZE;
 use rustfs_kms::{KeyImpactReport, KmsAuditOperation, KmsError, types::*};
 use rustfs_policy::policy::action::{Action, KmsAction};
@@ -838,49 +837,8 @@ mod tests {
    }

    fn describe_by_path(key_id: &str) -> String {
-        path_key_id_at(&format!("/rustfs/admin/v3/kms/keys/{key_id}"))
-            .expect("path key id should decode")
-            .unwrap_or_default()
-    }
-
-    /// The key id `DescribeKmsKeyHandler` sees for a request path, through the
-    /// same route pattern the handler is registered under.
-    fn path_key_id_at(path: &str) -> rustfs_s3_types::S3Result<Option<String>> {
-        let mut router = matchit::Router::new();
-        router
-            .insert("/rustfs/admin/v3/kms/keys/{key_id}", ())
-            .expect("route should register");
-        let matched = router.at(path).expect("path should match the describe route");
-        super::path_key_id(&matched.params)
-    }
-
-    /// An AWS key ARN only fits in one path segment escaped. Read verbatim it
-    /// was looked up under its escaped spelling and reported as not found, so
-    /// a key could be described by ARN through the query alias but not through
-    /// the documented path route.
-    #[test]
-    fn path_key_id_is_percent_decoded() {
-        let arn = "arn:aws:kms:eu-north-1:111122223333:key/74e5921e-107a-4584-bfda-a5dadeb3e13e";
-        let escaped = "arn%3Aaws%3Akms%3Aeu-north-1%3A111122223333%3Akey%2F74e5921e-107a-4584-bfda-a5dadeb3e13e";
-        assert_eq!(
-            path_key_id_at(&format!("/rustfs/admin/v3/kms/keys/{escaped}"))
-                .expect("an escaped ARN is a valid key id")
-                .as_deref(),
-            Some(arn)
-        );
-        assert_eq!(
-            path_key_id_at("/rustfs/admin/v3/kms/keys/plain-key")
-                .expect("a plain id is a valid key id")
-                .as_deref(),
-            Some("plain-key")
-        );
-        assert_eq!(
-            path_key_id_at("/rustfs/admin/v3/kms/keys/has%20space")
-                .expect("an escaped space is a valid key id")
-                .as_deref(),
-            Some("has space")
-        );
-        path_key_id_at("/rustfs/admin/v3/kms/keys/%FF").expect_err("bytes that are not UTF-8 must be refused, not looked up");
+        // `DescribeKmsKeyHandler` reads the router path parameter verbatim.
+        key_id.to_string()
    }

    fn generate_data_key_body(key_id: &str) -> String {
@@ -2448,24 +2406,6 @@ async fn key_impact_if_requested(requested: bool, key_id: &str, default_key_id:
    Some(current_key_impact(key_id, default_key_id).await)
 }

-/// The key id named by the `{key_id}` path segment, percent-decoded.
-///
-/// A route parameter arrives exactly as the client sent it, so an id that has
-/// to be escaped to fit in one path segment — an AWS key ARN, with its `:` and
-/// `/` — would otherwise be looked up, authorized and audited under its
-/// escaped spelling and never found.
-fn path_key_id(params: &Params<'_, '_>) -> S3Result<Option<String>> {
-    params
-        .get("key_id")
-        .map(|raw| {
-            percent_decode_str(raw)
-                .decode_utf8()
-                .map(|decoded| decoded.into_owned())
-                .map_err(|_| s3_error!(InvalidArgument, "invalid key id path parameter"))
-        })
-        .transpose()
-}
-
 /// Describe a KMS key
 pub struct DescribeKmsKeyHandler;

@@ -2480,7 +2420,6 @@ impl Operation for DescribeKmsKeyHandler {
            check_key_valid(get_session_token(req.uri(), req.headers()).unwrap_or_default(), cred.access_key()).await?;

        let audit = KmsAdminAudit::from_envelope(req.envelope(), &cred);
-        let path_key_id = path_key_id(&params)?;

        audit.gate(
            validate_admin_request_with_kms_key(
@@ -2490,14 +2429,14 @@ impl Operation for DescribeKmsKeyHandler {
                false,
                kms_describe_key_actions(),
                req.policy_source_addr(),
-                path_key_id.as_deref().unwrap_or_default(),
+                params.get("key_id").unwrap_or_default(),
            )
            .await,
            KmsAuditOperation::DescribeKey,
-            path_key_id.as_deref(),
+            params.get("key_id"),
        )?;

-        let Some(key_id) = path_key_id.as_deref() else {
+        let Some(key_id) = params.get("key_id") else {
            let response = DescribeKmsKeyResponse {
                success: false,
                message: "missing required parameter: 'keyId'".to_string(),
@@ -223,6 +223,22 @@ fn map_add_user_create_error(err: rustfs_iam::error::Error) -> S3Error {
    }
 }

+/// The caller's current secret when the request rotates the caller's own key,
+/// `None` when it creates or updates another user (which does not read it).
+///
+/// # Errors
+///
+/// `AccessDenied` for a self-rotation whose verified credential carries no
+/// secret key.
+fn self_rotation_secret(check_deny_only: bool, secret: Option<&str>) -> S3Result<Option<&str>> {
+    if !check_deny_only {
+        return Ok(None);
+    }
+    secret
+        .map(Some)
+        .ok_or_else(|| S3Error::with_message(S3ErrorCode::AccessDenied, "the request credential carries no secret key"))
+}
+
 #[async_trait::async_trait]
 impl Operation for AddUser {
    async fn call(&self, mut req: AdminRequest, _params: Params<'_, '_>) -> S3Result<AdminResponse> {
@@ -242,11 +258,9 @@ impl Operation for AddUser {

        let (cred, owner) =
            check_key_valid(get_session_token(req.uri(), req.headers()).unwrap_or_default(), input_cred.access_key()).await?;
-        // The caller's current secret outlives the body read below.
-        let input_secret = input_cred
-            .require_secret_key()
-            .map_err(|err| S3Error::with_message(S3ErrorCode::InternalError, err.to_string()))?
-            .to_owned();
+        // The caller's current secret outlives the body read below. Only a
+        // self-rotation reads it; creating another user does not.
+        let input_secret = input_cred.secret_key().map(str::to_owned);

        let ak = query.access_key.as_deref().unwrap_or_default();

@@ -327,9 +341,10 @@ impl Operation for AddUser {
        )
        .await?;

-        let updated_at = if check_deny_only {
+        let rotation_secret = self_rotation_secret(check_deny_only, input_secret.as_deref())?;
+        let updated_at = if let Some(current_secret) = rotation_secret {
            let (updated_at, status, _) = iam_store
-                .change_own_secret_key(ak, &input_secret, &args.secret_key)
+                .change_own_secret_key(ak, current_secret, &args.secret_key)
                .await
                .map_err(map_add_user_create_error)?;
            args.status = status;
@@ -1396,6 +1411,23 @@ mod tests {
    use serde_json::Value;
    use std::collections::HashMap;

+    #[test]
+    fn self_rotation_without_a_secret_key_is_refused() {
+        let err = super::self_rotation_secret(true, None).expect_err("self-rotation needs the current secret");
+        assert_eq!(err.code(), &S3ErrorCode::AccessDenied);
+    }
+
+    #[test]
+    fn creating_another_user_does_not_read_the_secret_key() {
+        assert_eq!(super::self_rotation_secret(false, None).expect("create path proceeds"), None);
+        assert_eq!(super::self_rotation_secret(false, Some("current")).expect("create path"), None);
+    }
+
+    #[test]
+    fn self_rotation_uses_the_presented_secret_key() {
+        assert_eq!(super::self_rotation_secret(true, Some("current")).expect("rotation"), Some("current"));
+    }
+
    fn credential_less_request(method: Method, uri: &'static str) -> AdminRequest {
        crate::admin::operation::test_support::TestRequest::new(method, Uri::from_static(uri))
            .headers(HeaderMap::new())
@@ -217,6 +217,15 @@ pub(crate) mod test_support {
            self
        }

+        /// A presented identity whose verified credential carries no secret key.
+        pub(crate) fn credentials_without_secret(mut self, access_key: &str, session_token: Option<&str>) -> Self {
+            self.credentials = Some(RequestCredentials::without_secret_key(
+                access_key.to_owned(),
+                session_token.map(str::to_owned),
+            ));
+            self
+        }
+
        pub(crate) fn extensions(mut self, extensions: Extensions) -> Self {
            self.extensions = extensions;
            self
@@ -33,7 +33,6 @@ use crate::storage_api::kms::{ECStore, StorageError, get_bucket_sse_config};
 use async_trait::async_trait;
 use rustfs_gateway_types::persistence::PersistedBucketEncryptionConfiguration as ServerSideEncryptionConfiguration;
 use rustfs_kms::{DeletionReferenceChecker, KeyImpactReport, KeyReference, KeyReferenceKind};
-use rustfs_policy::policy::statement::RequestedKmsKey;
 use std::sync::Arc;
 use tracing::warn;

@@ -72,7 +71,7 @@ pub(crate) async fn current_key_impact(key_id: &str, default_key_id: Option<&str
 async fn collect_key_impact(key_id: &str, default_key_id: Option<&str>, store: Option<Arc<ECStore>>) -> KeyImpactReport {
    let mut report = KeyImpactReport::configuration_layer(key_id);

-    if default_key_id.is_some_and(|default_key_id| names_same_key(default_key_id, key_id)) {
+    if default_key_id == Some(key_id) {
        report.push_reference(KeyReference {
            kind: KeyReferenceKind::ServiceDefaultKey,
            id: key_id.to_string(),
@@ -168,18 +167,10 @@ fn sse_config_references_key(config: &ServerSideEncryptionConfiguration, key_id:
        rule.apply_server_side_encryption_by_default
            .as_ref()
            .and_then(|sse| sse.kms_master_key_id.as_deref())
-            .is_some_and(|configured| names_same_key(configured, key_id))
+            .is_some_and(|configured| configured == key_id)
    })
 }

-/// Whether a configured key identifier and the one being deleted name the same
-/// key. An AWS key answers to its id and to its ARN, so the two can differ in
-/// spelling while a deletion of one still takes the other away. An alias cannot
-/// be resolved here and is compared literally.
-fn names_same_key(configured: &str, key_id: &str) -> bool {
-    configured == key_id || RequestedKmsKey::same_key(configured, key_id)
-}
-
 #[cfg(test)]
 mod tests {
    use super::*;
@@ -201,29 +192,6 @@ mod tests {
        }
    }

-    /// A bucket or the service default can name an AWS key by ARN while the
-    /// deletion names it by id, or the other way round. Compared literally,
-    /// the report said nothing referenced a key that was in fact the default.
-    #[test]
-    fn references_are_found_across_spellings_of_one_aws_key() {
-        let key_id = "74e5921e-107a-4584-bfda-a5dadeb3e13e";
-        let arn = "arn:aws:kms:eu-north-1:111122223333:key/74e5921e-107a-4584-bfda-a5dadeb3e13e";
-        let other = "arn:aws:kms:eu-north-1:111122223333:key/00000000-0000-4000-8000-000000000000";
-
-        for (configured, deleted) in [(arn, key_id), (key_id, arn), (arn, arn), (key_id, key_id)] {
-            assert!(names_same_key(configured, deleted), "{configured} and {deleted} name one key");
-            assert!(
-                bucket_reference("sse-bucket", Ok(sse_kms_config(Some(configured))), deleted).is_some(),
-                "a bucket configured with {configured} references {deleted}"
-            );
-        }
-        assert!(!names_same_key(other, key_id));
-        assert!(bucket_reference("sse-bucket", Ok(sse_kms_config(Some(other))), key_id).is_none());
-        // An alias is only ever equal to its own spelling.
-        assert!(names_same_key("alias/payroll", "alias/payroll"));
-        assert!(!names_same_key("alias/payroll", key_id));
-    }
-
    #[test]
    fn referencing_bucket_blocks_deletion() {
        let reference = bucket_reference("sse-bucket", Ok(sse_kms_config(Some("kms-key-1"))), "kms-key-1")
@@ -1449,8 +1449,8 @@ fn record_managed_kms_outcome(
 /// Master-key version recorded in a managed-SSE data-key envelope, if the
 /// wrapping backend recorded one.
 ///
-/// `None` is the honest answer for every other shape: Transit and AWS keep
-/// the version inside their own opaque ciphertext, Local records no version,
+/// `None` is the honest answer for every other shape: Transit and AWS wrap
+/// into opaque ciphertext that is not an envelope, Local records no version,
 /// and pre-versioning envelopes never carried the field. The single field is
 /// read through `serde_json::Value` rather than a full `DataKeyEnvelope`
 /// parse, so the audit path cannot double-count the envelope's unknown-field
@@ -3211,15 +3211,21 @@ async fn apply_managed_decryption_material_inner(
    // Persisted wrapping format is the read-side source of truth. The
    // advertised SSE scheme and current KMS availability are write policy
    // and runtime state, neither of which identifies the historical provider.
-    //
-    // The one shape that cannot name its provider is a backend-native blob
-    // stored without an envelope. It is offered to the KMS only when the
-    // local provider could not have written those bytes and the active
-    // backend is one whose released versions persisted them that way.
-    let bare_kms = if is_data_key_envelope(&encrypted_data_key) || is_local_sse_dek(&encrypted_data_key) {
-        None
+    let provider: Arc<dyn SseDekProvider> = if is_data_key_envelope(&encrypted_data_key) {
+        // When a test-injected provider is registered via set_sse_dek_provider_for_test,
+        // use it instead of creating a fresh KmsSseDekProvider which cannot resolve
+        // a KMS service without an AppContext.
+        //
+        // Reads GLOBAL_KMS_DEK_PROVIDER (populated only by set_sse_dek_provider_for_test),
+        // never GLOBAL_SSE_DEK_PROVIDER, so a local provider cached by a prior
+        // get_local_sse_dek_provider call cannot be selected to unwrap a KMS envelope.
+        if let Some(cached) = GLOBAL_KMS_DEK_PROVIDER.read().ok().and_then(|guard| guard.as_ref().cloned()) {
+            cached
+        } else {
+            Arc::new(KmsSseDekProvider::new().await?)
+        }
    } else {
-        bare_data_key_kms_provider().await
+        get_local_sse_dek_provider().await?
    };
    // A MinIO sealed key alone does not mean MinIO wrote the object: RustFS's own
    // writer fills MinIO's metadata slots too, while still storing a RustFS
@@ -3229,50 +3235,24 @@ async fn apply_managed_decryption_material_inner(
    // recognizing RustFS positively — and treating only the remainder as MinIO —
    // keeps a RustFS envelope from ever reaching MinIO's decoder.
    #[cfg(feature = "rio-v2")]
-    let unenveloped_sealed_dek = if minio_sealed_key.is_some() && !is_rustfs_managed_data_key(&encrypted_data_key) {
-        Some(
-            decrypt_unenveloped_sealed_dek(
-                bare_kms.clone(),
-                get_local_sse_dek_provider().await,
-                &encrypted_data_key,
-                &kms_key_id,
-                &object_context,
-            )
-            .await?,
-        )
+    let decrypted_data_key = if minio_sealed_key.is_some() && !is_rustfs_managed_data_key(&encrypted_data_key) {
+        provider
+            .decrypt_minio_sse_dek(&encrypted_data_key, &kms_key_id, &object_context)
+            .await
+    } else if is_legacy_rustfs_managed_metadata(&normalized_metadata) {
+        provider
+            .decrypt_legacy_sse_dek(&encrypted_data_key, &kms_key_id, &object_context)
+            .await
    } else {
-        None
+        provider
+            .decrypt_sse_dek(&encrypted_data_key, &kms_key_id, &object_context)
+            .await
    };
    #[cfg(not(feature = "rio-v2"))]
-    let unenveloped_sealed_dek: Option<[u8; 32]> = None;
-
-    let decrypted_data_key = match unenveloped_sealed_dek {
-        Some(data_key) => data_key,
-        None => {
-            let provider: Arc<dyn SseDekProvider> = if is_data_key_envelope(&encrypted_data_key) {
-                kms_sse_dek_provider().await?
-            } else if let Some(kms_provider) = bare_kms {
-                kms_provider
-            } else {
-                get_local_sse_dek_provider().await?
-            };
-            #[cfg(feature = "rio-v2")]
-            let decrypted_data_key = if is_legacy_rustfs_managed_metadata(&normalized_metadata) {
-                provider
-                    .decrypt_legacy_sse_dek(&encrypted_data_key, &kms_key_id, &object_context)
-                    .await
-            } else {
-                provider
-                    .decrypt_sse_dek(&encrypted_data_key, &kms_key_id, &object_context)
-                    .await
-            };
-            #[cfg(not(feature = "rio-v2"))]
-            let decrypted_data_key = provider
-                .decrypt_sse_dek(&encrypted_data_key, &kms_key_id, &object_context)
-                .await;
-            decrypted_data_key?
-        }
-    };
+    let decrypted_data_key = provider
+        .decrypt_sse_dek(&encrypted_data_key, &kms_key_id, &object_context)
+        .await;
+    let decrypted_data_key = decrypted_data_key?;
    #[cfg(feature = "rio-v2")]
    let (key_bytes, base_nonce, key_kind) = if let Some(sealed_key) = minio_sealed_key {
        (
@@ -3411,8 +3391,14 @@ pub(crate) async fn rewrap_object_encryption_metadata(
    };
    let object_context = build_object_encryption_context(bucket, key, kms_context.as_ref());

-    // The same provider selection as the managed decrypt path.
-    let provider = kms_sse_dek_provider().await?;
+    // The same provider selection as the managed decrypt path: the
+    // test-injected provider when registered, the KMS-backed one otherwise.
+    let provider: Arc<dyn SseDekProvider> =
+        if let Some(cached) = GLOBAL_KMS_DEK_PROVIDER.read().ok().and_then(|guard| guard.as_ref().cloned()) {
+            cached
+        } else {
+            Arc::new(KmsSseDekProvider::new().await?)
+        };
    let response = provider.rewrap_sse_dek(&encrypted_data_key, &object_context).await?;
    if !response.rewrapped {
        return Ok(ObjectDekRewrapOutcome::AlreadyCurrent);
@@ -3472,12 +3458,6 @@ pub trait SseDekProvider: Send + Sync {
        context: &ObjectEncryptionContext,
    ) -> Result<[u8; 32], ApiError>;

-    /// Whether this provider may own a stored data key that is not a RustFS
-    /// envelope — see `KmsBackend::reads_bare_data_key_ciphertext`.
-    async fn reads_bare_data_key_ciphertext(&self) -> bool {
-        false
-    }
-
    /// Re-wrap a KMS-wrapped DEK envelope onto its master key's current
    /// version without exposing the plaintext DEK to the caller.
    ///
@@ -3636,12 +3616,6 @@ impl SseDekProvider for KmsSseDekProvider {
        Ok((data_key, encrypted_data_key))
    }

-    async fn reads_bare_data_key_ciphertext(&self) -> bool {
-        self.current_service()
-            .await
-            .is_some_and(|service| service.reads_bare_data_key_ciphertext())
-    }
-
    async fn rewrap_sse_dek(
        &self,
        encrypted_dek: &[u8],
@@ -3881,76 +3855,6 @@ struct LocalSseDekEnvelope<'a> {
    ciphertext: &'a str,
 }

-/// Whether [`LocalSseDekProvider`] could have written these bytes.
-///
-/// Covers both shapes [`LocalSseDekProvider::decrypt_dek`] accepts — anything
-/// presented as a JSON object, so a damaged local envelope keeps its own
-/// diagnosis, and the deprecated `nonce:ciphertext` base64 pair. Everything
-/// else is bytes the local provider can only reject.
-fn is_local_sse_dek(encrypted_dek: &[u8]) -> bool {
-    let Ok(text) = std::str::from_utf8(encrypted_dek) else {
-        return false;
-    };
-    if text.trim_start().starts_with('{') {
-        return true;
-    }
-    let is_base64 = |part: &str| BASE64_STANDARD.decode_to_vec(part).is_ok();
-    text.split_once(':')
-        .is_some_and(|(nonce, ciphertext)| is_base64(nonce) && is_base64(ciphertext))
-}
-
-/// Unwrap the data key of an object that carries a sealed object key but no
-/// RustFS envelope.
-///
-/// Two writers produce that shape, and nothing in the bytes tells them apart:
-/// MinIO, whose data key is sealed under the static master key, and a RustFS
-/// `rio-v2` build on a backend that persisted its native ciphertext bare.
-/// MinIO's sealing is authenticated, so opening it under the local master key
-/// is a proof rather than a guess: the bare blob is offered to the KMS only
-/// once that has failed, or when no local master key exists to try. Without a
-/// backend that reads bare ciphertext the MinIO route is the only one, exactly
-/// as before.
-#[cfg(feature = "rio-v2")]
-async fn decrypt_unenveloped_sealed_dek(
-    bare_kms: Option<Arc<dyn SseDekProvider>>,
-    local: Result<Arc<dyn SseDekProvider>, ApiError>,
-    encrypted_dek: &[u8],
-    kms_key_id: &str,
-    context: &ObjectEncryptionContext,
-) -> Result<[u8; 32], ApiError> {
-    let Some(kms) = bare_kms else {
-        return local?.decrypt_minio_sse_dek(encrypted_dek, kms_key_id, context).await;
-    };
-    if let Ok(local) = local
-        && let Ok(data_key) = local.decrypt_minio_sse_dek(encrypted_dek, kms_key_id, context).await
-    {
-        return Ok(data_key);
-    }
-    kms.decrypt_sse_dek(encrypted_dek, kms_key_id, context).await
-}
-
-/// The provider that unwraps KMS-wrapped data keys.
-///
-/// A provider registered through `set_sse_dek_provider_for_test` stands in for
-/// the fresh [`KmsSseDekProvider`], which cannot resolve a KMS service without
-/// an `AppContext`. Only `GLOBAL_KMS_DEK_PROVIDER` is consulted, never
-/// `GLOBAL_SSE_DEK_PROVIDER`, so a local provider cached by an earlier
-/// `get_local_sse_dek_provider` call cannot be selected to unwrap a KMS envelope.
-async fn kms_sse_dek_provider() -> Result<Arc<dyn SseDekProvider>, ApiError> {
-    if let Some(cached) = GLOBAL_KMS_DEK_PROVIDER.read().ok().and_then(|guard| guard.as_ref().cloned()) {
-        return Ok(cached);
-    }
-    Ok(Arc::new(KmsSseDekProvider::new().await?))
-}
-
-/// The KMS provider, when its backend may own a stored data key that is not a
-/// RustFS envelope. `None` without a running KMS, which leaves such a key to
-/// the local provider exactly as before.
-async fn bare_data_key_kms_provider() -> Option<Arc<dyn SseDekProvider>> {
-    let provider = kms_sse_dek_provider().await.ok()?;
-    provider.reads_bare_data_key_ciphertext().await.then_some(provider)
-}
-
 /// Test-only alias so existing test code that references `TestSseDekProvider`
 /// continues to compile without changes.
 #[cfg(test)]
@@ -7607,480 +7511,6 @@ pub(crate) mod tests {
        reset_sse_dek_provider();
    }

-    /// A loopback stand-in for AWS KMS, so the real AWS backend — SDK client,
-    /// service manager and all — sits behind the SSE entry points under test.
-    ///
-    /// It issues data keys the way AWS does: a binary blob that is not UTF-8
-    /// and carries nothing RustFS could recognize, decryptable only with the
-    /// encryption context it was generated under.
-    struct FakeAwsKms {
-        endpoint: String,
-        state: Arc<FakeAwsKmsState>,
-        server: tokio::task::JoinHandle<()>,
-    }
-
-    /// Plaintext data key and the encryption context it was generated under.
-    type FakeAwsDataKey = ([u8; 32], serde_json::Value);
-
-    #[derive(Default)]
-    struct FakeAwsKmsState {
-        data_keys: std::sync::Mutex<HashMap<Vec<u8>, FakeAwsDataKey>>,
-        decrypt_calls: std::sync::atomic::AtomicUsize,
-    }
-
-    impl Drop for FakeAwsKms {
-        fn drop(&mut self) {
-            self.server.abort();
-        }
-    }
-
-    impl FakeAwsKms {
-        const KEY_ARN: &'static str = "arn:aws:kms:us-east-1:111122223333:key/test-key";
-
-        async fn start() -> Self {
-            let state = Arc::new(FakeAwsKmsState::default());
-            let listener = tokio::net::TcpListener::bind("127.0.0.1:0")
-                .await
-                .expect("bind fake AWS KMS listener");
-            let endpoint = format!("http://{}", listener.local_addr().expect("fake AWS KMS address"));
-            let app = axum::Router::new().fallback(Self::handle).with_state(state.clone());
-            let server = tokio::spawn(async move {
-                axum::serve(listener, app).await.expect("serve fake AWS KMS");
-            });
-            Self { endpoint, state, server }
-        }
-
-        fn decrypt_calls(&self) -> usize {
-            self.state.decrypt_calls.load(std::sync::atomic::Ordering::SeqCst)
-        }
-
-        async fn handle(
-            axum::extract::State(state): axum::extract::State<Arc<FakeAwsKmsState>>,
-            headers: axum::http::HeaderMap,
-            body: axum::body::Bytes,
-        ) -> axum::response::Response {
-            let operation = headers
-                .get("x-amz-target")
-                .and_then(|target| target.to_str().ok())
-                .and_then(|target| target.rsplit('.').next())
-                .unwrap_or_default();
-            let request: serde_json::Value = serde_json::from_slice(&body).unwrap_or_default();
-            let context = request
-                .get("EncryptionContext")
-                .cloned()
-                .unwrap_or_else(|| serde_json::json!({}));
-
-            let reply = match operation {
-                "ListKeys" => Ok(serde_json::json!({ "Keys": [], "Truncated": false })),
-                "DescribeKey" => Ok(serde_json::json!({
-                    "KeyMetadata": {
-                        "KeyId": "test-key",
-                        "Arn": Self::KEY_ARN,
-                        "CreationDate": 1_700_000_000,
-                        "Enabled": true,
-                        "KeyUsage": "ENCRYPT_DECRYPT",
-                        "KeyState": "Enabled",
-                        "Origin": "AWS_KMS",
-                        "KeyManager": "CUSTOMER",
-                    }
-                })),
-                "GenerateDataKey" => {
-                    let mut plaintext = [0u8; 32];
-                    rand::Rng::fill_bytes(&mut rand::rng(), &mut plaintext);
-                    // A continuation byte straight after ASCII is never valid
-                    // UTF-8, like the real blob in practice.
-                    let mut blob = vec![0x01, 0x02, 0x02, 0x00, 0x78];
-                    let mut tail = [0u8; 48];
-                    rand::Rng::fill_bytes(&mut rand::rng(), &mut tail);
-                    blob.extend(tail.map(|byte| 0x80 | (byte & 0x3f)));
-                    state
-                        .data_keys
-                        .lock()
-                        .expect("fake AWS KMS state")
-                        .insert(blob.clone(), (plaintext, context));
-                    Ok(serde_json::json!({
-                        "KeyId": Self::KEY_ARN,
-                        "Plaintext": BASE64_STANDARD.encode_to_string(plaintext),
-                        "CiphertextBlob": BASE64_STANDARD.encode_to_string(&blob),
-                    }))
-                }
-                "Decrypt" => {
-                    state.decrypt_calls.fetch_add(1, std::sync::atomic::Ordering::SeqCst);
-                    let blob = request
-                        .get("CiphertextBlob")
-                        .and_then(serde_json::Value::as_str)
-                        .and_then(|blob| BASE64_STANDARD.decode_to_vec(blob).ok())
-                        .unwrap_or_default();
-                    match state.data_keys.lock().expect("fake AWS KMS state").get(&blob) {
-                        Some((plaintext, sealed_context)) if *sealed_context == context => Ok(serde_json::json!({
-                            "KeyId": Self::KEY_ARN,
-                            "Plaintext": BASE64_STANDARD.encode_to_string(plaintext),
-                            "EncryptionAlgorithm": "SYMMETRIC_DEFAULT",
-                        })),
-                        _ => Err("InvalidCiphertextException"),
-                    }
-                }
-                _ => Err("UnsupportedOperationException"),
-            };
-
-            let (status, body) = match reply {
-                Ok(body) => (axum::http::StatusCode::OK, body),
-                Err(code) => (
-                    axum::http::StatusCode::BAD_REQUEST,
-                    serde_json::json!({ "__type": code, "message": code }),
-                ),
-            };
-            axum::response::Response::builder()
-                .status(status)
-                .header("content-type", "application/x-amz-json-1.1")
-                .body(axum::body::Body::from(body.to_string()))
-                .expect("fake AWS KMS response")
-        }
-    }
-
-    /// Run `test` with the global KMS service on the AWS backend, pointed at a
-    /// [`FakeAwsKms`], and no SSE provider injected — the wiring a deployment
-    /// with `RUSTFS_KMS_BACKEND=aws` and no local master key has.
-    async fn with_aws_kms_backend<F, Fut>(test: F)
-    where
-        F: FnOnce(Arc<FakeAwsKms>) -> Fut,
-        Fut: std::future::Future<Output = ()>,
-    {
-        let _guard = lock_sse_test_state().await;
-        reset_sse_dek_provider();
-        let fake = Arc::new(FakeAwsKms::start().await);
-
-        async_with_vars(
-            [
-                ("AWS_ACCESS_KEY_ID", Some("AKIDTEST")),
-                ("AWS_SECRET_ACCESS_KEY", Some("secret")),
-                ("AWS_SESSION_TOKEN", None),
-                ("AWS_PROFILE", None),
-                ("AWS_EC2_METADATA_DISABLED", Some("true")),
-                ("HTTP_PROXY", None),
-                ("HTTPS_PROXY", None),
-                ("http_proxy", None),
-                ("https_proxy", None),
-                ("RUSTFS_SSE_S3_MASTER_KEY", None),
-                ("__RUSTFS_SSE_SIMPLE_CMK", None),
-            ],
-            async {
-                let mut config = rustfs_kms::KmsConfig::aws(Some("us-east-1".to_string())).with_insecure_development_defaults();
-                let rustfs_kms::BackendConfig::Aws(aws) = &mut config.backend_config else {
-                    panic!("the AWS constructor must create an AWS backend configuration");
-                };
-                aws.endpoint_url = Some(fake.endpoint.clone());
-                let manager = rustfs_kms::init_global_kms_service_manager();
-                manager.reconfigure(config).await.expect("start the AWS KMS backend");
-                let provider = KmsSseDekProvider::new_with_service_manager(manager.clone())
-                    .await
-                    .expect("kms provider should initialize from the AWS-backed manager");
-                super::set_sse_dek_provider_for_test(Arc::new(provider));
-
-                test(fake.clone()).await;
-
-                manager.stop().await.expect("stop the AWS KMS backend");
-            },
-        )
-        .await;
-
-        reset_sse_dek_provider();
-    }
-
-    fn aws_sse_kms_write() -> EncryptionRequest<'static> {
-        EncryptionRequest {
-            bucket: "bucket",
-            key: "dir/object",
-            server_side_encryption: Some(ServerSideEncryption::from_static(ServerSideEncryption::AWS_KMS)),
-            ssekms_key_id: Some("test-key".to_string()),
-            ssekms_context: None,
-            sse_customer_algorithm: None,
-            sse_customer_key: None,
-            sse_customer_key_md5: None,
-            content_size: 4096,
-            principal: None,
-        }
-    }
-
-    async fn read_managed_object(metadata: &HashMap<String, String>) -> Result<super::DecryptionMaterial, ApiError> {
-        sse_decryption(DecryptionRequest {
-            bucket: "bucket",
-            key: "dir/object",
-            metadata,
-            sse_customer_key: None,
-            sse_customer_key_md5: None,
-            principal: None,
-        })
-        .await
-        .map(|material| material.expect("an SSE-KMS object must yield decryption material"))
-    }
-
-    /// An SSE-KMS object written through the AWS backend must be readable.
-    ///
-    /// It was not: the AWS data key reached storage as a bare blob, the read
-    /// path took the unrecognized bytes for a locally wrapped key, and every
-    /// GET failed asking for `RUSTFS_SSE_S3_MASTER_KEY` without AWS ever being
-    /// asked to decrypt.
-    #[tokio::test]
-    async fn sse_kms_object_written_through_the_aws_backend_reads_back() {
-        with_aws_kms_backend(|aws| async move {
-            let material = sse_encryption(aws_sse_kms_write())
-                .await
-                .expect("sse-kms encryption")
-                .expect("managed sse-kms material");
-            let metadata = encryption_material_to_metadata(&material).expect("kms metadata should serialize");
-            // The rio-v2 writer keeps the wrapped data key in MinIO's slot only.
-            let stored_key = metadata
-                .get(INTERNAL_ENCRYPTION_KEY_HEADER)
-                .or_else(|| metadata.get(MINIO_INTERNAL_ENCRYPTION_KMS_DATA_KEY_HEADER))
-                .expect("stored data key");
-            let stored_key = BASE64_STANDARD.decode_to_vec(stored_key).expect("stored data key is base64");
-            assert!(
-                rustfs_kms::is_data_key_envelope(&stored_key),
-                "the stored data key must identify itself as KMS-wrapped"
-            );
-
-            let decrypted = read_managed_object(&metadata)
-                .await
-                .expect("an object written through the AWS backend must decrypt");
-
-            assert_eq!(decrypted.key_bytes, material.key_bytes);
-            assert_eq!(decrypted.sse_type, SSEType::SseKms);
-            assert_eq!(aws.decrypt_calls(), 1, "the read must be served by AWS Decrypt");
-        })
-        .await;
-    }
-
-    /// Objects written by releases through 1.0.5 keep the AWS blob bare in
-    /// their metadata. They are intact and must decrypt as they are, without
-    /// being rewritten. A `rio-v2` build stored the same bare blob beside a
-    /// sealed object key, which is the shape MinIO's own objects have, so the
-    /// route has to hold there too.
-    #[tokio::test]
-    async fn sse_kms_object_with_a_bare_aws_data_key_reads_back() {
-        with_aws_kms_backend(|aws| async move {
-            let material = sse_encryption(aws_sse_kms_write())
-                .await
-                .expect("sse-kms encryption")
-                .expect("managed sse-kms material");
-            let mut metadata = encryption_material_to_metadata(&material).expect("kms metadata should serialize");
-
-            // Rebuild what those releases persisted: the same headers, with
-            // the blob AWS issued in place of the envelope around it.
-            let envelope_b64 = metadata
-                .get(INTERNAL_ENCRYPTION_KEY_HEADER)
-                .or_else(|| metadata.get(MINIO_INTERNAL_ENCRYPTION_KMS_DATA_KEY_HEADER))
-                .expect("stored data key")
-                .clone();
-            let envelope: serde_json::Value = serde_json::from_slice(
-                &BASE64_STANDARD
-                    .decode_to_vec(&envelope_b64)
-                    .expect("stored data key is base64"),
-            )
-            .expect("stored data key is an envelope");
-            let bare: Vec<u8> = serde_json::from_value(envelope["encrypted_key"].clone()).expect("envelope carries the AWS blob");
-            let bare_b64 = BASE64_STANDARD.encode_to_string(&bare);
-            let mut replaced = 0;
-            for value in metadata.values_mut().filter(|value| **value == envelope_b64) {
-                *value = bare_b64.clone();
-                replaced += 1;
-            }
-            assert!(replaced > 0, "the stored data key must have been replaced");
-            assert!(!rustfs_kms::is_data_key_envelope(&bare));
-
-            let decrypted = read_managed_object(&metadata)
-                .await
-                .expect("an object carrying a bare AWS data key must decrypt");
-
-            assert_eq!(decrypted.key_bytes, material.key_bytes);
-            assert_eq!(aws.decrypt_calls(), 1, "the read must be served by AWS Decrypt");
-        })
-        .await;
-    }
-
-    /// The bare-blob route exists for AWS alone. No other backend claims an
-    /// unrecognized data key, and without a KMS it still belongs to the local
-    /// provider, which is the one that can say what is wrong with it.
-    #[tokio::test]
-    async fn bare_data_key_stays_with_the_local_provider_without_an_aws_backend() {
-        let _guard = lock_sse_test_state().await;
-        reset_sse_dek_provider();
-        let local_kms = KmsSseDekProvider::new_with_service_manager(configure_test_global_local_kms().await)
-            .await
-            .expect("kms provider should initialize from the Local-backed manager");
-        assert!(!local_kms.reads_bare_data_key_ciphertext().await);
-        assert!(
-            !TestSseDekProvider::new_with_key([9u8; 32])
-                .reads_bare_data_key_ciphertext()
-                .await
-        );
-
-        async_with_vars([("RUSTFS_SSE_S3_MASTER_KEY", None::<&str>), ("__RUSTFS_SSE_SIMPLE_CMK", None)], async {
-            let metadata = HashMap::from([
-                ("x-amz-server-side-encryption".to_string(), ServerSideEncryption::AWS_KMS.to_string()),
-                (
-                    INTERNAL_ENCRYPTION_KEY_HEADER.to_string(),
-                    BASE64_STANDARD.encode_to_string([0x01, 0x02, 0x02, 0x00, 0x78, 0x80, 0x81]),
-                ),
-                (INTERNAL_ENCRYPTION_IV_HEADER.to_string(), BASE64_STANDARD.encode_to_string([0x14; 12])),
-                (INTERNAL_ENCRYPTION_KEY_ID_HEADER.to_string(), "test-key".to_string()),
-            ]);
-
-            let error = read_managed_object(&metadata)
-                .await
-                .expect_err("a bare blob has no owner without a KMS");
-
-            assert!(
-                error.message.contains("RUSTFS_SSE_S3_MASTER_KEY"),
-                "the local provider must still own the diagnosis: {}",
-                error.message
-            );
-        })
-        .await;
-
-        reset_sse_dek_provider();
-    }
-
-    /// A KMS stand-in that owns bare data keys and records what it is asked to unwrap.
-    #[cfg(feature = "rio-v2")]
-    struct BareKmsProvider {
-        data_key: [u8; 32],
-        unwrapped: std::sync::Mutex<Vec<Vec<u8>>>,
-    }
-
-    #[cfg(feature = "rio-v2")]
-    #[async_trait::async_trait]
-    impl SseDekProvider for BareKmsProvider {
-        async fn generate_sse_dek(
-            &self,
-            _context: &ObjectEncryptionContext,
-            _kms_key_id: &str,
-        ) -> Result<(DataKey, Vec<u8>), ApiError> {
-            Err(ApiError::from(StorageError::other("not exercised")))
-        }
-
-        async fn decrypt_sse_dek(
-            &self,
-            encrypted_dek: &[u8],
-            _kms_key_id: &str,
-            _context: &ObjectEncryptionContext,
-        ) -> Result<[u8; 32], ApiError> {
-            self.unwrapped.lock().expect("unwrap log").push(encrypted_dek.to_vec());
-            Ok(self.data_key)
-        }
-
-        async fn reads_bare_data_key_ciphertext(&self) -> bool {
-            true
-        }
-    }
-
-    /// Seal a data key the way MinIO's builtin KMS does, in its raw layout:
-    /// `sealed || iv[16] || nonce[12]`, AES-256-GCM under `HMAC-SHA256(master, iv)`.
-    #[cfg(feature = "rio-v2")]
-    fn seal_minio_kms_data_key(data_key: &[u8; 32], master_key: &[u8; 32], context: &ObjectEncryptionContext) -> Vec<u8> {
-        use aes_gcm::{Aes256Gcm, KeyInit, aead::Aead};
-        use hmac::Mac;
-        let (iv, nonce) = ([0x21u8; 16], [0x42u8; 12]);
-        let mut mac = super::HmacSha256::new_from_slice(master_key).expect("hmac key");
-        mac.update(&iv);
-        let sealing_key: [u8; 32] = mac.finalize().into_bytes().into();
-        let aad = super::minio_kms_associated_data(context).expect("associated data");
-        let mut sealed = Aes256Gcm::new_from_slice(&sealing_key)
-            .expect("sealing key")
-            .encrypt(
-                &aes_gcm::Nonce::from(nonce),
-                aes_gcm::aead::Payload {
-                    msg: data_key,
-                    aad: &aad,
-                },
-            )
-            .expect("seal");
-        sealed.extend_from_slice(&iv);
-        sealed.extend_from_slice(&nonce);
-        sealed
-    }
-
-    /// A sealed object key with no RustFS envelope comes from MinIO or from a
-    /// `rio-v2` build on a backend that stored its ciphertext bare. Opening it
-    /// as MinIO's is an authenticated proof, so the genuine MinIO object keeps
-    /// its route and only the remainder reaches the KMS.
-    #[cfg(feature = "rio-v2")]
-    #[tokio::test]
-    async fn unenveloped_sealed_data_key_goes_to_minio_first_and_the_kms_otherwise() {
-        let context = ObjectEncryptionContext::new("bucket".to_string(), "dir/object".to_string());
-        let master_key = [0x5Au8; 32];
-        let minio_data_key = [0x11u8; 32];
-        let minio_ciphertext = seal_minio_kms_data_key(&minio_data_key, &master_key, &context);
-        let aws_blob = vec![
-            0x01, 0x02, 0x02, 0x00, 0x78, 0x80, 0x81, 0x82, 0x83, 0x84, 0x85, 0x86, 0x87, 0x88, 0x89,
-        ];
-        let kms = || {
-            Arc::new(BareKmsProvider {
-                data_key: [0x22u8; 32],
-                unwrapped: std::sync::Mutex::new(Vec::new()),
-            })
-        };
-        let local =
-            || -> Result<Arc<dyn SseDekProvider>, ApiError> { Ok(Arc::new(TestSseDekProvider::new_with_key(master_key))) };
-        let no_local = || -> Result<Arc<dyn SseDekProvider>, ApiError> {
-            Err(ApiError {
-                code: S3ErrorCode::InvalidRequest,
-                message: "no local master key".to_string(),
-                source: None,
-            })
-        };
-
-        // MinIO's own object on a deployment whose backend also reads bare keys.
-        let provider = kms();
-        let data_key = super::decrypt_unenveloped_sealed_dek(Some(provider.clone()), local(), &minio_ciphertext, "key", &context)
-            .await
-            .expect("a MinIO-sealed data key must open under the local master key");
-        assert_eq!(data_key, minio_data_key);
-        assert!(
-            provider.unwrapped.lock().expect("unwrap log").is_empty(),
-            "a MinIO-sealed data key must never be sent to the KMS"
-        );
-
-        // A bare backend blob: with a local master key that cannot open it, and with none at all.
-        for local in [local(), no_local()] {
-            let provider = kms();
-            let data_key = super::decrypt_unenveloped_sealed_dek(Some(provider.clone()), local, &aws_blob, "key", &context)
-                .await
-                .expect("a bare backend blob must be unwrapped by the KMS");
-            assert_eq!(data_key, [0x22u8; 32]);
-            assert_eq!(*provider.unwrapped.lock().expect("unwrap log"), vec![aws_blob.clone()]);
-        }
-
-        // Without a backend that reads bare keys, the MinIO route is the only one.
-        let data_key = super::decrypt_unenveloped_sealed_dek(None, local(), &minio_ciphertext, "key", &context)
-            .await
-            .expect("the MinIO route is unchanged");
-        assert_eq!(data_key, minio_data_key);
-        super::decrypt_unenveloped_sealed_dek(None, local(), &aws_blob, "key", &context)
-            .await
-            .expect_err("bytes MinIO did not seal have no owner without such a backend");
-        let error = super::decrypt_unenveloped_sealed_dek(None, no_local(), &minio_ciphertext, "key", &context)
-            .await
-            .expect_err("a missing local master key is still reported as such");
-        assert_eq!((error.code, error.message.as_str()), (S3ErrorCode::InvalidRequest, "no local master key"));
-    }
-
-    #[test]
-    fn local_sse_dek_recognition_covers_exactly_the_local_formats() {
-        let wrapped = TestSseDekProvider::encrypt_dek([7u8; 32], [9u8; 32]).expect("local wrap");
-        assert!(super::is_local_sse_dek(wrapped.as_bytes()));
-        assert!(super::is_local_sse_dek(b"bm9uY2U=:Y2lwaGVydGV4dA=="));
-        // A damaged local envelope is still the local provider's to diagnose.
-        assert!(super::is_local_sse_dek(br#"  {"version": 9"#));
-
-        // An AWS blob: binary, and not UTF-8.
-        assert!(!super::is_local_sse_dek(&[0x01, 0x02, 0x02, 0x00, 0x78, 0x80, 0x81]));
-        // Text that merely contains a colon is not the legacy pair.
-        assert!(!super::is_local_sse_dek(b"vault:v1:Y2lwaGVydGV4dA=="));
-        assert!(!super::is_local_sse_dek(b"not base64:Y2lwaGVydGV4dA=="));
-        assert!(!super::is_local_sse_dek(b""));
-    }
-
    /// Regression test for "local provider cached → dynamically enable KMS → decrypt KMS envelope".
    ///
    /// Verifies that a `TestSseDekProvider` previously cached in `GLOBAL_SSE_DEK_PROVIDER`