ci(parity): run dual-stack differential against pinned release (#9315 )

* feat(server): add the RUSTFS_S3_STACK selector and dual-stack e2e lane (#8411) * refactor(server): group the s3s-compat tower layers under legacy_compat (#8410) Move the ten tower layers that patch s3s behaviour out of rustfs/src/server/layer.rs into rustfs/src/server/legacy_compat/, one file per layer with its tests. Each file header names the s3s behaviour it patches, the gateway mechanism that replaces it (Replaced by:), and the task that removes it (Removed by: T2.12, rustfs/backlog#2771). The layers keep their code, their order in external_service_stack!, and their behaviour; no layer is deleted. Add server::http::tests::stack_census, which renders the ordered service chain of the real external_service_stack! expansion and compares it with a committed constant, so a reorder, insertion or removal is a visible diff. Route the two s3s error types the moved layers use through a storage_api::server::legacy_compat shim so the split adds no s3s-importing file, and lower the s3s footprint baselines to the measured values (files 209 to 208, s3_error! lines 1586 to 1585). Keep the server runtime-source bypass guard covering the moved code. Refs rustfs/backlog#2787, rustfs/backlog#2734 * feat(s3-types): add the RustFS-owned S3 error type and s3_error! macro (#8406) `rustfs-s3-types` now provides `S3ErrorCode` (generated by build.rs from src/codes.txt, the grep-derived call surface, plus `Custom`), `S3Error`, `S3Result`, `StdError` and the `s3_error` macro with the same call syntax as s3s, so the T1.2 codemod is a `use` swap. Statuses are hand-written and pinned to the s3s behaviour RustFS ships today by an oracle test that uses s3s as a dev-dependency only. The `From` conversions for the legacy edge live behind the off-by-default `compat-s3s` feature (the orphan rule forbids them in rustfs/src/error.rs); `rustfs` enables it. Refs rustfs/backlog#2742, rustfs/backlog#2734 * ci: add the per-crate test-count ratchet (#8404) Quick Checks now fails any PR that lowers the number of test attributes in a workspace crate or adds #[ignore], measured per crate against the exact rows of .config/test-count-baseline.txt (rustfs/backlog#2734 section 1 rule 3). The counter widens the spec's literal #[(tokio::)?test] to the tokio attributes that carry arguments and both spellings of #[test_case(...)], and anchors at line start so commented-out attributes do not count. The initial baseline covers 52 crates, 21077 tests and 164 ignored. Refs rustfs/backlog#2738 Refs rustfs/backlog#2734 * test(ecstore): pin the persisted bucket-config XML bytes for 13 families (#8408) Before any DTO or codec is replaced, pin the exact bytes that deserialize followed by serialize produces for every persisted bucket-configuration XML family in its three source shapes (AWS documentation, MinIO-written, RustFS-written), and the bytes RustFS writes today for a typed value. Byte equality only: a structural comparison would let a codec replacement change the persisted encoding unnoticed. The generator script refuses to overwrite existing goldens without --force, refuses on dirty codec paths, cannot report success from its write mode, and records the generating commit in the fixtures README. Refs rustfs/backlog#2744, rustfs/backlog#2734 * ci: raise the test-count baseline to the integration tree (#8426) * ci: make the s3s footprint baseline exact and lower-only (#8403) * ci: make the s3s footprint baseline exact and lower-only The s3s footprint guard compared three counters against ceilings held in the script itself, so a ceiling could sit above the tree and a drift was fixable by raising it (rustfs/rustfs#5964 raised the s3_error! ceiling to go green). The three values now live in .config/s3s-footprint-baseline.txt as files, s3_error_lines and ecstore_files, the script reads only that file, and a measurement that differs from its baseline in either direction fails: growth prints "s3s footprint grew: <measured> > <baseline>", shrinkage prints "lower the baseline to <measured> in this PR". The counting method is unchanged; the baseline is the current measurement of gateway/integration (209 / 1585 / 36). The header records the end state: once all three reach 0, T4.2 switches the script to failing on any occurrence. Refs rustfs/backlog#2739, rustfs/backlog#2734 * ci: add the s3s edge allowlist mode to the footprint guard (#8405) `scripts/check_s3s_footprint.sh --mode allowlist` fails on every Rust file that references s3s (`s3s::` or `s3s_sigv4`) and is not matched by an entry of the new `.config/s3s-edge-allowlist.txt`; `--dry-run` lists those files and exits 0. The allowlist carries the initial edge from the spec, and the script rejects an entry that names or matches anything under crates/ other than crates/e2e_test. The baseline mode and the argument-less CI invocation are unchanged. Refs rustfs/backlog#2741, rustfs/backlog#2734 * test(ecstore): reach the bucket-config codec through the test facade (#8428) * refactor(s3-types): own the HTTP Body and drop s3s from three crates (#8431) Add rustfs_s3_types::Body, a BoxBody<Bytes, StdError> wrapper with empty() and From<Bytes | Vec<u8> | String | &'static str>, and the legacy From conversions to and from s3s::Body behind compat-s3s. The signer is now generic over the request body: it only reads and writes the request head, so it needs no body type and its callers keep theirs. s3-client sends its requests with the owned Body, madmin signs a bodiless head, and targets drops its two S3Request/S3Response header wrappers in favour of extract_params_header. rustfs-signer, rustfs-madmin and rustfs-targets no longer depend on s3s; the s3s footprint baseline drops from 209 to 202 files. Refs rustfs/backlog#2747, rustfs/backlog#2734 * refactor(ecstore): move S3ErrorCode to rustfs-s3-types by codemod (#8432) * test(ecstore): pin the InvalidRange responses of new_getobjectreader Both error branches of new_getobjectreader build an ErrorResponse with S3ErrorCode::InvalidRange, and no test observed either code. Pin the code, its wire name and the location fields of each branch before the code type moves to rustfs-s3-types (rustfs/backlog#2743), so a changed code on either branch fails a test. The ecstore test-count row rises by the two tests. * chore(scripts): add the s3s-to-rustfs-s3-types error codemod Task T1.2 of rustfs/backlog#2734 (rustfs/backlog#2743) moves S3Error, S3ErrorCode, S3Result and s3_error! from s3s to rustfs-s3-types by script, one crate closure per PR. The codemod rewrites use trees (splitting mixed s3s imports) and qualified code paths, never comments or literals, adds the rustfs-s3-types dependency, drops s3s only from a crate that no longer names it, refuses every file of the s3s edge allowlist, takes --hold for a file whose type is fixed outside the closure, and prints a per-crate census. A second run changes nothing. Its self-test runs under make script-tests. * refactor(ecstore): take S3ErrorCode from rustfs-s3-types Output of scripts/codemods/s3_error_to_s3_types.py over crates/ecstore, crates/lifecycle, crates/scanner and crates/replication (rustfs/backlog#2743 batches 1 and 2). The ErrorResponse code field, the two InvalidRange constructions in new_getobjectreader and an unused tier import now use rustfs_s3_types::S3ErrorCode; the variant names and wire names are unchanged. lifecycle, scanner and replication reference none of the four error names. warm_backend.rs and warm_backend_s3.rs are held: they compare against rustfs_s3_client::ErrorResponse.code, which stays the s3s type until s3-client moves. * ci(footprint): count only s3_error! lines still on s3s's macro s3_error_lines counted every line holding the text `s3_error!`, which also matches the RustFS-owned `rustfs_s3_types::s3_error!`. A codemod that moves call sites onto rustfs-s3-types therefore could not lower it (rustfs/backlog#2743, decision 3; rustfs/rustfs#8432, gotcha 2). The counter now applies a textual, per-file and per-line rule that errs on the side of counting (full rule in the script header): - occurrences spelled `rustfs_s3_types::s3_error!` never count; - a line holding `s3s::s3_error!` always counts; - any other `s3_error!` line counts unless its file is cleared, that is, it imports the macro from rustfs_s3_types by name on a line starting with the use item (or defines it with `macro_rules! s3_error`) and does not also bring s3s's macro in (`use s3s::s3_error`, a `use s3s::{..}` tree naming it or a crate-root `*`, `use s3s::*`, or `#[macro_use] extern crate s3s`); - fail closed: a file matching neither form (`use super::*`, a crate-local re-export, a renamed or glob rustfs_s3_types import, a commented-out import, a comment-only mention) counts. Census of the 1585 lines on 96fc06776: 1478 in 64 files that import the macro from s3s; 107 in 15 files matching neither form (105 through `use super::*`, 1 qualified `s3s::s3_error!`, 1 comment line); none in files that import it from rustfs_s3_types or define it. Every line stays counted, so the baseline remains s3_error_lines=1585 (files=202 and ecstore_files=36 unchanged). Rewriting rustfs/src/auth.rs's import to rustfs_s3_types lowers the new counter to 1567; the old one stayed 1585. `--self-test` (wired into `make script-tests`) copies the script into 22 fixture trees: 6 positive cases and 16 negative ones (lines that must not count, forms that must not clear a file, and a missing baseline, missing key or failing classification search that must fail). Each tree has a control line that must count, and each run gets s3s text on stdin. Seven cases were red against the old counter. 19 mutants each turn their case red: old definition, flipped import test, dropped fail-closed branch, dropped e2e_test exclusion, dropped s3s veto, dropped qualified s3s rule, dropped rustfs-qualified strip, unanchored rustfs import, renamed import accepted, nested glob taken as root glob, single-line s3s trees only, dropped s3s glob forms, dropped macro_use form, dropped definition clearing, ignored classification rg errors, error-line search without its path, classification searches without their path, missing baseline skipped, missing key skipped. * test(e2e): compare two RUSTFS_S3_STACK servers over every S3 operation (#8435) Add dual_stack_differential to e2e_test. It starts two rustfs processes from the same binary, stack A on RUSTFS_S3_STACK=legacy and stack B on RUSTFS_E2E_STACK_B (default legacy until rustfs/backlog#2734 T2.5), sends both the same aws-sdk-s3 script covering the 73 methods of impl S3 for FS (a normal request and a 4xx negative each), and compares status, error code, an allowlist of response headers and a few typed body fields. Error message text is never compared. Each run writes <target>/dual-stack/<run>.json with compared_ops and the (op, field, a, b) differences; any difference fails the test. differential_detects_an_injected_difference starts B with RUSTFS_REGION=eu-west-1 and requires GetBucketLocation to be reported. The comparator is pure and unit-tested without a server. Wire the module into the e2e-smoke filter and update the e2e-smoke and e2e-full selection digests and the e2e_test test-count row. Refs rustfs/backlog#2740 * feat(app): add the RustFS-owned RequestEnvelope and its s3s constructor (#8437) * feat(app): add the RustFS-owned RequestEnvelope and its s3s constructor * ci: classify the s3s edge adapters in storage/s3_compat as interface * feat(s3-types): add Body::from_stream and move Swift off s3s::Body (#8438) rustfs/backlog#2747 batch 3. `Body::from_stream` wraps a chunk stream of unknown length over `StreamBody`/`BoxBody`: one data frame per chunk, the first error ends the body, the size hint stays unknown, and dropping the body drops the stream. Parity tests pin it to the s3s `StreamingBlob::wrap` it replaces, at the `http_body` level and byte for byte through hyper. The Swift handlers in rustfs-protocols build their responses as the owned `Body`. `SwiftService` converts them into the wrapped S3 service's response body type through `From<Body>`, so rustfs keeps serving `s3s::Body` at the legacy edge unchanged. Request bodies stay Send-only (`UnsyncBoxBody`); the two storage writers and the DLO/SLO segment streams get `Sync` from `sync_wrapper::SyncStream`, which is what s3s's `http_body_unsync` mutex provided before. The DTO and error uses of s3s in protocols are untouched (T1.5 and T1.2). * refactor(protocols): move error types to rustfs-s3-types by codemod (#8439) T1.2 batch 3 (rustfs/backlog#2743): s3select-api, s3select-query and the error items of protocols move off s3s by scripts/codemods, run with --hold crates/protocols/src/sftp/errors.rs. That file stays on s3s: it downcasts the backend error, whose runtime type is still s3s::S3Error. - rustfs/src/protocols/client.rs: the two webdav session methods now return rustfs_s3_types::S3Result and convert the s3s error with From. - codemod: a child module that calls s3_error! through `use super::*` gets its own `use rustfs_s3_types::s3_error;` once its parent imports that macro by name; otherwise it is listed as GLOB. - s3-types: a table test pins code, status and the s3s error the legacy edge renders for every variant a moved call site names. - s3_error_lines 1585 -> 1584 (a lifecycle comment reworded). * test(connect): restore one-second drive request budgets Apply the exact test-only repair already landed on main in 5972a09cc3. The filesystem probe and production CLI success test request one second and keep their existing one-second duration and parameter checks. All outcome, I/O, error, cleanup and resource-limit assertions remain. Scope: rustfs/tests/connect_perf_drive.rs only. Patch-id: cb0a1dfc8f68a67727525adea7458e2cee7d09f8. Validation environment: LANG=en_US.UTF-8, CARGO_BUILD_JOBS=1, CARGO_INCREMENTAL=0, exclusive APFS-cloned CARGO_TARGET_DIR. Quick Checks: every command from .github/actions/quick-checks/action.yml, with captured output and exit status below. $ shellcheck --version && actionlint ShellCheck - shell script analysis tool version: 0.11.0 license: GNU General Public License, version 3 website: https://www.shellcheck.net exit_code=0 $ cargo fmt --all --check exit_code=0 $ ./scripts/check_unsafe_code_allowances.sh exit_code=0 $ ./scripts/check_layer_dependencies.sh Layer dependency guard passed (no new reverse dependencies/cycles). exit_code=0 $ ./scripts/check_architecture_migration_rules.sh Architecture migration rules passed. exit_code=0 $ ./scripts/check_logging_guardrails.sh ✅ Logging guardrails check passed exit_code=0 $ ./scripts/check_error_other_format_ratchet.sh error(format!) ratchet passed (total call sites: 582). exit_code=0 $ ./scripts/check_no_tokio_io_uring.sh exit_code=0 $ ./scripts/check_extension_schema_boundaries.sh Extension schema boundaries passed. exit_code=0 $ ./scripts/check_body_cache_whitelist.sh Body-cache whitelist guard passed. exit_code=0 $ ./scripts/check_s3s_footprint.sh s3s footprint OK: files importing s3s is 195 (baseline files=195) s3s footprint OK: s3s s3_error! lines is 1584 (baseline s3_error_lines=1584) s3s footprint OK: ecstore files referencing s3s is 36 (baseline ecstore_files=36) ✅ s3s footprint baseline check passed exit_code=0 $ ./scripts/check_test_count_ratchet.sh test-count ratchet OK: 52 crates, 21222 tests, 164 ignored match .config/test-count-baseline.txt exit_code=0 $ ./scripts/check_fips_wording.sh FIPS wording guard passed (README.md and CHANGELOG.md contain no forbidden claims). KMS wording guard passed (crates/kms claims no Vault KV2 Transit wrapping). exit_code=0 $ ./scripts/check_embedded_secrets.sh Embedded secret guard passed (no private key material or provider credential literal in the tree). exit_code=0 $ make script-tests Running script tests... entrypoint credential policy tests passed internode grpc A/B bench env tests passed object batch benchmark enhanced tests passed PASS package service script contract hotpath warp A/B gate tests passed hotpath warp ABBA tests passed Scanner validation artifacts written to /var/folders/0w/s4wxsn2n4w9b6f3cd4xrz53w0000gn/T/tmp.1pFe6Kl1Yu/out Scanner validation artifacts written to /var/folders/0w/s4wxsn2n4w9b6f3cd4xrz53w0000gn/T/tmp.1pFe6Kl1Yu/out-missing-pid ............................. ---------------------------------------------------------------------- Ran 29 tests in 181.286s OK {"decision":"verified","pending_gates":["G01","G03","G04","G05","G06","G07","G08","G09","G10","G11","G12","G13","G14","P1","P2","P3","P4","R-D","R-E","R-L"],"pending_lanes":["authority-coverage","checkpoint-and-crash","ec8-4-multiset","maintenance-producers","mixed-version-rollback","mrf-responsibility","scheduler-pressure","status-and-outcome"],"rejected_errors":[],"rejected_gate":null,"release_approved":false,"release_schema_capable":true,"schema":1,"verified_gate":"G02"} {"decision":"verified","pending_gates":["G01","G02","G03","G04","G05","G06","G07","G08","G09","G10","G11","G12","G13","G14","P1","P2","P3","P4","R-D","R-L"],"pending_lanes":["authority-coverage","checkpoint-and-crash","ec8-4-multiset","maintenance-producers","mixed-version-rollback","mrf-responsibility","scheduler-pressure","status-and-outcome"],"rejected_errors":[],"rejected_gate":null,"release_approved":false,"release_schema_capable":true,"schema":1,"verified_gate":"R-E"} {"decision":"verified","pending_gates":["G01","G03","G04","G05","G06","G07","G08","G09","G10","G11","G12","G13","G14","P1","P2","P3","P4","R-D","R-E","R-L"],"pending_lanes":["authority-coverage","checkpoint-and-crash","ec8-4-multiset","maintenance-producers","mixed-version-rollback","mrf-responsibility","scheduler-pressure","status-and-outcome"],"rejected_errors":[],"rejected_gate":null,"release_approved":false,"release_schema_capable":true,"schema":1,"verified_gate":"G02"} {"decision":"verified","pending_gates":["G01","G02","G03","G04","G05","G06","G07","G08","G09","G10","G11","G12","G13","G14","P1","P2","P3","P4","R-D","R-L"],"pending_lanes":["authority-coverage","checkpoint-and-crash","ec8-4-multiset","maintenance-producers","mixed-version-rollback","mrf-responsibility","scheduler-pressure","status-and-outcome"],"rejected_errors":[],"rejected_gate":null,"release_approved":false,"release_schema_capable":true,"schema":1,"verified_gate":"R-E"} {"decision":"verified","pending_gates":["G02","G03","G04","G05","G06","G07","G08","G09","G10","G11","G12","G13","G14","P1","P2","P3","P4","R-D","R-E","R-L"],"pending_lanes":["authority-coverage","checkpoint-and-crash","ec8-4-multiset","maintenance-producers","mixed-version-rollback","mrf-responsibility","scheduler-pressure","status-and-outcome"],"rejected_errors":[],"rejected_gate":null,"release_approved":false,"release_schema_capable":true,"schema":1,"verified_gate":"G01"} {"decision":"verified","pending_gates":["G01","G02","G04","G05","G06","G07","G08","G09","G10","G11","G12","G13","G14","P1","P2","P3","P4","R-D","R-E","R-L"],"pending_lanes":["authority-coverage","checkpoint-and-crash","ec8-4-multiset","maintenance-producers","mixed-version-rollback","mrf-responsibility","scheduler-pressure","status-and-outcome"],"rejected_errors":[],"rejected_gate":null,"release_approved":false,"release_schema_capable":true,"schema":1,"verified_gate":"G03"} {"decision":"verified","pending_gates":["G01","G02","G03","G04","G05","G06","G07","G08","G09","G10","G11","G12","G13","G14","P1","P2","P3","P4","R-D","R-E"],"pending_lanes":["authority-coverage","checkpoint-and-crash","ec8-4-multiset","maintenance-producers","mixed-version-rollback","mrf-responsibility","scheduler-pressure","status-and-outcome"],"rejected_errors":[],"rejected_gate":null,"release_approved":false,"release_schema_capable":true,"schema":1,"verified_gate":"R-L"} {"decision":"verified","pending_gates":["G01","G02","G03","G04","G05","G06","G07","G08","G09","G10","G11","G12","G13","P1","P2","P3","P4","R-D","R-E","R-L"],"pending_lanes":["authority-coverage","checkpoint-and-crash","ec8-4-multiset","maintenance-producers","mixed-version-rollback","mrf-responsibility","scheduler-pressure","status-and-outcome"],"rejected_errors":[],"rejected_gate":null,"release_approved":false,"release_schema_capable":true,"schema":1,"verified_gate":"G14"} {"decision":"verified","pending_gates":["G01","G02","G03","G04","G05","G06","G07","G08","G09","G10","G11","G12","G13","P1","P2","P3","P4","R-D","R-E","R-L"],"pending_lanes":["authority-coverage","checkpoint-and-crash","ec8-4-multiset","maintenance-producers","mixed-version-rollback","mrf-responsibility","scheduler-pressure","status-and-outcome"],"rejected_errors":[],"rejected_gate":null,"release_approved":false,"release_schema_capable":true,"schema":1,"verified_gate":"G14"} {"decision":"verified","pending_gates":["G01","G02","G03","G04","G05","G06","G07","G08","G09","G11","G12","G13","G14","P1","P2","P3","P4","R-D","R-E","R-L"],"pending_lanes":["authority-coverage","checkpoint-and-crash","ec8-4-multiset","maintenance-producers","mixed-version-rollback","mrf-responsibility","scheduler-pressure","status-and-outcome"],"rejected_errors":[],"rejected_gate":null,"release_approved":false,"release_schema_capable":true,"schema":1,"verified_gate":"G10"} {"decision":"verified","pending_gates":["G01","G02","G03","G04","G05","G06","G07","G08","G09","G10","G11","G12","G13","G14","P2","P3","P4","R-D","R-E","R-L"],"pending_lanes":["authority-coverage","checkpoint-and-crash","ec8-4-multiset","maintenance-producers","mixed-version-rollback","mrf-responsibility","scheduler-pressure","status-and-outcome"],"rejected_errors":[],"rejected_gate":null,"release_approved":false,"release_schema_capable":true,"schema":1,"verified_gate":"P1"} {"decision":"verified","pending_gates":["G01","G02","G03","G04","G05","G06","G07","G08","G09","G10","G11","G12","G13","G14","P1","P3","P4","R-D","R-E","R-L"],"pending_lanes":["authority-coverage","checkpoint-and-crash","ec8-4-multiset","maintenance-producers","mixed-version-rollback","mrf-responsibility","scheduler-pressure","status-and-outcome"],"rejected_errors":[],"rejected_gate":null,"release_approved":false,"release_schema_capable":true,"schema":1,"verified_gate":"P2"} {"decision":"verified","pending_gates":["G01","G02","G03","G04","G05","G06","G07","G08","G09","G10","G11","G12","G13","G14","P1","P2","P4","R-D","R-E","R-L"],"pending_lanes":["authority-coverage","checkpoint-and-crash","ec8-4-multiset","maintenance-producers","mixed-version-rollback","mrf-responsibility","scheduler-pressure","status-and-outcome"],"rejected_errors":[],"rejected_gate":null,"release_approved":false,"release_schema_capable":true,"schema":1,"verified_gate":"P3"} {"decision":"verified","pending_gates":["G01","G02","G03","G04","G06","G07","G08","G09","G10","G11","G12","G13","G14","P1","P2","P3","P4","R-D","R-E","R-L"],"pending_lanes":["authority-coverage","checkpoint-and-crash","ec8-4-multiset","maintenance-producers","mixed-version-rollback","mrf-responsibility","scheduler-pressure","status-and-outcome"],"rejected_errors":[],"rejected_gate":null,"release_approved":false,"release_schema_capable":true,"schema":1,"verified_gate":"G05"} {"decision":"verified","pending_gates":["G01","G02","G03","G04","G05","G07","G08","G09","G10","G11","G12","G13","G14","P1","P2","P3","P4","R-D","R-E","R-L"],"pending_lanes":["authority-coverage","checkpoint-and-crash","ec8-4-multiset","maintenance-producers","mixed-version-rollback","mrf-responsibility","scheduler-pressure","status-and-outcome"],"rejected_errors":[],"rejected_gate":null,"release_approved":false,"release_schema_capable":true,"schema":1,"verified_gate":"G06"} {"decision":"verified","pending_gates":["G01","G02","G03","G04","G05","G06","G07","G08","G09","G10","G11","G12","G13","G14","P1","P2","P3","P4","R-E","R-L"],"pending_lanes":["authority-coverage","checkpoint-and-crash","ec8-4-multiset","maintenance-producers","mixed-version-rollback","mrf-responsibility","scheduler-pressure","status-and-outcome"],"rejected_errors":[],"rejected_gate":null,"release_approved":false,"release_schema_capable":true,"schema":1,"verified_gate":"R-D"} Status-and-outcome release descriptor verified: /private/var/folders/0w/s4wxsn2n4w9b6f3cd4xrz53w0000gn/T/tmp8rwn1pe0/descriptor/release-bundle-status-outcome.json {"decision":"verified","pending_gates":["G01","G02","G03","G04","G06","G07","G08","G09","G10","G11","G12","G13","G14","P1","P2","P3","P4","R-D","R-E","R-L"],"pending_lanes":["authority-coverage","checkpoint-and-crash","ec8-4-multiset","maintenance-producers","mixed-version-rollback","mrf-responsibility","scheduler-pressure","status-and-outcome"],"rejected_errors":[],"rejected_gate":null,"release_approved":false,"release_schema_capable":true,"schema":1,"verified_gate":"G05"} {"decision":"verified","pending_gates":["G01","G02","G03","G04","G05","G07","G08","G09","G10","G11","G12","G13","G14","P1","P2","P3","P4","R-D","R-E","R-L"],"pending_lanes":["authority-coverage","checkpoint-and-crash","ec8-4-multiset","maintenance-producers","mixed-version-rollback","mrf-responsibility","scheduler-pressure","status-and-outcome"],"rejected_errors":[],"rejected_gate":null,"release_approved":false,"release_schema_capable":true,"schema":1,"verified_gate":"G06"} {"decision":"verified","pending_gates":["G01","G02","G03","G04","G05","G06","G07","G08","G09","G10","G11","G12","G13","G14","P1","P2","P3","P4","R-E","R-L"],"pending_lanes":["authority-coverage","checkpoint-and-crash","ec8-4-multiset","maintenance-producers","mixed-version-rollback","mrf-responsibility","scheduler-pressure","status-and-outcome"],"rejected_errors":[],"rejected_gate":null,"release_approved":false,"release_schema_capable":true,"schema":1,"verified_gate":"R-D"} {"decision":"verified","pending_gates":["G01","G02","G03","G04","G05","G06","G08","G09","G10","G11","G12","G13","G14","P1","P2","P3","P4","R-D","R-E","R-L"],"pending_lanes":["authority-coverage","checkpoint-and-crash","ec8-4-multiset","maintenance-producers","mixed-version-rollback","mrf-responsibility","scheduler-pressure","status-and-outcome"],"rejected_errors":[],"rejected_gate":null,"release_approved":false,"release_schema_capable":true,"schema":1,"verified_gate":"G07"} {"decision":"verified","pending_gates":["G01","G02","G03","G04","G05","G06","G07","G09","G10","G11","G12","G13","G14","P1","P2","P3","P4","R-D","R-E","R-L"],"pending_lanes":["authority-coverage","checkpoint-and-crash","ec8-4-multiset","maintenance-producers","mixed-version-rollback","mrf-responsibility","scheduler-pressure","status-and-outcome"],"rejected_errors":[],"rejected_gate":null,"release_approved":false,"release_schema_capable":true,"schema":1,"verified_gate":"G08"} {"decision":"verified","pending_gates":["G01","G02","G03","G04","G05","G06","G07","G08","G09","G10","G11","G12","G13","G14","P1","P2","P3","R-D","R-E","R-L"],"pending_lanes":["authority-coverage","checkpoint-and-crash","ec8-4-multiset","maintenance-producers","mixed-version-rollback","mrf-responsibility","scheduler-pressure","status-and-outcome"],"rejected_errors":[],"rejected_gate":null,"release_approved":false,"release_schema_capable":true,"schema":1,"verified_gate":"P4"} exact 1MiB handoff ABBA tests passed pinned paired ABBA bench tests passed ==> /var/folders/0w/s4wxsn2n4w9b6f3cd4xrz53w0000gn/T/tmp.mYmuI01hNq/bin/path_containment (-max_total_time=7, -seed=123456789, -artifact_prefix=/var/folders/0w/s4wxsn2n4w9b6f3cd4xrz53w0000gn/T/tmp.mYmuI01hNq/explicit-artifacts/path_containment/, corpus=/private/tmp/gw-orchestrator-2734/ci-drive/fuzz/corpus/path_containment) -seed=123456789 target=path_containment seed=123456789 max_total_time=7 git_revision=4d15b816236630b2c42f321ffc9ed3654edc94e3 git_dirty=true runner_mode=prebuilt ==> /var/folders/0w/s4wxsn2n4w9b6f3cd4xrz53w0000gn/T/tmp.mYmuI01hNq/bin/path_containment (-max_total_time=1, -seed=2322617706, -artifact_prefix=/var/folders/0w/s4wxsn2n4w9b6f3cd4xrz53w0000gn/T/tmp.mYmuI01hNq/auto-artifacts/path_containment/, corpus=/private/tmp/gw-orchestrator-2734/ci-drive/fuzz/corpus/path_containment) -seed=2322617706 ==> cargo +nightly fuzz run path_containment (-max_total_time=9, -seed=987654321, -artifact_prefix=/var/folders/0w/s4wxsn2n4w9b6f3cd4xrz53w0000gn/T/tmp.mYmuI01hNq/cargo-artifacts/path_containment/) +nightly fuzz run path_containment -seed=987654321 seed=987654321 runner_mode=cargo-fuzz fuzz runner seed manifest tests passed ✅ scripts/python_bin.sh resolver checks passed self-test ok: clean fixture with every exemption present passes self-test ok: all 11 pattern families fire self-test ok: moving a pinned source is reported Embedded secret guard self-test passed. self-test ok: p1-s3s-import: 'use s3s::s3_error;' counts every line, the rustfs_s3_types module's too (s3_error_lines=3) self-test ok: p2-s3s-mixed-tree: a multi-line s3s use tree naming s3_error counts (s3_error_lines=3) self-test ok: p3-s3s-glob: a crate-root glob of s3s, bare or inside a tree, counts (s3_error_lines=5) self-test ok: p4-macro-use: '#[macro_use] extern crate s3s' counts (s3_error_lines=3) self-test ok: p5-qualified-s3s: a qualified s3s::s3_error! line counts in a cleared file (s3_error_lines=2) self-test ok: p6-exact-pass: a baseline equal to the tree passes (exit 0) self-test ok: n1-rustfs-import: 'use rustfs_s3_types::s3_error;' alone counts nothing (s3_error_lines=1) self-test ok: n2-rustfs-mixed-tree: a rustfs_s3_types tree plus an s3s tree without the macro counts nothing (s3_error_lines=1) self-test ok: n3-defines: the file defining the macro counts nothing (s3_error_lines=1) self-test ok: n4-qualified-rustfs: a qualified rustfs_s3_types::s3_error! line counts nothing (s3_error_lines=1) self-test ok: n5-renamed-call: a call through a renamed macro is not text-matched (out of scope) (s3_error_lines=1) self-test ok: n6-comment-string: a comment or string mentioning s3_error! in a cleared file counts nothing (s3_error_lines=1) self-test ok: n7-e2e-excluded: crates/e2e_test is excluded (s3_error_lines=1) self-test ok: n8-super-glob: an unrecognised import form ('use super::*;') counts (s3_error_lines=3) self-test ok: n9-facade: a crate-local re-export counts (s3_error_lines=2) self-test ok: n10-renamed-import: a renamed rustfs_s3_types import does not clear s3_error! lines (s3_error_lines=2) self-test ok: n11-commented-import: a commented-out rustfs_s3_types import does not clear (s3_error_lines=2) self-test ok: n12-rustfs-glob: a glob import of rustfs_s3_types does not clear (s3_error_lines=2) self-test ok: n13-comment-only: a comment mentioning s3_error! in a file with no import counts (s3_error_lines=2) self-test ok: f1-baseline-missing: a missing baseline file fails (exit 1) self-test ok: f2-key-missing: a baseline without s3_error_lines fails (exit 1) self-test ok: f3-rg-fails: a failing classification search fails the guard (exit 1) ✅ s3s footprint self-test passed test_core_gate_rejects_missing_ignored_filtered_and_corrupt_inputs (__main__.SelfTests.test_core_gate_rejects_missing_ignored_filtered_and_corrupt_inputs) ... ok test_e2e_requires_registration (__main__.SelfTests.test_e2e_requires_registration) ... ok test_fuzz_runtime_uses_matrix_target (__main__.SelfTests.test_fuzz_runtime_uses_matrix_target) ... ok test_ilm_lane_keeps_the_measured_cargo_build_budget (__main__.SelfTests.test_ilm_lane_keeps_the_measured_cargo_build_budget) ... ok test_profile_listing_binds_platform_digest (__main__.SelfTests.test_profile_listing_binds_platform_digest) ... ok test_profile_listing_enforces_selection (__main__.SelfTests.test_profile_listing_enforces_selection) ... ok test_quick_checks_commands_propagate_failure (__main__.SelfTests.test_quick_checks_commands_propagate_failure) ... ok test_quick_checks_rejects_caller_and_execution_bypasses (__main__.SelfTests.test_quick_checks_rejects_caller_and_execution_bypasses) ... ok test_runner_readiness_fails_closed (__main__.SelfTests.test_runner_readiness_fails_closed) ... ok test_s3_runner_rejects_unbounded_failure_locals (__main__.SelfTests.test_s3_runner_rejects_unbounded_failure_locals) ... ok test_scanner_heal_begin_requires_embedded_source_provenance (__main__.SelfTests.test_scanner_heal_begin_requires_embedded_source_provenance) ... ok test_scanner_heal_case_does_not_approve_pending_release (__main__.SelfTests.test_scanner_heal_case_does_not_approve_pending_release) ... ok test_scanner_heal_case_lane_is_required_for_runner_profile (__main__.SelfTests.test_scanner_heal_case_lane_is_required_for_runner_profile) ... ok test_scanner_heal_case_only_schema_cannot_approve_release (__main__.SelfTests.test_scanner_heal_case_only_schema_cannot_approve_release) ... ok test_scanner_heal_crash_case_rejects_restart_oracle (__main__.SelfTests.test_scanner_heal_crash_case_rejects_restart_oracle) ... ok test_scanner_heal_ec84_case_rejects_wrong_erasure_geometry (__main__.SelfTests.test_scanner_heal_ec84_case_rejects_wrong_erasure_geometry) ... ok test_scanner_heal_finish_collects_oracles_from_registry (__main__.SelfTests.test_scanner_heal_finish_collects_oracles_from_registry) ... ok test_scanner_heal_multipool_case_accepts_deferred_optional_outage_write (__main__.SelfTests.test_scanner_heal_multipool_case_accepts_deferred_optional_outage_write) ... ok test_scanner_heal_pending_gate_cannot_map_to_implemented_lane (__main__.SelfTests.test_scanner_heal_pending_gate_cannot_map_to_implemented_lane) ... ok test_scanner_heal_receipts_reject_reuse_and_missing_builds (__main__.SelfTests.test_scanner_heal_receipts_reject_reuse_and_missing_builds) ... ok test_scanner_heal_rejects_boolean_fractional_and_out_of_geometry_integers (__main__.SelfTests.test_scanner_heal_rejects_boolean_fractional_and_out_of_geometry_integers) ... ok test_scanner_heal_rejects_broken_execution_and_artifacts (__main__.SelfTests.test_scanner_heal_rejects_broken_execution_and_artifacts) ... ok test_scanner_heal_rejects_copied_junit_and_wrong_suite_build (__main__.SelfTests.test_scanner_heal_rejects_copied_junit_and_wrong_suite_build) ... ok test_scanner_heal_release_bundle_accepts_complete_measured_evidence (__main__.SelfTests.test_scanner_heal_release_bundle_accepts_complete_measured_evidence) ... ok test_scanner_heal_release_bundle_assembler_copies_measured_artifacts (__main__.SelfTests.test_scanner_heal_release_bundle_assembler_copies_measured_artifacts) ... ok test_scanner_heal_release_bundle_assembler_merges_measured_descriptors (__main__.SelfTests.test_scanner_heal_release_bundle_assembler_merges_measured_descriptors) ... ok test_scanner_heal_release_bundle_assembler_rejects_duplicate_descriptor_gate (__main__.SelfTests.test_scanner_heal_release_bundle_assembler_rejects_duplicate_descriptor_gate) ... ok test_scanner_heal_release_bundle_assembler_rejects_fixture_descriptor (__main__.SelfTests.test_scanner_heal_release_bundle_assembler_rejects_fixture_descriptor) ... ok test_scanner_heal_release_bundle_assembler_rejects_hash_mismatch (__main__.SelfTests.test_scanner_heal_release_bundle_assembler_rejects_hash_mismatch) ... ok test_scanner_heal_release_bundle_assembler_rejects_mixed_descriptor_revisions (__main__.SelfTests.test_scanner_heal_release_bundle_assembler_rejects_mixed_descriptor_revisions) ... ok test_scanner_heal_release_bundle_enforces_topology_duration_profile_and_versions (__main__.SelfTests.test_scanner_heal_release_bundle_enforces_topology_duration_profile_and_versions) ... ok test_scanner_heal_release_bundle_fixture_writer_is_not_release_evidence (__main__.SelfTests.test_scanner_heal_release_bundle_fixture_writer_is_not_release_evidence) ... ok test_scanner_heal_release_bundle_gate_accepts_single_measured_gate (__main__.SelfTests.test_scanner_heal_release_bundle_gate_accepts_single_measured_gate) ... ok test_scanner_heal_release_bundle_gate_rejects_missing_field (__main__.SelfTests.test_scanner_heal_release_bundle_gate_rejects_missing_field) ... ok test_scanner_heal_release_bundle_gate_rejects_unknown_gate (__main__.SelfTests.test_scanner_heal_release_bundle_gate_rejects_unknown_gate) ... ok test_scanner_heal_release_bundle_rejects_synthetic_or_missing_fields (__main__.SelfTests.test_scanner_heal_release_bundle_rejects_synthetic_or_missing_fields) ... ok test_scanner_heal_release_bundle_requires_domain_evidence (__main__.SelfTests.test_scanner_heal_release_bundle_requires_domain_evidence) ... ok test_scanner_heal_release_bundle_requires_field_provenance (__main__.SelfTests.test_scanner_heal_release_bundle_requires_field_provenance) ... ok test_scanner_heal_release_bundle_requires_json_artifact_provenance (__main__.SelfTests.test_scanner_heal_release_bundle_requires_json_artifact_provenance) ... ok test_scanner_heal_release_bundle_requires_mixed_version_field_roles (__main__.SelfTests.test_scanner_heal_release_bundle_requires_mixed_version_field_roles) ... ok test_scanner_heal_release_bundle_requires_same_gate_measurement_window (__main__.SelfTests.test_scanner_heal_release_bundle_requires_same_gate_measurement_window) ... ok test_scanner_heal_release_consumes_multiple_registry_oracles (__main__.SelfTests.test_scanner_heal_release_consumes_multiple_registry_oracles) ... ok test_scanner_heal_release_matrix_lanes_remain_pending_without_real_evidence (__main__.SelfTests.test_scanner_heal_release_matrix_lanes_remain_pending_without_real_evidence) ... ok test_scanner_heal_release_requirements_cannot_be_cleared (__main__.SelfTests.test_scanner_heal_release_requirements_cannot_be_cleared) ... ok test_scanner_heal_release_status_rejects_focused_case_run (__main__.SelfTests.test_scanner_heal_release_status_rejects_focused_case_run) ... ok test_scanner_heal_release_status_rejects_synthetic_case (__main__.SelfTests.test_scanner_heal_release_status_rejects_synthetic_case) ... ok test_scanner_heal_required_evidence_fields_cannot_be_removed (__main__.SelfTests.test_scanner_heal_required_evidence_fields_cannot_be_removed) ... ok test_scheduled_alerts_require_completion_watchdog (__main__.SelfTests.test_scheduled_alerts_require_completion_watchdog) ... ok test_update_profile_selection_changes_only_requested_platform (__main__.SelfTests.test_update_profile_selection_changes_only_requested_platform) ... ok test_validate_includes_quick_checks (__main__.SelfTests.test_validate_includes_quick_checks) ... ok test_vault_tests_require_cross_process_serialization (__main__.SelfTests.test_vault_tests_require_cross_process_serialization) ... ok test_workflow_readiness_requires_dependency_probe (__main__.SelfTests.test_workflow_readiness_requires_dependency_probe) ... ok ---------------------------------------------------------------------- Ran 52 tests in 14.301s OK e2e-full selection OK: 1 tests, sha256=29e30e6ed2ed38d9e5b0a7cdd9536abf5c719f7021a02b578ae1381563a3c273 ................................... ---------------------------------------------------------------------- Ran 35 tests in 1.161s OK ................... ---------------------------------------------------------------------- Ran 19 tests in 17.397s OK .......... ---------------------------------------------------------------------- Ran 10 tests in 0.183s OK test_actual_selector_bootstrap_uses_base_policy_and_fails_closed (__main__.SelfTests.test_actual_selector_bootstrap_uses_base_policy_and_fails_closed) ... ok test_documentation_paths_do_not_hide_build_or_fixture_changes (__main__.SelfTests.test_documentation_paths_do_not_hide_build_or_fixture_changes) ... ok test_e2e_scope_keeps_production_and_shared_configuration_full (__main__.SelfTests.test_e2e_scope_keeps_production_and_shared_configuration_full) ... ok test_e2e_shortcut_rejects_new_direct_renamed_and_target_dependencies (__main__.SelfTests.test_e2e_shortcut_rejects_new_direct_renamed_and_target_dependencies) ... ok test_e2e_shortcut_traverses_implicit_path_dependencies (__main__.SelfTests.test_e2e_shortcut_traverses_implicit_path_dependencies) ... ok test_event_contract_requires_complete_candidate_and_optional_lanes (__main__.SelfTests.test_event_contract_requires_complete_candidate_and_optional_lanes) ... ok test_every_wrong_result_missing_job_or_selection_fails_closed (__main__.SelfTests.test_every_wrong_result_missing_job_or_selection_fails_closed) ... ok test_full_e2e_gate_preserves_workflow_branch_and_event_scope (__main__.SelfTests.test_full_e2e_gate_preserves_workflow_branch_and_event_scope) ... ok test_git_range_includes_deleted_source_and_rename_origins (__main__.SelfTests.test_git_range_includes_deleted_source_and_rename_origins) ... ok test_job_ids_and_display_names_cannot_hide_validation (__main__.SelfTests.test_job_ids_and_display_names_cannot_hide_validation) ... ok test_report_only_stack_lane_mirrors_e2e_smoke_outside_the_aggregate (__main__.SelfTests.test_report_only_stack_lane_mirrors_e2e_smoke_outside_the_aggregate) ... ok test_repository_wiring_and_missing_dependency_regression (__main__.SelfTests.test_repository_wiring_and_missing_dependency_regression) ... ok test_verify_command_preserves_failures (__main__.SelfTests.test_verify_command_preserves_failures) ... ok ---------------------------------------------------------------------- Ran 13 tests in 3.280s OK security coverage self-test passed test_api_failures_preserve_other_evidence_and_never_enter_grace (__main__.SelfTests.test_api_failures_preserve_other_evidence_and_never_enter_grace) ... ok test_attempt_outcome_does_not_replace_recent_success (__main__.SelfTests.test_attempt_outcome_does_not_replace_recent_success) ... ok test_cancelled_attempt_cannot_refresh_expired_success (__main__.SelfTests.test_cancelled_attempt_cannot_refresh_expired_success) ... ok test_cli_requires_the_repository_default_branch (__main__.SelfTests.test_cli_requires_the_repository_default_branch) ... ok test_config_rejects_duplicate_and_invalid_entries (__main__.SelfTests.test_config_rejects_duplicate_and_invalid_entries) ... ok test_freshness_boundaries (__main__.SelfTests.test_freshness_boundaries) ... ok test_grace_requires_two_successful_queries_with_no_history (__main__.SelfTests.test_grace_requires_two_successful_queries_with_no_history) ... ok test_invalid_api_evidence_fails_closed (__main__.SelfTests.test_invalid_api_evidence_fails_closed) ... ok test_report_retains_every_workflow (__main__.SelfTests.test_report_retains_every_workflow) ... ok test_requests_filter_schedule_default_branch_and_success_on_server (__main__.SelfTests.test_requests_filter_schedule_default_branch_and_success_on_server) ... ok ---------------------------------------------------------------------- Ran 10 tests in 0.051s OK .............................. ---------------------------------------------------------------------- Ran 30 tests in 22.354s OK .......... ---------------------------------------------------------------------- Ran 10 tests in 2.969s OK ................................. ---------------------------------------------------------------------- Ran 33 tests in 0.160s OK Online functional runner labels: pf-testing ....... ---------------------------------------------------------------------- Ran 7 tests in 0.002s OK ....... ---------------------------------------------------------------------- Ran 7 tests in 0.001s OK .... ---------------------------------------------------------------------- Ran 4 tests in 0.128s OK ... ---------------------------------------------------------------------- Ran 3 tests in 0.547s OK PASS: cold stampede script self-test exit_code=0 $ python3 ./scripts/test_object_generation_protocol_model.py test_a_recovery_barrier_cannot_choose_an_empty_value (__main__.ObjectGenerationProtocolModel.test_a_recovery_barrier_cannot_choose_an_empty_value) ... ok test_all_bounded_two_proposer_interleavings_choose_at_most_one_value (__main__.ObjectGenerationProtocolModel.test_all_bounded_two_proposer_interleavings_choose_at_most_one_value) ... ok test_an_unavailable_decision_quorum_cannot_choose_a_candidate (__main__.ObjectGenerationProtocolModel.test_an_unavailable_decision_quorum_cannot_choose_a_candidate) ... ok test_counter_allocation_requires_persistence_and_is_consumed_once (__main__.ObjectGenerationProtocolModel.test_counter_allocation_requires_persistence_and_is_consumed_once) ... ok test_counter_restart_burns_persisted_allocations_before_or_after_issue (__main__.ObjectGenerationProtocolModel.test_counter_restart_burns_persisted_allocations_before_or_after_issue) ... ok test_crash_after_promise_before_reply_does_not_restore_an_older_ballot (__main__.ObjectGenerationProtocolModel.test_crash_after_promise_before_reply_does_not_restore_an_older_ballot) ... ok test_duplicate_or_unconfigured_voters_cannot_form_a_quorum (__main__.ObjectGenerationProtocolModel.test_duplicate_or_unconfigured_voters_cannot_form_a_quorum) ... ok test_empty_or_unobserved_minority_slot_can_retry_without_allocating_another_revision (__main__.ObjectGenerationProtocolModel.test_empty_or_unobserved_minority_slot_can_retry_without_allocating_another_revision) ... ok test_equal_ballot_cannot_accept_different_bytes_and_lower_ballot_is_rejected (__main__.ObjectGenerationProtocolModel.test_equal_ballot_cannot_accept_different_bytes_and_lower_ballot_is_rejected) ... ok test_forgetting_tombstone_authority_allows_an_aba_create (__main__.ObjectGenerationProtocolModel.test_forgetting_tombstone_authority_allows_an_aba_create) ... ok test_issuing_before_persistence_reuses_a_ballot_after_restart (__main__.ObjectGenerationProtocolModel.test_issuing_before_persistence_reuses_a_ballot_after_restart) ... ok test_lost_ack_and_restart_preserve_the_chosen_operation (__main__.ObjectGenerationProtocolModel.test_lost_ack_and_restart_preserve_the_chosen_operation) ... ok test_minority_accepted_work_is_adopted_and_finished (__main__.ObjectGenerationProtocolModel.test_minority_accepted_work_is_adopted_and_finished) ... ok test_omitting_highest_accepted_adoption_has_a_double_choice_counterexample (__main__.ObjectGenerationProtocolModel.test_omitting_highest_accepted_adoption_has_a_double_choice_counterexample) ... ok test_quorum_choice_does_not_require_an_uncontacted_disk_to_learn_it (__main__.ObjectGenerationProtocolModel.test_quorum_choice_does_not_require_an_uncontacted_disk_to_learn_it) ... ok test_recovery_failure_or_stale_expected_head_does_not_allocate_a_successor (__main__.ObjectGenerationProtocolModel.test_recovery_failure_or_stale_expected_head_does_not_allocate_a_successor) ... ok test_replacing_voters_with_empty_state_can_choose_a_conflicting_value (__main__.ObjectGenerationProtocolModel.test_replacing_voters_with_empty_state_can_choose_a_conflicting_value) ... ok test_skipping_predecessor_recovery_chooses_a_broken_revision_chain (__main__.ObjectGenerationProtocolModel.test_skipping_predecessor_recovery_chooses_a_broken_revision_chain) ... ok test_stale_publication_rollback_and_gc_cannot_change_the_winner (__main__.ObjectGenerationProtocolModel.test_stale_publication_rollback_and_gc_cannot_change_the_winner) ... ok test_superseded_operation_identity_cannot_be_reused_for_different_input (__main__.ObjectGenerationProtocolModel.test_superseded_operation_identity_cannot_be_reused_for_different_input) ... ok test_superseded_operation_retry_returns_its_original_successor (__main__.ObjectGenerationProtocolModel.test_superseded_operation_retry_returns_its_original_successor) ... ok test_tombstone_and_null_marker_payloads_do_not_reset_authority_revision (__main__.ObjectGenerationProtocolModel.test_tombstone_and_null_marker_payloads_do_not_reset_authority_revision) ... ok test_tombstone_rejects_a_delayed_never_present_create (__main__.ObjectGenerationProtocolModel.test_tombstone_rejects_a_delayed_never_present_create) ... ok test_unlearned_predecessor_is_recovered_before_the_second_slot (__main__.ObjectGenerationProtocolModel.test_unlearned_predecessor_is_recovered_before_the_second_slot) ... ok ---------------------------------------------------------------------- Ran 24 tests in 0.958s OK model: states=51023, transitions=167520, terminal=732, chosen_candidates=2 negative control: double-choice trace=((0, 'prepare', 0), (0, 'prepare', 1), (0, 'prepare', 2), (0, 'accept', 0), (0, 'accept', 1), (0, 'accept', 2), (1, 'prepare', 0), (1, 'prepare', 1), (1, 'prepare', 2), (1, 'accept', 0), (1, 'accept', 1), (1, 'accept', 2)) multi-slot round schedules: 640 exit_code=0 $ python3 ./scripts/check_test_wiring.py OK: e2e modules, runner selection, fuzz matrices, profiles, scheduled alerts, and Quick Checks are wired exit_code=0 $ ./scripts/check_no_planning_docs.sh exit_code=0 $ ./scripts/check_uring_lane_lib_only.sh OK: no 'uring_'-matching test functions under crates/ecstore/tests; --lib is safe exit_code=0 Initial complete gate $ cargo nextest run --profile ci -p rustfs --test connect_perf_drive --no-fail-fast Nextest run ID e07fc5b9-c6c8-4cdd-ae61-54839bf24437 with nextest profile: ci Starting 10 tests across 1 binary PASS [ 0.016s] rustfs::connect_perf_drive cancellation_and_single_collector_leave_no_scratch PASS [ 0.011s] rustfs::connect_perf_drive cleanup_failure_overrides_cancellation_and_remains_attributed PASS [ 0.009s] rustfs::connect_perf_drive consent_and_resource_budgets_fail_before_scratch_io PASS [ 0.009s] rustfs::connect_perf_drive invalid_roots_and_existing_scratch_fail_without_clobbering PASS [ 16.950s] rustfs::connect_perf_drive production_cli_measures_and_signs_exact_binary_provenance PASS [ 4.025s] rustfs::connect_perf_drive production_cli_rejects_missing_l1_consent_and_limits_before_artifacts PASS [ 0.008s] rustfs::connect_perf_drive read_only_root_and_full_drive_errors_are_classified PASS [ 0.015s] rustfs::connect_perf_drive real_local_write_read_is_measured_and_scratch_is_removed PASS [ 0.007s] rustfs::connect_perf_drive runtime_timeout_and_full_drive_use_schema_valid_failed_reason PASS [ 0.035s] rustfs::connect_perf_drive successful_result_has_signed_bounded_private_offline_export ──────────── Summary [ 21.087s] 10 tests run: 10 passed, 0 skipped exit_code=0 Restored complete gate $ cargo nextest run --profile ci -p rustfs --test connect_perf_drive --no-fail-fast Nextest run ID 233b7307-5361-4eb7-94d5-b384f17d2d6a with nextest profile: ci Starting 10 tests across 1 binary PASS [ 0.009s] rustfs::connect_perf_drive cancellation_and_single_collector_leave_no_scratch PASS [ 0.008s] rustfs::connect_perf_drive cleanup_failure_overrides_cancellation_and_remains_attributed PASS [ 0.008s] rustfs::connect_perf_drive consent_and_resource_budgets_fail_before_scratch_io PASS [ 0.008s] rustfs::connect_perf_drive invalid_roots_and_existing_scratch_fail_without_clobbering PASS [ 19.518s] rustfs::connect_perf_drive production_cli_measures_and_signs_exact_binary_provenance PASS [ 4.146s] rustfs::connect_perf_drive production_cli_rejects_missing_l1_consent_and_limits_before_artifacts PASS [ 0.009s] rustfs::connect_perf_drive read_only_root_and_full_drive_errors_are_classified PASS [ 0.018s] rustfs::connect_perf_drive real_local_write_read_is_measured_and_scratch_is_removed PASS [ 0.009s] rustfs::connect_perf_drive runtime_timeout_and_full_drive_use_schema_valid_failed_reason PASS [ 0.054s] rustfs::connect_perf_drive successful_result_has_signed_bounded_private_offline_export ──────────── Summary [ 23.787s] 10 tests run: 10 passed, 0 skipped exit_code=0 Mutation M1: success_measurement reports max(actual elapsed, requested budget + 1), forcing 1001 ms for the one-second request. The retained duration assertion at rustfs/tests/connect_perf_drive.rs:105 fails. $ cargo nextest run --profile ci -p rustfs --test connect_perf_drive --no-fail-fast -E 'test(real_local_write_read_is_measured_and_scratch_is_removed)' info: experimental features enabled: setup-scripts Compiling rustfs v1.0.1 (/private/tmp/gw-orchestrator-2734/ci-drive/rustfs) warning: linker stderr: ld: __eh_frame section too large (max 16MB) to encode dwarf unwind offsets in compact unwind table, performance of exception handling might be affected | = note: `#[warn(linker_messages)]` on by default warning: `rustfs` (bin "rustfs") generated 1 warning warning: `rustfs` (bin "rustfs-cli") generated 1 warning (1 duplicate) Finished `test` profile [unoptimized + debuginfo] target(s) in 7m 06s ──────────── Nextest run ID 9f618777-7bf6-4cbc-8803-dd7f50f449ed with nextest profile: ci Starting 1 test across 1 binary (9 tests skipped) FAIL [ 0.016s] rustfs::connect_perf_drive real_local_write_read_is_measured_and_scratch_is_removed stdout ─── running 1 test test real_local_write_read_is_measured_and_scratch_is_removed ... FAILED failures: failures: real_local_write_read_is_measured_and_scratch_is_removed test result: FAILED. 0 passed; 1 failed; 0 ignored; 0 measured; 9 filtered out; finished in 0.01s stderr ─── thread 'real_local_write_read_is_measured_and_scratch_is_removed' (106986846) panicked at rustfs/tests/connect_perf_drive.rs:105:5: assertion failed: (1..=1_000).contains(&data.duration_millis) note: run with `RUST_BACKTRACE=1` environment variable to display a backtrace ──────────── Summary [ 0.017s] 1 test run: 0 passed, 1 failed, 9 skipped FAIL [ 0.016s] rustfs::connect_perf_drive real_local_write_read_is_measured_and_scratch_is_removed error: test run failed exit_code=100 Mutation M2: target_parameters reports requested duration + 1. The in-process assertion at rustfs/tests/connect_perf_drive.rs:111 and CLI JSON assertion at rustfs/tests/connect_perf_drive.rs:527 both reject 1001 against 1000. $ cargo nextest run --profile ci -p rustfs --test connect_perf_drive --no-fail-fast -E 'test(real_local_write_read_is_measured_and_scratch_is_removed) | test(production_cli_measures_and_signs_exact_binary_provenance)' info: experimental features enabled: setup-scripts Compiling rustfs v1.0.1 (/private/tmp/gw-orchestrator-2734/ci-drive/rustfs) warning: linker stderr: ld: __eh_frame section too large (max 16MB) to encode dwarf unwind offsets in compact unwind table, performance of exception handling might be affected | = note: `#[warn(linker_messages)]` on by default warning: `rustfs` (bin "rustfs") generated 1 warning warning: `rustfs` (bin "rustfs-cli") generated 1 warning (1 duplicate) Finished `test` profile [unoptimized + debuginfo] target(s) in 7m 20s ──────────── Nextest run ID b73d46ab-c910-4e1e-8063-7fc7e332ec15 with nextest profile: ci Starting 2 tests across 1 binary (8 tests skipped) FAIL [ 13.968s] rustfs::connect_perf_drive production_cli_measures_and_signs_exact_binary_provenance stdout ─── running 1 test test production_cli_measures_and_signs_exact_binary_provenance ... FAILED failures: failures: production_cli_measures_and_signs_exact_binary_provenance test result: FAILED. 0 passed; 1 failed; 0 ignored; 0 measured; 9 filtered out; finished in 13.96s stderr ─── thread 'production_cli_measures_and_signs_exact_binary_provenance' (107015462) panicked at rustfs/tests/connect_perf_drive.rs:527:5: assertion `left == right` failed left: Number(1001) right: 1000 note: run with `RUST_BACKTRACE=1` environment variable to display a backtrace FAIL [ 0.015s] rustfs::connect_perf_drive real_local_write_read_is_measured_and_scratch_is_removed stdout ─── running 1 test test real_local_write_read_is_measured_and_scratch_is_removed ... FAILED failures: failures: real_local_write_read_is_measured_and_scratch_is_removed test result: FAILED. 0 passed; 1 failed; 0 ignored; 0 measured; 9 filtered out; finished in 0.01s stderr ─── thread 'real_local_write_read_is_measured_and_scratch_is_removed' (107016667) panicked at rustfs/tests/connect_perf_drive.rs:111:5: assertion `left == right` failed left: 1001 right: 1000 note: run with `RUST_BACKTRACE=1` environment variable to display a backtrace ──────────── Summary [ 13.985s] 2 tests run: 0 passed, 2 failed, 8 skipped FAIL [ 13.968s] rustfs::connect_perf_drive production_cli_measures_and_signs_exact_binary_provenance FAIL [ 0.015s] rustfs::connect_perf_drive real_local_write_read_is_measured_and_scratch_is_removed error: test run failed exit_code=100 Transient production mutations were restored byte-for-byte before the complete green rerun. Original/restored SHA256: 8d389e6d8836c041aab8c6a93e56addd65ba3f70b33264e1c74f639d0572a2da. $ git diff --check exit_code=0 $ git diff --cached --check exit_code=0 Independent orchestrator acceptance Tested code tree: e67c72fb01 Tested tree: 16fbbd2ea4 shellcheck --version && actionlint -> exit 0 cargo fmt --all --check -> exit 0 ./scripts/check_unsafe_code_allowances.sh -> exit 0 ./scripts/check_layer_dependencies.sh -> exit 0 ./scripts/check_architecture_migration_rules.sh -> exit 0 ./scripts/check_logging_guardrails.sh -> exit 0 ./scripts/check_error_other_format_ratchet.sh -> exit 0 ./scripts/check_no_tokio_io_uring.sh -> exit 0 ./scripts/check_extension_schema_boundaries.sh -> exit 0 ./scripts/check_body_cache_whitelist.sh -> exit 0 ./scripts/check_s3s_footprint.sh -> exit 0 ./scripts/check_test_count_ratchet.sh -> exit 0 ./scripts/check_fips_wording.sh -> exit 0 ./scripts/check_embedded_secrets.sh -> exit 0 make script-tests -> exit 0 python3 ./scripts/test_object_generation_protocol_model.py -> exit 0 python3 ./scripts/check_test_wiring.py -> exit 0 ./scripts/check_no_planning_docs.sh -> exit 0 ./scripts/check_uring_lane_lib_only.sh -> exit 0 $ LANG=en_US.UTF-8 CARGO_BUILD_JOBS=1 CARGO_INCREMENTAL=0 cargo nextest run --profile ci -p rustfs --test connect_perf_drive --no-fail-fast info: experimental features enabled: setup-scripts warning: linker stderr: ld: __eh_frame section too large (max 16MB) to encode dwarf unwind offsets in compact unwind table, performance of exception handling might be affected | = note: `#[warn(linker_messages)]` on by default warning: `rustfs` (bin "rustfs-cli") generated 1 warning warning: `rustfs` (bin "rustfs") generated 1 warning (1 duplicate) Finished `test` profile [unoptimized + debuginfo] target(s) in 0.69s ──────────── Nextest run ID cd1edf4c-f1c0-456d-ad1a-76cc00ce05c9 with nextest profile: ci Starting 10 tests across 1 binary PASS [ 0.012s] rustfs::connect_perf_drive cancellation_and_single_collector_leave_no_scratch PASS [ 0.010s] rustfs::connect_perf_drive cleanup_failure_overrides_cancellation_and_remains_attributed PASS [ 0.010s] rustfs::connect_perf_drive consent_and_resource_budgets_fail_before_scratch_io PASS [ 0.009s] rustfs::connect_perf_drive invalid_roots_and_existing_scratch_fail_without_clobbering PASS [ 20.328s] rustfs::connect_perf_drive production_cli_measures_and_signs_exact_binary_provenance PASS [ 5.527s] rustfs::connect_perf_drive production_cli_rejects_missing_l1_consent_and_limits_before_artifacts PASS [ 0.013s] rustfs::connect_perf_drive read_only_root_and_full_drive_errors_are_classified PASS [ 0.019s] rustfs::connect_perf_drive real_local_write_read_is_measured_and_scratch_is_removed PASS [ 0.011s] rustfs::connect_perf_drive runtime_timeout_and_full_drive_use_schema_valid_failed_reason PASS [ 0.049s] rustfs::connect_perf_drive successful_result_has_signed_bounded_private_offline_export ──────────── Summary [ 25.991s] 10 tests run: 10 passed, 0 skipped exit_code=0 All 54 tracked Cargo.toml files parse without duplicate keys. git diff --check passed. The production source remains unchanged; the two deliberate implementation mutations above were restored before this complete green run. * fix(test): compare legacy body wire parity as raw bytes Lossy UTF-8 comparison concealed a changed binary response byte. Compare the already captured Vec<u8> values directly. Validation: - Binary mutant: migrated payload 0x84 vs legacy 0x83; old comparator passed 1/1, raw comparator failed 0/1 (exit 100); restored green. - Non-Swift body-drain mutant: untouched-body delegation test failed with empty vs payload (exit 100); restored handler without a diff. - cargo nextest run --profile ci -p rustfs-s3-types -p rustfs-protocols --features rustfs-protocols/swift --no-fail-fast: 543 tests run, 543 passed, 0 skipped after both mutants restored. - cargo fmt --all --check and git diff --check passed. Orchestrator acceptance (LANG=en_US.UTF-8): cargo nextest run --profile ci -p rustfs-s3-types -p rustfs-protocols --features rustfs-protocols/swift --no-fail-fast Summary [2.445s] 543 tests run: 543 passed, 0 skipped. All shared quick checks on the final code tree: shellcheck --version && actionlint -> exit 0 cargo fmt --all --check -> exit 0 ./scripts/check_unsafe_code_allowances.sh -> exit 0 ./scripts/check_layer_dependencies.sh -> exit 0 ./scripts/check_architecture_migration_rules.sh -> exit 0 ./scripts/check_logging_guardrails.sh -> exit 0 ./scripts/check_error_other_format_ratchet.sh -> exit 0 ./scripts/check_no_tokio_io_uring.sh -> exit 0 ./scripts/check_extension_schema_boundaries.sh -> exit 0 ./scripts/check_body_cache_whitelist.sh -> exit 0 ./scripts/check_s3s_footprint.sh -> exit 0 ./scripts/check_test_count_ratchet.sh -> exit 0 ./scripts/check_fips_wording.sh -> exit 0 ./scripts/check_embedded_secrets.sh -> exit 0 make script-tests -> exit 0 python3 ./scripts/test_object_generation_protocol_model.py -> exit 0 python3 ./scripts/check_test_wiring.py -> exit 0 ./scripts/check_no_planning_docs.sh -> exit 0 ./scripts/check_uring_lane_lib_only.sh -> exit 0 Final independent acceptance after the drive-test CI repair Parent: a99d8f4f89 Tested code tree: 93cbd96bf0 Tested tree: 075ae5161c2b62a96812bc1e3c904a7663b4f2f8 shellcheck --version && actionlint -> exit 0 cargo fmt --all --check -> exit 0 ./scripts/check_unsafe_code_allowances.sh -> exit 0 ./scripts/check_layer_dependencies.sh -> exit 0 ./scripts/check_architecture_migration_rules.sh -> exit 0 ./scripts/check_logging_guardrails.sh -> exit 0 ./scripts/check_error_other_format_ratchet.sh -> exit 0 ./scripts/check_no_tokio_io_uring.sh -> exit 0 ./scripts/check_extension_schema_boundaries.sh -> exit 0 ./scripts/check_body_cache_whitelist.sh -> exit 0 ./scripts/check_s3s_footprint.sh -> exit 0 ./scripts/check_test_count_ratchet.sh -> exit 0 ./scripts/check_fips_wording.sh -> exit 0 ./scripts/check_embedded_secrets.sh -> exit 0 make script-tests -> exit 0 python3 ./scripts/test_object_generation_protocol_model.py -> exit 0 python3 ./scripts/check_test_wiring.py -> exit 0 ./scripts/check_no_planning_docs.sh -> exit 0 ./scripts/check_uring_lane_lib_only.sh -> exit 0 $ LANG=en_US.UTF-8 CARGO_BUILD_JOBS=1 CARGO_INCREMENTAL=0 cargo nextest run --profile ci -p rustfs-s3-types -p rustfs-protocols --features rustfs-protocols/swift --no-fail-fast Finished `test` profile [unoptimized + debuginfo] target(s) in 10m 08s Nextest run ID 5c7ff102-ec6d-479a-86e0-810859b28bbf with nextest profile: ci Starting 543 tests across 8 binaries Summary [ 6.729s] 543 tests run: 543 passed, 0 skipped exit_code=0 The only source change is the raw-byte wire comparator. Cargo manifests and the ratchet baselines remain unchanged from the independently validated parent. The invalid-byte and consumed-request mutations remain recorded above; their implementation changes were restored. * fix: repair integration checks and IPC test readiness Wait for the RPC/locks test sockets to pass the existing owner and mode checks inside the original 5s readiness window and 10ms poll interval. Use the existing admin S3 carrier for the new recovery and quota error calls. Regenerate the test-count baseline for main's 118 existing tests across seven crates. Pair the private pagination helper's raw-scan inputs and remove the redundant Copy clone from the retirement event. Keep the original assertions, responses, authorization, locks and storage operations. The pagination body and raw-scan expressions at all five calls are byte-identical; reinserting only the Copy clone restores the entire base heal file. All ignored baseline values and s3s counts stay unchanged. Validation base: c47432e353 Validation tree: f081084bc5 Patch SHA-256: 2dcf0028691bfb2772edfb38e262829b04f4151832dcce823c9dd9f7434b52de Trace production-prefix SHA-256: fd994f218a21429d62866f4efb8eee1ee57b58f4a51865aa35d05a54ef1ccb45 All commands used LANG=en_US.UTF-8, CARGO_BUILD_JOBS=1, CARGO_INCREMENTAL=0 and CARGO_NET_OFFLINE=true in one exclusive target. Every gate retained the same tree and six source hashes before and after. 54 Cargo.toml files parse without duplicate keys. cargo clippy -p rustfs --all-targets -- -D warnings Finished `dev` profile [unoptimized + debuginfo] target(s) in 12m 50s Exit: 0. Wall time: 770.855s. The original 19 quick-check steps: shellcheck --version && actionlint -> exit 0 cargo fmt --all --check -> exit 0 ./scripts/check_unsafe_code_allowances.sh -> exit 0 ./scripts/check_layer_dependencies.sh -> exit 0 ./scripts/check_architecture_migration_rules.sh -> exit 0 ./scripts/check_logging_guardrails.sh -> exit 0 ./scripts/check_error_other_format_ratchet.sh -> exit 0 ./scripts/check_no_tokio_io_uring.sh -> exit 0 ./scripts/check_extension_schema_boundaries.sh -> exit 0 ./scripts/check_body_cache_whitelist.sh -> exit 0 ./scripts/check_s3s_footprint.sh -> exit 0 ./scripts/check_test_count_ratchet.sh -> exit 0 ./scripts/check_fips_wording.sh -> exit 0 ./scripts/check_embedded_secrets.sh -> exit 0 make script-tests -> exit 0 python3 ./scripts/test_object_generation_protocol_model.py -> exit 0 python3 ./scripts/check_test_wiring.py -> exit 0 ./scripts/check_no_planning_docs.sh -> exit 0 ./scripts/check_uring_lane_lib_only.sh -> exit 0 Wall time: 286.337s. cargo nextest run --profile ci -p rustfs --no-fail-fast Summary [ 217.017s] 5382 tests run: 5382 passed (4 leaky), 40 skipped Exit: 0. Wall time: 1324.414s. cargo nextest run --profile ci -p rustfs-s3-types -p rustfs-protocols --features rustfs-protocols/swift --no-fail-fast Summary [ 2.189s] 543 tests run: 543 passed, 0 skipped Exit: 0. Wall time: 123.192s. cargo nextest run --profile ci -p rustfs-ecstore list_objects_paginate --no-fail-fast Summary [ 0.170s] 8 tests run: 8 passed, 5921 skipped Exit: 0. Wall time: 261.324s. cargo nextest run --profile ci -p rustfs-heal durable_mrf_anchor_retires_only_after_exact_bucket_generation_is_deleted --no-fail-fast Summary [ 0.805s] 1 test run: 1 passed, 835 skipped Exit: 0. Wall time: 199.801s. Baseline regeneration: ./scripts/check_test_count_ratchet.sh --update wrote .config/test-count-baseline.txt: 52 crates, 21340 tests, 164 ignored Test-only increases: ecstore48, heal3, iam14, io-metrics2, obs3, policy14, rustfs34. Existing ignored values and all other rows are byte-identical. s3s footprint OK: files importing s3s is 195 (baseline files=195) s3s footprint OK: s3s s3_error! lines is 1584 (baseline s3_error_lines=1584) s3s footprint OK: ecstore files referencing s3s is 36 (baseline ecstore_files=36) Historical runtime red controls on f622 (same retained two assertions): cargo nextest run --profile ci -p rustfs --lib -E 'test(=connect::diagnostics::trace_runtime::tests::local_top_rpc_reads_service_http_completions_and_signs_offline) | test(=connect::diagnostics::trace_runtime::tests::local_top_locks_reads_service_manager_and_signs_offline)' --no-fail-fast Original path readiness with a temporary 500ms bind/seal gap: Summary [ 0.863s] 2 tests run: 0 passed, 2 failed, 5113 skipped Exit: 100. Sealed readiness in that same gap: Summary [ 5.621s] 2 tests run: 2 passed, 5113 skipped Exit: 0. Compiled M1: replace the helper predicate with path.exists(): Summary [ 0.870s] 2 tests run: 0 passed, 2 failed, 5113 skipped Exit: 100. Restore the sealed predicate byte-for-byte: Summary [ 5.640s] 2 tests run: 2 passed, 5113 skipped Exit: 0. Both reds observe mode755 with owner_matches=true and fail at the original RPC/locks unwraps with StateSecurity. M1 failure locations: trace_runtime.rs 2547:29 and 2713:29. No new assertions were introduced. cargo nextest run --profile ci -p rustfs --lib -E 'test(connect::diagnostics::trace_runtime::tests::)' --no-fail-fast Summary [ 20.748s] 28 tests run: 28 passed (1 leaky), 5087 skipped Exit: 0. This security sweep is historical; the full final unit gate above covers the actual c474 candidate. The first disk-stopped compile ran no tests and is excluded from red credit. All temporary seal instrumentation is removed from the final source. Final unit cleanup markers: 4 leaky, 40 original skips (exit0). The ld __eh_frame linker warning also appears verbatim in the original f622 run and the earlier c474 run; there are no new source warnings. * fix: remove redundant pending closure in disk test Pass the capture-free pending future factory as a function item in the existing slow-operation test. This fixes the redundant_closure error at disk_store.rs:2914 that stopped strict Clippy before the feature tests could run. The single-line test edit preserves the factory's result, warning and timeout behavior. Reversing the replacement reproduces the parent file byte for byte. All 220 assertion lines and the production prefix are unchanged; no tests, assertions, guards, profiles or baselines were added or relaxed. Original Linux CI evidence (run 37888552268 at 53245be9f): - RIO job113685483992: strict Clippy exit101, redundant closure at disk_store.rs:2914; feature tests skipped. - Workspace Test and Lint job113685484058: same strict Clippy exit101; nextest and doc tests skipped. These are lint failures, not runtime mutation kills or known flakes. Native macOS arm64 validation on tree7a5bf47e7533b29479bc85207416102f1337c566, LANG=en_US.UTF-8, one build job, CARGO_INCREMENTAL=0 and offline dependencies: shellcheck --version && actionlint -> exit 0 cargo fmt --all --check -> exit 0 ./scripts/check_unsafe_code_allowances.sh -> exit 0 ./scripts/check_layer_dependencies.sh -> exit 0 ./scripts/check_architecture_migration_rules.sh -> exit 0 ./scripts/check_logging_guardrails.sh -> exit 0 ./scripts/check_error_other_format_ratchet.sh -> exit 0 ./scripts/check_no_tokio_io_uring.sh -> exit 0 ./scripts/check_extension_schema_boundaries.sh -> exit 0 ./scripts/check_body_cache_whitelist.sh -> exit 0 ./scripts/check_s3s_footprint.sh -> exit 0 ./scripts/check_test_count_ratchet.sh -> exit 0 ./scripts/check_fips_wording.sh -> exit 0 ./scripts/check_embedded_secrets.sh -> exit 0 make script-tests -> exit 0 python3 ./scripts/test_object_generation_protocol_model.py -> exit 0 python3 ./scripts/check_test_wiring.py -> exit 0 ./scripts/check_no_planning_docs.sh -> exit 0 ./scripts/check_uring_lane_lib_only.sh -> exit 0 All 19 checks: exit0, 312.161s. $ cargo clippy -p rustfs -p rustfs-ecstore --all-targets --features rio-v2 -- -D warnings Finished `dev` profile [unoptimized + debuginfo] target(s) in 20m 11s exit0, 1213.210s total. $ cargo nextest run --profile ci -p rustfs-ecstore --features rio-v2 slow_disk_operation_warns_once_while_pending_and_preserves_result --no-fail-fast Finished `test` profile [unoptimized + debuginfo] target(s) in 4m 34s PASS [0.026s] rustfs-ecstore disk::disk_store::tests::slow_disk_operation_warns_once_while_pending_and_preserves_result Summary [ 0.170s] 1 test run: 1 passed, 5943 skipped exit0, 281.920s total; 5943 existing filter skips. warning: linker stderr: ld: __eh_frame section too large (max 16MB) to encode dwarf unwind offsets in compact unwind table, performance of exception handling might be affected This linker warning is retained; flags, profiles and allowances are unchanged. 54 tracked Cargo.toml files parsed without duplicate keys. git diff --check passed. No new assertions were introduced, so no new runtime mutation is claimed. Native compilation and the existing helper test do not substitute for actual Linux RIO execution; the new integration CI run still needs to recover before the RPC merge. Preservation and output hashes: parent file SHA256: 8e59ee745e3aaaa22848b943d3b10f6353b515956282cde1f3af2c394200c9ac final file SHA256: b69ba3237061b161448ae8b62b615885d5770b98a5b4a2479bcfbd70f1bd1f17 unchanged production prefix SHA256: 7903a6e288252c420231242d7b58a25fa2a7140ea3dd430cbe7f0dde11369dcb RIO Linux failure log SHA256:421addb9868fd88e4109a922311e54f91f0dfd70862f8ba7fcaba193482ed90b Workspace Linux failure log SHA256:e0aea8d0f014c022252a9534968e083c98cfc1fd760da445ecd04ec3f522e26a 19 output SHA256:35a54f28d707956ed1edb29d492c4dfe8a9fe753d103b111b5a2f1b5c41c2b98 RIO Clippy output SHA256:d6046e3868f2b3992fe11cdd04e9e98eb50a87d1234d9a70dbfeb154ff61638d slow-operation output SHA256:605b7b258d2fd09c6ef2ab2ffc90d433d0509480f1e45125d20c823759e83608 Refs rustfs/backlog#2734. * test(ecstore): isolate tier-free recovery fixture startup The fixture constructor can take the production recovery lock and wait for 60 seconds before this test installs its queue and seeds metadata. Disable its background recovery only while creating this fixture, then restore the environment before calling the existing production recovery entrypoint. Only the existing test setup changes. Its production entrypoint, default lock parameters, original assertions, 30-second timeout, cancellation and cleanup remain intact. No tests or assertions were added. Production and store/init.rs are byte-identical to the parent. Independent acceptance ran on merge tree 8ec358cb42ba9ebb3bcfeafcef048a3f0366355b; the runner header reports parent HEAD, while before/after index and source hashes confirm this candidate. Worker task branch d8055766b8 has the same tree. Every command used LANG=en_US.UTF-8. Task compilation also used CARGO_BUILD_JOBS=1, CARGO_INCREMENTAL=0, CARGO_NET_OFFLINE=true, an independent target, and empty RUSTC_WRAPPER/RUSTC_WORKSPACE_WRAPPER to keep compilers in its owned process group without changing repository configuration. Command: shellcheck --version && actionlint Exit 0; observed output excerpt: version: 0.11.0 Log SHA-256: fc9eec16b4191d058a295813ca9afbe9d2b37b59975da860170726be162a3924 Command: cargo fmt --all --check Exit 0; observed output excerpt: (no stdout or stderr) Log SHA-256: e3b0c44298fc1c149afbf4c8996fb92427ae41e4649b934ca495991b7852b855 Command: ./scripts/check_unsafe_code_allowances.sh Exit 0; observed output excerpt: (no stdout or stderr) Log SHA-256: e3b0c44298fc1c149afbf4c8996fb92427ae41e4649b934ca495991b7852b855 Command: ./scripts/check_layer_dependencies.sh Exit 0; observed output excerpt: Layer dependency guard passed (no new reverse dependencies/cycles). Log SHA-256: 69c6cb1896c0586b28d1167380d3260d99d8017a4265c8df5a2348f289e93a12 Command: ./scripts/check_architecture_migration_rules.sh Exit 0; observed output excerpt: Architecture migration rules passed. Log SHA-256: 28790e70d070607ed4de0704f292a65030be92ec224ef106d4b72fd8296f896e Command: ./scripts/check_logging_guardrails.sh Exit 0; observed output excerpt: ✅ Logging guardrails check passed Log SHA-256: 7adf41abff8ed8f7617decf86498888ddde78937099dd9ce47f6d359dfc2d317 Command: ./scripts/check_error_other_format_ratchet.sh Exit 0; observed output excerpt: error(format!) ratchet passed (total call sites: 579). Log SHA-256: 66172aef18f150ba0605af122e26ffcf91f6dd86f43e38b05e2e982ce8d783ee Command: ./scripts/check_no_tokio_io_uring.sh Exit 0; observed output excerpt: (no stdout or stderr) Log SHA-256: e3b0c44298fc1c149afbf4c8996fb92427ae41e4649b934ca495991b7852b855 Command: ./scripts/check_extension_schema_boundaries.sh Exit 0; observed output excerpt: Extension schema boundaries passed. Log SHA-256: f13d0acd0e2ce5d9815241b12b4dab7d01cc11b9b65bcf6dfb7812c878532a51 Command: ./scripts/check_body_cache_whitelist.sh Exit 0; observed output excerpt: Body-cache whitelist guard passed. Log SHA-256: 6b828847b0383950ee6eef9cd846d0c322c945e47efd4c7d1ad881178810d6bb Command: ./scripts/check_s3s_footprint.sh Exit 0; observed output excerpt: s3s footprint OK: files importing s3s is 195 (baseline files=195) s3s footprint OK: s3s s3_error! lines is 1584 (baseline s3_error_lines=1584) s3s footprint OK: ecstore files referencing s3s is 36 (baseline ecstore_files=36) ✅ s3s footprint baseline check passed Log SHA-256: 47f53d04ef645618563fe1f22abc64667f19078f1974feedcca46e4f2c016844 Command: ./scripts/check_test_count_ratchet.sh Exit 0; observed output excerpt: test-count ratchet OK: 52 crates, 21340 tests, 164 ignored match .config/test-count-baseline.txt Log SHA-256: 618a5e330e933de0c6ff484546df401457024a88bb5986d11d588794e33dab60 Command: ./scripts/check_fips_wording.sh Exit 0; observed output excerpt: FIPS wording guard passed (README.md and CHANGELOG.md contain no forbidden claims). KMS wording guard passed (crates/kms claims no Vault KV2 Transit wrapping). Log SHA-256: 99221cc9979c480ec4f46f50d76c52b18c72fe499f12e15e4f4a22a1b8bfb140 Command: ./scripts/check_embedded_secrets.sh Exit 0; observed output excerpt: Embedded secret guard passed (no private key material or provider credential literal in the tree). Log SHA-256: 437a3ceba32094d2901889965ad191e411aad32cdb18bf0b8659e818011c406b Command: make script-tests Exit 0; observed output excerpt: test_cli_requires_the_repository_default_branch (__main__.SelfTests.test_cli_requires_the_repository_default_branch) ... ok test_config_rejects_duplicate_and_invalid_entries (__main__.SelfTests.test_config_rejects_duplicate_and_invalid_entries) ... ok test_freshness_boundaries (__main__.SelfTests.test_freshness_boundaries) ... ok test_grace_requires_two_successful_queries_with_no_history (__main__.SelfTests.test_grace_requires_two_successful_queries_with_no_history) ... ok test_invalid_api_evidence_fails_closed (__main__.SelfTests.test_invalid_api_evidence_fails_closed) ... ok test_report_retains_every_workflow (__main__.SelfTests.test_report_retains_every_workflow) ... ok test_requests_filter_schedule_default_branch_and_success_on_server (__main__.SelfTests.test_requests_filter_schedule_default_branch_and_success_on_server) ... ok Ran 10 tests in 0.042s OK Ran 30 tests in 20.627s OK Ran 10 tests in 2.268s OK Ran 33 tests in 0.127s OK Ran 7 tests in 0.002s OK Ran 7 tests in 0.001s OK Ran 4 tests in 0.108s OK Ran 3 tests in 0.564s OK PASS: cold stampede script self-test Log SHA-256: 75fda2d069a373c63667b908baaf99e6591aab9099a739b0c4c22ceb6da8d68c Command: python3 ./scripts/test_object_generation_protocol_model.py Exit 0; observed output excerpt: test_an_unavailable_decision_quorum_cannot_choose_a_candidate (__main__.ObjectGenerationProtocolModel.test_an_unavailable_decision_quorum_cannot_choose_a_candidate) ... ok test_counter_allocation_requires_persistence_and_is_consumed_once (__main__.ObjectGenerationProtocolModel.test_counter_allocation_requires_persistence_and_is_consumed_once) ... ok test_counter_restart_burns_persisted_allocations_before_or_after_issue (__main__.ObjectGenerationProtocolModel.test_counter_restart_burns_persisted_allocations_before_or_after_issue) ... ok test_crash_after_promise_before_reply_does_not_restore_an_older_ballot (__main__.ObjectGenerationProtocolModel.test_crash_after_promise_before_reply_does_not_restore_an_older_ballot) ... ok test_duplicate_or_unconfigured_voters_cannot_form_a_quorum (__main__.ObjectGenerationProtocolModel.test_duplicate_or_unconfigured_voters_cannot_form_a_quorum) ... ok test_empty_or_unobserved_minority_slot_can_retry_without_allocating_another_revision (__main__.ObjectGenerationProtocolModel.test_empty_or_unobserved_minority_slot_can_retry_without_allocating_another_revision) ... ok test_equal_ballot_cannot_accept_different_bytes_and_lower_ballot_is_rejected (__main__.ObjectGenerationProtocolModel.test_equal_ballot_cannot_accept_different_bytes_and_lower_ballot_is_rejected) ... ok test_forgetting_tombstone_authority_allows_an_aba_create (__main__.ObjectGenerationProtocolModel.test_forgetting_tombstone_authority_allows_an_aba_create) ... ok test_issuing_before_persistence_reuses_a_ballot_after_restart (__main__.ObjectGenerationProtocolModel.test_issuing_before_persistence_reuses_a_ballot_after_restart) ... ok test_lost_ack_and_restart_preserve_the_chosen_operation (__main__.ObjectGenerationProtocolModel.test_lost_ack_and_restart_preserve_the_chosen_operation) ... ok test_minority_accepted_work_is_adopted_and_finished (__main__.ObjectGenerationProtocolModel.test_minority_accepted_work_is_adopted_and_finished) ... ok test_omitting_highest_accepted_adoption_has_a_double_choice_counterexample (__main__.ObjectGenerationProtocolModel.test_omitting_highest_accepted_adoption_has_a_double_choice_counterexample) ... ok test_quorum_choice_does_not_require_an_uncontacted_disk_to_learn_it (__main__.ObjectGenerationProtocolModel.test_quorum_choice_does_not_require_an_uncontacted_disk_to_learn_it) ... ok test_recovery_failure_or_stale_expected_head_does_not_allocate_a_successor (__main__.ObjectGenerationProtocolModel.test_recovery_failure_or_stale_expected_head_does_not_allocate_a_successor) ... ok test_replacing_voters_with_empty_state_can_choose_a_conflicting_value (__main__.ObjectGenerationProtocolModel.test_replacing_voters_with_empty_state_can_choose_a_conflicting_value) ... ok test_skipping_predecessor_recovery_chooses_a_broken_revision_chain (__main__.ObjectGenerationProtocolModel.test_skipping_predecessor_recovery_chooses_a_broken_revision_chain) ... ok test_stale_publication_rollback_and_gc_cannot_change_the_winner (__main__.ObjectGenerationProtocolModel.test_stale_publication_rollback_and_gc_cannot_change_the_winner) ... ok test_superseded_operation_identity_cannot_be_reused_for_different_input (__main__.ObjectGenerationProtocolModel.test_superseded_operation_identity_cannot_be_reused_for_different_input) ... ok test_superseded_operation_retry_returns_its_original_successor (__main__.ObjectGenerationProtocolModel.test_superseded_operation_retry_returns_its_original_successor) ... ok test_tombstone_and_null_marker_payloads_do_not_reset_authority_revision (__main__.ObjectGenerationProtocolModel.test_tombstone_and_null_marker_payloads_do_not_reset_authority_revision) ... ok test_tombstone_rejects_a_delayed_never_present_create (__main__.ObjectGenerationProtocolModel.test_tombstone_rejects_a_delayed_never_present_create) ... ok test_unlearned_predecessor_is_recovered_before_the_second_slot (__main__.ObjectGenerationProtocolModel.test_unlearned_predecessor_is_recovered_before_the_second_slot) ... ok Ran 24 tests in 0.929s OK Log SHA-256: 0b4be926efb0609d497ae58930340153f01957b606206cbbdb3e0d47eb29d3a2 Command: python3 ./scripts/check_test_wiring.py Exit 0; observed output excerpt: OK: e2e modules, runner selection, fuzz matrices, profiles, scheduled alerts, and Quick Checks are wired Log SHA-256: 21f676f458451525535ab0ef73d6a9c0d9774e16d76a4d19e396ad34936dd39c Command: ./scripts/check_no_planning_docs.sh Exit 0; observed output excerpt: (no stdout or stderr) Log SHA-256: e3b0c44298fc1c149afbf4c8996fb92427ae41e4649b934ca495991b7852b855 Command: ./scripts/check_uring_lane_lib_only.sh Exit 0; observed output excerpt: OK: no 'uring_'-matching test functions under crates/ecstore/tests; --lib is safe Log SHA-256: 43f3f851e27ec2ab9e34d6b05fefdede38af992699c473904bfe6f53a46d52e3 Command: cargo nextest run --profile ci -p rustfs-ecstore --lib --features rio-v2 -E 'test(=bucket::lifecycle::bucket_lifecycle_ops::tests::tier_free_version_recovery_production_entrypoint_enqueues_seeded_item)' --no-fail-fast Exit 0; wall 234.399s. Observed output: Finished `test` profile [unoptimized + debuginfo] target(s) in 3m 49s Starting 1 test across 1 binary (5871 tests skipped) SETUP PASS [ 0.006s] ecstore-base-stack: sh -c 'echo RUST_MIN_STACK=4194304 >> "$NEXTEST_ENV"' PASS [ 0.440s] rustfs-ecstore bucket::lifecycle::bucket_lifecycle_ops::tests::tier_free_version_recovery_production_entrypoint_enqueues_seeded_item Summary [ 0.586s] 1 test run: 1 passed, 5871 skipped Log SHA-256: 34e973f86e8dc2102c1a0e483e0ac802ce7df3d1b40a603932f62c0805c4de27 Command: cargo nextest run --profile ci -p rustfs-ecstore --lib --features rio-v2 -E 'test(=bucket::lifecycle::bucket_lifecycle_ops::tests::tier_free_version_recovery_eight_coordinators_hold_one_leader_and_take_over) | test(=store::bucket::tests::creation_commit_backfill_runs_under_the_config_write_fence) | test(=store::bucket::tests::orphaned_bucket_recovery_requires_every_volume_absent_and_is_retryable)' --no-fail-fast Exit 0; wall 4.010s. Observed output: Finished `test` profile [unoptimized + debuginfo] target(s) in 0.43s Starting 3 tests across 1 binary (5869 tests skipped) SETUP PASS [ 0.005s] ecstore-base-stack: sh -c 'echo RUST_MIN_STACK=4194304 >> "$NEXTEST_ENV"' PASS [ 0.068s] rustfs-ecstore bucket::lifecycle::bucket_lifecycle_ops::tests::tier_free_version_recovery_eight_coordinators_hold_one_leader_and_take_over PASS [ 0.637s] rustfs-ecstore store::bucket::tests::creation_commit_backfill_runs_under_the_config_write_fence PASS [ 0.846s] rustfs-ecstore store::bucket::tests::orphaned_bucket_recovery_requires_every_volume_absent_and_is_retryable Summary [ 0.987s] 3 tests run: 3 passed, 5869 skipped Log SHA-256: b5da2a927ff18ca9aa217762f7186a6d4371bd9e8feedd7b77940e10b64dffac Command: cargo clippy -p rustfs -p rustfs-ecstore --all-targets --features rio-v2 -- -D warnings Exit 0; wall 1084.508s. Observed output: Finished `dev` profile [unoptimized + debuginfo] target(s) in 18m 03s Log SHA-256: 28f98be7970faaa71cd118da1da4e6e2e3d23b9dda7e6449eaa7a37e741260b5 Compiled mutation command (unchanged for each red and restored run): cargo nextest run --profile ci -p rustfs-ecstore --lib --features rio-v2 -E 'test(=bucket::lifecycle::bucket_lifecycle_ops::tests::tier_free_version_recovery_production_entrypoint_enqueues_seeded_item)' --no-fail-fast M1: reverse only this test-local constructor flag from false to true. The mutant compiled, then failed the original recovered-item queue assertion after 30.481s. Restoring the entire observed candidate passed in 0.460s. Red log 55bf3b1535529ac0bcfb8071f9b56f9f40631007a85c315367e7e93786d65293; restored log 204d5fbe4574a7917f5e740a795d864fe28a7e7fb5e33d27b0a9cf68e2f17f52. M2: replace only the actual awaited enqueue with false, consuming its original inputs. The mutant compiled, then failed the same original queue assertion after 30.516s. Restoring the entire observed candidate passed in 0.457s. Red log 89186330413215e231e2b06bf013cc31097ea8f4a106d91421a36d6c8986a7fd; restored log e848b42d2302401afa5dbcd9e37bf76d2e56a96ab6de5a488513b997f53c5e9b. All temporary observation and mutation code was removed before final acceptance. Historical baseline and diagnostic failures remain recorded; they are not mutation credit. The separate Object Lock intent-retry diagnostic still fails, and these native macOS checks do not establish Linux CI recovery. * refactor(rpc): serve internode rpc bodies with the owned Body (#8441) * test(rpc): pin the wire bytes of every internode rpc response body rustfs/backlog#2747 batch 4, step 1 of 2, on the s3s bodies the rpc paths serve today. Each response body builder in storage/rpc/http_service.rs is served through hyper's HTTP/1.1 server and the raw bytes the peer reads are compared with goldens captured from this commit: status, content-length or chunked framing, chunk boundaries, the terminating chunk, HEAD, and the cut connection a failure after data produces. The capability responses carry a per-process proof, so their length header and payload are checked against the body they encode instead. The read_file_stream response builder moves into read_file_stream_response so the test reaches the same code the handler runs; behaviour is unchanged. * refactor(rpc): serve internode rpc bodies with the owned Body rustfs/backlog#2747 batch 4, step 2 of 2. The internode rpc HTTP paths build their responses with rustfs_s3_types::Body: Body::from_stream replaces Body::from(StreamingBlob::wrap(..)) at the six stream sites, and the in-memory and empty bodies keep their constructors. InternodeRpcService converts its responses into the wrapped service's response body through From<Body>, so the server keeps composing it with the s3s service as before. The wire-byte goldens from the previous commit pass unchanged; storage/rpc no longer names s3s. * fix(rpc): simplify wire test response builder type Name the boxed response builder used by the capability wire test so all-target Clippy can check it without a type-complexity warning. Validation (LANG=en_US.UTF-8, CARGO_INCREMENTAL=0): - cargo clippy -p rustfs --all-targets -- -D warnings: exit 0. - cargo nextest run --profile ci -p rustfs --lib -E 'test(/storage::rpc::/)': 279 tests run, 279 passed, 4842 skipped. - All 19 quick-check commands, including script-tests, both Python scripts and cargo fmt --all --check: exit 0. - Footprint baseline regenerated: files=194, s3_error_lines=1584, ecstore_files=36; test-count baseline regenerated: 52 crates, 21228 tests, 164 ignored. - All 54 Cargo.toml files parse without duplicate keys. No assertion was added or changed by this lint fix. R1-R8 are retained in the existing PR verification record. * fix(ecstore): resume fenced Object Lock creation intents * fix(ecstore): bound creation retry futures on the stack * fix(ecstore): require native proof for creation retries * fix(ecstore): bound bucket creation stack usage (#8466) * fix(ecstore): retain creation fences through rollback Keep lifecycle, metadata-transaction, and namespace guards around rollback and metadata cleanup. Preserve the original publication error when a guard is lost. Build the timestamp retry fixture through the real interrupted producer so it checks a valid pending creation. Validated source tree: 4e8e207999. Two independent source reviews found no blocking regression. Validation (LANG=en_US.UTF-8): - bash -c 'shellcheck --version && actionlint' — exit 0 - cargo fmt --all --check — exit 0 - ./scripts/check_unsafe_code_allowances.sh — exit 0 - ./scripts/check_layer_dependencies.sh — exit 0 - ./scripts/check_architecture_migration_rules.sh — exit 0 - ./scripts/check_logging_guardrails.sh — exit 0 - ./scripts/check_error_other_format_ratchet.sh — exit 0 - ./scripts/check_no_tokio_io_uring.sh — exit 0 - ./scripts/check_extension_schema_boundaries.sh — exit 0 - ./scripts/check_body_cache_whitelist.sh — exit 0 - ./scripts/check_s3s_footprint.sh — exit 0 - ./scripts/check_test_count_ratchet.sh — exit 0 - ./scripts/check_fips_wording.sh — exit 0 - ./scripts/check_embedded_secrets.sh — exit 0 - make script-tests — exit 0 - python3 ./scripts/test_object_generation_protocol_model.py — exit 0 - python3 ./scripts/check_test_wiring.py — exit 0 - ./scripts/check_no_planning_docs.sh — exit 0 - ./scripts/check_uring_lane_lib_only.sh — exit 0 - cargo nextest run --profile ci -p rustfs-ecstore --lib --locked --no-fail-fast — 5894 passed, 0 skipped - cargo check -p rustfs-ecstore --all-targets --locked — exit 0 - cargo nextest run --profile ci -p rustfs --lib --locked --no-fail-fast -E 'test(=admin::handlers::table_catalog::tests::cold_strong_registration_completes_while_other_bucket_backup_waits) | test(=admin::handlers::table_catalog::tests::commit_backend_borrowed_put_preserves_owner_authorization_and_precondition) | test(=admin::handlers::table_catalog::tests::disable_table_bucket_preserves_objects_and_can_be_reenabled) | test(=admin::handlers::table_catalog::tests::load_table_handler_negotiates_vended_credentials_without_breaking_metadata_only_callers) | test(=admin::handlers::table_catalog::tests::namespace_property_handler_updates_object_backed_catalog_and_maps_errors) | test(=app::object::get::tests::execute_get_object_resumes_from_relocated_pool_without_splicing_body) | test(=table_catalog::store::tests::catalog_writer_loss_prevents_recovery_storage_commit)' — 7 passed - cargo nextest run --profile ci -p rustfs --lib --locked --no-fail-fast -E 'test(unsupported_remote_disk_)' — 2 passed Additional fault-injection evidence remains incomplete. Timestamp M1 reached its original assertion; its restore run is being collected. M2–M5 and the three planned rollback faults have not run. An automated risk check interrupted the timestamp proof task; no assertions or validation settings were weakened. * fix(ecstore): box nested bucket rollback state Keep the guarded rollback future behind a heap allocation so bucket creation does not store its nested state inline. The rollback body, all three fence references and pending-proof revocation order are unchanged. The same three original Linux tests overflowed on both e611 CI attempts. Local format, ecstore test compilation and all 19 quick-check steps pass; the original Linux feature lanes still need to validate this candidate. * test(ecstore): isolate rollback guard loss fixtures Run the three rollback metadata-cleanup cases in separate test processes so reused /proc/self/fd aliases from earlier cases cannot invalidate their creation witnesses. Keep the original fixture, guards, assertions and io_uring test selection, and verify that each child runs the intended case successfully. Update the generated test-count baseline for the added macro arm. Validation: cargo check -p rustfs-ecstore --tests; cargo fmt --all --check; all 19 quick-check commands, including make script-tests and the two Python checks. Linux io_uring acceptance and fault controls remain pending. * refactor(replication): move the replication config onto gateway types (#8443) * wip: T1.4 replication family * refactor(replication): move the replication config onto gateway types The replication configuration family leaves s3s: rustfs-replication no longer depends on s3s, and ECStore bucket metadata, the scanner and the rustfs app/admin layers hold rustfs_gateway_types::persistence::PersistedReplicationConfiguration. The persisted XML goes through the gateway persistence codec behind a new BucketConfigXml trait (the 39 byte goldens pass with unchanged fixtures); the legacy s3s edge converts once in storage/ecfs.rs through storage/s3_compat/replication.rs; the scanner data usage cache keeps its MessagePack bytes through a serde mirror pinned by bytes recorded from s3s 5761ddfe. rustfs-gateway-types 0.32.0 is pinned once (rev 772d3385). Refs rustfs/backlog#2745. * test: use S3 serializer for list response tests * fix(ci): restore integration test compilation Gate the diagnostic test import with its feature, initialize the conditional MetadataWriteStore fields explicitly, and decode the scanner replication fixture through its existing cache boundary. Preserve the original fixture and assertions. Validation was run on tree 544acb80c0 at 6003d4b8. LANG=en_US.UTF-8; no source changes follow validation. Quick checks: 19/19 passed. - shellcheck --version && actionlint: exit 0 - cargo fmt --all --check: exit 0 - ./scripts/check_unsafe_code_allowances.sh: exit 0 - ./scripts/check_layer_dependencies.sh: exit 0 - ./scripts/check_architecture_migration_rules.sh: exit 0 - ./scripts/check_logging_guardrails.sh: exit 0 - ./scripts/check_error_other_format_ratchet.sh: exit 0 - ./scripts/check_no_tokio_io_uring.sh: exit 0 - ./scripts/check_extension_schema_boundaries.sh: exit 0 - ./scripts/check_body_cache_whitelist.sh: exit 0 - ./scripts/check_s3s_footprint.sh: exit 0 - ./scripts/check_test_count_ratchet.sh: exit 0 - ./scripts/check_fips_wording.sh: exit 0 - ./scripts/check_embedded_secrets.sh: exit 0 - make script-tests: exit 0 - python3 ./scripts/test_object_generation_protocol_model.py: exit 0 - python3 ./scripts/check_test_wiring.py: exit 0 - ./scripts/check_no_planning_docs.sh: exit 0 - ./scripts/check_uring_lane_lib_only.sh: exit 0 Darwin validation: - cargo clippy --locked -p rustfs-ecstore -p rustfs-scanner --all-targets -- -D warnings: exit 0 - cargo nextest run --locked -p rustfs-ecstore -p rustfs-scanner --lib -E 'test(config::com::tests::save_config_preserves_retryable_system_volume_errors) | test(scanner_io::tests::replication_maintenance_refreshes_compacted_stats_across_cycles_and_retries)' --test-threads 1 --no-fail-fast: exit 0, 2/2 passed - cargo clippy --locked -p rustfs-ecstore -p rustfs-scanner --all-targets --features rustfs-ecstore/diagnostic-read-version-caller,rustfs-scanner/diagnostic-read-version-caller -- -D warnings: exit 101 - cargo nextest run --locked -p rustfs-ecstore -p rustfs-scanner --lib --features rustfs-ecstore/diagnostic-read-version-caller,rustfs-scanner/diagnostic-read-version-caller -E 'test(config::com::tests::save_config_preserves_retryable_system_volume_errors) | test(scanner_io::tests::replication_maintenance_refreshes_compacted_stats_across_cycles_and_retries) | test(cluster::rpc::runtime_sources::tests::) | test(config::com::tests::diagnostic_config_write_retains_origin_and_api_for_results_and_cancellation)' --test-threads 1 --no-fail-fast: exit 0, 14/14 passed Diagnostic strict Clippy remains red on four unchanged production lints: runtime_sources.rs if_same_then_else and three redundant_locals reports in set_disk/core/metadata_read.rs and set_disk/read.rs. Their source ranges and feature wiring match the base; the base was not compiled separately. Linux aarch64 validation (Rust/Clippy 1.99.0, UID 501, jobs=1): - cargo clippy --all-targets -- -D warnings: exit 0; Finished dev profile in 68m 12s - cargo clippy -p rustfs -p rustfs-ecstore --all-targets --features rio-v2 -- -D warnings: exit 0; Finished dev profile in 49m 45s Source HEAD/tree and clean status matched before and after both commands. No compiler or stack overrides were used. protoc 35.1 was installed from its verified ARM64 release; flatc was absent and not invoked by the workspace build scripts. The local environment differs from hosted CI. * fix(ecstore): recover delete batches with missing versions Treat an explicit non-null version as a no-op when it is absent from both recovery sources. Keep rejecting missing null versions, delete-marker mismatches and conflicting local generations. Add returning-disk and partial-receipt regressions, extend journal boundary checks, and regenerate the ecstore test-count baseline. Production faults exercise the 24 added assertion sites and 7 semantic checks: 33 faults failed at the expected sites and passed after full restoration. Validation (LANG=en_US.UTF-8, one build job): cargo nextest run --locked -p rustfs-ecstore --lib -E 'test(delete_recovery) | test(quorum_delete_cleanup_coordinator_binds_single_and_batch) | test(crash_delete_retry_batch_reclaims_inline_and_reserved_external)' --test-threads 1 --no-fail-fast Summary [ 34.580s] 69 tests run: 69 passed, 6058 skipped; exit 0 cargo clippy --locked -p rustfs-ecstore --all-targets -- -D warnings Finished `dev` profile [unoptimized + debuginfo] target(s) in 5m 37s; exit 0 All 19 final Quick Checks completed with exit 0: - shellcheck --version && actionlint - cargo fmt --all --check - ./scripts/check_unsafe_code_allowances.sh - ./scripts/check_layer_dependencies.sh - ./scripts/check_architecture_migration_rules.sh - ./scripts/check_logging_guardrails.sh - ./scripts/check_error_other_format_ratchet.sh - ./scripts/check_no_tokio_io_uring.sh - ./scripts/check_extension_schema_boundaries.sh - ./scripts/check_body_cache_whitelist.sh - ./scripts/check_s3s_footprint.sh - ./scripts/check_test_count_ratchet.sh - ./scripts/check_fips_wording.sh - ./scripts/check_embedded_secrets.sh - make script-tests - python3 ./scripts/test_object_generation_protocol_model.py - python3 ./scripts/check_test_wiring.py - ./scripts/check_no_planning_docs.sh - ./scripts/check_uring_lane_lib_only.sh test-count ratchet OK: 52 crates, 22009 tests, 165 ignored match .config/test-count-baseline.txt PASS: cold stampede script self-test model: states=51023, transitions=167520, terminal=732, chosen_candidates=2 multi-slot round schedules: 640 OK: e2e modules, runner selection, fuzz matrices, profiles, scheduled alerts, and Quick Checks are wired * fix(ecstore): retry tier commit intent lock contention Retry transient namespace lock timeouts within the existing three-attempt CAS budget. Reload and verify the original mutation identity after contention, and preserve the final typed error when attempts are exhausted. Cover save/read contention, exhausted retries, replacement identities and CAS reconciliation without changing the existing timeout or lock policy. Validation: - cargo nextest run --locked -p rustfs-ecstore --features test-util --lib -E 'test(tier_mutation)' --test-threads 1 --no-fail-fast: 52 passed. - cargo nextest run --locked -p rustfs-ecstore --features test-util --lib -E 'test(intent_advance_) | test(=services::tier::tier::tests::caller_cancellation_after_durable_prepare_does_not_hide_the_mutation)' --test-threads 1 --no-fail-fast: 5 passed. - cargo clippy --locked -p rustfs-ecstore --all-targets -- -D warnings: passed. - cargo clippy --locked -p rustfs-ecstore --all-targets --features test-util -- -D warnings: passed. - 27 mutation pairs failed at the intended assertions and passed after restoring the production code. - bash scripts/check_test_count_ratchet.sh --update, then bash scripts/check_test_count_ratchet.sh: 52 crates, 22013 tests, 165 ignored; ecstore increases from 6581 to 6585. * refactor(lifecycle): replace s3s domain types with Gateway DTOs Keep lifecycle rules, restore requests, Object Lock, and scanner cached tags in domain types. Convert at the existing HTTP boundary while preserving legacy XML, scanner cache bytes, date ordering, and tombstones. Linux test validation before the final import-path-only adjustment: cargo nextest run -p rustfs-ecstore bucket_config_goldens 39 passed; 6211 skipped by selector. cargo nextest run --lib -p rustfs -E 'test(/test_zero_rule_lifecycle_tombstone_recognition/) | test(/test_local_lifecycle_staleness_axis_selection/) | test(/test_lifecycle_staleness_preserves_legacy_submillisecond_axis/) | test(/test_merge_incoming_lifecycle/) | test(/test_lifecycle_expiry_statement_matrix/) | test(/test_lifecycle_expiry_subset_xml/) | test(/put_bucket_lifecycle_validation_errors_keep_their_s3_code/) | test(/execute_put_bucket_lifecycle_configuration_rejects_missing_configuration/) | test(/execute_restore_object_rejects_missing_restore_request/) | test(/execute_restore_object_rejects_select_type/) | test(/execute_restore_object_maps_failures_to_typed_s3_errors/) | test(/execute_restore_object_returns_internal_error_when_store_uninitialized/) | test(/restore_request_conversion_preserves_legacy_output_values/) | test(/restore_request_conversion_preserves_presence_and_raw_enums/) | test(/restore_request_conversion_distinguishes_absent_and_empty_containers/)' 25 passed; 5318 skipped by selector. cargo nextest run --profile ci -p rustfs-ecstore -p rustfs-lifecycle -p rustfs-replication -p rustfs-scanner 7872 passed; 21 skipped under the existing CI profile. Previously verified mutation controls: 39 production faults, 41 RED/GREEN pairs. cargo clippy --locked -p rustfs -p rustfs-ecstore -p rustfs-lifecycle -p rustfs-scanner --all-targets -- -D warnings Passed in 40m 45s before the import-path adjustment, then in 1m 59s on the final source with warnings denied. The final adjustment imports the existing helper through the object domain facade; fmt and the architecture check also passed. Part of rustfs/backlog#2745. The remaining ecstore configuration families follow separately. * refactor(admin): move error types to rustfs-s3-types (#8440) * refactor(protocols): move error types to rustfs-s3-types by codemod T1.2 batch 3 (rustfs/backlog#2743): s3select-api, s3select-query and the error items of protocols move off s3s by scripts/codemods, run with --hold crates/protocols/src/sftp/errors.rs. That file stays on s3s: it downcasts the backend error, whose runtime type is still s3s::S3Error. - rustfs/src/protocols/client.rs: the two webdav session methods now return rustfs_s3_types::S3Result and convert the s3s error with From. - codemod: a child module that calls s3_error! through `use super::*` gets its own `use rustfs_s3_types::s3_error;` once its parent imports that macro by name; otherwise it is listed as GLOB. - s3-types: a table test pins code, status and the s3s error the legacy edge renders for every variant a moved call site names. - s3_error_lines 1585 -> 1584 (a lifecycle comment reworded). * refactor(admin): move error types to rustfs-s3-types (WIP, no build) T1.2 batch 4 (rustfs/backlog#2743), work in progress: rustfs check still fails with 16 errors (listed in the PR body). - codemod: --facade PATH treats a crate-local re-export of s3s's error items like s3s for the four names (9 new self-tests, 9 mutants red). - codemod output for rustfs/src/admin and rustfs/src/site_replication (--facade crate::storage_api::site_replication::s3); 6 glob children get an explicit rustfs_s3_types::s3_error import. - admin/router.rs: the Operation trait returns rustfs_s3_types::S3Result. - error.rs: From<ApiError> builds the rustfs-s3-types error; the s3s conversion is derived from it. * fix(admin): convert error types at the s3s boundary Batch 4 of T1.2 left the admin and site_replication modules returning rustfs_s3_types::S3Error while their callers and callees on the storage and auth side still speak s3s::S3Error. Convert at each edge with the existing From impls in rustfs-s3-types (compat_s3s.rs) instead of changing any code, message or status: - admin/auth.rs, admin/handlers/object_zip_download.rs, admin/handlers/table_catalog/mod.rs: map the s3s result returned by check_key_valid / authorize_request / authorize_internal_object_request into the admin error type with .map_err(Into::into). - storage/ecfs.rs: the s3s-facing side maps the admin site_replication gate results the other way. - admin/handlers/pools.rs: ApiError.code stays s3s::S3ErrorCode, so the rustfs code is converted with .into() when the struct is built, in production and in the two tests that build or compare it. - admin/handlers/heal.rs, admin/runtime_sources.rs: the router returns s3s errors (S3Route impl); the expected rustfs code is converted with .into() before the comparison. - admin/handlers/table_catalog/mod.rs: iceberg_rest_error takes &'static str so the Cow<'static, str> custom code borrows it; every caller already passes an ICEBERG_ERROR_* constant or a &'static str. - admin/handlers/quota.rs: drop a now same-type .into() after s3_error!. - storage_api.rs, site_replication/mod.rs: drop re-exports and an import that no longer have users (clippy -D warnings). cargo check -p rustfs --all-targets and cargo clippy -p rustfs --all-targets -- -D warnings are clean; the s3s footprint baseline is unchanged at files=172, s3_error_lines=303, ecstore_files=28. Refs rustfs/backlog#2743, rustfs/backlog#2734. * refactor(ecstore): move bucket configs onto gateway persisted types (#8709) * refactor(ecstore): move bucket configs onto gateway persisted types Finish the ECStore batch of T1.4 (rustfs/backlog#2745). The remaining eleven bucket configuration families (Website, CORS, Tagging, Versioning, ObjectLock, Encryption, Notification, Logging, Accelerate, RequestPayment, PublicAccessBlock) now read and write their stored XML through the rustfs-gateway-types persistence and cors_tagging codecs instead of the s3s codec, so the legacy s3s bucket-config codec and its s3gate-metadata-xml compat marker are removed. - ecstore: BucketConfigXml is implemented by the gateway codec for every family; metadata, metadata_sys, versioning, object lock, replication and lifecycle consumers take the persisted types. Region moves to rustfs_s3_types::Region, and the ETag conversion to s3s moves to the rustfs edge with its tests. Only warm_backend.rs and warm_backend_s3.rs still reference s3s, held for T1.5 (rustfs/backlog#2746). - scanner: object lock cache, item actions and data usage use the persisted types. - rustfs: the legacy S3 edge converts each family between s3s DTOs and the persisted types in storage/storage_api.rs by writing one codec's XML and reading it with the other; Tagging converts field by field. - Bump the rustfs-gateway-types pin to 21283e87. The 39 persisted-byte goldens are unchanged and pass, so a document the old codec wrote reads back and re-encodes to the same bytes. s3s footprint: files 181 -> 146, ecstore_files 28 -> 2, s3_error_lines unchanged. * test(app): prepare the PutBucketEncryption config from the request The uninitialized-store test passed an empty persisted configuration, which the use case now validates before it reaches the store check, so it answered MalformedXML instead of InternalError. Convert the request's configuration the way the S3 edge does. * style(app): format the PutBucketEncryption test * refactor(s3-client): move remote error codes and DTO enums off s3s (#8783) Part of T1.5 (rustfs/backlog#2746), batch 1. * refactor(s3select): run queries on gateway DTO request shapes (#8794) * refactor(s3select): run queries on an engine request built from gateway DTO shapes Part of T1.5 (rustfs/backlog#2746), batch 2. * fix(s3select): keep the select request conversion in the app layer The layer guard forbids app -> storage; the conversion moves from storage/s3_compat/select.rs to app/select_request.rs unchanged. * fix(heal): retry transient metadata conflicts (#8812) * fix(heal): retry transient metadata conflicts * chore: update heal test count baseline * test: reserve nextest capacity for progress probes (#8818) * test(e2e): use a non-default region for the differential negative control Since main's region-aware CreateBucket and GetBucketLocation (#8739, #8743), us-east-1 answers an empty LocationConstraint and 200 for an owned re-create, so the control no longer reported its own region. Also refresh the e2e-full linux selection from the CI listing (682 tests). * refactor(protocols): move StorageBackend to gateway types (#8825) * refactor(protocols): move storage backend to gateway types * test: update e2e test count baseline * fix: preserve copy source keys and unknown lengths * test: update protocols test count baseline * fix(protocols): preserve typed SFTP backend errors * test(protocols): use gateway list input in policy tests * fix(protocols): clear clippy failures after T1.5 merge (#8831) fix(protocols): clear clippy failures after merge * fix(ecstore): finalize restore metadata after local copy-back (#8836) * fix(ecstore): finalize restore metadata after local copy-back * test(ecstore): ratchet added restore finalization cases * fix(ecstore): keep restore copyback compatible with older peers * fix(ecstore): gate restore copyback on fleet compatibility * fix(ecstore): retain transition identity during restore copyback * fix(ecstore): route committed restore copyback locally * fix restore copyback cleanup * fix(ecstore): honor restore probe test fallback * refactor(storage): migrate option errors to domain types (#8839) * refactor(s3): migrate private multipart validation errors (#8844) * refactor(auth): derive policy conditions from AuthzFacts (#8442) * refactor(auth): extract authorization facts * test(e2e): update smoke selection baseline * test(e2e): update full selection baseline * refactor(admin): migrate health and prefix usage to owned requests (#8887) * refactor(admin): add owned health router boundary * refactor(admin): migrate prefix usage to owned request * refactor(admin): complete the owned admin request boundary (#8949) * refactor(admin): complete the owned admin request boundary Every type implementing the owned admin `Operation` now serves the legacy router through one blanket impl that converts the s3s request and response at that single edge, so handlers register with `AdminOperation(&Handler {})` and need no adapter statics; `LegacyAdminAdapter` is removed. `AdminRequest` gains the accessors handlers read today (`remote_addr`, `client_info`, `request_context`, `req_info`, `req_info_mut`, `install_req_info`, `envelope_mut`, `take_body`), the request helpers gain envelope forms (`authenticate_envelope_credentials`, `object_store_from_envelope`, the table-catalog `*_from_envelope` helpers, `KmsAdminAudit::from_envelope`, `AccountAuditContext::from_envelope`, `CallerIdentity::resolve_envelope`), `read_compatible_admin_body` accepts either body type, and the owned `Body` gains `store_all_limited` and `Default`. `admin::operation::test_support::TestRequest` builds owned requests for handler tests. Unused accessors carry a per-method dead-code expectation that each migrating handler batch removes. No handler behavior changes; the remaining migrations follow in batches. * chore(admin): add the T1.12 admin handler codemod The script rewrites an admin handler file from the s3s request and response types to the owned boundary (imports, signatures, request field and helper calls, body access, response construction, and test request literals) and leaves a TODO comment at every site it cannot rewrite mechanically. It is idempotent and is removed once T1.12 completes. * refactor(admin): migrate replication and KMS handlers to owned requests (#8951) Move the T1.12 batch 2 admin handlers (site_replication, kms_keys, event, replication, diagnostics, on_demand_migration, audit, account, kms_rekey, kms_key_lifecycle, object_data_cache, tls_debug, integrity) from the s3s request/response types to the owned AdminRequest/AdminResponse boundary. Handler logic, authorization order, error codes and response bytes are unchanged; the request/response plumbing now goes through the envelope accessors and helpers introduced by the owned boundary. The shared response helpers (utils::json_response, utils::empty_response, handlers::admin_json_response, target_descriptor::build_json_response, profile::authorize_profile_request) and the site replication repair executor still answer with the s3s types, so the migrated files carry owned-boundary forms of those helpers and convert the repair response at its call edge. The streaming console-log body and the two devnull drains move from the s3s byte stream to the owned Body stream/frame API. Drop the dead_code expectations for the owned accessors and envelope helpers this batch starts using, and lower the s3s footprint baseline to the measured 99 files. Part of rustfs/backlog#2788 (rustfs/backlog#2734 T1.12). * refactor(admin): add owned forms of the shared admin response helpers Handler batches were copying `json_response`, `empty_response`, `admin_json_response`, and `build_json_response` into each migrated file because the shared helpers still return the s3s response. Provide owned forms next to each legacy helper (`owned_json_response`, `owned_empty_response`, `owned_admin_json_response`, `owned_build_json_response`, `authorize_profile_envelope`) with parity tests against the legacy shape, derive `Debug` for `AdminResponse` so tests can use `expect_err`, and fix three codemod rules: strip owned string conversions in test credentials, annotate non-literal URI parses, and leave a rebound request parameter immutable. * refactor(admin): use the shared owned response helpers in batch 2 (#8954) Replace the per-file owned-boundary copies of json_response, empty_response, admin_json_response, build_json_response and authorize_profile_request in the batch 2 admin handlers with the shared owned_* helpers and authorize_profile_envelope that the boundary branch now provides, and restore expect_err in the handler tests now that AdminResponse implements Debug; the `.err().expect(..)` stand-ins were failing the base branch's clippy step (clippy::err_expect under -D warnings). The site replication repair call edge keeps its conversion because the repair executor still answers with the s3s response type. Drop the dead_code expectations on the shared owned helpers this batch now uses in non-test code. Part of rustfs/backlog#2788 (rustfs/backlog#2734 T1.12). * fix(admin): preserve legacy body limit error responses * refactor(admin): move policy, ILM and batch 3 handlers to owned requests (#8956) Batch 3 of T1.12 (rustfs/backlog#2788, parent rustfs/backlog#2734) moves 14 admin handler files to AdminRequest/AdminResponse and the owned Operation trait: ilm_transition, policies, idp_compat, config_admin, plugins_instances, profile_admin, group, heal, object_zip_download, kms_key_metadata, durability, inspect_archive, sts and gateway_key_inventory. Handler logic, authorization order, error codes and response bytes are unchanged; test request literals use test_support::TestRequest. The handlers use the shared owned helpers (owned_json_response, owned_admin_json_response, owned_build_json_response, authorize_profile_envelope). The dead_code expectations on the AdminRequest accessors and RequestEnvelope::uri_mut they now exercise are removed. The legacy target_descriptor::build_json_response keeps only its equivalence test caller and carries the matching expectation until it retires. object_zip_download builds the s3s request the S3 gate still reads from the owned request facts; heal authenticates through authenticate_envelope_credentials; the trace, inspect-archive and zip bodies use Body::from_stream. extract_plugin_instance_filters takes the URI so extensions.rs keeps calling it, and the two idp_compat delegations stay on the legacy trait until their delegates migrate. The s3s footprint baseline drops from 99 to 90 files. * test(admin): anchor the replication diff body contract on the call The source-contract test looked for `let body = read_compatible_admin_body(` on one line; rustfmt wrapped that binding after the handler moved to the owned request boundary, so the marker was no longer found. Anchor on the call itself so formatting cannot change what the contract checks. * refactor(admin): migrate table catalog and batch 1 to owned requests (#8955) * fix(admin): preserve catalog body limit response bytes * refactor(admin): migrate identity and cluster handlers to owned requests (#8966) * refactor(admin): migrate identity and cluster handlers to owned requests Move the T1.12 batch 4 admin handlers (user, service_account, oidc, tier, system, pools, quota, kms_dynamic, kms_backup, rebalance, module_switch, bucket_meta, cluster_snapshot, realtime, bucket_recovery, profile) from the s3s request/response types to the owned AdminRequest/AdminResponse boundary. All 90 impl Operation blocks now take AdminRequest and return AdminResponse; handler logic, authorization order, error codes and response bytes are unchanged. The handlers use the shared owned helpers (owned_json_response, owned_admin_json_response, authorize_profile_envelope, the envelope-form runtime sources and audit constructors) instead of local copies. profile.rs drops the s3s forms of authorize_profile_request and profile_not_implemented_response now that profile_admin.rs is on the envelope gate too; profile_not_implemented_response answers with an AdminResponse. The realtime metrics stream moves from the s3s ByteStream adapter to Body::from_stream, and the rebalance idle-404 document still renders through s3s::S3Error::to_http_response with its body converted at the edge. The idp_compat delegators ImportIamV2 and AddServiceAccountLdap move to the owned trait as well: with their targets on both traits, the method-call delegation became ambiguous between the owned impl and the router blanket impl, so they now take the owned request directly. KmsAdminAudit::from_request loses its last production caller with this batch and is deleted; its comparison test now asserts the envelope facts directly. Lower the s3s footprint baseline to the measured 68 files. Part of rustfs/backlog#2788 (rustfs/backlog#2734 T1.12). * refactor(admin): remove the legacy admin helpers batch 4 superseded Batch 4 moved the last production callers of the s3s-typed admin helpers to their owned forms, which left `admin_json_response`, `json_response`, `authenticate_request`, `authorize_admin_request`, `object_store_from_req` and `object_store_from_extensions` unused, along with the `Body`, `S3Request` and `S3Response` re-exports of the admin `s3` facade. Remove them. The owned forms keep their documentation; the behavior tests of the removed `json_response` now exercise `owned_json_response`, and the compat-prefix test asserts the owned response directly. Two tests existed only to compare a removed legacy helper with its owned twin (`owned_json_response_matches_the_legacy_shape`) or to cover the removed legacy gate (`authorize_admin_request_without_credentials_is_rejected`); they go with the helpers, and the owned equivalents stay. The rustfs test-count row drops from 5957 to 5955 and the s3s footprint from 68 to 66 files. * refactor(admin): drop the legacy target descriptor JSON response helper (#8968) `target_descriptor::build_json_response` returned the s3s response and had no production caller left once the event, audit and plugin instance handlers moved to `owned_build_json_response`; it survived only for the legacy/owned equivalence test. Remove it, give the owned helper its own documentation, and make the test assert the owned response directly: status, `application/json`, the echoed `x-request-id` when present and only then, and the body bytes. The file no longer imports s3s, so the footprint baseline drops from 66 to 65 files. * test(ecstore): pin native history on cancellation under the cancellable queue main moved the local write path onto the disk blocking queue, which skips an operation once its waiter is dropped, so a borrowed write cancelled while it is queued no longer creates its file. The integration-only native publication test still expected the queued open to run. Assert the new execution model instead (the open never runs) while keeping its safety claims: the Started record is taken before queueing and the cancelled publication never regrants native history. * chore(gateway): raise test-count rows for tests merged from main * test(s3select): use the owned JSON input types in merged planner tests * fix(ecstore): scope the merged direct-writer test hook to test builds * fix(ecstore): create parents before admitted truncate opens Align the heal tests with removal-disabled preservation and local-only outcome fields, and refresh the e2e-full selection for the synced tests. * chore(gateway): update test-count baseline after main sync Main #9241 removed the node pool-metadata readiness cache with 8 tests and added 6, so the rustfs row drops from 6440 to 6438. The ecstore and s3select rows rise for tests added by #9237, #9239 and #9241. * feat(scripts): add junit_diff.py for dual-stack JUnit comparison (#9275) * test(e2e): allow a pinned release binary as differential reference side RUSTFS_E2E_REFERENCE_BINARY makes side A of the dual-stack differential a released rustfs binary, started without RUSTFS_S3_STACK. A missing, non-file or non-executable path fails the run instead of falling back to the same-binary mode. The report records which side is the reference and each side's --version output. * ci(gateway): make the test-count ratchet a floor, not exact (#9278) ci(gateway): make the test-count ratchet a floor instead of an exact baseline Tests below a row still fail; tests above it, a member without a row and ignores below the row now pass with a note. Ignores above a row still fail. Adds contract tests wired into make script-tests. * chore(gateway): record ignored microbenchmarks merged from main (#9290) chore(gateway): record the ignored microbenchmarks merged from main main added three throughput microbenchmarks that are ignored by design (#9217): one each in rustfs-checksums, rustfs-rio and rustfs-utils. The test-count ratchet treats a higher ignored count as a failure until the row is raised, so raise those three rows to 1 and tighten the test rows to the merged tree with --update. * test(e2e): update smoke selection for reference-binary unit tests * ci(s3-tests): add reference parity lane against pinned 1.0.5 (#9288) * ci(parity): run the dual-stack differential against the pinned reference Adds a differential job to reference-parity.yml: side A is the pinned release, side B the candidate artifact built for the s3-tests comparison. The job prints reference_side, version_a and version_b from the report, fails when side A is not the reference or both versions are equal, and lists every difference in the job summary. * ci(parity): print the first version line in the differential summary * chore: restore main's Cargo and baseline files after merging main --------- Co-authored-by: rustfs-cla-assistant[bot] <273121464+rustfs-cla-assistant[bot]@users.noreply.github.com>
ci(e2e): exclude reference unit tests from e2e-full selection (#9342 )
2026-11-11 20:54:09 +00:00 · 2026-11-11 15:15:12 -05:00 · 2026-11-11 14:13:10 -05:00 · 2026-11-11 14:12:54 -05:00 · 2026-11-11 14:07:20 -05:00
22 changed files with 977 additions and 114 deletions
@@ -946,6 +946,10 @@ path = "junit.xml"
 # parallel-safe — the same property e2e-smoke relies on. The exceptions are the
 # 4-disk reliability / degraded-read fault-injection tests, multi-node heal
 # clusters, and the fixed-port Vault tests, all serialized below.
+# `dual_stack_differential::reference::` holds hermetic unit tests of how the
+# differential picks side A (shell-script stand-ins, no server, no reference
+# binary). They already run on every pull request in e2e-smoke, so e2e-full
+# excludes them to keep its selection digest unchanged.
 [profile.e2e-full]
 default-filter = """
    package(e2e_test)
@@ -956,6 +960,7 @@ default-filter = """
    & !test(/^replication_target_matrix_test::/)
    & !test(/^on_demand_migration::(concurrency_test|fault_test|interop_test|real_source_test)::/)
    & !test(/^upgrade_compatibility_test::upgrade_write_readiness_tests::/)
+    & !test(/^dual_stack_differential::reference::/)
 """
 fail-fast = false

@@ -30,6 +30,16 @@
 # result and honours .config/reference-parity-tolerated.txt, whose entries must
 # each match a real difference.
 #
+# The `differential` job adds the 73-operation dual-stack differential
+# (crates/e2e_test/src/dual_stack_differential, rustfs/backlog#2740) against the
+# same pinned reference: side A is the released binary (RUSTFS_E2E_REFERENCE_BINARY,
+# started without RUSTFS_S3_STACK), side B is the candidate built above on its
+# legacy stack. It reuses the candidate build artifact, never builds the candidate
+# twice, and prints reference_side, version_a and version_b from the report the test
+# wrote; it fails when side A was not the reference or both sides report the same
+# version. Every (op, field, a, b) difference goes to the job summary. Report-only
+# until the first result has been reviewed.
+#
 # Report-only: not part of the Test and Lint aggregate. Scheduled failures are
 # owned by the scheduled-validation watchdog (.github/scheduled-validations.json).

@@ -45,6 +55,7 @@ on:
      - ".config/reference-parity-tolerated.txt"
      - "scripts/s3-tests/reference_parity.py"
      - "scripts/junit_diff.py"
+      - "crates/e2e_test/src/dual_stack_differential/**"
  schedule:
    - cron: "37 1 * * *"
  workflow_dispatch:
@@ -253,6 +264,131 @@ jobs:
          if-no-files-found: error
          retention-days: 14

+  differential:
+    name: Dual-stack differential (reference vs candidate)
+    needs: [ build-candidate ]
+    # Compiles the e2e_test harness against the ci-dev cache like the other
+    # sm-standard-4 jobs; the candidate server itself is the downloaded artifact.
+    runs-on: sm-standard-4
+    timeout-minutes: 90
+    env:
+      FORCE_JAVASCRIPT_ACTIONS_TO_NODE24: "true"
+      NO_PROXY: 127.0.0.1,localhost,::1
+    steps:
+      - name: Checkout repository
+        uses: actions/checkout@f548e57e544e1ff5a4c46bf1e1b8685f8e4a348a # v7
+        with:
+          persist-credentials: false
+
+      - name: Setup Rust environment
+        uses: ./.github/actions/setup
+        with:
+          rust-version: stable
+          cache-shared-key: ci-dev
+          cache-save-if: 'false'
+          install-build-packaging-tools: 'false'
+
+      - name: Fetch pinned reference binary
+        id: reference
+        uses: ./.github/actions/reference-binary
+
+      - name: Download candidate binary
+        uses: actions/download-artifact@3e5f45b2cfb9172054b4087a40e8e0b5a5461e7c # v8.0.1
+        with:
+          name: reference-parity-candidate-binary
+          path: target/debug
+
+      - name: Run dual-stack differential
+        id: differential
+        continue-on-error: true
+        env:
+          REFERENCE_PATH: ${{ steps.reference.outputs.path }}
+          RUSTFS_E2E_LOG_DIR: ${{ runner.temp }}/rustfs-differential-logs
+        run: |
+          set -euo pipefail
+          export no_proxy="$NO_PROXY"
+          chmod +x target/debug/rustfs
+          test -x "$REFERENCE_PATH"
+          mkdir -p "$RUSTFS_E2E_LOG_DIR"
+          # Side A: the released binary. Side B: the candidate on its legacy stack.
+          export RUSTFS_E2E_REFERENCE_BINARY="$REFERENCE_PATH"
+          test_name='dual_stack_differential::tests::both_stacks_answer_every_operation_identically'
+          python3 scripts/e2e_binary.py run --features e2e-test-hooks -- \
+            cargo test --locked -p e2e_test --lib "$test_name" -- --exact --nocapture \
+            2>&1 | tee "$RUNNER_TEMP/differential-test.log"
+
+      - name: Summarize report and verify the reference side
+        if: always()
+        run: |
+          set -euo pipefail
+          python3 - <<'PY'
+          import glob, json, os, sys
+
+          summary = os.environ["GITHUB_STEP_SUMMARY"]
+          reports = sorted(glob.glob("target/dual-stack/self-comparison-*.json"))
+          lines = ["## Dual-stack differential: reference vs candidate", ""]
+          failures = []
+          if len(reports) != 1:
+              failures.append(f"expected exactly one self-comparison report, found {len(reports)}: {reports}")
+          else:
+              report = json.load(open(reports[0]))
+              side = report.get("reference_side")
+              version_a = report.get("version_a")
+              version_b = report.get("version_b")
+              lines += [
+                  f"- report: `{reports[0]}`",
+                  f"- reference_side: `{side}`",
+                  f"- version_a: `{(version_a or '').splitlines()[0] if version_a else version_a}`",
+                  f"- version_b: `{(version_b or '').splitlines()[0] if version_b else version_b}`",
+                  f"- stack_a: `{report.get('stack_a')}`, stack_b: `{report.get('stack_b')}`",
+                  f"- compared_ops: {report.get('compared_ops')}, compared_cases: {report.get('compared_cases')}",
+                  "",
+              ]
+              print(f"reference_side={side}\nversion_a={version_a}\nversion_b={version_b}")
+              lines += ["```", "version_a:", version_a or "", "", "version_b:", version_b or "", "```", ""]
+              if side != "a":
+                  failures.append(f"reference_side is {side!r}, expected 'a'")
+              if not version_a or not version_b:
+                  failures.append("version_a or version_b is missing")
+              elif version_a == version_b:
+                  failures.append("version_a equals version_b: the reference was replaced by the candidate")
+              diffs = report.get("differences") or []
+              lines.append(f"### Differences: {len(diffs)}")
+              by_op = {}
+              for d in diffs:
+                  by_op.setdefault(d["op"], []).append(d)
+              for op in sorted(by_op):
+                  lines += ["", f"#### {op}", "", "| case | field | a (reference) | b (candidate) |", "| --- | --- | --- | --- |"]
+                  for d in by_op[op]:
+                      cell = lambda v: "`" + str(v).replace("|", "\\|").replace("`", "'") + "`"
+                      lines.append(f"| {d['case']} | {d['field']} | {cell(d['a'])} | {cell(d['b'])} |")
+          if failures:
+              lines += ["", "### Observation check failed", ""] + [f"- {f}" for f in failures]
+          open(summary, "a").write("\n".join(lines) + "\n")
+          print("\n".join(lines))
+          for f in failures:
+              print(f"::error::{f}")
+          sys.exit(1 if failures else 0)
+          PY
+
+      - name: Upload differential reports
+        if: always()
+        uses: actions/upload-artifact@043fb46d1a93c77aae656e7c1c64a875d1fc6a0a # v7.0.1
+        with:
+          name: reference-differential-report
+          overwrite: true
+          path: |
+            target/dual-stack/*.json
+            ${{ runner.temp }}/differential-test.log
+          if-no-files-found: warn
+          retention-days: 14
+
+      - name: Fail when the differential found differences
+        if: steps.differential.outcome == 'failure'
+        run: |
+          echo "::error::the dual-stack differential failed; see the job summary for every (op, field, a, b)"
+          exit 1
+
  compare:
    name: Compare targets
    needs: [ plan, s3tests ]
@@ -37,6 +37,7 @@ and this project adheres to [Semantic Versioning](https://semver.org/spec/v2.0.0
 - **Presigned URLs honour only signed headers** (GHSA-g8w9-qw9q-fghr): a SigV4 presigned request that carries an `x-amz-*` request header not listed in `X-Amz-SignedHeaders` is now rejected with `403 AccessDenied` ("There were headers present in the request which were not signed"), matching AWS S3. Previously the holder of a presigned `PutObject` URL could add unsigned `x-amz-tagging`, `x-amz-storage-class`, `x-amz-website-redirect-location`, ACL, metadata, Object Lock or SSE headers and have them applied. Presigners that intend a property must set it before signing so the SDK lists the header in `SignedHeaders`; `x-amz-cf-id` (CloudFront) remains tolerated unsigned. Header-signed SigV4 and SigV2 requests are unchanged.

 ### Fixed
+- **Immediate KMS key deletion no longer reports success without destroying the key** (behavior change): with `RUSTFS_KMS_ALLOW_IMMEDIATE_DELETION=true`, a `DELETE /rustfs/admin/v3/kms/keys/delete` request with `force_immediate` against an `Enabled` or `Disabled` key on the Vault KV2 or Vault Transit backend returned success but only moved the key to `PendingDeletion`; the key material stayed in Vault and objects encrypted under it remained readable, while the Local backend destroyed the key in that one request. Key destruction is now the same two explicit steps on every backend: schedule the deletion, then either let the waiting window elapse or send `force_immediate` to destroy the pending key early. `force_immediate` against a key that is not `PendingDeletion` is refused with `400` and changes nothing, so a success response always means the key is gone and no single request can destroy a key in use. Automation that destroyed Local keys with one `force_immediate` request must schedule the deletion first. On Vault Transit, a read under a key that no longer exists in Vault now fails with `KMS.NotFoundException` instead of `503 ServiceUnavailable`. No persisted format changes. See `docs/operations/kms-key-destruction.md`.
 - **`mc admin info` and `mc admin replicate resync` work again**: since the cluster snapshot view (#3803), `GET /v3/info` and `GET /v3/storageinfo` returned the madmin `InfoMessage` / `StorageInfo` nested under an `info` key next to `admin_discovery` (and, for `/info`, `bitrotSelftest`). madmin decodes those bodies directly, so `mc admin info` printed empty values and `mc admin replicate resync start/status` rejected every alias with "alias provided is not part of cluster replication" because the local `deploymentID` read as empty. Requests on the MinIO-compatible `/minio/admin/v3` prefix now receive the bare madmin body again; `/rustfs/admin/v3`, which rc and the console use, keeps the envelope unchanged.
 - **Replication status writebacks no longer invalidate scanner cycles**: every replication status writeback (`persist_replication_state_if_current` in `Update` mode, including the heals the scanner queues during its own walk) marked the bucket dirty on the namespace path, advanced the dirty-usage generation and woke the scanner, so a site-replication resync or any steadily replicated bucket ended every cycle `superseded`, the authoritative usage baseline was never certified, and the console reported usage freshness as stale for as long as replication ran (#8375). Those writebacks are now status class: they do not touch the dirty-usage generation, the activity snapshot, the dirty-usage wake or the durable dirty journal, and only the process-local producer coverage is recorded. They need no rescan queue, because a deployment with an active replication rule already walks every bucket on every cycle and that walk refreshes `replication_stats`. Tier-expiration cleanup, which removes data, stays on the namespace path. No persisted format, protocol, admin API or configuration changes. Design: `docs/architecture/scanner-per-bucket-convergence.md`.
 - **A bucket written during a scanner walk no longer supersedes the whole cycle**: the post-walk cluster activity proof required the entire activity snapshot and the dirty-usage generation to be unchanged, so a single-node deployment with one bucket written every few seconds never reached a `Complete` cycle, `usage_snapshot_converged` never became true and the authoritative usage baseline was never certified (#8494). Dirty usage is now certified per bucket: a structurally complete walk that the guardrail did not veto is `Complete`, buckets whose dirty generation still matches the start snapshot are cleared, and buckets written during the walk stay dirty and are published as the pending set, so `usage_snapshot_converged` means "no bucket pending" and the next cycle is scoped to the pending buckets. Such a cycle writes the authoritative usage object; the observed object is now selected by an explicit observational mark that vetoed cycles carry. The guardrail itself vetoes only on the fields that change which namespace the walk described (peer set, process instance, protocol version, topology, maintenance generation, data movement, movement generation, publication blocked) and reports the field and host in the scanner log and in the new `usage_freshness` fields `guardrail_veto_total`, `last_guardrail_veto_unix_secs` and `last_guardrail_veto_reason`. In a distributed deployment a remote peer whose dirty usage moved during the walk still supersedes the cycle until scoped acknowledgements can name its buckets. `.usage.v2.json` gains the additive, default-when-absent fields `usage_snapshot_pending_buckets`, `usage_snapshot_pending_bucket_count` and `usage_snapshot_observational`; older readers ignore them and treat the snapshot as an authoritative but unconverged publication. Quota is unchanged: an uncertified bucket keeps its last usage value as the floor and live writes do not advance it. No protocol or configuration changes. Design: `docs/architecture/scanner-per-bucket-convergence.md`.
@@ -586,7 +586,18 @@ async fn test_reads_under_a_deleted_kms_key_report_key_not_found() -> Result<(),
        .await?;
    assert_eq!(put.ssekms_key_id(), Some(doomed_key_id.as_str()));

-    info!("🗑️ deleting {doomed_key_id} immediately");
+    // Destruction is two requests: schedule the deletion, then destroy the
+    // pending key without waiting out its window.
+    info!("🗑️ scheduling {doomed_key_id} for deletion, then destroying it");
+    kms_admin_request(
+        &base.url,
+        http::Method::DELETE,
+        "/rustfs/admin/v3/kms/keys/delete",
+        Some(&serde_json::json!({ "key_id": doomed_key_id }).to_string()),
+        &base.access_key,
+        &base.secret_key,
+    )
+    .await?;
    kms_admin_request(
        &base.url,
        http::Method::DELETE,
@@ -16,8 +16,8 @@

 use crate::backends::{
    BackendCapabilities, ExpiredKeyRemoval, KmsBackend, ListedKeyFailure, StateGatedOperation, UnreadableKeys,
-    classify_listed_key_failure, ensure_key_status_permits, ensure_tag_keys_are_mutable, paginate_keys, started_at_the_first_key,
-    validate_key_id_segment,
+    classify_listed_key_failure, destruction_requires_pending_deletion, ensure_key_status_permits, ensure_tag_keys_are_mutable,
+    paginate_keys, started_at_the_first_key, validate_key_id_segment,
 };
 use crate::config::KmsConfig;
 use crate::config::LocalConfig;
@@ -2085,8 +2085,6 @@ impl KmsBackend for LocalKmsBackend {
    }

    async fn delete_key(&self, request: DeleteKeyRequest) -> Result<DeleteKeyResponse> {
-        // For local backend, we'll implement immediate deletion by default
-        // unless a pending window is specified
        let key_id = &request.key_id;

        // Deletion is a read-modify-write (or read-then-remove) cycle, so hold
@@ -2101,6 +2099,13 @@ impl KmsBackend for LocalKmsBackend {
            .map_err(|_| KmsError::key_not_found(format!("Key {key_id} not found")))?;

        let (deletion_date_str, deletion_date_dt) = if request.force_immediate.unwrap_or(false) {
+            // Expedited destruction only skips the rest of the waiting window:
+            // the key must already be pending deletion (or be a tombstone left
+            // by a crashed removal, which may simply be completed).
+            if matches!(master_key.status, KeyStatus::Active | KeyStatus::Disabled) {
+                return Err(destruction_requires_pending_deletion(key_id));
+            }
+
            // Tombstone first: mark the record Deleted before removing the
            // file, so a crash between the two steps leaves a key that is
            // already unusable and whose removal can simply be re-run.
@@ -3617,6 +3622,13 @@ mod tests {
        assert!(fsync_recorder::dir_sync_count(dir) > dirs_before, "update must fsync the key directory");

        let backend = LocalKmsBackend { client };
+        backend
+            .delete_key(DeleteKeyRequest {
+                key_id: "durable-key".to_string(),
+                ..Default::default()
+            })
+            .await
+            .expect("schedule deletion");
        let dirs_before = fsync_recorder::dir_sync_count(dir);
        backend
            .delete_key(DeleteKeyRequest {
@@ -129,6 +129,19 @@ pub(crate) fn ensure_key_status_permits(key_id: &str, status: &KeyStatus, operat
    ensure_key_state_permits(key_id, &state, operation)
 }

+/// Refusal for an expedited destruction of a key that was never scheduled.
+///
+/// Destruction always passes through `PendingDeletion`, the one state a
+/// deletion can still be cancelled from; `force_immediate` only skips the rest
+/// of the waiting window. Every backend that destroys keys refuses with this
+/// error instead of scheduling the key on the caller's behalf, so a success
+/// response always means the key is gone.
+pub(crate) fn destruction_requires_pending_deletion(key_id: &str) -> KmsError {
+    KmsError::invalid_key_state(format!(
+        "Key {key_id} is not pending deletion: schedule the deletion before requesting immediate destruction"
+    ))
+}
+
 /// Tag key that carries the key's identity rather than user metadata.
 ///
 /// The key-creation path lifts `name` out of the caller's tag map and uses it
@@ -20,9 +20,9 @@ use crate::backends::vault_credentials::{
 };
 use crate::backends::{
    BackendCapabilities, ExpiredKeyRemoval, KmsBackend, ListedKeyFailure, StateGatedOperation, UnreadableKeys,
-    classify_listed_key_failure, empty_key_page, ensure_key_state_permits, ensure_key_status_permits,
-    ensure_rewrap_context_matches, ensure_tag_keys_are_mutable, list_keys_page_size, paginate_keys, started_at_the_first_key,
-    validate_key_id_segment,
+    classify_listed_key_failure, destruction_requires_pending_deletion, empty_key_page, ensure_key_state_permits,
+    ensure_key_status_permits, ensure_rewrap_context_matches, ensure_tag_keys_are_mutable, list_keys_page_size, paginate_keys,
+    started_at_the_first_key, validate_key_id_segment,
 };
 use crate::config::{KmsConfig, VaultConfig};
 use crate::encryption::{
@@ -2135,8 +2135,6 @@ impl KmsBackend for VaultKmsBackend {
    }

    async fn delete_key(&self, request: DeleteKeyRequest) -> Result<DeleteKeyResponse> {
-        // For Vault backend, we'll mark keys for deletion but not physically delete them
-        // This allows for recovery during the pending window
        let key_id = &request.key_id;

        // First, check if the key exists and get its metadata
@@ -2149,41 +2147,32 @@ impl KmsBackend for VaultKmsBackend {
        };

        let deletion_date = if request.force_immediate.unwrap_or(false) {
-            // Check if key is already in PendingDeletion state (or a tombstone
-            // left by a crashed removal, which may simply be completed)
-            if key_metadata.key_state == KeyState::PendingDeletion || key_metadata.key_state == KeyState::Unavailable {
-                // Tombstone first: mark the record Deleted before removing it,
-                // so a crash between the two steps leaves a key that is already
-                // unusable and whose removal can simply be re-run. Written
-                // check-and-set with re-validation: a concurrent cancellation
-                // that commits first wins and fails this removal instead of
-                // being overwritten.
-                self.client
-                    .update_key_data_with_cas(key_id, |key_data| match key_data.status {
-                        KeyStatus::Deleted => Ok(CasMutation::Skip(())),
-                        KeyStatus::PendingDeletion => {
-                            key_data.status = KeyStatus::Deleted;
-                            Ok(CasMutation::Write(()))
-                        }
-                        KeyStatus::Active | KeyStatus::Disabled => {
-                            Err(KmsError::invalid_key_state(format!("Key {key_id} is no longer pending deletion")))
-                        }
-                    })
-                    .await?;
-                // Force immediate deletion: physically delete the key from Vault storage
-                self.client.delete_key(key_id).await?;
+            // Expedited destruction only skips the rest of the waiting window:
+            // the key must already be pending deletion (or be a tombstone left
+            // by a crashed removal, which may simply be completed).
+            //
+            // Tombstone first: mark the record Deleted before removing it, so
+            // a crash between the two steps leaves a key that is already
+            // unusable and whose removal can simply be re-run. Written
+            // check-and-set with re-validation: a concurrent cancellation
+            // that commits first wins and fails this removal instead of
+            // being overwritten.
+            self.client
+                .update_key_data_with_cas(key_id, |key_data| match key_data.status {
+                    KeyStatus::Deleted => Ok(CasMutation::Skip(())),
+                    KeyStatus::PendingDeletion => {
+                        key_data.status = KeyStatus::Deleted;
+                        Ok(CasMutation::Write(()))
+                    }
+                    KeyStatus::Active | KeyStatus::Disabled => Err(destruction_requires_pending_deletion(key_id)),
+                })
+                .await?;
+            self.client.delete_key(key_id).await?;
+            key_metadata.key_state = KeyState::PendingDeletion;
+            key_metadata.deletion_date = Some(Zoned::now());

-                // Return empty deletion_date to indicate key was permanently deleted
-                None
-            } else {
-                // For non-pending keys, mark as PendingDeletion
-                let marked_at = Zoned::now();
-                self.mark_key_pending_deletion(key_id, &marked_at).await?;
-                key_metadata.key_state = KeyState::PendingDeletion;
-                key_metadata.deletion_date = Some(marked_at);
-
-                None
-            }
+            // No deletion date: the key is permanently gone, not scheduled.
+            None
        } else {
            // Schedule for deletion (default 30 days)
            ensure_key_state_permits(key_id, &key_metadata.key_state, StateGatedOperation::ScheduleDeletion)?;
@@ -4029,6 +4018,101 @@ mod tests {
        );
    }

+    /// Destruction always passes through the cancellable pending state. An
+    /// immediate deletion of a key that was never scheduled is refused without
+    /// writing anything: parking it in PendingDeletion and reporting success
+    /// would tell the caller a key is gone while every object under it stays
+    /// readable.
+    #[tokio::test]
+    async fn wired_immediate_deletion_of_an_enabled_key_is_refused_without_writing() {
+        let vault = ScriptedVault::serve(vec![
+            // describe_key: key info plus stored metadata.
+            ScriptedResponse::ok(kv2_read_data(&healthy_key_data())),
+            ScriptedResponse::ok(kv2_read_data(&healthy_key_data())),
+            // The check-and-set read observes a key that is not pending.
+            ScriptedResponse::ok(kv2_metadata_read_data(1)),
+            ScriptedResponse::ok(kv2_read_data(&healthy_key_data())),
+        ])
+        .await;
+        let config = KmsConfig::vault(
+            url::Url::parse(&vault.address).expect("scripted vault address should parse"),
+            "scripted-token".to_string(),
+        )
+        .with_insecure_development_defaults();
+        let backend = VaultKmsBackend::new(config).await.expect("vault kv2 backend should build");
+
+        let error = backend
+            .delete_key(DeleteKeyRequest {
+                key_id: "wired-key".to_string(),
+                pending_window_in_days: None,
+                force_immediate: Some(true),
+                confirm_key_id: Some("wired-key".to_string()),
+            })
+            .await
+            .expect_err("a key that is not pending deletion must not be destroyed in one request");
+        assert!(
+            matches!(&error, KmsError::InvalidOperation { message } if message.contains("not pending deletion")),
+            "got {error:?}"
+        );
+
+        let requests = vault.requests();
+        assert!(
+            requests.iter().all(|line| line.starts_with("GET ")),
+            "a refused immediate deletion must neither write nor delete: {requests:?}"
+        );
+    }
+
+    /// The other half: once the key is pending deletion, one immediate request
+    /// tombstones the record and removes it.
+    #[tokio::test]
+    async fn wired_immediate_deletion_of_a_pending_key_destroys_it() {
+        let mut pending = healthy_key_data();
+        pending.status = KeyStatus::PendingDeletion;
+        pending.deletion_date = Some(Zoned::now() + Duration::from_secs(7 * 86400));
+
+        let vault = ScriptedVault::serve(vec![
+            // describe_key: key info plus stored metadata.
+            ScriptedResponse::ok(kv2_read_data(&pending)),
+            ScriptedResponse::ok(kv2_read_data(&pending)),
+            // Check-and-set tombstone.
+            ScriptedResponse::ok(kv2_metadata_read_data(1)),
+            ScriptedResponse::ok(kv2_read_data(&pending)),
+            ScriptedResponse::ok(kv2_write_ack()),
+            // The key was never rotated: no version records to purge.
+            ScriptedResponse::empty_list_404(),
+            // Removal of the key record itself.
+            ScriptedResponse::ok(serde_json::Value::Null),
+        ])
+        .await;
+        let config = KmsConfig::vault(
+            url::Url::parse(&vault.address).expect("scripted vault address should parse"),
+            "scripted-token".to_string(),
+        )
+        .with_insecure_development_defaults();
+        let backend = VaultKmsBackend::new(config).await.expect("vault kv2 backend should build");
+
+        let response = backend
+            .delete_key(DeleteKeyRequest {
+                key_id: "wired-key".to_string(),
+                pending_window_in_days: None,
+                force_immediate: Some(true),
+                confirm_key_id: Some("wired-key".to_string()),
+            })
+            .await
+            .expect("immediate deletion of a pending key should succeed");
+        assert!(response.deletion_date.is_none(), "a destroyed key has no future deadline to report");
+
+        let requests = vault.requests();
+        assert_eq!(requests.len(), 7, "{requests:?}");
+        let tombstone = parse_write_body(&vault.request_bodies()[4]);
+        assert_eq!(tombstone["options"]["cas"], serde_json::json!(1), "{tombstone}");
+        assert_eq!(tombstone["data"]["status"], serde_json::json!("Deleted"), "{tombstone}");
+        assert_eq!(
+            requests[6], "DELETE /v1/secret/metadata/rustfs/kms/keys/wired-key",
+            "the key record must be removed: {requests:?}"
+        );
+    }
+
    #[tokio::test]
    async fn wired_schedule_deletion_retries_after_cas_conflict() {
        let vault = ScriptedVault::serve(vec![
@@ -21,8 +21,9 @@ use crate::backends::vault_credentials::{
 };
 use crate::backends::{
    BackendCapabilities, ExpiredKeyRemoval, KmsBackend, ListedKeyFailure, StateGatedOperation, UnreadableKeys,
-    classify_listed_key_failure, empty_key_page, ensure_key_state_permits, ensure_rewrap_context_matches,
-    ensure_tag_keys_are_mutable, list_keys_page_size, paginate_keys, started_at_the_first_key, validate_key_id_segment,
+    classify_listed_key_failure, destruction_requires_pending_deletion, empty_key_page, ensure_key_state_permits,
+    ensure_rewrap_context_matches, ensure_tag_keys_are_mutable, list_keys_page_size, paginate_keys, started_at_the_first_key,
+    validate_key_id_segment,
 };
 use crate::config::{KmsConfig, VaultTransitConfig};
 use crate::encryption::{DataKeyEnvelope, generate_key_material};
@@ -61,6 +62,10 @@ use zeroize::Zeroizing;
 /// replay of the stale snapshot.
 const METADATA_CAS_ATTEMPTS: usize = 3;

+/// The sole error message of the 400 Vault Transit returns when a data-path
+/// call names a key that does not exist.
+const TRANSIT_KEY_NOT_FOUND_MESSAGE: &str = "encryption key not found";
+
 /// TTL bound on cached metadata records. This caps how long one node can keep
 /// acting on lifecycle state another node has since changed (disable,
 /// schedule-deletion): the divergence window is one TTL instead of "until
@@ -486,6 +491,14 @@ impl VaultTransitKmsClient {
        match error {
            vaultrs::error::ClientError::ResponseWrapError => KmsError::key_not_found(key_id),
            vaultrs::error::ClientError::APIError { code: 404, .. } => KmsError::key_not_found(key_id),
+            // Transit answers decrypt and rewrap for a key that does not exist
+            // with a 400, not the 404 a key read gets. Left as a backend error
+            // it would surface as a retryable outage instead of a missing key.
+            vaultrs::error::ClientError::APIError { code: 400, ref errors }
+                if errors.iter().any(|message| message == TRANSIT_KEY_NOT_FOUND_MESSAGE) =>
+            {
+                KmsError::key_not_found(key_id)
+            }
            other => KmsError::backend_error(format!("Vault Transit {operation} failed for key {key_id}: {other}")),
        }
    }
@@ -1709,25 +1722,35 @@ impl KmsBackend for VaultTransitKmsBackend {
        let mut key_metadata = self.client.key_metadata_response(&key_id).await?;

        let deletion_date = if request.force_immediate.unwrap_or(false) {
-            if key_metadata.key_state == KeyState::PendingDeletion {
-                if !self.client.read_transit_key(&key_id).await?.deletion_allowed {
-                    self.client.allow_transit_key_deletion(&key_id).await?;
-                }
-                self.client.delete_transit_key(&key_id).await?;
-                self.client.delete_key_metadata(&key_id).await?;
-                None
-            } else {
-                let now = Zoned::now();
-                self.client
-                    .mutate_key_metadata(&key_id, |metadata| {
-                        metadata.key_state = KeyState::PendingDeletion;
-                        metadata.deletion_date = Some(now.clone());
+            // Expedited destruction only skips the rest of the waiting window:
+            // the key must already be pending deletion (or be a tombstone left
+            // by a crashed removal, which may simply be completed).
+            //
+            // Tombstone first, as the expired-key sweep does: an Unavailable
+            // record is rejected by every state gate, so a crash before the
+            // Transit key is destroyed leaves a key that is already unusable.
+            // Re-checked against every fresh snapshot: a concurrent
+            // cancellation that commits first fails this removal.
+            let now = Zoned::now();
+            self.client
+                .mutate_key_metadata(&key_id, |metadata| match metadata.key_state {
+                    KeyState::PendingDeletion | KeyState::Unavailable => {
+                        metadata.key_state = KeyState::Unavailable;
                        Ok(())
-                    })
-                    .await?;
-                key_metadata = self.client.key_metadata_response(&key_id).await?;
-                None
+                    }
+                    KeyState::Enabled | KeyState::Disabled | KeyState::PendingImport => {
+                        Err(destruction_requires_pending_deletion(&key_id))
+                    }
+                })
+                .await?;
+            if !self.client.read_transit_key(&key_id).await?.deletion_allowed {
+                self.client.allow_transit_key_deletion(&key_id).await?;
            }
+            self.client.delete_transit_key(&key_id).await?;
+            self.client.delete_key_metadata(&key_id).await?;
+            key_metadata.key_state = KeyState::PendingDeletion;
+            key_metadata.deletion_date = Some(now);
+            None
        } else {
            ensure_key_state_permits(&key_id, &key_metadata.key_state, StateGatedOperation::ScheduleDeletion)?;

@@ -3005,6 +3028,149 @@ mod tests {
        );
    }

+    /// Transit reports a missing key on the data path as a 400. Reads of
+    /// objects under a destroyed key must see a missing key, not a backend
+    /// outage the client is told to retry.
+    #[tokio::test]
+    async fn wired_decrypt_under_a_destroyed_transit_key_reports_key_not_found() {
+        let (vault, client) = scripted_client(vec![ScriptedResponse::error(400, TRANSIT_KEY_NOT_FOUND_MESSAGE)]).await;
+        let envelope = DataKeyEnvelope {
+            key_id: "dek".to_string(),
+            master_key_id: "wired-key".to_string(),
+            key_spec: "AES_256".to_string(),
+            encrypted_key: b"vault:v1:AAAA".to_vec(),
+            nonce: Vec::new(),
+            encryption_context: HashMap::new(),
+            created_at: Zoned::now(),
+            master_key_version: Some(1),
+            context_binding: None,
+        };
+
+        let error = client
+            .decrypt(
+                &DecryptRequest {
+                    ciphertext: serde_json::to_vec(&envelope).expect("serialize envelope"),
+                    encryption_context: HashMap::new(),
+                    grant_tokens: Vec::new(),
+                },
+                None,
+            )
+            .await
+            .expect_err("decrypting under a destroyed key must fail");
+        assert!(matches!(&error, KmsError::KeyNotFound { .. }), "got {error:?}");
+        assert_eq!(
+            vault.requests(),
+            ["POST /v1/transit/decrypt/wired-key"],
+            "a missing key must not be retried"
+        );
+    }
+
+    /// Destruction always passes through the cancellable pending state. An
+    /// immediate deletion of a key that was never scheduled is refused without
+    /// writing anything: parking it in PendingDeletion and reporting success
+    /// would tell the caller a key is gone while every object under it stays
+    /// readable.
+    #[tokio::test]
+    async fn wired_immediate_deletion_of_an_enabled_key_is_refused_without_writing() {
+        let enabled = TransitKeyMetadata::from_create_request(&CreateKeyRequest::default());
+        let vault = ScriptedVault::serve(vec![
+            // Current key metadata.
+            ScriptedResponse::ok(transit_key_read_data("wired-key")),
+            ScriptedResponse::ok(metadata_read_data(&enabled)),
+            // The check-and-set read observes a key that is not pending.
+            ScriptedResponse::ok(kv2_metadata_read_data(1)),
+            ScriptedResponse::ok(metadata_read_data(&enabled)),
+        ])
+        .await;
+        let config = KmsConfig::vault_transit(
+            url::Url::parse(&vault.address).expect("scripted vault address should parse"),
+            "scripted-token".to_string(),
+        )
+        .with_insecure_development_defaults();
+        let backend = VaultTransitKmsBackend::new(config)
+            .await
+            .expect("vault transit backend should build");
+
+        let error = backend
+            .delete_key(DeleteKeyRequest {
+                key_id: "wired-key".to_string(),
+                pending_window_in_days: None,
+                force_immediate: Some(true),
+                confirm_key_id: Some("wired-key".to_string()),
+            })
+            .await
+            .expect_err("a key that is not pending deletion must not be destroyed in one request");
+        assert!(
+            matches!(&error, KmsError::InvalidOperation { message } if message.contains("not pending deletion")),
+            "got {error:?}"
+        );
+
+        let requests = vault.requests();
+        assert!(
+            requests.iter().all(|line| line.starts_with("GET ")),
+            "a refused immediate deletion must neither write nor delete: {requests:?}"
+        );
+    }
+
+    /// The other half: once the key is pending deletion, one immediate request
+    /// tombstones the metadata, destroys the Transit key and removes the record.
+    #[tokio::test]
+    async fn wired_immediate_deletion_of_a_pending_key_destroys_it() {
+        let mut pending = TransitKeyMetadata::from_create_request(&CreateKeyRequest::default());
+        pending.key_state = KeyState::PendingDeletion;
+        pending.deletion_date = Some(Zoned::now() + Duration::from_secs(7 * 86400));
+        let vault = ScriptedVault::serve(vec![
+            // Current key metadata for the response.
+            ScriptedResponse::ok(transit_key_read_data("wired-key")),
+            ScriptedResponse::ok(metadata_read_data(&pending)),
+            // Check-and-set tombstone.
+            ScriptedResponse::ok(kv2_metadata_read_data(1)),
+            ScriptedResponse::ok(metadata_read_data(&pending)),
+            ScriptedResponse::ok(kv2_write_ack()),
+            // Transit refuses deletion until the key is configured to allow it.
+            ScriptedResponse::ok(transit_key_read_data("wired-key")),
+            ScriptedResponse::ok(serde_json::Value::Null),
+            // Transit key, then its metadata record.
+            ScriptedResponse::ok(serde_json::Value::Null),
+            ScriptedResponse::ok(serde_json::Value::Null),
+        ])
+        .await;
+        let config = KmsConfig::vault_transit(
+            url::Url::parse(&vault.address).expect("scripted vault address should parse"),
+            "scripted-token".to_string(),
+        )
+        .with_insecure_development_defaults();
+        let backend = VaultTransitKmsBackend::new(config)
+            .await
+            .expect("vault transit backend should build");
+
+        let response = backend
+            .delete_key(DeleteKeyRequest {
+                key_id: "wired-key".to_string(),
+                pending_window_in_days: None,
+                force_immediate: Some(true),
+                confirm_key_id: Some("wired-key".to_string()),
+            })
+            .await
+            .expect("immediate deletion of a pending key should succeed");
+        assert!(response.deletion_date.is_none(), "a destroyed key has no future deadline to report");
+
+        let requests = vault.requests();
+        assert_eq!(requests.len(), 9, "{requests:?}");
+        let tombstone: serde_json::Value = serde_json::from_str(&vault.request_bodies()[4]).expect("tombstone body is JSON");
+        assert_eq!(tombstone["options"]["cas"], serde_json::json!(1), "{tombstone}");
+        assert_eq!(tombstone["data"]["key_state"], serde_json::json!("Unavailable"), "{tombstone}");
+        assert_eq!(
+            requests[7], "DELETE /v1/transit/keys/wired-key",
+            "the transit key must be destroyed: {requests:?}"
+        );
+        assert_eq!(
+            requests[8],
+            format!("DELETE {METADATA_VERSION_PATH}"),
+            "the metadata record must be removed: {requests:?}"
+        );
+    }
+
    #[tokio::test]
    async fn wired_expired_sweep_backs_off_when_cancel_wins_the_cas_race() {
        let mut pending = TransitKeyMetadata::from_create_request(&CreateKeyRequest::default());
@@ -466,6 +466,8 @@ impl KmsManager {
    /// opt-in and a per-request confirmation that echoes the key id; without
    /// either, the request is refused rather than downgraded to a scheduled
    /// deletion, so a caller never believes a key is gone when it is not.
+    /// The backend adds the last condition under its own synchronization: the
+    /// key must already be pending deletion.
    fn check_deletion_request(&self, request: &DeleteKeyRequest) -> Result<()> {
        if !request.force_immediate.unwrap_or(false) {
            let days = request.pending_window_in_days.unwrap_or(DEFAULT_PENDING_DELETION_WINDOW_DAYS);
@@ -493,7 +495,7 @@ impl KmsManager {

        warn!(
            key_id = %request.key_id,
-            "immediate KMS key deletion accepted; key material is destroyed without a waiting window and cannot be recovered"
+            "immediate KMS key deletion authorized; a key pending deletion is destroyed without waiting out its window and cannot be recovered"
        );
        Ok(())
    }
@@ -1468,6 +1470,7 @@ mod tests {
        let temp_dir = tempdir().expect("Failed to create temp dir");
        let manager = deletion_manager(&temp_dir, true).await;
        let key_id = create_named_key(&manager, "confirmed-force-delete").await;
+        schedule_deletion(&manager, &key_id).await;

        manager
            .delete_key(DeleteKeyRequest {
@@ -1486,6 +1489,43 @@ mod tests {
        assert!(matches!(error, KmsError::KeyNotFound { .. }), "expected KeyNotFound, got {error:?}");
    }

+    async fn schedule_deletion(manager: &KmsManager, key_id: &str) {
+        manager
+            .delete_key(DeleteKeyRequest {
+                key_id: key_id.to_string(),
+                ..Default::default()
+            })
+            .await
+            .expect("scheduling the deletion must succeed");
+    }
+
+    /// Destruction always passes through the cancellable pending state: a
+    /// fully authorized immediate deletion of a key that was never scheduled
+    /// is refused, and leaves the key exactly as it was.
+    #[tokio::test]
+    async fn immediate_deletion_of_a_key_that_was_never_scheduled_is_refused() {
+        let temp_dir = tempdir().expect("Failed to create temp dir");
+        let manager = deletion_manager(&temp_dir, true).await;
+        let key_id = create_named_key(&manager, "unscheduled-force-delete").await;
+        let probe = data_key_probe(&manager, &key_id).await;
+
+        let error = manager
+            .delete_key(DeleteKeyRequest {
+                key_id: key_id.clone(),
+                force_immediate: Some(true),
+                confirm_key_id: Some(key_id.clone()),
+                ..Default::default()
+            })
+            .await
+            .expect_err("a key that is not pending deletion must not be destroyed in one request");
+        assert!(
+            matches!(&error, KmsError::InvalidOperation { message } if message.contains("not pending deletion")),
+            "got {error:?}"
+        );
+
+        assert_key_material_intact(&manager, &key_id, &probe).await;
+    }
+
    /// Reference checker whose answer is fixed, standing in for the server's
    /// bucket-configuration gate.
    struct StaticReferences(Vec<String>);
@@ -1624,6 +1664,7 @@ mod tests {
            .await
            .with_deletion_reference_checker(Some(Arc::new(StaticReferences(Vec::new()))));
        let key_id = create_named_key(&manager, "unreferenced-confirmed-force-delete").await;
+        schedule_deletion(&manager, &key_id).await;

        manager
            .delete_key(DeleteKeyRequest {
@@ -27,7 +27,7 @@

 mod common;

-use common::{TestKms, assert_invalid_operation, ctx};
+use common::{TestKms, assert_invalid_operation, ctx, schedule_deletion};
 use rustfs_kms::{
    CancelKeyDeletionRequest, DeleteKeyRequest, DescribeKeyRequest, GenerateDataKeyRequest, KeySpec, KeyState, KmsManager,
 };
@@ -190,6 +190,7 @@ async fn a_destroyed_key_cannot_be_served_from_cache() {

    // Warm the cache, then destroy the key outright.
    assert_eq!(describe_state(&manager, &key_id).await, KeyState::Enabled);
+    schedule_deletion(&manager, &key_id).await;
    manager
        .delete_key(DeleteKeyRequest {
            key_id: key_id.clone(),
@@ -32,7 +32,7 @@ use std::collections::HashSet;
 use std::sync::Arc;
 use std::time::Duration;

-use common::{TestKms, assert_invalid_operation, ctx};
+use common::{TestKms, assert_invalid_operation, ctx, schedule_deletion};
 use rustfs_kms::{
    CreateKeyRequest, DecryptRequest, DeleteKeyRequest, DescribeKeyRequest, GenerateDataKeyRequest, KeySpec, KeyState, KmsConfig,
    KmsError, KmsServiceManager, KmsServiceStatus,
@@ -519,6 +519,7 @@ async fn a_destroyed_key_stays_destroyed_across_a_restart() {
    kms.create_key("gone-for-good").await;
    kms.create_key("kept").await;

+    schedule_deletion(&manager, "gone-for-good").await;
    manager
        .delete_key(DeleteKeyRequest {
            key_id: "gone-for-good".to_string(),
@@ -33,7 +33,7 @@

 mod common;

-use common::{BackendCase, TestKms, assert_context_mismatch, ctx, flip_middle_bit, for_each_backend, payload};
+use common::{BackendCase, TestKms, assert_context_mismatch, ctx, flip_middle_bit, for_each_backend, payload, schedule_deletion};
 use rustfs_kms::{
    DecryptRequest, EncryptRequest, GenerateDataKeyRequest, KeySpec, KmsError, ObjectEncryptionContext, is_data_key_envelope,
 };
@@ -412,6 +412,7 @@ async fn a_data_key_is_not_transferable_between_master_keys() {

    // Deleting wrapper-a makes its blobs undecryptable while wrapper-b's keep
    // working — the blobs are genuinely bound to distinct material.
+    schedule_deletion(&manager, "wrapper-a").await;
    manager
        .delete_key(rustfs_kms::DeleteKeyRequest {
            key_id: "wrapper-a".to_string(),
@@ -39,7 +39,7 @@ mod common;

 use common::{
    BackendCase, BackendKind, TestKms, assert_invalid_operation, assert_key_already_exists, assert_key_not_found,
-    assert_unsupported_capability, ctx, for_each_backend, without_probe_key,
+    assert_unsupported_capability, ctx, for_each_backend, schedule_deletion, without_probe_key,
 };
 use rustfs_kms::{
    CancelKeyDeletionRequest, CreateKeyRequest, DecryptRequest, DeleteKeyRequest, DescribeKeyRequest, EncryptRequest,
@@ -417,11 +417,27 @@ async fn deletion_pending_window_is_bounded() {
 }

 #[tokio::test]
-async fn forced_immediate_deletion_removes_the_key() {
+async fn forced_immediate_deletion_removes_a_pending_key_and_refuses_any_other() {
    let kms = TestKms::local_with(|config| config.allow_immediate_deletion = true).await;
    let manager = kms.kms().await;
    let key_id = kms.create_key("burn-now").await;

+    // Destruction always passes through the cancellable pending state: asked
+    // of a key that was never scheduled, it is refused and changes nothing.
+    assert_invalid_operation(
+        manager
+            .delete_key(DeleteKeyRequest {
+                key_id: key_id.clone(),
+                pending_window_in_days: None,
+                force_immediate: Some(true),
+                confirm_key_id: Some(key_id.clone()),
+            })
+            .await,
+        "not pending deletion",
+    );
+    assert_eq!(describe_state(&manager, &key_id).await, KeyState::Enabled);
+
+    schedule_deletion(&manager, &key_id).await;
    let deleted = manager
        .delete_key(DeleteKeyRequest {
            key_id: key_id.clone(),
@@ -281,6 +281,17 @@ impl TestKms {
    }
 }

+/// Move `key_id` to `PendingDeletion` with the default waiting window — the
+/// step every destruction has to pass through first.
+pub async fn schedule_deletion(kms: &KmsManager, key_id: &str) {
+    kms.delete_key(DeleteKeyRequest {
+        key_id: key_id.to_string(),
+        ..Default::default()
+    })
+    .await
+    .unwrap_or_else(|error| panic!("scheduling the deletion of {key_id} should succeed: {error:?}"));
+}
+
 /// Best-effort removal of the Vault keys a harness instance created, so a
 /// persistent dev Vault does not accumulate `behavior-*` keys across runs
 /// (rustfs/backlog#1774). A no-op for the Local and Static backends, whose
@@ -306,26 +317,29 @@ pub async fn cleanup_vault_keys(kind: BackendKind, config: &KmsConfig, created_k
    let manager = start_manager(&config).await;
    let kms = manager.get_manager().await.expect("KMS manager should be running");
    for key_id in key_ids {
-        // The Transit backend deletes in two steps (first call parks the key in
-        // PendingDeletion, the next call destroys it); KV2 destroys on the
-        // first call and reports KeyNotFound on the second.
-        for _ in 0..2 {
-            match kms
-                .delete_key(DeleteKeyRequest {
-                    key_id: key_id.clone(),
-                    pending_window_in_days: None,
-                    force_immediate: Some(true),
-                    confirm_key_id: Some(key_id.clone()),
-                })
-                .await
-            {
-                Ok(_) => continue,
-                Err(KmsError::KeyNotFound { .. }) => break,
-                Err(error) => {
-                    eprintln!("vault key cleanup: could not delete {key_id}: {error:?}");
-                    break;
-                }
-            }
+        // Destruction is two requests: park the key in PendingDeletion, then
+        // destroy it. A spec may have left the key scheduled already (the
+        // schedule is refused, the destruction still applies) or destroyed it.
+        let scheduled = kms
+            .delete_key(DeleteKeyRequest {
+                key_id: key_id.clone(),
+                ..Default::default()
+            })
+            .await;
+        if matches!(scheduled, Err(KmsError::KeyNotFound { .. })) {
+            continue;
+        }
+        match kms
+            .delete_key(DeleteKeyRequest {
+                key_id: key_id.clone(),
+                pending_window_in_days: None,
+                force_immediate: Some(true),
+                confirm_key_id: Some(key_id.clone()),
+            })
+            .await
+        {
+            Ok(_) | Err(KmsError::KeyNotFound { .. }) => {}
+            Err(error) => eprintln!("vault key cleanup: could not delete {key_id}: {error:?}"),
        }
    }
    if let Err(error) = manager.stop().await {
@@ -22,7 +22,7 @@ The wire prefix is `/rustfs/admin/v3`. `GET /kms/status` and `GET /kms/service-s
 | `POST /kms/keys` | `kms:Configure` | high | no | Key creation shares the configure action. On the Local, Vault KV2 and Vault Transit backends the name must be a single path segment: empty names, names containing `/`, `\\` or NUL, and the dot segments `.` and `..` are refused with `400` before any backend request; the AWS backend takes ARNs and aliases and does not apply this rule |
 | `GET /kms/keys` | `kms:ListKeys` | sensitive | no | See the key listing contract below |
 | `GET /kms/keys/{key_id}` | `kms:DescribeKey` | sensitive | yes | `?impact=true` opts into the configuration-reference report |
-| `DELETE /kms/keys/delete` | `kms:DeleteKey` | critical | yes | JSON body; `force_immediate` also requires `confirm_key_id` and the server-side `RUSTFS_KMS_ALLOW_IMMEDIATE_DELETION` gate |
+| `DELETE /kms/keys/delete` | `kms:DeleteKey` | critical | yes | JSON body; schedules the deletion. `force_immediate` destroys a key that is already `PendingDeletion` and also requires `confirm_key_id` and the server-side `RUSTFS_KMS_ALLOW_IMMEDIATE_DELETION` gate; see [KMS key destruction](kms-key-destruction.md) |
 | `POST /kms/keys/cancel-deletion` | `kms:DeleteKey` | high | yes | |
 | `POST /kms/keys/enable` | `kms:EnableKey` | high | yes | |
 | `POST /kms/keys/disable` | `kms:DisableKey` | high | yes | |
@@ -3,7 +3,7 @@
 **Use this when:** choosing a KMS backend, scheduling or debugging master key rotation, planning a rolling upgrade of a cluster with KMS enabled, or auditing where master key material lives and who can read it.
 **Source of truth:** `crates/kms/src/config.rs` (`KmsBackend`, `ENV_KMS_*` constants), `crates/kms/src/backends/{local,vault,vault_transit,aws}.rs`, `crates/kms/src/encryption/dek.rs` (`DataKeyEnvelope`), `rustfs/src/admin/route_policy.rs` (KMS route actions).

-RustFS ships several KMS backends. They differ not only in deployment effort but in **where master key material lives and who can read it**. Pick a backend based on the confidentiality boundary you need, not on the name alone. Related: [Vault KMS authentication runbook](vault-kms-authentication.md) (credential sources, refresh, fail-closed window), [Cryptographic compliance positioning](kms-cryptographic-compliance.md), [Per-key KMS authorization](kms-per-key-authorization.md), [KMS admin API contract](kms-admin-contract.md), [KMS observability runbook](kms-observability-runbook.md).
+RustFS ships several KMS backends. They differ not only in deployment effort but in **where master key material lives and who can read it**. Pick a backend based on the confidentiality boundary you need, not on the name alone. Related: [Vault KMS authentication runbook](vault-kms-authentication.md) (credential sources, refresh, fail-closed window), [Cryptographic compliance positioning](kms-cryptographic-compliance.md), [Per-key KMS authorization](kms-per-key-authorization.md), [KMS key destruction](kms-key-destruction.md), [KMS admin API contract](kms-admin-contract.md), [KMS observability runbook](kms-observability-runbook.md).

 ## Backend comparison

@@ -113,7 +113,7 @@ Decryption loads exactly the version recorded in the envelope and fails closed w

 - Every version record that any stored DEK envelope references must remain readable. The bulk rekey sweep (`POST /rustfs/admin/v3/kms/keys/rekey`, `kms:Rekey`) rewraps stored envelopes onto the current version; until a sweep has completed with zero failures after the last rotation, assume **every** version of a rotated key is referenced. A completed sweep is evidence, not authority — the deletion gate stays the decision point. Replication strips encryption metadata in transit, so each replica site runs its own sweep.
 - Version records are ordinary KV v2 secrets under the key subtree. Never run `kv metadata delete` or `kv destroy` against `{prefix}/{key_id}/versions/*`, and do not apply `delete-version-after` or retention tooling to that subtree. Each version record has a single KV revision, so KV `max-versions` settings neither protect nor endanger history — but metadata deletion removes a record entirely.
- Permanent key deletion through RustFS (`force_immediate` after `PendingDeletion`) purges the key's version records together with the key record; that is the only supported way to remove them. It requires `RUSTFS_KMS_ALLOW_IMMEDIATE_DELETION=true` on the server and a `DELETE` with a JSON body that sets `force_immediate` and echoes the key id as `confirm_key_id`; the query-parameter form is refused outright. Leave the gate off except while actively destroying keys — the pending-deletion window plus `CancelKeyDeletion` is the only recovery path for objects encrypted under the key.
+- Key destruction through RustFS purges the key's version records together with the key record; that is the only supported way to remove them. A key is destroyed either by the background sweep once a scheduled deletion's window has elapsed, or by `force_immediate` on a key that is already `PendingDeletion`; see [KMS key destruction](kms-key-destruction.md) for the full lifecycle and safeguards. It requires `RUSTFS_KMS_ALLOW_IMMEDIATE_DELETION=true` on the server and a `DELETE` with a JSON body that sets `force_immediate` and echoes the key id as `confirm_key_id`; the query-parameter form is refused outright. Leave the gate off except while actively destroying keys — the pending-deletion window plus `CancelKeyDeletion` is the only recovery path for objects encrypted under the key.
 - `force_immediate` is refused with `409 Conflict` while any bucket's default encryption configuration names the key or the key is the KMS service default key. A scheduled deletion is not refused for that reason: it destroys nothing and stays cancellable, and the background sweep re-checks the same references before destroying material.
 - For Vault Transit, retention is governed by the Transit key's `min_decryption_version`: never raise it above the oldest version that may still protect live ciphertext.

@@ -0,0 +1,130 @@
+# KMS key destruction
+
+English | [简体中文](kms-key-destruction.zh-CN.md)
+
+**Use this when:** retiring a KMS key, deciding whether data protected by a key can be made permanently unrecoverable, answering an erasure or key-compromise request, or explaining why a key scheduled for deletion still exists.
+**Source of truth:** `crates/kms/src/manager.rs` (`delete_key_inner`, `check_deletion_request`, `refuse_referenced_immediate_deletion`), `crates/kms/src/deletion_worker.rs` (the sweep and its metrics), `crates/kms/src/types.rs` (`*_PENDING_DELETION_WINDOW_DAYS`), `crates/kms/src/backends/{local,vault,vault_transit,aws}.rs` (`delete_key`, `remove_expired_key`), `rustfs/src/kms_deletion_gate.rs` (configuration references), `rustfs/src/admin/handlers/kms_keys.rs` (`DeleteKmsKeyHandler`).
+
+Destroying a KMS key is a product feature, not a maintenance side effect: it is how RustFS makes data permanently unrecoverable without rewriting the data itself. This page states what the feature guarantees, the steps a key goes through, and the safeguards around it. Related: [KMS admin API contract](kms-admin-contract.md) (routes and IAM actions), [KMS backend security properties](kms-backend-security.md) (where key material lives, version retention), [Per-key KMS authorization](kms-per-key-authorization.md), [KMS observability runbook](kms-observability-runbook.md).
+
+## What destruction means
+
+Every SSE-KMS object, and every SSE-S3 object on a deployment whose SSE-S3 keys come from the KMS, is encrypted with its own data key, and that data key is stored next to the object only in wrapped form — encrypted under a KMS key. Destroying the KMS key removes the only means of unwrapping those data keys. The object bytes stay on disk, and stay replicated and tiered wherever they already are, but nothing can decrypt them again. This is cryptographic erasure.
+
+The feature serves three needs that logical deletion cannot:
+
+- **Erasure obligations.** A tenant leaves, or a deletion request has to cover every copy, including versions, replicas and tiered data that are expensive or impossible to locate and overwrite one by one.
+- **Key compromise.** Material that may have leaked is retired for good once the data under it has been rewrapped or written off.
+- **Decommissioning.** Media that cannot be wiped block by block is made unreadable by destroying the keys instead.
+
+Because one key usually protects many objects, destroying it is the most consequential operation the KMS admin API offers. Every rule below follows from that.
+
+## Guarantees
+
+- **A key is never destroyed in one step.** Destruction always passes through `PendingDeletion`, a state in which the key still exists, objects under it are still readable, and the deletion can still be cancelled. No single request takes an `Enabled` or `Disabled` key to destroyed.
+- **Success means what it says.** A scheduling request that succeeds has destroyed nothing. A destruction request that succeeds has destroyed the key: the key record and every retained version of its material are gone from the backend when the response is returned. A request that cannot do what it was asked is refused; it is never quietly turned into the other kind.
+- **Destruction is final.** RustFS keeps no copy of destroyed material and offers no undelete. Recovery after destruction is possible only from a backup taken outside this feature (see [What destruction does not cover](#what-destruction-does-not-cover)).
+- **Configuration that still names the key blocks destruction.** A key that is the KMS service default key, or that any bucket's default encryption configuration names, is not destroyed — neither by the background sweep nor on request.
+- **Every decision is recorded.** Scheduling, cancellation, destruction and each refusal are written to the KMS audit log with the caller's identity; destruction carried out by the background sweep is recorded as an internal operation.
+
+## Lifecycle
+
+| State | Entered by | New encryption | Reads of existing objects | Leaves by |
+| --- | --- | --- | --- | --- |
+| `Enabled` | Key creation, enable, cancelled deletion | Allowed | Allowed | Disable, schedule deletion |
+| `Disabled` | Disable | Refused | Allowed | Enable, schedule deletion |
+| `PendingDeletion` | Schedule deletion | Refused | Allowed | Cancel deletion, destruction |
+| Destroyed | Window elapsed, or expedited destruction | — | Fail permanently | — |
+
+Reads keep working through `Disabled` and `PendingDeletion` on purpose: the waiting window is only a safety net if the data is still there to notice missing. The AWS KMS backend is the exception — AWS itself refuses decryption for disabled and pending keys; see [Backend differences](#backend-differences).
+
+After destruction, any read of an object encrypted under the key fails with `400 KMS.NotFoundException`, and the key id answers `404` on the admin API. The name is free again, but a key created under it later is a new key with new material: it does not make the old objects readable.
+
+## Step 1: schedule deletion
+
+`DELETE /rustfs/admin/v3/kms/keys/delete` with a JSON body naming `key_id` moves an `Enabled` or `Disabled` key to `PendingDeletion` and records a deadline.
+
+- `pending_window_in_days` sets the waiting window. It must be between 7 and 30 days and defaults to 30; anything outside that range is refused with `400`.
+- The response carries `deletion_date`, the deadline after which the key becomes eligible for destruction, and `impact`, the configuration references found at that moment.
+- Scheduling a key that configuration still references is allowed. It destroys nothing, and the references are checked again before anything is destroyed. Treat a non-empty `impact` as a list of things to clear before the deadline.
+- Scheduling a key that is already `PendingDeletion` is refused with `400`; cancel first to change the deadline.
+
+Use the window to confirm nothing still depends on the key: watch for reads that would fail without it, move bucket defaults to another key, and run the [bulk rekey sweep](../architecture/kms-bulk-rekey-contract.md) if the data must outlive the key.
+
+## Step 2: cancel, or let the window elapse
+
+`POST /rustfs/admin/v3/kms/keys/cancel-deletion` returns a `PendingDeletion` key to `Enabled` and clears the deadline. It works at any point before the key is destroyed, and fails once destruction has begun.
+
+If the deadline passes without a cancellation, the background sweep destroys the key. The sweep runs on every node about once a minute, needs no operator action, and is safe to run concurrently and to repeat after a restart. For each expired key it re-checks the configuration references and the key's state at that moment:
+
+- **Still referenced:** the key is left in `PendingDeletion`, still readable and still cancellable, and is reconsidered on every later sweep. It is destroyed once the references are gone. A key that never gets destroyed is therefore a signal to act on, not a failure to retry.
+- **Cancelled in the meantime:** the cancellation wins and nothing is destroyed.
+- **Destruction interrupted** (crash, backend outage): the key is left unusable rather than half-present, and the next sweep completes the removal.
+
+## Expedited destruction
+
+When waiting for the deadline is not acceptable — an erasure deadline, a confirmed compromise — a key that is already `PendingDeletion` can be destroyed on request, without waiting for its deadline. The same `DELETE` route takes `force_immediate: true`.
+
+Expedited destruction is refused unless every one of these holds:
+
+| Condition | Refusal |
+| --- | --- |
+| The server runs with `RUSTFS_KMS_ALLOW_IMMEDIATE_DELETION=true` | `400` |
+| The request is a JSON body; the query-string form is not accepted | `400` |
+| `confirm_key_id` repeats `key_id` exactly | `400` |
+| The key is already `PendingDeletion` | `400`; schedule the deletion first |
+| No bucket default encryption configuration names the key, and it is not the KMS service default key | `409 Conflict` |
+| The backend can physically delete keys | `501` |
+
+Expedited destruction shortens the wait; it does not remove the cancellable step. The scheduling request and the destruction request are two separate, audited actions, so a single mistaken or malicious call can at worst park a key in a state that `cancel-deletion` undoes.
+
+Keep `RUSTFS_KMS_ALLOW_IMMEDIATE_DELETION` off in normal operation and turn it on only for the duration of a planned destruction. With the gate off, the waiting window is the only path to destruction, whatever the caller's permissions.
+
+## Safeguards and their limits
+
+| Safeguard | What it protects against | What it does not do |
+| --- | --- | --- |
+| Mandatory `PendingDeletion` step | A single request destroying data | Stop two deliberate requests |
+| Waiting window (7–30 days) | Discovering a dependency only after the key is gone | Tell you which objects use the key |
+| Configuration reference check | Destroying a key that new writes are still directed at | See objects already encrypted under the key |
+| Server-side gate for expedited destruction | A credential with `kms:DeleteKey` skipping the window on its own | Apply once an operator has turned it on |
+| `confirm_key_id` | Acting on the wrong key through a mistyped or replayed request | Authorize anything by itself |
+| `kms:DeleteKey`, checked against the named key | Callers without a grant on that key | Separate scheduling from destroying; both, and cancellation, use the same action |
+
+The reference check is deliberately limited to configuration. RustFS does not keep an inventory of which objects are encrypted under which key, so an empty `impact` report never means the key is unused. The objects already written under a key are exactly what destruction is meant to make unreadable; whether that is the intended outcome is the operator's decision, and the waiting window is the time to verify it.
+
+## What destruction does not cover
+
+- **Copies of the key outside RustFS's control.** Vault snapshots and storage backups, backups of the Local backend's key directory, and KMS backups taken through the admin API may still hold the material. Erasure is only complete once those copies are expired or destroyed under their own procedures.
+- **Other keys.** Objects rewrapped onto another key by the rekey sweep before destruction remain readable under that key. Each replication site holds its own wrapped data keys and its own KMS; destroying a key at one site does not affect another.
+- **Data keys already in use.** A request that had unwrapped its data key before the destruction completes can finish. Nodes may also serve cached key metadata for a short time; clear the KMS cache when the cut-off must be immediate.
+- **Plaintext and other encryption modes.** SSE-C objects depend on client-held keys, and unencrypted objects on no key at all.
+- **The bytes on disk.** Ciphertext is not removed. Delete the objects as usual to reclaim space.
+
+## Backend differences
+
+| Backend | Schedule and cancel | Destruction after the window | Expedited destruction | What is removed |
+| --- | --- | --- | --- | --- |
+| Local | Yes | Background sweep | Yes | The key file |
+| Vault KV2 | Yes | Background sweep | Yes | The key record and every version record under it |
+| Vault Transit | Yes | Background sweep | Yes | The Transit key and its metadata record; RustFS sets `deletion_allowed` on the Transit key immediately before deleting it |
+| AWS KMS | Yes; AWS owns the window | Performed by AWS | No (`501`) | Whatever AWS removes; a cancelled key returns as `Disabled`, and reads are refused for the whole pending window |
+| Static | No | — | — | Nothing; the key is configuration, not a managed key |
+
+On the Vault backends the RustFS credential needs delete permission on the key paths for destruction to succeed; see [KMS backend security properties](kms-backend-security.md). Without it, scheduling works and destruction fails at the sweep, which keeps retrying.
+
+## Observing it
+
+- `rustfs_kms_pending_deletion_keys` — keys scheduled and not yet due. A value that never falls after a deadline means destruction is blocked or failing.
+- `rustfs_kms_deletion_tombstone_keys` — keys whose destruction was interrupted and awaits the next sweep. Expected to return to zero within a few sweeps.
+- `rustfs_kms_deletion_sweep_keys_total{outcome}` — `removed`, `blocked` (still referenced), `skipped` (cancelled or not yet due), `failed`, `unreadable`. Alert on sustained `blocked` or `failed`.
+
+The server log names the key and the references whenever the sweep refuses to destroy an expired key. See the [KMS observability runbook](kms-observability-runbook.md) for how the gauges behave when part of the key set is unreadable.
+
+## Procedures
+
+**Retire a key that is no longer needed.** Move every bucket default off the key and change the service default if it is this key. Rekey the data that must survive. Schedule the deletion and read `impact` in the response. Watch for failed reads during the window. Let the sweep destroy the key, then confirm the key id answers `404`.
+
+**Erase data by a deadline.** Schedule the deletion as above and clear every reference. On the day, enable `RUSTFS_KMS_ALLOW_IMMEDIATE_DELETION`, send the expedited request with `confirm_key_id`, confirm the key id answers `404` and that a read of an affected object fails with `KMS.NotFoundException`, then turn the gate off again. Expire the backups that still hold the key.
+
+**Undo a scheduled deletion.** Send `cancel-deletion` before the deadline. On AWS KMS, enable the key afterwards.
@@ -0,0 +1,130 @@
+# KMS 密钥销毁
+
+[English](kms-key-destruction.md) | 简体中文
+
+**适用场景：** 退役一把 KMS 密钥；判断某把密钥保护的数据能否被永久变为不可恢复；响应数据擦除请求或密钥泄露事件；解释为什么一把已计划删除的密钥仍然存在。
+**事实来源：** `crates/kms/src/manager.rs`（`delete_key_inner`、`check_deletion_request`、`refuse_referenced_immediate_deletion`）、`crates/kms/src/deletion_worker.rs`（后台清扫及其指标）、`crates/kms/src/types.rs`（`*_PENDING_DELETION_WINDOW_DAYS`）、`crates/kms/src/backends/{local,vault,vault_transit,aws}.rs`（`delete_key`、`remove_expired_key`）、`rustfs/src/kms_deletion_gate.rs`（配置引用）、`rustfs/src/admin/handlers/kms_keys.rs`（`DeleteKmsKeyHandler`）。
+
+销毁 KMS 密钥是一项产品特性，而不是运维操作的副作用：RustFS 靠它在不重写数据本身的前提下，让数据永久不可恢复。本页说明这项特性保证什么、一把密钥要经过哪些步骤，以及围绕它的保护措施。相关文档（英文）：[KMS admin API contract](kms-admin-contract.md)（路由与 IAM 动作）、[KMS backend security properties](kms-backend-security.md)（密钥材料存放位置、版本保留）、[Per-key KMS authorization](kms-per-key-authorization.md)、[KMS observability runbook](kms-observability-runbook.md)。
+
+## 销毁意味着什么
+
+每个 SSE-KMS 对象，以及 SSE-S3 密钥由 KMS 提供的部署中的每个 SSE-S3 对象，都用各自的数据密钥加密；这把数据密钥只以被包裹的形式（用某把 KMS 密钥加密后）保存在对象旁边。销毁 KMS 密钥，就去掉了解开这些数据密钥的唯一手段。对象的字节仍在磁盘上，已有的副本和分层数据也仍在原处，但再也没有任何东西能解密它们。这就是密码学擦除。
+
+这项特性服务于逻辑删除做不到的三类需求：
+
+- **擦除义务。** 租户退场，或者一次删除请求必须覆盖每一份拷贝——包括逐个定位并覆写代价高昂甚至不可能的历史版本、副本和分层数据。
+- **密钥泄露。** 可能已泄露的密钥材料，在其下的数据完成重新包裹或被判定放弃之后，被永久退役。
+- **介质退役。** 无法逐块擦除的介质，改为销毁密钥使其不可读。
+
+一把密钥通常保护着大量对象，所以销毁它是 KMS 管理 API 中后果最重的操作。下面每一条规则都由此而来。
+
+## 保证
+
+- **密钥永远不会一步销毁。** 销毁必须先经过 `PendingDeletion`：在这个状态下密钥仍然存在，其下的对象仍然可读，删除仍然可以取消。没有任何单个请求能把 `Enabled` 或 `Disabled` 的密钥直接变为已销毁。
+- **成功即如实。** 计划删除的请求成功，意味着什么都没有被销毁。销毁请求成功，意味着密钥确实已销毁：响应返回时，密钥记录及其保留的每一个材料版本都已从后端消失。做不到所请求之事的请求会被拒绝，绝不会被悄悄改成另一种操作。
+- **销毁不可逆。** RustFS 不保留已销毁材料的任何副本，也不提供撤销删除。销毁之后只可能从本特性之外的备份中恢复（见[销毁不覆盖什么](#销毁不覆盖什么)）。
+- **仍被配置引用的密钥不会被销毁。** 作为 KMS 服务默认密钥的密钥，或者被任意桶的默认加密配置指名的密钥，不会被销毁——后台清扫不会，按请求销毁也不会。
+- **每个决定都有记录。** 计划删除、取消、销毁以及每一次拒绝，都连同调用者身份写入 KMS 审计日志；后台清扫执行的销毁记录为内部操作。
+
+## 生命周期
+
+| 状态 | 进入方式 | 新的加密 | 读取已有对象 | 离开方式 |
+| --- | --- | --- | --- | --- |
+| `Enabled` | 创建密钥、启用、取消删除 | 允许 | 允许 | 禁用、计划删除 |
+| `Disabled` | 禁用 | 拒绝 | 允许 | 启用、计划删除 |
+| `PendingDeletion` | 计划删除 | 拒绝 | 允许 | 取消删除、销毁 |
+| 已销毁 | 等待期届满，或加急销毁 | — | 永久失败 | — |
+
+在 `Disabled` 和 `PendingDeletion` 状态下读取照常工作，这是有意为之：只有数据还在、缺失才会被察觉，等待期才算得上安全网。AWS KMS 后端是例外——AWS 自身会拒绝用已禁用和待删除的密钥解密，见[后端差异](#后端差异)。
+
+销毁之后，读取任何用该密钥加密的对象都会以 `400 KMS.NotFoundException` 失败，该密钥 ID 在管理 API 上返回 `404`。这个名字可以再次使用，但之后用同名创建的是一把拥有全新材料的新密钥：它不会让旧对象重新可读。
+
+## 第一步：计划删除
+
+`DELETE /rustfs/admin/v3/kms/keys/delete` 携带指名 `key_id` 的 JSON 请求体，把 `Enabled` 或 `Disabled` 的密钥置为 `PendingDeletion`，并记录截止时间。
+
+- `pending_window_in_days` 设定等待期，必须在 7 到 30 天之间，默认 30 天；超出范围以 `400` 拒绝。
+- 响应携带 `deletion_date`（此时间之后密钥才有资格被销毁）和 `impact`（当时发现的配置引用）。
+- 允许对仍被配置引用的密钥计划删除。这不销毁任何东西，而且在真正销毁之前会重新检查引用。请把非空的 `impact` 当作截止时间之前需要清理的事项清单。
+- 对已处于 `PendingDeletion` 的密钥再次计划删除会以 `400` 拒绝；要修改截止时间请先取消。
+
+请利用等待期确认没有任何东西仍然依赖这把密钥：留意那些没有它就会失败的读取，把桶默认加密改到别的密钥上；如果数据必须比密钥活得更久，运行[批量 rekey 清扫](../architecture/kms-bulk-rekey-contract.md)。
+
+## 第二步：取消，或等待期届满
+
+`POST /rustfs/admin/v3/kms/keys/cancel-deletion` 把 `PendingDeletion` 的密钥恢复为 `Enabled` 并清除截止时间。在密钥被销毁之前的任何时刻都有效；销毁一旦开始即失败。
+
+如果截止时间已过而没有取消，后台清扫会销毁密钥。清扫在每个节点上大约每分钟运行一次，无需运维介入，可以并发运行，也可以在重启后重复运行。对每把到期的密钥，它都会重新检查当时的配置引用和密钥状态：
+
+- **仍被引用：** 密钥保持 `PendingDeletion`，仍可读、仍可取消，并在之后每一轮清扫中被重新考虑；引用消失后即被销毁。因此，一把迟迟没有被销毁的密钥是需要处理的信号，而不是需要重试的故障。
+- **期间被取消：** 取消优先，不销毁任何东西。
+- **销毁被中断**（崩溃、后端故障）：密钥会处于不可用状态而不是半存在状态，下一轮清扫完成移除。
+
+## 加急销毁
+
+当等不到截止时间时——擦除期限将至、泄露已确认——可以按请求销毁一把已处于 `PendingDeletion` 的密钥，无需等待其截止时间。同一个 `DELETE` 路由接受 `force_immediate: true`。
+
+除非以下条件全部满足，否则加急销毁会被拒绝：
+
+| 条件 | 不满足时 |
+| --- | --- |
+| 服务端以 `RUSTFS_KMS_ALLOW_IMMEDIATE_DELETION=true` 运行 | `400` |
+| 请求使用 JSON 请求体；不接受查询字符串形式 | `400` |
+| `confirm_key_id` 与 `key_id` 完全一致 | `400` |
+| 密钥已处于 `PendingDeletion` | `400`；请先计划删除 |
+| 没有任何桶的默认加密配置指名该密钥，且它不是 KMS 服务默认密钥 | `409 Conflict` |
+| 后端支持物理删除密钥 | `501` |
+
+加急销毁缩短的是等待，而不是去掉可取消的那一步。计划删除和销毁是两个独立的、各自被审计的操作，所以单次误操作或恶意调用最坏也只能把密钥停放在一个 `cancel-deletion` 可以撤销的状态。
+
+正常运行时请保持 `RUSTFS_KMS_ALLOW_IMMEDIATE_DELETION` 关闭，只在一次有计划的销毁期间打开。开关关闭时，无论调用者拥有什么权限，等待期都是通往销毁的唯一途径。
+
+## 保护措施及其边界
+
+| 保护措施 | 防范什么 | 做不到什么 |
+| --- | --- | --- |
+| 强制的 `PendingDeletion` 步骤 | 单个请求销毁数据 | 阻止两次有意为之的请求 |
+| 等待期（7–30 天） | 密钥没了之后才发现依赖 | 告诉你哪些对象在用这把密钥 |
+| 配置引用检查 | 销毁一把新写入仍在指向的密钥 | 看到已经用该密钥加密的对象 |
+| 加急销毁的服务端开关 | 持有 `kms:DeleteKey` 的凭据自行跳过等待期 | 运维打开之后继续生效 |
+| `confirm_key_id` | 因输错或重放的请求而操作了错误的密钥 | 自身授予任何权限 |
+| 针对指名密钥校验的 `kms:DeleteKey` | 对该密钥没有授权的调用者 | 区分计划删除与销毁；二者以及取消都使用同一个动作 |
+
+引用检查有意只覆盖配置层。RustFS 不维护"哪些对象由哪把密钥加密"的清单，所以空的 `impact` 报告绝不意味着密钥没人用。已经写在某把密钥之下的对象，恰恰就是销毁要使其不可读的东西；这是不是期望的结果，由运维决定，等待期就是用来核实这一点的。
+
+## 销毁不覆盖什么
+
+- **RustFS 控制范围之外的密钥副本。** Vault 快照与存储备份、Local 后端密钥目录的备份，以及通过管理 API 生成的 KMS 备份，都可能仍然持有密钥材料。只有这些副本按各自的流程过期或销毁之后，擦除才算完成。
+- **其他密钥。** 销毁之前已被 rekey 清扫重新包裹到另一把密钥上的对象，仍可凭那把密钥读取。每个复制站点各自持有自己的被包裹数据密钥和自己的 KMS；在一个站点销毁密钥不影响其他站点。
+- **已在使用中的数据密钥。** 在销毁完成之前已经解开数据密钥的请求可以执行完毕。节点也可能在短时间内继续提供缓存的密钥元数据；如果需要立刻切断，请清除 KMS 缓存。
+- **明文与其他加密模式。** SSE-C 对象依赖客户端持有的密钥，未加密对象不依赖任何密钥。
+- **磁盘上的字节。** 密文不会被移除。要回收空间请照常删除对象。
+
+## 后端差异
+
+| 后端 | 计划与取消 | 等待期届满后的销毁 | 加急销毁 | 移除的内容 |
+| --- | --- | --- | --- | --- |
+| Local | 支持 | 后台清扫 | 支持 | 密钥文件 |
+| Vault KV2 | 支持 | 后台清扫 | 支持 | 密钥记录及其下的每一条版本记录 |
+| Vault Transit | 支持 | 后台清扫 | 支持 | Transit 密钥及其元数据记录；RustFS 在删除前一刻为该 Transit 密钥设置 `deletion_allowed` |
+| AWS KMS | 支持；等待期由 AWS 掌管 | 由 AWS 执行 | 不支持（`501`） | 由 AWS 决定；取消后密钥回到 `Disabled`，且整个待删除期间读取都会被拒绝 |
+| Static | 不支持 | — | — | 无；该密钥是配置，不是受管密钥 |
+
+在 Vault 后端上，RustFS 使用的凭据需要对密钥路径拥有删除权限，销毁才能成功，见 [KMS backend security properties](kms-backend-security.md)。缺少该权限时，计划删除可以成功，而销毁会在清扫时失败并持续重试。
+
+## 可观测性
+
+- `rustfs_kms_pending_deletion_keys`——已计划且尚未到期的密钥数。截止时间之后数值迟迟不降，说明销毁被阻止或正在失败。
+- `rustfs_kms_deletion_tombstone_keys`——销毁被中断、等待下一轮清扫的密钥数。预期在几轮清扫内回到零。
+- `rustfs_kms_deletion_sweep_keys_total{outcome}`——`removed`、`blocked`（仍被引用）、`skipped`（已取消或尚未到期）、`failed`、`unreadable`。请对持续出现的 `blocked` 或 `failed` 告警。
+
+每当清扫拒绝销毁一把到期的密钥，服务端日志都会写明该密钥及其引用。部分密钥不可读时各个 gauge 的表现，见 [KMS observability runbook](kms-observability-runbook.md)。
+
+## 操作流程
+
+**退役一把不再需要的密钥。** 把所有桶的默认加密从该密钥上移走；如果它是服务默认密钥，更换服务默认密钥。对必须保留的数据执行 rekey。计划删除，并阅读响应中的 `impact`。等待期内留意失败的读取。让清扫销毁密钥，然后确认该密钥 ID 返回 `404`。
+
+**在期限前擦除数据。** 按上面的方式计划删除并清除所有引用。到期当天，打开 `RUSTFS_KMS_ALLOW_IMMEDIATE_DELETION`，发送携带 `confirm_key_id` 的加急请求，确认该密钥 ID 返回 `404`、读取受影响对象以 `KMS.NotFoundException` 失败，然后重新关闭开关。让仍持有该密钥的备份过期。
+
+**撤销一次计划删除。** 在截止时间之前发送 `cancel-deletion`。在 AWS KMS 上，之后还需启用该密钥。
@@ -166,7 +166,7 @@ impl Operation for ChangeOwnPasswordHandler {
            .credentials()
            .ok_or_else(|| s3::error(S3ErrorCode::AccessDenied, "authentication required"))?
            .require_secret_key()
-            .map_err(|err| s3::error(S3ErrorCode::InternalError, err.to_string()))?
+            .map_err(|_| s3::error(S3ErrorCode::AccessDenied, "the request credential carries no secret key"))?
            .to_string();
        let caller = CallerIdentity::resolve_envelope(req.envelope()).await?;
        let audit = AccountAuditContext::from_envelope(req.envelope());
@@ -541,6 +541,23 @@ fn validate_new_secret_key(request: &ChangePasswordRequest) -> S3Result<()> {
 mod tests {
    use super::*;
    use crate::server::ADMIN_PREFIX;
+
+    #[tokio::test]
+    async fn change_own_password_without_a_secret_key_is_access_denied() {
+        let req = crate::admin::operation::test_support::TestRequest::new(
+            Method::POST,
+            format!("{ADMIN_PREFIX}/v3/change-own-password")
+                .parse::<http::Uri>()
+                .expect("uri"),
+        )
+        .credentials_without_secret("account-user", None)
+        .build();
+        let err = ChangeOwnPasswordHandler {}
+            .call(req, Params::new())
+            .await
+            .expect_err("a credential without a secret key cannot rotate its password");
+        assert_eq!(err.code(), &S3ErrorCode::AccessDenied);
+    }
    use rustfs_madmin::account::{AccountMutability, CredentialsSource};

    #[test]
@@ -61,12 +61,11 @@ pub fn register_bucket_recovery_route(router: &mut S3Router<AdminOperation>) ->
 impl Operation for RecoverOrphanedBucketHandler {
    async fn call(&self, mut req: AdminRequest, params: Params<'_, '_>) -> S3Result<AdminResponse> {
        authorize_admin_envelope(req.envelope(), vec![Action::AdminAction(AdminAction::RecoverOrphanedBucketAction)]).await?;
-        let secret_key = req
+        let credentials = req
            .credentials()
-            .ok_or_else(|| S3Error::with_message(S3ErrorCode::InvalidRequest, "authentication required"))?
-            .require_secret_key()
-            .map(|secret| Zeroizing::new(secret.to_owned()))
-            .map_err(|err| S3Error::with_message(S3ErrorCode::InternalError, err.to_string()))?;
+            .ok_or_else(|| S3Error::with_message(S3ErrorCode::InvalidRequest, "authentication required"))?;
+        // Checked before the recovery runs, so a refusal leaves the bucket untouched.
+        let response_secret = compat_response_secret(req.uri().path(), credentials.secret_key())?;
        let bucket = params
            .get("bucket")
            .filter(|bucket| !bucket.is_empty())
@@ -125,22 +124,66 @@ impl Operation for RecoverOrphanedBucketHandler {
            "orphaned bucket metadata recovery completed"
        );

-        super::owned_admin_json_response(
-            req.uri().path(),
-            &secret_key,
-            StatusCode::OK,
-            &RecoverOrphanedBucketResponse {
-                bucket: bucket.to_owned(),
-                retired_incarnation_id: request.expected_incarnation_id,
-            },
-        )
+        let response = RecoverOrphanedBucketResponse {
+            bucket: bucket.to_owned(),
+            retired_incarnation_id: request.expected_incarnation_id,
+        };
+        match response_secret {
+            Some(secret_key) => super::owned_admin_json_response(req.uri().path(), &secret_key, StatusCode::OK, &response),
+            None => crate::admin::utils::owned_json_response(StatusCode::OK, &response),
+        }
    }
 }

+/// The key the response body is sealed with. Only the MinIO compat prefix
+/// encrypts admin responses, so only that path needs the caller's secret.
+///
+/// # Errors
+///
+/// `AccessDenied` on the compat prefix when the verified credential carries no
+/// secret key.
+fn compat_response_secret(path: &str, secret: Option<&str>) -> S3Result<Option<Zeroizing<String>>> {
+    if !crate::admin::utils::is_compat_admin_request(path) {
+        return Ok(None);
+    }
+    secret
+        .map(|secret| Some(Zeroizing::new(secret.to_owned())))
+        .ok_or_else(|| S3Error::with_message(S3ErrorCode::AccessDenied, "the request credential carries no secret key"))
+}
+
 #[cfg(test)]
 mod tests {
    use super::*;

+    const OWNED_PATH: &str = "/rustfs/admin/v3/recover-orphaned-bucket/orphan";
+    const COMPAT_PATH: &str = "/minio/admin/v3/recover-orphaned-bucket/orphan";
+
+    #[test]
+    fn compat_recovery_without_a_secret_key_is_refused_before_it_runs() {
+        let err = compat_response_secret(COMPAT_PATH, None).expect_err("compat response needs a sealing key");
+        assert_eq!(err.code(), &S3ErrorCode::AccessDenied);
+    }
+
+    #[test]
+    fn owned_recovery_without_a_secret_key_proceeds_unsealed() {
+        assert!(
+            compat_response_secret(OWNED_PATH, None)
+                .expect("owned path needs no key")
+                .is_none()
+        );
+        assert!(
+            compat_response_secret(OWNED_PATH, Some("secret"))
+                .expect("owned path")
+                .is_none()
+        );
+    }
+
+    #[test]
+    fn compat_recovery_with_a_secret_key_seals_with_it() {
+        let secret = compat_response_secret(COMPAT_PATH, Some("secret")).expect("compat path with key");
+        assert_eq!(secret.as_deref().map(String::as_str), Some("secret"));
+    }
+
    #[test]
    fn recovery_request_rejects_missing_or_nil_generation_identity() {
        assert!(serde_json::from_slice::<RecoverOrphanedBucketRequest>(b"{}").is_err());
@@ -223,6 +223,22 @@ fn map_add_user_create_error(err: rustfs_iam::error::Error) -> S3Error {
    }
 }

+/// The caller's current secret when the request rotates the caller's own key,
+/// `None` when it creates or updates another user (which does not read it).
+///
+/// # Errors
+///
+/// `AccessDenied` for a self-rotation whose verified credential carries no
+/// secret key.
+fn self_rotation_secret(check_deny_only: bool, secret: Option<&str>) -> S3Result<Option<&str>> {
+    if !check_deny_only {
+        return Ok(None);
+    }
+    secret
+        .map(Some)
+        .ok_or_else(|| S3Error::with_message(S3ErrorCode::AccessDenied, "the request credential carries no secret key"))
+}
+
 #[async_trait::async_trait]
 impl Operation for AddUser {
    async fn call(&self, mut req: AdminRequest, _params: Params<'_, '_>) -> S3Result<AdminResponse> {
@@ -242,11 +258,9 @@ impl Operation for AddUser {

        let (cred, owner) =
            check_key_valid(get_session_token(req.uri(), req.headers()).unwrap_or_default(), input_cred.access_key()).await?;
-        // The caller's current secret outlives the body read below.
-        let input_secret = input_cred
-            .require_secret_key()
-            .map_err(|err| S3Error::with_message(S3ErrorCode::InternalError, err.to_string()))?
-            .to_owned();
+        // The caller's current secret outlives the body read below. Only a
+        // self-rotation reads it; creating another user does not.
+        let input_secret = input_cred.secret_key().map(str::to_owned);

        let ak = query.access_key.as_deref().unwrap_or_default();

@@ -327,9 +341,10 @@ impl Operation for AddUser {
        )
        .await?;

-        let updated_at = if check_deny_only {
+        let rotation_secret = self_rotation_secret(check_deny_only, input_secret.as_deref())?;
+        let updated_at = if let Some(current_secret) = rotation_secret {
            let (updated_at, status, _) = iam_store
-                .change_own_secret_key(ak, &input_secret, &args.secret_key)
+                .change_own_secret_key(ak, current_secret, &args.secret_key)
                .await
                .map_err(map_add_user_create_error)?;
            args.status = status;
@@ -1396,6 +1411,23 @@ mod tests {
    use serde_json::Value;
    use std::collections::HashMap;

+    #[test]
+    fn self_rotation_without_a_secret_key_is_refused() {
+        let err = super::self_rotation_secret(true, None).expect_err("self-rotation needs the current secret");
+        assert_eq!(err.code(), &S3ErrorCode::AccessDenied);
+    }
+
+    #[test]
+    fn creating_another_user_does_not_read_the_secret_key() {
+        assert_eq!(super::self_rotation_secret(false, None).expect("create path proceeds"), None);
+        assert_eq!(super::self_rotation_secret(false, Some("current")).expect("create path"), None);
+    }
+
+    #[test]
+    fn self_rotation_uses_the_presented_secret_key() {
+        assert_eq!(super::self_rotation_secret(true, Some("current")).expect("rotation"), Some("current"));
+    }
+
    fn credential_less_request(method: Method, uri: &'static str) -> AdminRequest {
        crate::admin::operation::test_support::TestRequest::new(method, Uri::from_static(uri))
            .headers(HeaderMap::new())
@@ -217,6 +217,15 @@ pub(crate) mod test_support {
            self
        }

+        /// A presented identity whose verified credential carries no secret key.
+        pub(crate) fn credentials_without_secret(mut self, access_key: &str, session_token: Option<&str>) -> Self {
+            self.credentials = Some(RequestCredentials::without_secret_key(
+                access_key.to_owned(),
+                session_token.map(str::to_owned),
+            ));
+            self
+        }
+
        pub(crate) fn extensions(mut self, extensions: Extensions) -> Self {
            self.extensions = extensions;
            self