fix(scanner): correct expiry test result type

* fix(sftp): allow bucket-scoped root listings

Listing the SFTP root required s3:ListAllMyBuckets. Without it a
bucket-scoped user got PermissionDenied on `ls /`; with it they saw every
bucket on the instance. S3 ListBuckets and the WebDAV root (#6298)
already fall back to a per-bucket filtered listing.

When the global ListAllMyBuckets check is denied, list buckets through
the session-aware ListBuckets path instead, so IAM filters the result to
buckets the principal may ListBucket or GetBucketLocation. Principals
with ListAllMyBuckets keep the unfiltered listing, and IAM outages and
timeouts still fail as before.

list_buckets_for_session and its request builder were WebDAV-only; they
are now also compiled with the sftp feature.

* fix(protocols): re-check the session key for bucket-scoped listings

list_buckets_for_session built its request from the identity captured at
login. Through the SFTP root-listing fallback, a service account disabled
or expired after login could keep listing bucket names, unfiltered when
its parent may list every bucket.

Validate the session key with check_key_valid_with_context, as
authorize_protocol_request does for every other protocol operation, build
the request from the fresh credentials and is_owner, map an invalid key
to AccessDenied, and attach the server context.

The SFTP fallback now passes the session's request headers and transport
security, like every other SFTP authorization. DummyBackend fails closed
when no session listing is queued, matching the trait default, and gains
a stall_authorization hook. New tests cover a service account disabled
mid-session, IAM outage and timeout on the global check (no fallback),
non-AccessDenied backend errors, truncation of fallback output and the
fail-closed default.

* test(protocols): compare error code without referencing s3s

The session-listing test compared against s3s::S3ErrorCode, which made
policy_test.rs a new s3s-referencing file and tripped the s3s footprint
ratchet (210 files against a baseline of 209). Compare the code's string
form instead.

* fix(protocols): refresh session bucket discovery without stale copies

Build bucket listing requests directly from freshly validated credentials
and server-scoped ownership instead of cloning and replacing login state.
Reuse the same authorization path for WebDAV capacity discovery.

Preserve typed access-denied checks through the protocol facade, and cover
request conditions, session-policy updates, revocation, listing bounds,
and fallback deadlines with regression tests.

Co-Authored-By: heihutu <heihutu@gmail.com>
Co-Authored-By: zhi22915 <qiuzgang@gmail.com>

---------

Co-authored-by: Hauser <housemecn@gmail.com>
Co-authored-by: heihutu <heihutu@gmail.com>
Co-authored-by: zhi22915 <qiuzgang@gmail.com>

.agents/skills/adversarial-validation/references/performance.md

@@ -6,7 +6,21 @@
 - Route every new sync/flush through the durability-mode and bucket override
   gates; mode `none` must not pay the new fsync.
 - Keep blocking filesystem/CPU work off async runtime threads, but do not split
-  one small operation into many `spawn_blocking` round trips.
+  one small operation into many `spawn_blocking` round trips; coalesce `open`,
+  `read`/`write`, `fsync`, and `close` into one blocking closure per disk. When
+  coalescing steps across execution lanes, route the combined closure to the
+  dedicated fsync lane whenever any step performs file or directory sync so
+  stalled syncs cannot starve foreground metadata reads.
+- Check for per-disk syscall amplification (`disks × parts/files`): pin a
+  directory once (`openat2` on Linux) and reuse the handle across sibling reads
+  or same-directory renames instead of re-walking ancestor components per file
+  or re-opening the same parent directory twice.
+- When the file size is already known from `fstat` metadata, pre-size the
+  destination buffer (`len + 1`) before reading to EOF so small metadata reads
+  complete in a single `read()` + EOF check rather than exponential-growth
+  reallocations and syscalls.
+- Avoid redundant disk re-reads or existence checks under the same exclusive
+  lock when the state or raw bytes were just loaded or persisted.
 - Measure lock hold time across I/O and compare acquisition order for ABBA.
 - Keep cleanup, extra stat/rename, and diagnostics out of the PUT commit critical
   section when they need not be there.

.config/e2e-distributed-selection.txt

@@ -1,2 +1,2 @@
-sha256-linux=14becc458b9eb24add510d94ee6cd04ba1fb19907d598effe1524e5242400d10
-sha256-darwin=14becc458b9eb24add510d94ee6cd04ba1fb19907d598effe1524e5242400d10
+sha256-linux=eb33067d90ee66c0a97e8f3658e46c152e47f897d2fbfdc6cb5ad93cfa55414b
+sha256-darwin=eb33067d90ee66c0a97e8f3658e46c152e47f897d2fbfdc6cb5ad93cfa55414b

.config/e2e-full-selection.txt

@@ -1,2 +1,2 @@
-sha256-darwin=596ca4ce6661a1ccd5c8278a7781fd46922ebe5e23377a254471ec5f4fabfeb0
-sha256-linux=1f3d1d167fbd58a72b46a255edb111c7ee0abdb1ae37ac00400d1239cacdd5e1
+sha256-darwin=a67a347f26be186c598b5beaf68e6bcbe1855350a35e1423dd5b8a81bcba06a5
+sha256-linux=ad4fabe811c6cca1aba89fad34ccd404b5832a6a955875c2826f6e0fa1ea4f1f

.config/e2e-repl-nightly-selection.txt

@@ -1 +1 @@
-sha256=4eac269f9a59b2f9be4e7a87ea6aa38454f5ebe10ede35c9b8d4218137d2cd35
+sha256=e4ee3898b840a8374cb02c64342cc15f8d7ecd1becaf4b4a2a4657290f9e7ff1

.config/e2e-smoke-selection.txt

@@ -1 +1 @@
-sha256=57ffdd620dec9cba08a411b8ad681f4eb906cf5764ffed3c7c992084f5fc6092
+sha256=f065e7071aac9d48a7c1b301f328fa7cc14f11c4e7dc46da3e55057c4ea29bd5

.config/ecstore-required-tests.json

@@ -229,7 +229,7 @@
     {
       "invariant": "list-objects-shared-corpus-orphan-cleanup-requires-authoritative-evidence",
       "suite": "rustfs-ecstore",
-      "name": "store::list_objects::differential_tests::list_objects_shared_corpus_orphan_cleanup_requires_authoritative_evidence"
+      "name": "store::list_objects::differential_tests::list_objects_shared_corpus_orphan_listing_preserves_inputs"
     },
     {
       "invariant": "list-objects-shared-corpus-consumer-drop-joins-real-disk-producer",

.config/make/lint-fmt.mak

@@ -70,11 +70,6 @@ s3s-footprint-check: ## Check the s3s dependency footprint ratchet stays frozen
 	@echo "📦 Checking s3s footprint ratchet..."
 	./scripts/check_s3s_footprint.sh
 
-.PHONY: test-count-ratchet-check
-test-count-ratchet-check: ## Check no workspace crate lost tests or gained #[ignore] against the committed baseline
-	@echo "🧪 Checking per-crate test-count ratchet..."
-	./scripts/check_test_count_ratchet.sh
-
 .PHONY: fips-wording-check
 fips-wording-check: ## Check docs and crates/kms do not over-claim crypto capabilities
 	@echo "📣 Checking cryptographic capability wording guard..."

.config/make/pre-commit.mak

@@ -20,11 +20,11 @@ planning-docs-check: ## Check that no planning-type documents are committed
 	./scripts/check_no_planning_docs.sh
 
 .PHONY: pre-commit
-pre-commit: fmt-check unsafe-code-check architecture-migration-check logging-guardrails-check error-other-ratchet-check tokio-io-uring-check extension-schema-check body-cache-whitelist-check s3s-footprint-check test-count-ratchet-check fips-wording-check embedded-secrets-check test-wiring-check doc-paths-check planning-docs-check quick-check ## Run fast pre-commit checks without clippy/full tests
+pre-commit: fmt-check unsafe-code-check architecture-migration-check logging-guardrails-check error-other-ratchet-check tokio-io-uring-check extension-schema-check body-cache-whitelist-check s3s-footprint-check fips-wording-check embedded-secrets-check test-wiring-check doc-paths-check planning-docs-check quick-check ## Run fast pre-commit checks without clippy/full tests
 	@echo "✅ All pre-commit checks passed!"
 
 .PHONY: pre-pr
-pre-pr: fmt-check unsafe-code-check architecture-migration-check logging-guardrails-check error-other-ratchet-check tokio-io-uring-check extension-schema-check body-cache-whitelist-check s3s-footprint-check test-count-ratchet-check fips-wording-check embedded-secrets-check test-wiring-check doc-paths-check planning-docs-check log-analyzer-rules-check offline-enrollment-e2e-check clippy-check test ## Run full pre-PR checks with clippy and tests
+pre-pr: fmt-check unsafe-code-check architecture-migration-check logging-guardrails-check error-other-ratchet-check tokio-io-uring-check extension-schema-check body-cache-whitelist-check s3s-footprint-check fips-wording-check embedded-secrets-check test-wiring-check doc-paths-check planning-docs-check log-analyzer-rules-check offline-enrollment-e2e-check clippy-check test ## Run full pre-PR checks with clippy and tests
 	@echo "✅ All pre-PR checks passed!"
 
 .PHONY: dev-check

.config/make/tests.mak

@@ -50,10 +50,9 @@ script-tests: ## Run shell script tests
 	./scripts/test_fuzz_runner.sh
 	./scripts/test_python_bin.sh
 	./scripts/check_embedded_secrets.sh --self-test
-	./scripts/check_s3s_footprint.sh --self-test
 	$(RUSTFS_PYTHON_BIN) ./scripts/check_test_wiring.py --self-test
-	$(RUSTFS_PYTHON_BIN) ./scripts/codemods/s3_error_to_s3_types.py --self-test
 	$(RUSTFS_PYTHON_BIN) ./scripts/test_e2e_binary.py
+	$(RUSTFS_PYTHON_BIN) ./scripts/test_inspect_btf_prototypes.py
 	$(RUSTFS_PYTHON_BIN) ./scripts/test_release_build_workflow.py
 	$(RUSTFS_PYTHON_BIN) ./scripts/test_release_glibc.py
 	$(RUSTFS_PYTHON_BIN) ./scripts/test_upgrade_workflow.py

.config/nextest.toml

@@ -345,12 +345,6 @@ threads-required = "num-test-threads"
 filter = 'package(rustfs-ecstore) & test(=store::init::tests::dispatch_manifest_rollback_bounded_concurrency_reaches_tail_behind_slow_member)'
 threads-required = "num-test-threads"
 
-# These short-deadline real-disk progress probes need the test run's capacity
-# for scanner and object-lock handoffs; keep their assertions and deadlines.
-[[profile.default.overrides]]
-filter = '(package(rustfs-scanner) & (test(=scanner::tests::cycle_persist_failure::later_cycle_post_commit_failure_reloads_durable_state) | test(=scanner::tests::running_main_loop_catches_up_pause_cleared_after_startup_observe))) | (package(rustfs-ecstore) & test(=store::init::tests::restored_transitioned_delete_causally_enqueues_free_version))'
-threads-required = "num-test-threads"
-
 # ---------------------------------------------------------------------------
 # ci profile — the strict CI gate (ci.yml `cargo nextest run --profile ci`)
 # ---------------------------------------------------------------------------
@@ -419,12 +413,6 @@ threads-required = "num-test-threads"
 filter = 'package(rustfs-scanner) & (test(/^scanner::backlog::tests::native_/) | test(=scanner_folder::tests::mrf_ownership::mrf_ownership_manager_completion_preserves_scanner_pending) | test(/^scanner_io::tests::service_cohort::/))'
 threads-required = "num-test-threads"
 
-# Keep short-deadline real-disk progress probes from competing with unrelated
-# tests for scanner and object-lock capacity. Assertions and deadlines stay on.
-[[profile.ci.overrides]]
-filter = '(package(rustfs-scanner) & (test(=scanner::tests::cycle_persist_failure::later_cycle_post_commit_failure_reloads_durable_state) | test(=scanner::tests::running_main_loop_catches_up_pause_cleared_after_startup_observe))) | (package(rustfs-ecstore) & test(=store::init::tests::restored_transitioned_delete_causally_enqueues_free_version))'
-threads-required = "num-test-threads"
-
 [[profile.ci.scripts]]
 filter = 'package(rustfs-ecstore) & test(/^(bucket::lifecycle::bucket_lifecycle_ops::tests::manual_transition_worker_result_recovery_marks_unknown_for_corrupt_marker|services::rebalance::entry::tests::real_rebalance_run_fence_loss_blocks_multipart_publication|store::init::tests::(batch_transitioned_delete_uses_free_version_per_item|decommission_entry_(allows_free_version_consumed_before_source_lock|rejects_subquorum_free_version_conflict_and_retains_source|skips_cleanup_only_marker_when_free_version_is_present)|dispatched_tier_delete_recovery_(checks_later_pool_then_commits_after_source_removal|finds_directory_source_on_encoded_set|retains_journal_on_source_metadata_error)|force_tier_remove_blocks_on_physical_free_version_hidden_by_other_pool|legacy_unknown_transition_delete_falls_back_for_single_batch_and_blocks_prefix|multi_pool_(recursive_prefix_rejects_legacy_or_hidden_merge_loser_before_delete|same_remote_tuple_(batch|single)_delete_waits_for_all_sources|same_tuple_recursive_prefix_uses_one_journal_owner|transitioned_delete_persists_one_free_version_per_remote_tuple)|recursive_prefix_partial_(pool|set)_failure_keeps_prepared_cleanup_owners|restored_transitioned_delete_uses_free_version_as_cleanup_owner|stable_transitioned_recursive_prefix_delete_uses_journal_owners|suspended_null_transition_delete_uses_free_version_as_sole_owner|tier_mutation_peer_handler_applies_prepare_commit_and_abort_idempotently|transition_response_loss_persists_unknown_outcome_for_provider_recovery|transition_transaction_recovery_(drops_record_after_confirmed_local_commit|keeps_cleanup_pending_local_commit)|transitioned_delete_(free_version_replays_after_store_restart|local_quorum_failure_rolls_back_without_cleanup_owner|uses_free_version_as_cleanup_owner)|versioned_delete_marker_keeps_transitioned_source_and_remote_object|versioned_explicit_transition_delete_preserves_other_version_then_allows_bucket_delete))$/)'
 setup = 'ecstore-large-stack'
@@ -684,14 +672,6 @@ test-group = 'ecstore-serial-flaky'
 # the PR-lane e2e coverage for scanner usage snapshots consumed by quota and
 # admin surfaces. It uses the same single-node, random-port, isolated-temp-dir
 # fixture as the existing smoke modules.
-#
-# Dual-stack differential (rustfs/backlog#2740): `dual_stack_differential::`
-# starts two single-node servers from the same binary, one per
-# RUSTFS_S3_STACK value, on random ports with isolated temp dirs, and runs one
-# SDK script covering every operation of `impl S3 for FS` against both. Its
-# comparator unit tests need no server. It meets the admission criteria
-# unchanged; the module clause admits a new differential test without another
-# filter edit.
 [profile.e2e-smoke]
 default-filter = """
     package(e2e_test) & (
@@ -700,7 +680,6 @@ default-filter = """
         | test(/^reliant::lifecycle::/)
         | test(/^conditional_delete_objects_test::/)
         | test(/^reliant::tiering::/)
-        | test(/^dual_stack_differential::/)
         | test(/^upgrade_compatibility_test::upgrade_write_readiness_tests::/)
         | test(/^on_demand_migration::(get_basic_test::(get_miss_pulls_inline_and_serves_locally_afterwards|head_miss_answers_from_the_source_without_persisting)|interaction_test::test_odm_admin_config_is_redacted_and_status_counts_match_the_source)$/)
     )

.config/s3s-edge-allowlist.txt

@@ -1,37 +0,0 @@
-# Files that may still reference s3s (`s3s::` or `s3s_sigv4`) when Phase 1 of
-# the gateway migration ends (rustfs/backlog#2734 task T0.6, rustfs/backlog#2741).
-# Consumed only by `scripts/check_s3s_footprint.sh --mode allowlist`.
-#
-# One entry per line: a repository-relative file path or a git `:(glob)`
-# pathspec. `**` matches across directories, `*` stays within one path
-# component, and a directory entry matches everything under it. Blank lines
-# and lines starting with `#` are ignored; an entry may not contain whitespace,
-# so a note belongs on its own comment line.
-#
-# Every Rust file listed by `git grep -l '\bs3s\(::\|_sigv4\)' -- '*.rs'` that
-# no entry matches fails the allowlist mode. Task T1.9 switches CI to that
-# mode; until then `--dry-run` prints the files still outside this list, which
-# is the Phase 1 tracking list.
-#
-# Never add crates/** other than crates/e2e_test: the engine crates leave s3s
-# entirely, and the script rejects such an entry. Only the s3s-facing edge of
-# the rustfs binary and the e2e oracle keep it.
-
-# rustfs binary: the s3s-facing edge that survives Phase 1
-rustfs/src/storage/ecfs.rs
-rustfs/src/storage/s3_compat/**
-rustfs/src/storage/access.rs
-rustfs/src/auth.rs
-rustfs/src/error.rs
-rustfs/src/storage_api.rs
-rustfs/src/app/storage_api.rs
-rustfs/src/admin/router.rs
-rustfs/src/app/metadata_route.rs
-rustfs/src/server/http.rs
-rustfs/src/server/layer.rs
-rustfs/src/server/hybrid.rs
-rustfs/src/protocols/client.rs
-
-# e2e oracle: the test target that verifies S3 behaviour against s3s
-crates/e2e_test/src/fake_s3_target/**
-crates/e2e_test/src/**

.config/s3s-footprint-baseline.txt

@@ -1,17 +0,0 @@
-# Exact baseline for the direct s3s footprint, consumed only by
-# scripts/check_s3s_footprint.sh (rustfs/backlog#1677; rustfs/backlog#2734 T0.4).
-#
-# Each value is the exact measurement of the tree, not a ceiling. CI fails when a
-# measured counter differs from its line here in either direction:
-#   - a larger measurement means new code widened the s3s surface: route it
-#     through the gateway abstractions instead; never raise a value here;
-#   - a smaller measurement means the PR shrank the surface: lower the matching
-#     value to the measured number in the same PR.
-#
-# files           Rust files referencing s3s paths, repo-wide minus crates/e2e_test
-# s3_error_lines  lines holding s3_error!, same scope, minus those the script
-#                 header's rule clears (rustfs_s3_types's macro); fails closed
-# ecstore_files   Rust files referencing s3s paths under crates/ecstore/src
-files=65
-s3_error_lines=289
-ecstore_files=0

.config/test-count-baseline.txt

@@ -1,64 +0,0 @@
-# Per-crate test-count baseline consumed by scripts/check_test_count_ratchet.sh
-# (rustfs/backlog#2734 section 1 rule 3; T0.3 in rustfs/backlog#2738).
-#
-# One row per workspace member: <package>, tests, ignored, tab-separated.
-# tests counts #[test], #[tokio::test], #[tokio::test(...)] and #[test_case(...)]
-# attribute lines under the package directory; ignored counts #[ignore] and
-# #[ignore = "..."]. Both columns are exact: CI fails when a crate's tests fall
-# below or rise above its row, or when its ignored count changes. Raise a row in
-# the PR that adds tests; lowering one is a reviewed statement that tests were
-# intentionally removed. Regenerate with:
-#   scripts/check_test_count_ratchet.sh --update
-e2e_test	988	20
-rustfs	5979	64
-rustfs-audit	70	0
-rustfs-checksums	26	0
-rustfs-common	36	0
-rustfs-concurrency	54	0
-rustfs-config	43	0
-rustfs-connect-symbol-catalog	6	0
-rustfs-credentials	24	0
-rustfs-crypto	67	0
-rustfs-data-usage	55	0
-rustfs-ecstore	7312	2
-rustfs-extension-schema	23	0
-rustfs-filemeta	352	0
-rustfs-heal	1001	3
-rustfs-heal-contracts	7	0
-rustfs-iam	394	0
-rustfs-io-core	49	0
-rustfs-io-metrics	151	0
-rustfs-keystone	29	0
-rustfs-kms	715	25
-rustfs-license	2	0
-rustfs-lifecycle	155	0
-rustfs-lock	138	0
-rustfs-log-analyzer	90	0
-rustfs-madmin	210	0
-rustfs-notify	137	0
-rustfs-object-capacity	92	0
-rustfs-object-data-cache	140	0
-rustfs-obs	381	0
-rustfs-policy	733	0
-rustfs-protocols	739	1
-rustfs-protos	73	0
-rustfs-replication	209	0
-rustfs-rio	206	0
-rustfs-rio-v2	34	6
-rustfs-s3-client	69	0
-rustfs-s3-ops	8	0
-rustfs-s3-types	103	0
-rustfs-s3select-api	207	0
-rustfs-s3select-query	189	2
-rustfs-scanner	1213	20
-rustfs-scanner-metrics	90	0
-rustfs-security-governance	20	0
-rustfs-signer	31	0
-rustfs-storage-api	34	0
-rustfs-targets	603	26
-rustfs-test-utils	1	0
-rustfs-tls-runtime	43	0
-rustfs-trusted-proxies	68	0
-rustfs-utils	247	0
-rustfs-zip	30	0
-zstd	11	0

.github/actions/quick-checks/action.yml

@@ -86,10 +86,6 @@ runs:
       shell: bash
       run: ./scripts/check_s3s_footprint.sh
 
-    - name: Check per-crate test-count ratchet
-      shell: bash
-      run: ./scripts/check_test_count_ratchet.sh
-
     - name: Check cryptographic capability wording
       shell: bash
       run: ./scripts/check_fips_wording.sh

.github/actions/schedule-failure-issue/README.md

@@ -74,7 +74,6 @@ Notes:
 - `.github/workflows/e2e-s3tests.yml` (weekly full s3-tests sweep)
 - `.github/workflows/mint.yml` (weekly multi-SDK mint run)
 - `.github/workflows/fuzz.yml` (nightly fuzz corpus jobs)
-- `.github/workflows/performance-ab.yml` (nightly warp A/B gate)
 - **Pending:** the ci-7 e2e nightly-full workflow does not exist yet
   (backlog#1149 ci-7). When it lands, wire it up with the snippet above.
 

.github/scheduled-validations.json

@@ -18,7 +18,6 @@
     "never_ran_grace_until": "2026-09-08T00:00:00Z"
   },
   { "workflow": ".github/workflows/nightly-gnu.yml", "max_age_hours": 36 },
-  { "workflow": ".github/workflows/performance-ab.yml", "max_age_hours": 36 },
   {
     "workflow": ".github/workflows/runner-hygiene.yml",
     "max_age_hours": 792,

.github/workflows/README.md

@@ -7,7 +7,7 @@
 
 This repository is public. For a public repository, GitHub's standard hosted runners (`ubuntu-latest`, `windows-latest`, `macos-latest`) cost nothing and have no minute cap, whatever plan the organisation is on. GitHub bills only two things: larger hosted runners, which have custom labels and are not used here, and standard runners in private repositories.
 
-Every other label in this repository is capacity RustFS pays for by the hour whether or not a job is running well:
+Every other label in this repository is capacity RustFS pays for. ARC pods and lab machines accrue cost while the machine exists. A Ubicloud label is metered for the minutes a job holds its VM:
 
 | Label | Paid by | What it is | Use it for |
 |---|---|---|---|
@@ -15,7 +15,7 @@ Every other label in this repository is capacity RustFS pays for by the hour whe
 | `windows-latest`, `macos-latest` | nobody | GitHub-hosted standard runners | Windows and macOS targets |
 | `sm-standard-2` | RustFS | ARC pod, Ubuntu 24.04, minimal image: no `gh`, no C toolchain; Docker only on the `dind-` label | only a job that a hosted runner cannot run (see the placement rule) |
 | `sm-standard-4` | RustFS | ARC pod, Ubuntu 24.04, 14 CPUs and 28 GB (the label is not the core count), large disk | workspace-scale Cargo builds |
-| `sm-standard-4-arm` | RustFS | ARC pod, aarch64; reached through the `build.yml` matrix, so `actionlint.yaml` does not list it | native aarch64 release builds |
+| `ubicloud-standard-16-arm` | RustFS | Ubicloud VM, alias of `ubicloud-standard-16-arm-ubuntu-2404`: Ubuntu 24.04 arm64, 16 vCPU, 48 GB, 300 GB disk. Reached through the `build.yml` matrix, so `actionlint.yaml` does not list it | native aarch64 release builds (`linux-aarch64-musl`, `linux-aarch64-gnu`) |
 | `dind-sm-standard-2` | RustFS | ARC pod with Docker-in-Docker | image builds that need `buildx` and a registry push |
 | `smoke-testing`, `pf-testing` | RustFS | long-lived lab machines | the packaged functional chain and performance suites |
 
@@ -40,7 +40,7 @@ A hosted runner is also the only choice for a few things the pods cannot do: `mo
 
 - Every `shared-key` has exactly one writer. `cache-warm.yml` owns `ci-dev`, `ci-feat-rio`, `ci-feat-proto` and `ci-uring`; every other `./.github/actions/setup` call states `cache-save-if` explicitly, and `scripts/security/check_cache_save_if.sh` fails the build when one does not.
 - All readers of a key run on the same image as its writer. Keys that live on hosted runners say so in their name (`ci-uring` is warmed on `ubuntu-latest`; `ci-e2e-distributed-hosted`, `e2e-upgrade-server-hosted`, `e2e-upgrade-tests-hosted`).
-- Moving a job between pools therefore means renaming its key. The new key is empty until a run from `main` (a scheduled run, or a manual dispatch of that workflow from `main`) saves it; until then that lane does a cold build.
+- Moving a job between pools therefore means renaming its key. The new key is empty until a run from `main` (a scheduled run, or a manual dispatch of that workflow from `main`) saves it; until then that lane does a cold build. The `build.yml` aarch64 release legs append `-ubicloud-arm` for this reason. `cache-warm.yml` does not write those keys.
 - `CARGO_BUILD_JOBS` and every other `CARGO*`, `CC*`, `CFLAGS*`, `CXX*`, `CMAKE*` and `RUST*` variable is hashed into the key. `ci.yml` sets `CARGO_BUILD_JOBS` at step level for that reason; promoting it to job level rotates the key for every lane that shares it.
 
 ## Where self-hosted minutes leak
@@ -62,7 +62,7 @@ gh api "repos/rustfs/rustfs/actions/runs/<run_id>/jobs?per_page=100" \
   --jq '.jobs[] | "\(.conclusion)\t\(.labels | join(","))\t\(.started_at)\t\(.completed_at)\t\(.name)"'
 ```
 
-Only rows whose label starts with `sm-`, `dind-`, `smoke-` or `pf-` cost anything. At the October 2026 review, a full pull-request run of `ci.yml` spent about 150 self-hosted minutes, a third of it in jobs that never compiled; treat that as the scale of what a placement mistake costs, not as a current figure.
+Only rows whose label starts with `sm-`, `dind-`, `ubicloud-`, `smoke-` or `pf-` cost anything. At the October 2026 review, a full pull-request run of `ci.yml` spent about 150 self-hosted minutes, a third of it in jobs that never compiled; treat that as the scale of what a placement mistake costs, not as a current figure.
 
 ## Checklist for a new or moved job
 

.github/workflows/audit.yml

@@ -29,7 +29,6 @@ on:
       - 'scripts/test_package_service_scripts.sh'
       - 'scripts/test_package_versions.sh'
       - 'scripts/test_docker_workflow.py'
-      - 'scripts/security/check_performance_ab_workflow.sh'
       - 'scripts/security/check_preview_release_workflow.sh'
       - 'scripts/security/check_tier_artifact_workflow.sh'
       - 'scripts/security/check_workflow_pins.sh'
@@ -48,7 +47,6 @@ on:
       - 'scripts/test_package_service_scripts.sh'
       - 'scripts/test_package_versions.sh'
       - 'scripts/test_docker_workflow.py'
-      - 'scripts/security/check_performance_ab_workflow.sh'
       - 'scripts/security/check_preview_release_workflow.sh'
       - 'scripts/security/check_tier_artifact_workflow.sh'
       - 'scripts/security/check_workflow_pins.sh'
@@ -155,9 +153,6 @@ jobs:
       - name: Check preview release workflow policy
         run: ./scripts/security/check_preview_release_workflow.sh
 
-      - name: Check performance A/B workflow trust boundary
-        run: ./scripts/security/check_performance_ab_workflow.sh
-
       - name: Check tier evidence workflow isolation
         run: ./scripts/security/check_tier_artifact_workflow.sh
 

.github/workflows/btf-prototype-reference.yml

@@ -0,0 +1,75 @@
+name: BTF prototype reference
+
+on:
+  pull_request:
+    paths:
+      - "scripts/inspect_btf_prototypes.py"
+      - "scripts/test_inspect_btf_prototypes.py"
+      - "scripts/fixtures/btf_prototypes.c"
+      - ".github/workflows/btf-prototype-reference.yml"
+  workflow_dispatch:
+
+permissions:
+  contents: read
+
+concurrency:
+  group: btf-prototype-reference-${{ github.ref }}
+  cancel-in-progress: true
+
+jobs:
+  reference:
+    runs-on: ubuntu-latest
+    timeout-minutes: 10
+    steps:
+      - uses: actions/checkout@f548e57e544e1ff5a4c46bf1e1b8685f8e4a348a # v7
+        with:
+          persist-credentials: false
+      - uses: actions/setup-python@5fda3b95a4ea91299a34e894583c3862153e4b97 # v7.0.0
+        with:
+          python-version: "3.12"
+      - name: Check prototype and CLI contracts
+        run: python3 -B scripts/test_inspect_btf_prototypes.py
+      - name: Install only missing native reference tools
+        run: |
+          select_bpftool() {
+            local candidate
+            local candidates=()
+            if command -v bpftool >/dev/null; then
+              candidates+=("$(command -v bpftool)")
+            fi
+            for candidate in /usr/lib/linux-tools-*/bpftool /usr/lib/linux-tools/*/bpftool; do
+              if [[ -x "$candidate" ]]; then
+                candidates+=("$candidate")
+              fi
+            done
+            while IFS= read -r candidate; do
+              if [[ -x "$candidate" ]] && file -Lb "$candidate" | grep -q 'ELF' \
+                && "$candidate" version >/dev/null 2>&1; then
+                printf '%s\n' "$candidate"
+                return 0
+              fi
+            done < <(printf '%s\n' "${candidates[@]}" | sort -Vr)
+            return 1
+          }
+          packages=()
+          command -v clang >/dev/null || packages+=(clang)
+          command -v llvm-objcopy >/dev/null || packages+=(llvm)
+          command -v gcc >/dev/null || packages+=(gcc)
+          select_bpftool >/dev/null || packages+=(linux-tools-generic)
+          if (( ${#packages[@]} )); then
+            sudo apt-get update
+            sudo apt-get install --no-install-recommends -y "${packages[@]}"
+          fi
+          # File parsing needs a working binary, not a host-kernel wrapper.
+          selected_bpftool="$(select_bpftool)"
+          "$selected_bpftool" version
+          printf 'BPFTOOL=%s\n' "$selected_bpftool" >> "$GITHUB_ENV"
+      - name: Compare declarations and layouts with independent references
+        run: |
+          clang -target bpf -O2 -g -c scripts/fixtures/btf_prototypes.c -o "$RUNNER_TEMP/prototypes.o"
+          llvm-objcopy --dump-section ".BTF=$RUNNER_TEMP/prototypes.btf" "$RUNNER_TEMP/prototypes.o"
+          gcc -std=c11 -Wall -Wextra -Werror -DRUSTFS_BTF_LAYOUT_REFERENCE \
+            scripts/fixtures/btf_prototypes.c -o "$RUNNER_TEMP/layout-reference"
+          "$RUNNER_TEMP/layout-reference" > "$RUNNER_TEMP/layout-reference.json"
+          python3 -B scripts/test_inspect_btf_prototypes.py --native-reference "$RUNNER_TEMP/prototypes.btf" \
+            --layout-reference "$RUNNER_TEMP/layout-reference.json"

.github/workflows/build.yml

@@ -181,12 +181,18 @@ jobs:
             selected="all"
           fi
 
-
+          # linux-aarch64-* need a real aarch64 host (placement rule: hardware)
+          # so the packaged binary and console smoke execute the artifact.
+          # ubicloud-standard-16-arm is that host: Ubuntu 24.04 arm64, alias of
+          # ubicloud-standard-16-arm-ubuntu-2404 (16 vCPU, 48 GB). linux-x86_64-* stay on the
+          # ARC sm-standard-4 pool. GNU rows keep cross=true so cargo-zigbuild
+          # pins the glibc 2.31 floor on the same architecture; musl stays a
+          # native cargo build.
           all='{"include":[
             {"target_id":"linux-x86_64-musl","os":"sm-standard-4","target":"x86_64-unknown-linux-musl","cross":false,"platform":"linux","rustflags":"","features":"","glibc":""},
-            {"target_id":"linux-aarch64-musl","os":"sm-standard-4-arm","target":"aarch64-unknown-linux-musl","cross":false,"platform":"linux","rustflags":"","features":"","glibc":""},
+            {"target_id":"linux-aarch64-musl","os":"ubicloud-standard-16-arm","target":"aarch64-unknown-linux-musl","cross":false,"platform":"linux","rustflags":"","features":"","glibc":""},
             {"target_id":"linux-x86_64-gnu","os":"sm-standard-4","target":"x86_64-unknown-linux-gnu","cross":true,"platform":"linux","rustflags":"","features":"pyroscope","glibc":"2.31"},
-            {"target_id":"linux-aarch64-gnu","os":"sm-standard-4-arm","target":"aarch64-unknown-linux-gnu","cross":true,"platform":"linux","rustflags":"","features":"pyroscope","glibc":"2.31"},
+            {"target_id":"linux-aarch64-gnu","os":"ubicloud-standard-16-arm","target":"aarch64-unknown-linux-gnu","cross":true,"platform":"linux","rustflags":"","features":"pyroscope","glibc":"2.31"},
             {"target_id":"macos-aarch64","os":"macos-latest","target":"aarch64-apple-darwin","cross":false,"platform":"macos","rustflags":"","features":"pyroscope","glibc":""},
             {"target_id":"windows-x86_64","os":"windows-latest","target":"x86_64-pc-windows-msvc","cross":false,"platform":"windows","rustflags":"","features":"","glibc":""}
           ]}'
@@ -244,7 +250,11 @@ jobs:
         with:
           rust-version: stable
           target: ${{ matrix.target }}
-          cache-shared-key: build-${{ matrix.target }}${{ matrix.glibc && format('-glibc-{0}', matrix.glibc) || '' }}
+          # rust-cache keys include OS and architecture but not the runner image.
+          # The Ubicloud aarch64 legs therefore use their own suffix so they
+          # cannot restore a cache written on a different arm64 image.
+          # x86_64, macOS, and Windows keys stay unsuffixed.
+          cache-shared-key: build-${{ matrix.target }}${{ matrix.glibc && format('-glibc-{0}', matrix.glibc) || '' }}${{ matrix.os == 'ubicloud-standard-16-arm' && '-ubicloud-arm' || '' }}
           # main only. A cache saved on refs/tags/X is scoped to that tag: no
           # other tag, no main run and no PR can restore it, so every release
           # cycle wrote up to 12 entries of 1-2GB (preview tag plus final tag,
@@ -764,9 +774,9 @@ jobs:
             exit 0
           fi
 
-          # The self-hosted Linux runners do not ship the aws CLI (GitHub-hosted
-          # images did). Install the official CLI bundle on demand so R2 uploads
-          # survive a fresh runner without relying on a distro package.
+          # ARC pods (the x86_64 legs) do not ship the aws CLI. Ubicloud arm64
+          # images follow the GitHub runner image and often do. Install the
+          # official CLI bundle when it is missing so R2 uploads work on either.
           if ! command -v aws >/dev/null 2>&1; then
             echo "aws CLI not found on runner; installing..."
             if command -v apt-get >/dev/null 2>&1; then

.github/workflows/ci.yml

@@ -16,9 +16,7 @@ name: Continuous Integration
 
 on:
   push:
-    # gateway/integration receives the daily merge of main as a direct push
-    # (rustfs/backlog#2734), so its pushes must prove the result like main's.
-    branches: [ main, gateway/integration ]
+    branches: [ main ]
     paths-ignore:
       - "**.md"
       - "docs/**"
@@ -739,8 +737,6 @@ jobs:
     needs: [ build-rustfs-debug-binary ]
     runs-on: sm-standard-2
     timeout-minutes: 30
-    env:
-      RUSTFS_S3_STACK: legacy
     steps:
       - name: Checkout repository
         uses: actions/checkout@f548e57e544e1ff5a4c46bf1e1b8685f8e4a348a # v7
@@ -859,105 +855,18 @@ jobs:
           path: ${{ runner.temp }}/rustfs-e2e-*/rustfs.log
           retention-days: 3
 
-  # The same e2e-smoke suite against the same debug binary with the gateway
-  # S3 stack selected (rustfs/backlog#2734). Report-only: it is outside the
-  # Test and Lint aggregate and becomes required at T2.10. The archive and run
-  # steps must stay identical to e2e-tests; scripts/ci_gate.py enforces that.
-  e2e-tests-gateway:
-    name: e2e-smoke (gateway, report-only until T2.10)
-    needs: [ build-rustfs-debug-binary ]
-    runs-on: sm-standard-2
-    timeout-minutes: 30
-    continue-on-error: true
-    env:
-      RUSTFS_S3_STACK: gateway
-    steps:
-      - name: Checkout repository
-        uses: actions/checkout@f548e57e544e1ff5a4c46bf1e1b8685f8e4a348a # v7
-        with:
-          persist-credentials: false
-
-      - name: Setup Rust environment
-        uses: ./.github/actions/setup
-        with:
-          rust-version: stable
-          cache-shared-key: ci-dev
-          cache-save-if: 'false'
-          install-build-packaging-tools: 'false'
-
-      - name: Set up Python
-        uses: actions/setup-python@5fda3b95a4ea91299a34e894583c3862153e4b97 # v7.0.0
-        with:
-          python-version: "3.12"
-
-      - name: Install awscurl
-        run: |
-          python3 -m pip install --user --upgrade pip "awscurl==0.44"
-          echo "AWSCURL_PATH=$HOME/.local/bin/awscurl" >> "$GITHUB_ENV"
-
-      - name: Verify awscurl
-        run: test -x "$AWSCURL_PATH"
-
-      - name: Download debug binary
-        uses: actions/download-artifact@3e5f45b2cfb9172054b4087a40e8e0b5a5461e7c # v8.0.1
-        with:
-          name: rustfs-debug-binary
-          path: target/debug
-
-      - name: Make binary executable
-        run: chmod +x ./target/debug/rustfs
-
-      - name: Archive e2e smoke test binaries
-        env:
-          NEXTEST_ARCHIVE: ${{ runner.temp }}/rustfs-e2e-smoke.tar.zst
-          NEXTEST_LISTING: ${{ runner.temp }}/rustfs-e2e-smoke-list.json
-        run: |
-          cargo nextest archive --profile e2e-smoke -p e2e_test --archive-file "${NEXTEST_ARCHIVE}"
-          cargo nextest list --profile e2e-smoke --archive-file "${NEXTEST_ARCHIVE}" --message-format json > "${NEXTEST_LISTING}"
-          python3 ./scripts/check_test_wiring.py --check-profile e2e-smoke "${NEXTEST_LISTING}"
-          ./scripts/check_security_smoke_count.sh check "${NEXTEST_LISTING}"
-
-      - name: Run e2e smoke suite
-        env:
-          NEXTEST_ARCHIVE: ${{ runner.temp }}/rustfs-e2e-smoke.tar.zst
-          RUSTFS_E2E_LOG_DIR: ${{ runner.temp }}/rustfs-e2e-smoke-logs
-          SAMPLER_INTERVAL_SECS: "15"
-        run: |
-          ./scripts/ci/resource_sampler.sh start e2e-smoke
-          trap './scripts/ci/resource_sampler.sh stop' EXIT
-          python3 scripts/e2e_binary.py run --features e2e-test-hooks -- cargo nextest run --profile e2e-smoke --archive-file "${NEXTEST_ARCHIVE}" \
-            --status-level all --final-status-level all --failure-output final
-
-      - name: Upload e2e smoke diagnostics
-        if: always()
-        uses: actions/upload-artifact@043fb46d1a93c77aae656e7c1c64a875d1fc6a0a # v7.0.1
-        with:
-          name: e2e-smoke-gateway-diagnostics-${{ github.run_number }}
-          path: |
-            ${{ runner.temp }}/rustfs-e2e-smoke-logs/
-            ${{ runner.temp }}/rustfs-e2e-smoke-list.json
-          if-no-files-found: warn
-
-      - name: Upload e2e smoke JUnit report
-        if: always()
-        uses: actions/upload-artifact@043fb46d1a93c77aae656e7c1c64a875d1fc6a0a # v7.0.1
-        with:
-          name: e2e-smoke-gateway-junit-${{ github.run_number }}
-          path: target/nextest/e2e-smoke/junit.xml
-          if-no-files-found: warn
-
   e2e-full:
     name: End-to-End Tests (full merge gate)
     # Merge gate only (backlog#1149 ci-5): the never-automated user-visible
     # suites — KMS, object_lock, multipart_auth, quota, checksum, encryption,
     # security-boundary, ... — via the e2e-full nextest profile. Too heavy for
-    # every PR, so it is gated to main and gateway/integration pushes, the merge
-    # queue, and manual dispatch. protocols / the 7 cluster suites / replication
-    # / #[ignore] are owned by other lanes (see .config/nextest.toml profile.e2e-full).
+    # every PR, so it is gated to main pushes, the merge queue, and manual
+    # dispatch. protocols / the 7 cluster suites / replication / #[ignore] are
+    # owned by other lanes (see .config/nextest.toml profile.e2e-full).
     if: >-
       github.event_name == 'workflow_dispatch' ||
       github.event_name == 'merge_group' ||
-      (github.event_name == 'push' && (github.ref == 'refs/heads/main' || github.ref == 'refs/heads/gateway/integration'))
+      (github.event_name == 'push' && github.ref == 'refs/heads/main')
     needs: [ build-rustfs-debug-binary ]
     runs-on: sm-standard-2
     timeout-minutes: 55

.github/workflows/coverage.yml

@@ -45,7 +45,7 @@ on:
     # 07:00 UTC Sunday — staggered clear of the other Sunday crons: ci (00:00),
     # build (01:00), e2e-s3tests (02:00), audit (03:00), nix-flake-update
     # (05:00), mint (06:00), and the daily fuzz (02:00), minio-interop (03:17),
-    # e2e-replication-nightly (04:00) and performance-ab (06:00) lanes.
+    # and e2e-replication-nightly (04:00) lanes.
     - cron: "43 7 * * 0"
 
 concurrency:

.github/workflows/e2e-replication-nightly.yml

@@ -38,8 +38,8 @@ name: e2e-nightly
 on:
   workflow_dispatch:
   schedule:
-    # 04:00 UTC nightly — staggered clear of fuzz/e2e-s3tests (02:00),
-    # stale (01:30) and performance-ab (06:00).
+    # 04:00 UTC nightly — staggered clear of fuzz/e2e-s3tests (02:00)
+    # and stale (01:30).
     - cron: "29 4 * * *"
 
 # Only alert-on-failure needs more than read access; it declares its own

.github/workflows/performance-ab.yml

@@ -1,384 +0,0 @@
-# Copyright 2024 RustFS Team
-#
-# Licensed under the Apache License, Version 2.0 (the "License");
-# you may not use this file except in compliance with the License.
-# You may obtain a copy of the License at
-#
-#     http://www.apache.org/licenses/LICENSE-2.0
-#
-# Unless required by applicable law or agreed to in writing, software
-# distributed under the License is distributed on an "AS IS" BASIS,
-# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
-# See the License for the specific language governing permissions and
-# limitations under the License.
-
-# Warp A/B budget gate for the hotpath series (rustfs/backlog#935 HP-14, item 4).
-#
-# Two entry points, honestly scoped:
-#   * schedule (nightly, on main): post-merge detection — catches a regression
-#     within 24h of landing, not before merge.
-#   * workflow_dispatch: an explicitly selected trusted ref.
-# The dispatch input can run the gate with --allow-regression so a deliberate
-# correctness cost (e.g. the #4221 fsync durability fix) is recorded, not
-# blocked (rustfs/backlog#935 correction 1).
-
-name: Performance A/B
-
-on:
-  schedule:
-    - cron: "31 6 * * *" # 06:31 UTC nightly, against main
-  workflow_dispatch:
-    inputs:
-      duration:
-        description: "warp duration per round"
-        required: false
-        default: "12s"
-        type: string
-      allow_regression:
-        description: "Pass the gate despite a FAIL (deliberate tradeoff)"
-        required: false
-        default: false
-        type: boolean
-permissions:
-  actions: read
-  contents: read
-
-env:
-  CARGO_TERM_COLOR: always
-  RUST_BACKTRACE: 1
-
-jobs:
-  warp-ab:
-    if: github.repository == 'rustfs/rustfs'
-    name: Warp A/B budget gate
-    runs-on: sm-standard-2
-    # A normal nightly restores the last successful binary and builds only the
-    # candidate; daily access keeps that cache warm. A cache miss may build both
-    # and needs room for the A/B run plus artifact and cache publication.
-    timeout-minutes: 180
-    steps:
-      - name: Checkout repository
-        uses: actions/checkout@f548e57e544e1ff5a4c46bf1e1b8685f8e4a348a # v7
-        with:
-          persist-credentials: false
-          fetch-depth: 0 # baseline may be an earlier successful scheduled head
-
-      - name: Setup Rust environment
-        uses: ./.github/actions/setup
-        with:
-          rust-version: stable
-          cache-shared-key: warp-ab-${{ hashFiles('**/Cargo.lock') }}
-          cache-save-if: ${{ github.ref == 'refs/heads/main' }}
-
-      - name: Install warp
-        run: |
-          set -euo pipefail
-          WARP_VERSION="1.0.0"
-          curl -fsSL "https://github.com/minio/warp/releases/download/v${WARP_VERSION}/warp_Linux_x86_64.tar.gz" \
-            | sudo tar -xz -C /usr/local/bin warp
-          warp --version
-
-      - name: Decide exemption
-        id: exempt
-        env:
-          INPUT_ALLOW_REGRESSION: ${{ github.event.inputs.allow_regression }}
-        run: |
-          allow="false"
-          if [[ "$INPUT_ALLOW_REGRESSION" == "true" ]]; then
-            allow="true"
-          fi
-          echo "allow_regression=$allow" >> "$GITHUB_OUTPUT"
-
-      # A failed regression run must keep comparing against the last known-good
-      # scheduled head. Otherwise the next nightly would absorb the regression
-      # into its baseline and turn green without a fix.
-      - name: Find last successful scheduled baseline
-        id: scheduled_baseline
-        if: github.event_name == 'schedule'
-        uses: actions/github-script@3a2844b7e9c422d3c10d287c895573f7108da1b3 # v9.0.0
-        with:
-          result-encoding: string
-          script: |
-            const { data } = await github.rest.actions.listWorkflowRuns({
-              owner: context.repo.owner,
-              repo: context.repo.repo,
-              workflow_id: "performance-ab.yml",
-              event: "schedule",
-              status: "success",
-              per_page: 1,
-            });
-            return data.workflow_runs[0]?.head_sha ?? "";
-
-      # Manual runs compare a selected ref with current main. Scheduled runs
-      # compare current main with the last successful scheduled head. With no
-      # history, the first run measures the candidate against itself and seeds
-      # that head only if the complete rig succeeds.
-      - name: Resolve baseline / candidate commits
-        id: commits
-        env:
-          SCHEDULED_BASELINE_SHA: ${{ steps.scheduled_baseline.outputs.result }}
-        run: |
-          set -euo pipefail
-          candidate_sha="$(git rev-parse HEAD)"
-          if [[ "${{ github.event_name }}" == "schedule" ]]; then
-            baseline_sha="${SCHEDULED_BASELINE_SHA:-$candidate_sha}"
-          else
-            baseline_sha="$(git rev-parse origin/main)"
-          fi
-          git cat-file -e "${baseline_sha}^{commit}"
-          if ! git merge-base --is-ancestor "$baseline_sha" "$candidate_sha"; then
-            echo "::error::baseline $baseline_sha is not an ancestor of candidate $candidate_sha; update the selected ref before comparing" >&2
-            exit 1
-          fi
-          echo "baseline_sha=$baseline_sha" >> "$GITHUB_OUTPUT"
-          echo "candidate_sha=$candidate_sha" >> "$GITHUB_OUTPUT"
-          echo "baseline commit: $baseline_sha"
-          echo "candidate commit: $candidate_sha"
-
-      # Exact-key restore of the candidate binary saved by its successful
-      # scheduled run. A miss leaves cache-hit unset and falls back to a source
-      # build of that known-good head.
-      - name: Restore cached baseline binary
-        id: baseline_cache
-        uses: actions/cache/restore@55cc8345863c7cc4c66a329aec7e433d2d1c52a9 # v6
-        with:
-          path: baseline-bin/rustfs
-          key: rustfs-baseline-${{ steps.commits.outputs.baseline_sha }}
-
-      # Self-heal: on a nightly/dispatch run where the candidate commit IS the
-      # baseline commit, a cache miss would make the rig build the same commit
-      # twice (~65min per side with the post-#4806 LTO profile — no job budget
-      # fits that). Build it once here, reuse it for both phases, and save it
-      # back to the cache so the next run hits.
-      - name: Build baseline on cache miss (same-commit self-heal)
-        id: selfheal
-        if: >-
-          steps.baseline_cache.outputs.cache-hit != 'true' &&
-          steps.commits.outputs.baseline_sha == steps.commits.outputs.candidate_sha
-        run: |
-          set -euo pipefail
-          cargo build --release --bin rustfs
-          mkdir -p baseline-bin
-          cp target/release/rustfs baseline-bin/rustfs
-          echo "built=true" >> "$GITHUB_OUTPUT"
-
-      - name: Build baseline on cache miss (different candidate)
-        id: baseline_build
-        if: >-
-          steps.baseline_cache.outputs.cache-hit != 'true' &&
-          steps.commits.outputs.baseline_sha != steps.commits.outputs.candidate_sha
-        run: |
-          set -euo pipefail
-          baseline_root="$RUNNER_TEMP/rustfs-baseline-${{ github.run_id }}"
-          baseline_target="$RUNNER_TEMP/rustfs-baseline-target-${{ github.run_id }}"
-          git worktree add --detach "$baseline_root" "${{ steps.commits.outputs.baseline_sha }}"
-          cargo build --release --manifest-path "$baseline_root/Cargo.toml" --bin rustfs --target-dir "$baseline_target"
-          mkdir -p baseline-bin
-          cp "$baseline_target/release/rustfs" baseline-bin/rustfs
-          git worktree remove --force "$baseline_root"
-          echo "built=true" >> "$GITHUB_OUTPUT"
-
-      - name: Build candidate binary
-        id: candidate_build
-        if: steps.commits.outputs.baseline_sha != steps.commits.outputs.candidate_sha
-        run: |
-          set -euo pipefail
-          cargo build --release --bin rustfs
-          mkdir -p candidate-bin
-          cp target/release/rustfs candidate-bin/rustfs
-          echo "built=true" >> "$GITHUB_OUTPUT"
-
-      - name: Save self-healed baseline to cache
-        if: steps.selfheal.outputs.built == 'true' || steps.baseline_build.outputs.built == 'true'
-        uses: actions/cache/save@55cc8345863c7cc4c66a329aec7e433d2d1c52a9 # v6
-        with:
-          path: baseline-bin/rustfs
-          key: rustfs-baseline-${{ steps.commits.outputs.baseline_sha }}
-
-      - name: Run warp A/B and gate
-        id: ab
-        env:
-          INPUT_DURATION: ${{ github.event.inputs.duration }}
-        run: |
-          set -euo pipefail
-          # The formal runner executes A1 baseline -> B1 candidate -> B2 candidate
-          # -> A2 baseline for each workload and drive-sync cell. It requires three
-          # rounds per leg to emit tail latency and error-rate evidence.
-          # --health-timeout 180 outlasts the server's own 120s startup-readiness
-          # budget, which the rig's previous 60s health poll undershot (the first
-          # two nightly failures). perf-6 recalibrates these once the noise study
-          # lands.
-          duration="${INPUT_DURATION:-12s}"
-          baseline_sha="${{ steps.commits.outputs.baseline_sha }}"
-          candidate_sha="${{ steps.commits.outputs.candidate_sha }}"
-          baseline_hit="${{ steps.baseline_cache.outputs.cache-hit }}"
-          selfheal_built="${{ steps.selfheal.outputs.built }}"
-          baseline_built="${{ steps.baseline_build.outputs.built }}"
-          candidate_built="${{ steps.candidate_build.outputs.built }}"
-
-          args=(--duration "$duration" --rounds 3 --cooldown 5 --health-timeout 180 --baseline-revision "$baseline_sha" --candidate-revision "$candidate_sha")
-
-          if [[ "$baseline_hit" == "true" || "$selfheal_built" == "true" || "$baseline_built" == "true" ]]; then
-            chmod +x baseline-bin/rustfs
-            base_bin="$PWD/baseline-bin/rustfs"
-            args+=(--baseline-bin "$base_bin")
-            if [[ "$baseline_hit" == "true" ]]; then
-              base_src="actions-cache (rustfs-baseline-$baseline_sha)"
-            elif [[ "$selfheal_built" == "true" ]]; then
-              base_src="source build (cache self-heal, saved as rustfs-baseline-$baseline_sha)"
-            else
-              base_src="isolated baseline source build (saved as rustfs-baseline-$baseline_sha)"
-            fi
-            if [[ "$candidate_sha" == "$baseline_sha" ]]; then
-              # No commits landed since the last successful baseline, so reuse
-              # the one binary for both phases and measure only rig drift.
-              args+=(--candidate-bin "$base_bin")
-              cand_src="same binary as baseline (same commit)"
-            elif [[ "$candidate_built" == "true" ]]; then
-              chmod +x candidate-bin/rustfs
-              args+=(--candidate-bin "$PWD/candidate-bin/rustfs")
-              cand_src="source build of the checked-out ref"
-            else
-              echo "::error::candidate binary was not built" >&2
-              exit 2
-            fi
-          else
-            echo "::error::baseline binary was not restored or built" >&2
-            exit 2
-          fi
-
-          echo "baseline binary: $base_src"
-          echo "candidate binary: $cand_src"
-
-          if [[ "${{ steps.exempt.outputs.allow_regression }}" == "true" ]]; then
-            args+=(--allow-regression --exemption-reason "workflow dispatch override")
-          fi
-          # Do not let a gate FAIL abort the job here; capture status and surface
-          # it after the step summary is written.
-          set +e
-          bash scripts/run_hotpath_warp_abba.sh "${args[@]}"
-          echo "status=$?" >> "$GITHUB_OUTPUT"
-          set -e
-          # Locate the newest run dir + gate.md for the summary/comment/artifact
-          # steps. On a startup failure there is no gate.md, but the run dir still
-          # holds server-logs/ for diagnosis.
-          # Run dirs are UTC-timestamp names (no special chars); ls is safe here.
-          # shellcheck disable=SC2012
-          run_dir="$(ls -td target/hotpath-abba/*/ 2>/dev/null | head -n1 || true)"
-          echo "run_dir=${run_dir%/}" >> "$GITHUB_OUTPUT"
-          # shellcheck disable=SC2012
-          gate_md="$(ls -t target/hotpath-abba/*/candidate_gate.md 2>/dev/null | head -n1 || true)"
-          echo "gate_md=$gate_md" >> "$GITHUB_OUTPUT"
-
-      - name: Upload A/B results
-        if: always()
-        uses: actions/upload-artifact@043fb46d1a93c77aae656e7c1c64a875d1fc6a0a # v7.0.1
-        with:
-          name: hotpath-warp-ab-${{ github.run_number }}
-          # Includes per-cell median_summary.csv / baseline_compare.csv, both gates,
-          # and server-logs/ (rustfs.log + startup env per phase) so a failed run
-          # is diagnosable. Short retention: this is churny nightly debug data.
-          path: target/hotpath-abba/
-          if-no-files-found: warn
-          retention-days: 14
-
-      - name: Write gate summary
-        if: always()
-        run: |
-          set -euo pipefail
-          status="${{ steps.ab.outputs.status }}"
-          gate_md="${{ steps.ab.outputs.gate_md }}"
-          run_dir="${{ steps.ab.outputs.run_dir }}"
-          {
-            echo "## Hotpath warp A/B — run ${{ github.run_number }}"
-            echo
-            if [[ "$status" == "0" ]]; then
-              echo "Rig/gate exit: \`0\` (pass or warn)."
-            else
-              echo "Rig/gate exit: \`${status:-unknown}\` — **FAILED**."
-            fi
-            echo
-            if [[ -n "$gate_md" && -f "$gate_md" ]]; then
-              cat "$gate_md"
-            else
-              echo "No \`gate.md\` produced — the rig failed **before** the gate"
-              echo "(most likely server startup / health). Failing phase(s) below;"
-              echo "full logs in the \`hotpath-warp-ab-${{ github.run_number }}\` artifact."
-              if [[ -n "$run_dir" && -d "$run_dir/server-logs" ]]; then
-                for f in "$run_dir"/server-logs/*.log; do
-                  [[ -f "$f" ]] || continue
-                  echo
-                  echo "<details><summary>$(basename "$f")</summary>"
-                  echo
-                  echo '```'
-                  tail -n 30 "$f"
-                  echo '```'
-                  echo "</details>"
-                done
-              fi
-            fi
-          } >> "$GITHUB_STEP_SUMMARY"
-
-      - name: Stage successful candidate baseline
-        if: >-
-          steps.ab.outputs.status == '0' &&
-          steps.commits.outputs.baseline_sha != steps.commits.outputs.candidate_sha
-        run: |
-          set -euo pipefail
-          cp candidate-bin/rustfs baseline-bin/rustfs
-
-      - name: Cache successful candidate baseline
-        if: >-
-          steps.ab.outputs.status == '0' &&
-          steps.commits.outputs.baseline_sha != steps.commits.outputs.candidate_sha
-        uses: actions/cache/save@55cc8345863c7cc4c66a329aec7e433d2d1c52a9 # v6
-        with:
-          path: baseline-bin/rustfs
-          key: rustfs-baseline-${{ steps.commits.outputs.candidate_sha }}
-
-      # Scheduled failure alerting is handled by the alert-on-failure job below
-      # (perf-2 consuming ci-8's schedule-failure-issue composite action).
-
-      - name: Enforce gate
-        if: always()
-        run: |
-          status="${{ steps.ab.outputs.status }}"
-          if [[ -z "$status" ]]; then
-            echo "::error::warp A/B setup failed before the rig ran. Check the first failed workflow step." >&2
-            exit 1
-          fi
-          if [[ "$status" != "0" ]]; then
-            echo "::error::warp A/B budget gate failed (exit $status). See the step summary / gate.md artifact." >&2
-            exit "$status"
-          fi
-          echo "warp A/B budget gate passed."
-
-  alert-on-failure:
-    name: Alert on scheduled failure
-    needs: [warp-ab]
-    # `always()` is required: without it this job is skipped when a needed
-    # job fails. Alerts only for scheduled (nightly) runs (backlog#1149
-    # ci-8); manual dispatch failures are already watched by a human.
-    # `cancelled` is included alongside `failure` on purpose: a job that hits
-    # timeout-minutes ends as `cancelled`, and the 2026-07-11..07-14 nightly
-    # timeouts went silent precisely because the guard was failure-only. The
-    # composite action already reports cancelled/timed-out jobs in the issue
-    # body.
-    if: >-
-      github.repository == 'rustfs/rustfs' && (
-        always() && github.event_name == 'schedule' &&
-        (contains(needs.*.result, 'failure') || contains(needs.*.result, 'cancelled'))
-      )
-    runs-on: ubuntu-latest
-    timeout-minutes: 10
-    permissions:
-      contents: read
-      issues: write
-    steps:
-      - uses: actions/checkout@f548e57e544e1ff5a4c46bf1e1b8685f8e4a348a # v7
-        with:
-          persist-credentials: false
-      - name: Open or update failure-tracking issue
-        uses: ./.github/actions/schedule-failure-issue
-        with:
-          github-token: ${{ secrets.BACKLOG_ISSUE_TOKEN }}

.github/workflows/schedule-failure-alert-drill.yml

@@ -16,8 +16,8 @@
 #
 # Forces a job failure and then runs .github/actions/schedule-failure-issue
 # through the exact `needs` + `always() && contains(needs.*.result,
-# 'failure')` wiring used by the real consumers (e2e-s3tests, mint, fuzz,
-# performance-ab). Dispatch it twice to verify both the issue-creation and
+# 'failure')` wiring used by the real consumers (e2e-s3tests, mint, fuzz).
+# Dispatch it twice to verify both the issue-creation and
 # the dedupe-comment paths, then close the resulting
 # "[scheduled-failure] Schedule Failure Alert Drill" issue in rustfs/backlog
 # (the default target repository). Requires the BACKLOG_ISSUE_TOKEN secret

.github/workflows/scheduled-validation-watchdog.yml

@@ -28,7 +28,6 @@ on:
       - "mint"
       - "minio-interop"
       - "Nightly GNU Build"
-      - "Performance A/B"
       - "Runner Hygiene"
     types: [completed]
 

ARCHITECTURE.md

@@ -155,7 +155,7 @@ default build (lifecycle:
    - ⚠️ PARTIALLY VIOLATED: serving-side `s3s` references remain in ecstore
      (bucket metadata/replication/lifecycle DTOs and error mapping). The
      count is ratcheted shrink-only by `scripts/check_s3s_footprint.sh`
-     (the `ecstore_files` line of `.config/s3s-footprint-baseline.txt`; the `object_lock` module was converted to
+     (`S3S_ECSTORE_FILES_BASELINE`; the `object_lock` module was converted to
      storage-level types as the first ratchet step). Target state: the
      baseline reaches zero and ecstore's `Cargo.toml` drops `s3s`.
 

Cargo.lock

@@ -8545,15 +8545,6 @@ version = "1.2.3"
 source = "registry+https://github.com/rust-lang/crates.io-index"
 checksum = "a1d01941d82fa2ab50be1e79e6714289dd7cde78eba4c074bc5a4374f650dfe0"
 
-[[package]]
-name = "quick-xml"
-version = "0.41.0"
-source = "registry+https://github.com/rust-lang/crates.io-index"
-checksum = "e660451e55124f798a69a5af3f49ccfbefbd41910eefd25caf2393e1f3473ec1"
-dependencies = [
- "memchr",
-]
-
 [[package]]
 name = "quick-xml"
 version = "0.42.0"
@@ -9418,7 +9409,7 @@ dependencies = [
  "proptest",
  "prost 0.14.4",
  "pyroscope",
- "quick-xml 0.42.0",
+ "quick-xml",
  "quinn",
  "rand 0.10.3",
  "rcgen",
@@ -9437,7 +9428,6 @@ dependencies = [
  "rustfs-ecstore",
  "rustfs-extension-schema",
  "rustfs-filemeta",
- "rustfs-gateway-types",
  "rustfs-heal",
  "rustfs-heal-contracts",
  "rustfs-iam",
@@ -9716,7 +9706,6 @@ dependencies = [
  "rustfs-data-usage",
  "rustfs-erasure-codec",
  "rustfs-filemeta",
- "rustfs-gateway-types",
  "rustfs-heal-contracts",
  "rustfs-io-metrics",
  "rustfs-lifecycle",
@@ -9822,48 +9811,6 @@ dependencies = [
  "xxhash-rust",
 ]
 
-[[package]]
-name = "rustfs-gateway-stream"
-version = "0.2.0"
-source = "git+https://github.com/rustfs/gateway.git?rev=21283e874c2381bab611e9347acbc5b92e9e2b3c#21283e874c2381bab611e9347acbc5b92e9e2b3c"
-dependencies = [
- "bitflags 2.13.2",
- "bytes",
- "http 1.5.0",
- "http-body",
-]
-
-[[package]]
-name = "rustfs-gateway-types"
-version = "0.32.0+aws.2026-10-02"
-source = "git+https://github.com/rustfs/gateway.git?rev=21283e874c2381bab611e9347acbc5b92e9e2b3c#21283e874c2381bab611e9347acbc5b92e9e2b3c"
-dependencies = [
- "bytes",
- "crc-fast",
- "futures-core",
- "hex",
- "http 1.5.0",
- "md-5 0.11.0",
- "percent-encoding",
- "rustfs-gateway-stream",
- "rustfs-gateway-xml",
- "s3s",
- "sha1 0.11.0",
- "sha2 0.11.1",
- "time",
- "twox-hash",
- "unicode-normalization",
- "zeroize",
-]
-
-[[package]]
-name = "rustfs-gateway-xml"
-version = "0.4.0"
-source = "git+https://github.com/rustfs/gateway.git?rev=21283e874c2381bab611e9347acbc5b92e9e2b3c#21283e874c2381bab611e9347acbc5b92e9e2b3c"
-dependencies = [
- "quick-xml 0.41.0",
-]
-
 [[package]]
 name = "rustfs-heal"
 version = "1.0.2"
@@ -10134,10 +10081,10 @@ dependencies = [
  "metrics-util",
  "proptest",
  "rustfs-config",
- "rustfs-gateway-types",
  "rustfs-replication",
  "rustfs-scanner-metrics",
  "rustfs-storage-api",
+ "s3s",
  "serial_test",
  "temp-env",
  "time",
@@ -10200,6 +10147,7 @@ dependencies = [
  "reqwest",
  "rmp-serde",
  "rustfs-signer",
+ "s3s",
  "serde",
  "serde_json",
  "sysinfo",
@@ -10239,7 +10187,7 @@ dependencies = [
  "jiff",
  "metrics",
  "percent-encoding",
- "quick-xml 0.42.0",
+ "quick-xml",
  "rayon",
  "rustc-hash",
  "rustfs-config",
@@ -10420,13 +10368,10 @@ dependencies = [
  "rustfs-config",
  "rustfs-credentials",
  "rustfs-ecstore",
- "rustfs-gateway-stream",
- "rustfs-gateway-types",
  "rustfs-iam",
  "rustfs-keystone",
  "rustfs-policy",
  "rustfs-rio",
- "rustfs-s3-types",
  "rustfs-storage-api",
  "rustfs-test-utils",
  "rustfs-tls-runtime",
@@ -10434,13 +10379,13 @@ dependencies = [
  "rustfs-trusted-proxies",
  "rustfs-utils",
  "rustls",
+ "s3s",
  "serde",
  "serde_json",
  "sha1 0.11.0",
  "sha2 0.11.1",
  "socket2",
  "subtle",
- "sync_wrapper",
  "tempfile",
  "thiserror 2.0.21",
  "time",
@@ -10490,7 +10435,7 @@ dependencies = [
  "regex",
  "rmp",
  "rmp-serde",
- "rustfs-gateway-types",
+ "s3s",
  "serde",
  "time",
  "tokio",
@@ -10585,13 +10530,11 @@ dependencies = [
  "hyper-util",
  "lazy_static",
  "md-5 0.11.0",
- "quick-xml 0.42.0",
+ "quick-xml",
  "rand 0.10.3",
  "rustfs-checksums",
  "rustfs-config",
- "rustfs-gateway-types",
  "rustfs-rio",
- "rustfs-s3-types",
  "rustfs-signer",
  "rustfs-storage-api",
  "rustfs-tls-runtime",
@@ -10625,19 +10568,9 @@ dependencies = [
 name = "rustfs-s3-types"
 version = "1.0.2"
 dependencies = [
- "bytes",
- "futures-core",
  "hotpath",
- "http 1.5.0",
- "http-body",
- "http-body-util",
- "hyper",
- "hyper-util",
- "pin-project-lite",
- "s3s",
  "serde",
  "serde_json",
- "tokio",
 ]
 
 [[package]]
@@ -10661,10 +10594,9 @@ dependencies = [
  "parking_lot 0.13.0",
  "rustfs-common",
  "rustfs-ecstore",
- "rustfs-gateway-types",
- "rustfs-s3-types",
  "rustfs-storage-api",
  "rustfs-test-utils",
+ "s3s",
  "serde_json",
  "serial_test",
  "thiserror 2.0.21",
@@ -10690,6 +10622,7 @@ dependencies = [
  "parking_lot 0.13.0",
  "rustfs-s3select-api",
  "rustfs-test-utils",
+ "s3s",
  "serde",
  "serde_json",
  "tokio",
@@ -10718,7 +10651,6 @@ dependencies = [
  "rustfs-data-usage",
  "rustfs-ecstore",
  "rustfs-filemeta",
- "rustfs-gateway-types",
  "rustfs-heal",
  "rustfs-heal-contracts",
  "rustfs-lifecycle",
@@ -10775,10 +10707,9 @@ dependencies = [
  "bytes",
  "hotpath",
  "http 1.5.0",
- "http-body",
  "hyper",
- "rustfs-s3-types",
  "rustfs-utils",
+ "s3s",
  "serde_urlencoded",
  "thiserror 2.0.21",
  "time",
@@ -10836,6 +10767,7 @@ dependencies = [
  "rustfs-utils",
  "rustls",
  "rustls-native-certs",
+ "s3s",
  "serde",
  "serde_json",
  "sha2 0.11.1",
@@ -11227,7 +11159,7 @@ dependencies = [
  "numeric_cast",
  "phf 0.14.0",
  "pin-project-lite",
- "quick-xml 0.42.0",
+ "quick-xml",
  "regex",
  "s3s-multipart",
  "s3s-rfc2047",

Cargo.toml

@@ -48,7 +48,7 @@ members = [
     "crates/replication", # Replication contracts and wire formats
     "crates/concurrency", # Concurrency management for RustFS - timeout, locking, backpressure, and I/O scheduling
     "crates/s3-client", # S3 client for engine-side consumption of remote S3 endpoints (tiering, transition targets)
-    "crates/s3-types", # S3 event and error type definitions
+    "crates/s3-types", # S3 event type definitions
     "crates/s3-ops", # S3 operation definitions and mapping
     "crates/s3select-api", # S3 Select API interface
     "crates/s3select-query", # S3 Select query engine
@@ -143,13 +143,6 @@ rustfs-tls-runtime = { path = "crates/tls-runtime", version = "1.0.2" }
 rustfs-utils = { path = "crates/utils", version = "1.0.2" }
 rustfs-zip = { path = "./crates/zip", version = "1.0.2" }
 
-# The gateway's S3 types: the DTOs and persisted bucket-configuration codecs the
-# engine crates move onto from s3s (rustfs/backlog#2745). The one pin of the
-# gateway in this workspace; no crate may enable a `compat-s3s*` feature except
-# the s3s-facing edge of the rustfs binary.
-rustfs-gateway-types = { git = "https://github.com/rustfs/gateway.git", rev = "21283e874c2381bab611e9347acbc5b92e9e2b3c", version = "0.32.0", default-features = false }
-rustfs-gateway-stream = { git = "https://github.com/rustfs/gateway.git", rev = "21283e874c2381bab611e9347acbc5b92e9e2b3c", version = "0.2.0", default-features = false }
-
 # Async Runtime and Networking
 async-channel = "2.5.0"
 async_zip = { default-features = false, version = "0.0.19" }
@@ -333,10 +326,6 @@ rust-embed = { version = "8.13.0" }
 rustc-hash = { version = "2.1.3" }
 s3s = { git = "https://github.com/s3s-project/s3s", rev = "5761ddfe4c509fcac87e3b99fd5bd9e176b442c0", features = ["minio"] }
 s3s-sigv4 = { git = "https://github.com/s3s-project/s3s", rev = "5761ddfe4c509fcac87e3b99fd5bd9e176b442c0" }
-# The same s3s pin under the name the legacy edge of the gateway migration uses
-# (crates/s3-types, feature `compat-s3s`). Keep `rev` identical to `s3s` above.
-# DELETE BY T4.2 (rustfs/backlog#2784).
-legacy-s3s = { package = "s3s", git = "https://github.com/s3s-project/s3s", rev = "5761ddfe4c509fcac87e3b99fd5bd9e176b442c0", features = ["minio"] }
 serial_test = "4.0.1"
 shadow-rs = { default-features = false, version = "2.0.0" }
 siphasher = "1.0.4"
@@ -345,7 +334,6 @@ compact_str = "0.10.0"
 snap = "1.1.2"
 starshard = { version = "2.4.0" }
 strum = { version = "0.28.0" }
-sync_wrapper = "1.0.2"
 sysinfo = "0.39.6"
 temp-env = "0.3.6"
 tempfile = "3.27.0"

crates/concurrency/Cargo.toml

@@ -26,7 +26,7 @@ rustfs-io-core = { workspace = true }
 serde = { workspace = true, features = ["derive"] }
 
 # Async runtime
-tokio = { workspace = true, features = ["sync", "fs", "rt-multi-thread"] }
+tokio = { workspace = true, features = ["sync", "fs", "rt-multi-thread", "time"] }
 
 # Logging
 tracing = { workspace = true }

crates/e2e_test/src/anonymous_access_test.rs

@@ -17,8 +17,7 @@
 //! when PublicAccessBlock configuration is missing or explicitly set.
 
 use crate::common::{RustFSTestEnvironment, init_logging, local_http_client};
-use crate::sts_query_compat_test::{OPA_AUTH_TOKEN, OpaMock};
-use aws_sdk_s3::types::{PublicAccessBlockConfiguration, Tag, Tagging};
+use aws_sdk_s3::types::PublicAccessBlockConfiguration;
 use tracing::info;
 
 async fn setup_public_bucket(
@@ -70,98 +69,6 @@ async fn anonymous_get_object(
     local_http_client().get(&url).send().await
 }
 
-#[tokio::test]
-async fn test_plain_http_matches_secure_transport_false_bucket_policy() -> Result<(), Box<dyn std::error::Error + Send + Sync>> {
-    init_logging();
-
-    let mut env = RustFSTestEnvironment::new().await?;
-    env.start_rustfs_server(vec![]).await?;
-    assert!(env.url.starts_with("http://"), "fixture must exercise plaintext HTTP: {}", env.url);
-
-    let bucket = "secure-transport-policy";
-    let key = "plain-http.txt";
-    let admin = env.create_s3_client();
-    admin.create_bucket().bucket(bucket).send().await?;
-    admin
-        .put_object()
-        .bucket(bucket)
-        .key(key)
-        .body(aws_sdk_s3::primitives::ByteStream::from_static(b"plain http"))
-        .send()
-        .await?;
-    admin
-        .put_bucket_policy()
-        .bucket(bucket)
-        .policy(
-            serde_json::json!({
-                "Version": "2012-10-17",
-                "Statement": [{
-                    "Effect": "Allow",
-                    "Principal": "*",
-                    "Action": "s3:GetObject",
-                    "Resource": format!("arn:aws:s3:::{bucket}/*"),
-                    "Condition": {"Bool": {"aws:SecureTransport": "false"}}
-                }]
-            })
-            .to_string(),
-        )
-        .send()
-        .await?;
-
-    let response = anonymous_get_object(&env, bucket, key).await?;
-    assert_eq!(response.status().as_u16(), 200);
-    assert_eq!(response.bytes().await?.as_ref(), b"plain http");
-    Ok(())
-}
-
-/// Root credentials have no attached IAM policy, so these requests depend on
-/// the owner flag reaching policy evaluation unchanged.
-#[tokio::test]
-async fn test_root_requests_keep_policy_owner_flag() -> Result<(), Box<dyn std::error::Error + Send + Sync>> {
-    init_logging();
-
-    let opa = OpaMock::start().await?;
-    let mut env = RustFSTestEnvironment::new().await?;
-    env.start_rustfs_server_with_env(
-        vec![],
-        &[
-            ("RUSTFS_POLICY_PLUGIN_URL", opa.url.as_str()),
-            ("RUSTFS_POLICY_PLUGIN_AUTH_TOKEN", OPA_AUTH_TOKEN),
-        ],
-    )
-    .await?;
-
-    let bucket = "root-owner-flag";
-    let key = "owner.txt";
-    let client = env.create_s3_client();
-    client.create_bucket().bucket(bucket).send().await?;
-    client
-        .put_object()
-        .bucket(bucket)
-        .key(key)
-        .body(aws_sdk_s3::primitives::ByteStream::from_static(b"owner access"))
-        .send()
-        .await?;
-
-    // Duplicate stored tags are ambiguous. An owner request must not enter the
-    // OPA tag-loading path, which rejects this metadata before evaluation.
-    client
-        .put_bucket_tagging()
-        .bucket(bucket)
-        .tagging(
-            Tagging::builder()
-                .tag_set(Tag::builder().key("department").value("finance").build()?)
-                .tag_set(Tag::builder().key("department").value("engineering").build()?)
-                .build()?,
-        )
-        .send()
-        .await?;
-    let object = client.get_object().bucket(bucket).key(key).send().await?;
-
-    assert_eq!(object.body.collect().await?.into_bytes().as_ref(), b"owner access");
-    Ok(())
-}
-
 /// Issue #2036: Anonymous GetObject should succeed when bucket policy allows it
 /// and no PublicAccessBlock configuration exists (ConfigNotFound).
 #[tokio::test]

crates/e2e_test/src/distributed/chaos_test.rs

@@ -13,18 +13,20 @@
 // limitations under the License.
 
 use super::harness::{
-    DistCluster, DistLayout, TestResult, assert_object_bytes, payload_for, put_object, retrying_get_equals, unique_bucket,
-    wait_for_ready, wait_until,
+    DistCluster, DistLayout, TestResult, assert_object_bytes, cluster_admin_ok, get_object_bytes, payload_for, put_object,
+    retrying_get_equals, retrying_put, s3_probe_client, s3_probe_error_code, unique_bucket, wait_for_ready, wait_until,
 };
 use crate::chaos::{census_object_version_on_disk, signed_admin_post};
 use crate::common::{build_test_s3_config, init_logging};
 use crate::fault_proxy::FaultMode;
 use aws_sdk_s3::Client;
+use aws_sdk_s3::error::SdkError;
+use aws_sdk_s3::operation::{get_object::GetObjectError, put_object::PutObjectError};
 use std::path::PathBuf;
 use std::sync::Arc;
 use std::time::Duration;
 use tokio::sync::{Barrier, mpsc};
-use tokio::time::timeout;
+use tokio::time::{Instant, timeout};
 
 #[tokio::test]
 async fn kill_and_restart_node_preserves_objects() -> TestResult {
@@ -44,6 +46,133 @@ async fn kill_and_restart_node_preserves_objects() -> TestResult {
     Ok(())
 }
 
+#[tokio::test]
+async fn ec4_partial_node_return_restores_read_write_quorum() -> TestResult {
+    init_logging();
+    let mut dist = DistCluster::start_with_env(
+        DistLayout::FourByFour,
+        &[
+            ("RUSTFS_ERASURE_SET_DRIVE_COUNT", "16"),
+            ("RUSTFS_STORAGE_CLASS_STANDARD", "EC:4"),
+        ],
+    )
+    .await?;
+    let storage = cluster_admin_ok(&dist.cluster, http::Method::GET, "/rustfs/admin/v3/storageinfo", None).await?;
+    let storage: serde_json::Value = serde_json::from_str(&storage)?;
+    let storage: rustfs_madmin::StorageInfo = serde_json::from_value(storage["info"].clone())?;
+    assert_eq!(storage.disks.len(), 16);
+    assert_eq!(storage.backend.total_sets, vec![1]);
+    assert_eq!(storage.backend.drives_per_set, vec![16]);
+    assert_eq!(storage.backend.standard_sc_parity, Some(4));
+
+    let bucket = unique_bucket("partial-ec4");
+    dist.create_bucket(&bucket).await?;
+    let client = s3_probe_client(&dist.client(0)?);
+    let baseline = payload_for("partial-recovery/baseline", 8 * 1024 * 1024);
+    put_object(&client, &bucket, "baseline.bin", baseline.clone()).await?;
+    wait_until(
+        Duration::from_secs(30),
+        || async {
+            for node in &dist.cluster.nodes {
+                for directory in &node.data_dirs {
+                    let census = census_object_version_on_disk(std::path::Path::new(directory), &bucket, "baseline.bin", None)?;
+                    if !census.is_complete() {
+                        return Ok(false);
+                    }
+                    assert_eq!(census.data_blocks, Some(12), "the object must require twelve data shards");
+                    assert_eq!(census.parity_blocks, Some(4), "the object must have four parity shards");
+                    assert!(!census.expected_part_numbers.is_empty(), "the baseline must use physical shards");
+                }
+            }
+            Ok(true)
+        },
+        "the baseline is complete on all sixteen EC:4 drives",
+    )
+    .await?;
+
+    dist.cluster.stop_node(2)?;
+    dist.cluster.stop_node(3)?;
+    let read_error = timeout(Duration::from_secs(60), get_object_bytes(&client, &bucket, "baseline.bin"))
+        .await?
+        .expect_err("eight unavailable drives must prevent the baseline GET");
+    assert!(
+        matches!(
+            s3_probe_error_code(read_error.as_ref()),
+            Some("SlowDown" | "SlowDownRead" | "ServiceUnavailable")
+        ),
+        "quorum loss must produce an availability error: {read_error:?}"
+    );
+    assert_eq!(
+        read_error
+            .downcast_ref::<SdkError<GetObjectError>>()
+            .and_then(SdkError::raw_response)
+            .map(|response| response.status().as_u16()),
+        Some(503),
+        "read quorum loss must remain a service availability failure"
+    );
+    let write_error = timeout(Duration::from_secs(60), put_object(&client, &bucket, "unavailable.bin", baseline.clone()))
+        .await?
+        .expect_err("eight unavailable drives must prevent a successful PUT");
+    assert!(
+        matches!(s3_probe_error_code(write_error.as_ref()), Some("SlowDown" | "ServiceUnavailable")),
+        "quorum loss must produce an availability error: {write_error:?}"
+    );
+    assert_eq!(
+        write_error
+            .downcast_ref::<SdkError<PutObjectError>>()
+            .and_then(SdkError::raw_response)
+            .map(|response| response.status().as_u16()),
+        Some(503),
+        "write quorum loss must remain a service availability failure"
+    );
+
+    // Recover only four drives. The remaining node must stay stopped while
+    // every live coordinator serves old data and acknowledges new writes.
+    dist.cluster.start_node_process(2).await?;
+    let recovery_deadline = Instant::now() + Duration::from_secs(90);
+    let mut recovered_objects = Vec::new();
+    for node in 0..3 {
+        assert!(dist.cluster.nodes[3].process.is_none(), "the fourth node must remain unavailable");
+        let client = s3_probe_client(&dist.client(node)?);
+        retrying_get_equals(
+            &client,
+            &bucket,
+            "baseline.bin",
+            &baseline,
+            recovery_deadline.saturating_duration_since(Instant::now()),
+        )
+        .await?;
+        let key = format!("partial-{node}.bin");
+        let body = payload_for(&key, 1024 * 1024);
+        retrying_put(
+            &client,
+            &bucket,
+            &key,
+            body.clone(),
+            recovery_deadline.saturating_duration_since(Instant::now()),
+        )
+        .await?;
+        recovered_objects.push((key, body));
+    }
+    for node in 0..3 {
+        assert!(dist.cluster.nodes[3].process.is_none(), "reads must complete before all drives return");
+        let client = s3_probe_client(&dist.client(node)?);
+        assert_object_bytes(&client, &bucket, "baseline.bin", &baseline).await?;
+        for (key, body) in &recovered_objects {
+            assert_object_bytes(&client, &bucket, key, body).await?;
+        }
+    }
+
+    dist.cluster.start_node(3).await?;
+    wait_for_ready(&dist.cluster).await?;
+    let client = s3_probe_client(&dist.client(3)?);
+    assert_object_bytes(&client, &bucket, "baseline.bin", &baseline).await?;
+    for (key, body) in &recovered_objects {
+        assert_object_bytes(&client, &bucket, key, body).await?;
+    }
+    Ok(())
+}
+
 #[tokio::test]
 async fn full_cluster_restart_preserves_objects() -> TestResult {
     init_logging();

crates/e2e_test/src/distributed/harness.rs

@@ -1181,11 +1181,16 @@ pub(crate) async fn retrying_get_equals(
                 )
                 .into()),
                 Err(error) => {
-                    if matches!(s3_probe_error_code(error.as_ref()), Some("SlowDown" | "ServiceUnavailable")) {
-                        Ok(false)
-                    } else {
-                        Err(error)
-                    }
+                    let retryable = match s3_probe_error_code(error.as_ref()) {
+                        Some("SlowDown" | "ServiceUnavailable") => true,
+                        // GET preserves this compatibility code for a lost data quorum.
+                        Some("SlowDownRead") => error
+                            .downcast_ref::<SdkError<GetObjectError>>()
+                            .and_then(SdkError::raw_response)
+                            .is_some_and(|response| response.status().as_u16() == 503),
+                        _ => false,
+                    };
+                    if retryable { Ok(false) } else { Err(error) }
                 }
             }
         },
@@ -1324,6 +1329,62 @@ mod retry_tests {
         Ok(())
     }
 
+    #[tokio::test(start_paused = true)]
+    async fn retrying_get_allows_read_quorum_recovery() -> TestResult {
+        use aws_sdk_s3::config::Region;
+        use aws_smithy_http_client::test_util::{ReplayEvent, StaticReplayClient};
+        use aws_smithy_types::body::SdkBody;
+
+        let endpoint = "http://read-quorum.example.invalid";
+        let event = |status: u16, body: &'static str| {
+            ReplayEvent::new(
+                http::Request::builder()
+                    .method(Method::GET)
+                    .uri(format!("{endpoint}/retry-probe/key?x-id=GetObject"))
+                    .body(SdkBody::empty())
+                    .expect("build replay request"),
+                http::Response::builder()
+                    .status(status)
+                    .header("content-type", "application/xml")
+                    .header("content-length", body.len())
+                    .body(SdkBody::from(body))
+                    .expect("build replay response"),
+            )
+        };
+        for status in [503, 403, 500] {
+            let transport = StaticReplayClient::new(vec![
+                event(status, "<Error><Code>SlowDownRead</Code></Error>"),
+                event(200, "expected bytes"),
+            ]);
+            let client = Client::from_conf(
+                aws_sdk_s3::Config::builder()
+                    .behavior_version_latest()
+                    .credentials_provider(Credentials::new(FAKE_ACCESS_KEY, FAKE_SECRET_KEY, None, None, "read-quorum-test"))
+                    .region(Region::new("us-east-1"))
+                    .endpoint_url(endpoint)
+                    .force_path_style(true)
+                    .retry_config(RetryConfig::disabled())
+                    .http_client(transport.clone())
+                    .build(),
+            );
+            let result = retrying_get_equals(&client, "retry-probe", "key", b"expected bytes", Duration::from_secs(5)).await;
+            if status == 503 {
+                result?;
+                assert_eq!(
+                    transport.actual_requests().count(),
+                    2,
+                    "the probe must retry the read-quorum error exactly once"
+                );
+                transport.assert_requests_match(&[]);
+            } else {
+                let error = result.expect_err("the compatibility code must not hide a non-503 failure");
+                assert_eq!(s3_probe_error_code(error.as_ref()), Some("SlowDownRead"));
+                assert_eq!(transport.actual_requests().count(), 1, "non-availability errors must fail immediately");
+            }
+        }
+        Ok(())
+    }
+
     #[tokio::test]
     async fn object_polls_distinguish_committed_and_replicated_keys() -> TestResult {
         let target = FakeS3Target::start().await?;

crates/e2e_test/src/distributed/mod.rs

@@ -40,5 +40,6 @@ mod scanner_heal_topology_fixture;
 mod scanner_heal_topology_test;
 mod scanner_heal_topology_tier_test;
 mod site_replication_test;
+mod table_catalog_test;
 mod upgrade_test;
 mod versioning_test;

crates/e2e_test/src/distributed/multipart_fence_acceptance_test.rs

@@ -1638,6 +1638,191 @@ async fn late_complete_packets_cannot_publish_after_abort_and_bucket_recreation(
             "refused late packets must not recreate staging"
         );
     }
+    late_part_packets_preserve_retired_and_fresh_uploads().await?;
+    Ok(())
+}
+
+async fn late_part_packets_preserve_retired_and_fresh_uploads() -> TestResult {
+    for (method, settles) in [("PreparePartTransaction", false), ("SettlePartTransaction", true)] {
+        for recreate_bucket in [false, true] {
+            let (cluster, proxies, clients, bucket) = fixture(1, "EC:2").await?;
+            let key = "late-part-transaction";
+            let previous = payload_for("committed-before-late-part", 4096);
+            clients[0]
+                .put_object()
+                .bucket(&bucket)
+                .key(key)
+                .body(ByteStream::from(previous.clone()))
+                .send()
+                .await?;
+            let upload = clients[0].create_multipart_upload().bucket(&bucket).key(key).send().await?;
+            let upload_id = upload.upload_id().ok_or("missing original upload ID")?;
+            clients[0]
+                .upload_part()
+                .bucket(&bucket)
+                .key(key)
+                .upload_id(upload_id)
+                .part_number(1)
+                .body(ByteStream::from(vec![0x41; 9 * 1024 * 1024]))
+                .send()
+                .await?;
+            let path = upload_path_scope(&bucket, key, upload_id)?;
+            let hash = sha256_hex(format!("{bucket}/{key}").as_bytes());
+            let roots = cluster
+                .nodes
+                .iter()
+                .flat_map(|node| &node.data_dirs)
+                .map(|root| Path::new(root).join(RUSTFS_META_MULTIPART_BUCKET).join(&path))
+                .collect::<Vec<_>>();
+            assert_eq!(roots.len(), 4);
+            let original = multipart_sources(&cluster, Some(&hash))?;
+            assert_eq!(original.len(), 4);
+            assert!(
+                original.values().all(|source| source
+                    .files
+                    .keys()
+                    .any(|path| path.file_name().and_then(|name| name.to_str()) == Some("part.1"))),
+                "all selected peers originally own the acknowledged part"
+            );
+            let scope = upload_scope(upload_id)?;
+            // Retain only the original attempt; recovery of the same upload
+            // must be able to issue fresh settlement requests through the gate.
+            let pauses = proxies[1..3]
+                .iter()
+                .map(|proxy| proxy.pause_late(&[method], 1, &scope))
+                .collect::<Result<Vec<_>, _>>()?;
+            let replacement = clients[0]
+                .upload_part()
+                .bucket(&bucket)
+                .key(key)
+                .upload_id(upload_id)
+                .part_number(1)
+                .body(ByteStream::from(vec![0x62; 8 * 1024 * 1024]));
+            let caller = tokio::spawn(async move { replacement.send().await });
+            wait_for_faults(&pauses, &[method]).await?;
+            let failure = tokio::time::timeout(DEADLINE, caller)
+                .await??
+                .expect_err("two retained peers cannot acknowledge part replacement");
+            assert_unavailable(&failure, "storage_write_quorum", None);
+            assert!(pauses.iter().all(|pause| pause.hits() == 1 && pause.responses() == 0));
+            let expected_actions = if settles { (1, 0) } else { (0, 0) };
+            assert!(
+                pauses.iter().all(|pause| pause.retained_settle_actions() == expected_actions),
+                "settlement capture must be a real Commit, not Rollback"
+            );
+            abort_after_fault(&clients[0], &bucket, key, upload_id, false).await?;
+            assert!(
+                roots.iter().all(|root| !root.exists()),
+                "ordinary Abort physically retires all original roots"
+            );
+            assert!(multipart_sources(&cluster, Some(&hash))?.is_empty());
+            for client in &clients {
+                assert_object_bytes_after_fault(client, &bucket, key, &previous).await?;
+            }
+            let protected = if recreate_bucket {
+                clients[0].delete_object().bucket(&bucket).key(key).send().await?;
+                clients[0].delete_bucket().bucket(&bucket).send().await?;
+                clients[1].create_bucket().bucket(&bucket).send().await?;
+                let fresh = payload_for("recreated-before-late-part", 4096);
+                clients[1]
+                    .put_object()
+                    .bucket(&bucket)
+                    .key(key)
+                    .body(ByteStream::from(fresh.clone()))
+                    .send()
+                    .await?;
+                fresh
+            } else {
+                previous
+            };
+            let fresh = clients[2].create_multipart_upload().bucket(&bucket).key(key).send().await?;
+            let fresh_id = fresh.upload_id().ok_or("missing fresh upload ID")?;
+            let fresh_body = payload_for("fresh-active-part", 4096);
+            let fresh_part = clients[2]
+                .upload_part()
+                .bucket(&bucket)
+                .key(key)
+                .upload_id(fresh_id)
+                .part_number(1)
+                .body(ByteStream::from(fresh_body.clone()))
+                .send()
+                .await?;
+            let before = multipart_sources(&cluster, Some(&hash))?;
+            assert_eq!(before.len(), 4, "the new upload exists on every original participant");
+            for pause in &pauses {
+                pause.release();
+            }
+            wait_until(
+                DEADLINE,
+                || async {
+                    Ok(pauses.iter().all(|pause| {
+                        pause.responses() == pause.hits()
+                            && if settles {
+                                pause.successful_mutation_replies() == pause.hits() && pause.refused_mutations() == 0
+                            } else {
+                                pause.refused_mutations() == pause.hits() && pause.successful_mutation_replies() == 0
+                            }
+                    }))
+                },
+                "every original part packet returns its typed authenticated terminal outcome",
+            )
+            .await?;
+            assert!(roots.iter().all(|root| !root.exists()), "late part packets cannot recreate retired roots");
+            assert!(
+                multipart_sources(&cluster, Some(&hash))? == before,
+                "late packets cannot change a fresh upload's shard or control bytes"
+            );
+            for client in &clients {
+                assert_object_bytes_after_fault(client, &bucket, key, &protected).await?;
+                let listed = client
+                    .list_parts()
+                    .bucket(&bucket)
+                    .key(key)
+                    .upload_id(fresh_id)
+                    .send()
+                    .await?;
+                assert_eq!(listed.parts().len(), 1);
+                assert_eq!(listed.parts()[0].e_tag(), fresh_part.e_tag());
+                let missing = client
+                    .list_parts()
+                    .bucket(&bucket)
+                    .key(key)
+                    .upload_id(upload_id)
+                    .send()
+                    .await
+                    .expect_err("old upload remains retired");
+                assert_eq!(missing.raw_response().expect("old upload service response").status().as_u16(), 404);
+                assert_eq!(missing.as_service_error().and_then(|error| error.meta().code()), Some("NoSuchUpload"));
+            }
+            clients[2]
+                .complete_multipart_upload()
+                .bucket(&bucket)
+                .key(key)
+                .upload_id(fresh_id)
+                .multipart_upload(
+                    CompletedMultipartUpload::builder()
+                        .parts(
+                            CompletedPart::builder()
+                                .part_number(1)
+                                .e_tag(fresh_part.e_tag().ok_or("missing fresh part ETag")?)
+                                .build(),
+                        )
+                        .build(),
+                )
+                .send()
+                .await?;
+            for client in &clients {
+                assert_object_bytes_after_fault(client, &bucket, key, &fresh_body).await?;
+            }
+            assert!(multipart_sources(&cluster, Some(&hash))?.is_empty());
+            assert_no_uploads_after_fault(&clients, &bucket).await?;
+            eprintln!(
+                "MPU late part: method={method}, recreate_bucket={recreate_bucket}, retired_roots=4, delivered=2, refused={}, successful_replies={}, fresh_sources_unchanged=4, verified_readers=4",
+                pauses.iter().map(RpcPause::refused_mutations).sum::<usize>(),
+                pauses.iter().map(RpcPause::successful_mutation_replies).sum::<usize>()
+            );
+        }
+    }
     Ok(())
 }
 

crates/e2e_test/src/distributed/table_catalog_test.rs

@@ -0,0 +1,140 @@
+// Copyright 2026 RustFS Team
+//
+// Licensed under the Apache License, Version 2.0 (the "License");
+// you may not use this file except in compliance with the License.
+// You may obtain a copy of the License at
+//
+//     http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing, software
+// distributed under the License is distributed on an "AS IS" BASIS,
+// WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+// See the License for the specific language governing permissions and
+// limitations under the License.
+
+use super::harness::{DistCluster, DistLayout, TestResult, get_object_bytes, put_object, unique_bucket};
+use crate::common::{init_logging, signed_request};
+use http::{Method, StatusCode};
+use serde_json::{Value, json};
+
+async fn catalog_request(
+    dist: &DistCluster,
+    node: usize,
+    method: Method,
+    path: &str,
+    body: Option<Value>,
+    expected: StatusCode,
+) -> TestResult<Value> {
+    let response = signed_request(
+        method.clone(),
+        &format!("{}{path}", dist.cluster.nodes[node].url),
+        &dist.cluster.access_key,
+        &dist.cluster.secret_key,
+        body.map(|value| serde_json::to_vec(&value)).transpose()?,
+        Some("application/json"),
+    )
+    .await?;
+    let status = response.status();
+    let text = response.text().await?;
+    assert_eq!(status, expected, "node {node}: {method} {path}: {text}");
+    if text.is_empty() {
+        Ok(Value::Null)
+    } else {
+        Ok(serde_json::from_str(&text)?)
+    }
+}
+
+async fn assert_bucket_state(dist: &DistCluster, bucket: &str, enabled: bool) -> TestResult {
+    for node in 0..dist.cluster.nodes.len() {
+        let response =
+            catalog_request(dist, node, Method::GET, &format!("/iceberg/v1/buckets/{bucket}"), None, StatusCode::OK).await?;
+        assert_eq!(response["enabled"], enabled, "node {node}: {response}");
+        assert_eq!(response["table-bucket"], bucket);
+        let expected = if enabled { StatusCode::OK } else { StatusCode::BAD_REQUEST };
+        catalog_request(dist, node, Method::GET, &format!("/iceberg/v1/{bucket}/namespaces"), None, expected).await?;
+        assert_eq!(
+            get_object_bytes(&dist.client(node)?, bucket, "ordinary.txt").await?,
+            b"ordinary bucket object",
+            "node {node}: toggling table support changed an ordinary object"
+        );
+    }
+    Ok(())
+}
+
+async fn assert_table_state(dist: &DistCluster, table_path: &str, created: &Value, pointer: &Value) -> TestResult {
+    for node in 0..dist.cluster.nodes.len() {
+        let loaded = catalog_request(dist, node, Method::GET, table_path, None, StatusCode::OK).await?;
+        assert_eq!(loaded["metadata-location"], created["metadata-location"], "node {node}");
+        assert_eq!(loaded["metadata"], created["metadata"], "node {node}");
+        let actual_pointer =
+            catalog_request(dist, node, Method::GET, &format!("{table_path}/metadata-location"), None, StatusCode::OK).await?;
+        assert_eq!(&actual_pointer, pointer, "node {node}: catalog pointer changed");
+    }
+    Ok(())
+}
+
+#[tokio::test]
+async fn table_bucket_toggles_reload_cached_metadata_on_every_peer() -> TestResult {
+    init_logging();
+    let dist = DistCluster::start_with_env(DistLayout::FourByFour, &[("RUSTFS_TABLE_CATALOG_BACKING", "object")]).await?;
+    let bucket = unique_bucket("table-peer-reload");
+    dist.create_bucket(&bucket).await?;
+    put_object(&dist.client(0)?, &bucket, "ordinary.txt", b"ordinary bucket object".to_vec()).await?;
+    let bucket_path = format!("/iceberg/v1/buckets/{bucket}");
+
+    // Warm each node's false cache before changing the marker. Do not poll:
+    // successful toggles must notify peers before their response is returned.
+    assert_bucket_state(&dist, &bucket, false).await?;
+    for (node, method, enabled) in [(0, Method::PUT, true), (1, Method::DELETE, false), (2, Method::PUT, true)] {
+        let response = catalog_request(&dist, node, method, &bucket_path, None, StatusCode::OK).await?;
+        assert_eq!(response["enabled"], enabled);
+        assert_bucket_state(&dist, &bucket, enabled).await?;
+    }
+
+    let namespaces = format!("/iceberg/v1/{bucket}/namespaces");
+    catalog_request(&dist, 3, Method::POST, &namespaces, Some(json!({"namespace": ["lab"]})), StatusCode::OK).await?;
+    let tables = format!("{namespaces}/lab/tables");
+    let created = catalog_request(
+        &dist,
+        0,
+        Method::POST,
+        &tables,
+        Some(json!({
+            "name": "events",
+            "schema": {
+                "type": "struct",
+                "fields": [{"id": 1, "name": "id", "required": false, "type": "long"}]
+            },
+            "stage-create": false
+        })),
+        StatusCode::OK,
+    )
+    .await?;
+    let uuid = created["metadata"]["table-uuid"]
+        .as_str()
+        .ok_or("create response omitted table UUID")?;
+    assert!(!uuid::Uuid::parse_str(uuid)?.is_nil());
+    assert!(
+        created["metadata-location"]
+            .as_str()
+            .is_some_and(|location| location.starts_with(&format!("s3://{bucket}/")))
+    );
+    let table_path = format!("{tables}/events");
+    let pointer =
+        catalog_request(&dist, 3, Method::GET, &format!("{table_path}/metadata-location"), None, StatusCode::OK).await?;
+    assert_eq!(pointer["metadata-location"], created["metadata-location"]);
+    assert!(pointer["version-token"].as_str().is_some_and(|token| !token.is_empty()));
+    assert!(pointer["generation"].as_u64().is_some());
+    assert_eq!(pointer["warehouse-location"], created["metadata"]["location"]);
+    assert_table_state(&dist, &table_path, &created, &pointer).await?;
+
+    // A rejected toggle must preserve both peer protections and catalog state.
+    catalog_request(&dist, 1, Method::DELETE, &bucket_path, None, StatusCode::CONFLICT).await?;
+    assert_bucket_state(&dist, &bucket, true).await?;
+    assert_table_state(&dist, &table_path, &created, &pointer).await?;
+
+    catalog_request(&dist, 2, Method::DELETE, &table_path, None, StatusCode::NO_CONTENT).await?;
+    catalog_request(&dist, 3, Method::DELETE, &format!("{namespaces}/lab"), None, StatusCode::NO_CONTENT).await?;
+    // The cluster fixture owns all remaining catalog objects and data roots.
+    Ok(())
+}

crates/e2e_test/src/dual_stack_differential/compare.rs

@@ -1,512 +0,0 @@
-// Copyright 2024 RustFS Team
-//
-// Licensed under the Apache License, Version 2.0 (the "License");
-// you may not use this file except in compliance with the License.
-// You may obtain a copy of the License at
-//
-//     http://www.apache.org/licenses/LICENSE-2.0
-//
-// Unless required by applicable law or agreed to in writing, software
-// distributed under the License is distributed on an "AS IS" BASIS,
-// WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
-// See the License for the specific language governing permissions and
-// limitations under the License.
-
-//! What one stack answered, and how two answers are compared.
-//!
-//! An [`Observation`] keeps only what a client acts on: the HTTP status, the S3 error code, the
-//! allowlisted response headers and a few typed fields decoded from the response. The error
-//! message is kept for diagnostics and never compared: message wording is judged by the wire
-//! corpus, not here. The comparator is pure so it can be tested without a server.
-
-use serde::Serialize;
-use std::collections::{BTreeMap, BTreeSet};
-use std::path::{Path, PathBuf};
-
-/// Response headers compared exactly (after [`normalized_header_value`]).
-const HEADER_ALLOWLIST: [&str; 7] = [
-    "content-type",
-    "x-amz-version-id",
-    "etag",
-    "x-amz-delete-marker",
-    "content-range",
-    "accept-ranges",
-    "x-amz-request-charged",
-];
-
-/// Every response header with this prefix is compared as well.
-const HEADER_ALLOWLIST_PREFIX: &str = "x-amz-server-side-encryption";
-
-/// Placeholder for a field one side did not send.
-const ABSENT: &str = "<absent>";
-
-/// The lowercase header name if the comparator looks at this header.
-pub(crate) fn compared_header(name: &str) -> Option<String> {
-    let name = name.to_ascii_lowercase();
-    (HEADER_ALLOWLIST.contains(&name.as_str()) || name.starts_with(HEADER_ALLOWLIST_PREFIX)).then_some(name)
-}
-
-/// A version ID is minted per process, so its value never matches across two servers; only
-/// whether it is the literal `null` (an unversioned write) carries meaning.
-fn normalized_header_value(name: &str, value: &str) -> String {
-    if name == "x-amz-version-id" && value != "null" {
-        "<version-id>".to_owned()
-    } else {
-        value.to_owned()
-    }
-}
-
-/// One request's outcome as a client sees it.
-#[derive(Clone, Debug, Serialize)]
-pub(crate) struct Observation {
-    /// The S3 operation, named as in the `s3s::S3` trait (`GetObject`).
-    pub op: &'static str,
-    /// Which request of the script this was (`normal`, `negative`, or a setup label).
-    pub case: &'static str,
-    /// HTTP status; 0 when no response arrived.
-    pub status: u16,
-    /// The S3 error code, or the SDK's error kind when there was no service error.
-    pub error_code: Option<String>,
-    /// Allowlisted headers only, lowercase names, normalized values.
-    pub headers: BTreeMap<String, String>,
-    /// Typed values decoded from the response body, never free text.
-    pub fields: BTreeMap<String, String>,
-    /// Diagnostic only; never compared.
-    #[serde(skip)]
-    pub message: Option<String>,
-}
-
-impl Observation {
-    /// Builds an observation, keeping only the allowlisted headers.
-    pub(crate) fn new<'h>(
-        op: &'static str,
-        case: &'static str,
-        status: u16,
-        error_code: Option<String>,
-        headers: impl IntoIterator<Item = (&'h str, &'h str)>,
-    ) -> Self {
-        let headers = headers
-            .into_iter()
-            .filter_map(|(name, value)| compared_header(name).map(|name| (name, value)))
-            .map(|(name, value)| {
-                let value = normalized_header_value(&name, value);
-                (name, value)
-            })
-            .collect();
-        Self {
-            op,
-            case,
-            status,
-            error_code,
-            headers,
-            fields: BTreeMap::new(),
-            message: None,
-        }
-    }
-
-    pub(crate) fn with_field(mut self, name: &str, value: impl Into<String>) -> Self {
-        self.fields.insert(name.to_owned(), value.into());
-        self
-    }
-
-    pub(crate) fn is_success(&self) -> bool {
-        (200..300).contains(&self.status)
-    }
-}
-
-/// One field on which the two stacks answered differently.
-#[derive(Clone, Debug, PartialEq, Eq, Serialize)]
-pub(crate) struct Difference {
-    pub op: String,
-    pub case: String,
-    pub field: String,
-    pub a: String,
-    pub b: String,
-}
-
-/// Every difference between two transcripts of the same script, matched by `(op, case)`.
-pub(crate) fn compare(a: &[Observation], b: &[Observation]) -> Vec<Difference> {
-    let mut differences = Vec::new();
-    let (index_a, duplicates_a) = index(a);
-    let (index_b, duplicates_b) = index(b);
-    for (op, case) in duplicates_a.union(&duplicates_b) {
-        let count = |side: &[Observation]| {
-            side.iter()
-                .filter(|observation| (observation.op, observation.case) == (*op, *case))
-                .count()
-                .to_string()
-        };
-        differences.push(difference(op, case, "duplicate_case", count(a), count(b)));
-    }
-    let keys: BTreeSet<_> = index_a.keys().chain(index_b.keys()).copied().collect();
-    for key @ (op, case) in keys {
-        match (index_a.get(&key), index_b.get(&key)) {
-            (Some(a), Some(b)) => compare_one(a, b, &mut differences),
-            (a, b) => {
-                let presence = |side: Option<&&Observation>| if side.is_some() { "present" } else { ABSENT }.to_owned();
-                differences.push(difference(op, case, "observation", presence(a), presence(b)));
-            }
-        }
-    }
-    differences
-}
-
-type Key = (&'static str, &'static str);
-
-fn index(side: &[Observation]) -> (BTreeMap<Key, &Observation>, BTreeSet<Key>) {
-    let mut index = BTreeMap::new();
-    let mut duplicates = BTreeSet::new();
-    for observation in side {
-        let key = (observation.op, observation.case);
-        if index.insert(key, observation).is_some() {
-            duplicates.insert(key);
-        }
-    }
-    (index, duplicates)
-}
-
-fn difference(op: &str, case: &str, field: &str, a: String, b: String) -> Difference {
-    Difference {
-        op: op.to_owned(),
-        case: case.to_owned(),
-        field: field.to_owned(),
-        a,
-        b,
-    }
-}
-
-fn compare_one(a: &Observation, b: &Observation, differences: &mut Vec<Difference>) {
-    let mut field = |name: String, value_a: Option<&str>, value_b: Option<&str>| {
-        if value_a != value_b {
-            let show = |value: Option<&str>| value.unwrap_or(ABSENT).to_owned();
-            differences.push(difference(a.op, a.case, &name, show(value_a), show(value_b)));
-        }
-    };
-    field(
-        "status".to_owned(),
-        Some(a.status.to_string().as_str()),
-        Some(b.status.to_string().as_str()),
-    );
-    field("error_code".to_owned(), a.error_code.as_deref(), b.error_code.as_deref());
-    let headers: BTreeSet<_> = a.headers.keys().chain(b.headers.keys()).collect();
-    for name in headers {
-        field(
-            format!("header.{name}"),
-            a.headers.get(name).map(String::as_str),
-            b.headers.get(name).map(String::as_str),
-        );
-    }
-    let fields: BTreeSet<_> = a.fields.keys().chain(b.fields.keys()).collect();
-    for name in fields {
-        field(
-            format!("body.{name}"),
-            a.fields.get(name).map(String::as_str),
-            b.fields.get(name).map(String::as_str),
-        );
-    }
-}
-
-/// Operations observed on both sides.
-pub(crate) fn compared_ops(a: &[Observation], b: &[Observation]) -> BTreeSet<&'static str> {
-    let ops_b: BTreeSet<_> = b.iter().map(|observation| observation.op).collect();
-    a.iter()
-        .map(|observation| observation.op)
-        .filter(|op| ops_b.contains(op))
-        .collect()
-}
-
-/// `(op, case)` pairs observed on both sides.
-pub(crate) fn compared_cases(a: &[Observation], b: &[Observation]) -> usize {
-    let cases_b: BTreeSet<_> = b.iter().map(|observation| (observation.op, observation.case)).collect();
-    a.iter()
-        .map(|observation| (observation.op, observation.case))
-        .collect::<BTreeSet<_>>()
-        .intersection(&cases_b)
-        .count()
-}
-
-/// The JSON report written for every run.
-#[derive(Debug, Serialize)]
-pub(crate) struct Report {
-    pub schema: u32,
-    pub run: String,
-    pub stack_a: String,
-    pub stack_b: String,
-    pub region_a: Option<String>,
-    pub region_b: Option<String>,
-    pub compared_ops: usize,
-    pub compared_cases: usize,
-    pub differences: Vec<Difference>,
-}
-
-impl Report {
-    /// Writes the report as `<directory>/<run>.json` and returns its path.
-    pub(crate) fn write_to(&self, directory: &Path) -> std::io::Result<PathBuf> {
-        std::fs::create_dir_all(directory)?;
-        let path = directory.join(format!("{}.json", self.run));
-        let json = serde_json::to_vec_pretty(self).map_err(std::io::Error::other)?;
-        std::fs::write(&path, json)?;
-        Ok(path)
-    }
-}
-
-/// One line per difference, for a failing assertion.
-pub(crate) fn render(differences: &[Difference]) -> String {
-    differences
-        .iter()
-        .map(|d| format!("({}, {}, {}, a={:?}, b={:?})", d.op, d.case, d.field, d.a, d.b))
-        .collect::<Vec<_>>()
-        .join("\n")
-}
-
-#[cfg(test)]
-mod tests {
-    use super::*;
-
-    fn ok(op: &'static str) -> Observation {
-        Observation::new(op, "normal", 200, None, [("content-type", "application/xml")])
-    }
-
-    fn fields(differences: &[Difference]) -> Vec<&str> {
-        differences.iter().map(|d| d.field.as_str()).collect()
-    }
-
-    #[test]
-    fn identical_transcripts_have_no_difference() {
-        let a = vec![
-            ok("HeadBucket"),
-            ok("GetBucketLocation").with_field("location_constraint", "us-east-1"),
-        ];
-        let b = a.clone();
-        assert_eq!(compare(&a, &b), Vec::new());
-        assert_eq!(compared_ops(&a, &b).len(), 2);
-    }
-
-    #[test]
-    fn a_status_difference_is_reported() {
-        let a = vec![ok("PutBucketAcl")];
-        let b = vec![Observation::new(
-            "PutBucketAcl",
-            "normal",
-            403,
-            None,
-            [("content-type", "application/xml")],
-        )];
-        assert_eq!(
-            compare(&a, &b),
-            vec![Difference {
-                op: "PutBucketAcl".into(),
-                case: "normal".into(),
-                field: "status".into(),
-                a: "200".into(),
-                b: "403".into(),
-            }]
-        );
-    }
-
-    #[test]
-    fn an_error_code_difference_is_reported() {
-        let a = vec![Observation::new("GetObject", "negative", 404, Some("NoSuchKey".into()), [])];
-        let b = vec![Observation::new(
-            "GetObject",
-            "negative",
-            404,
-            Some("NoSuchBucket".into()),
-            [],
-        )];
-        let differences = compare(&a, &b);
-        assert_eq!(fields(&differences), ["error_code"]);
-        assert_eq!((differences[0].a.as_str(), differences[0].b.as_str()), ("NoSuchKey", "NoSuchBucket"));
-    }
-
-    #[test]
-    fn an_error_code_on_one_side_only_is_reported() {
-        let a = vec![Observation::new("HeadBucket", "negative", 404, Some("NotFound".into()), [])];
-        let b = vec![Observation::new("HeadBucket", "negative", 404, None, [])];
-        let differences = compare(&a, &b);
-        assert_eq!(fields(&differences), ["error_code"]);
-        assert_eq!(differences[0].b, ABSENT);
-    }
-
-    #[test]
-    fn an_allowlisted_header_difference_is_reported() {
-        let a = vec![Observation::new("HeadObject", "normal", 200, None, [("ETag", "\"aa\"")])];
-        let b = vec![Observation::new("HeadObject", "normal", 200, None, [("etag", "\"bb\"")])];
-        assert_eq!(fields(&compare(&a, &b)), ["header.etag"]);
-    }
-
-    #[test]
-    fn a_header_sent_by_one_side_only_is_reported() {
-        let a = vec![Observation::new(
-            "DeleteObject",
-            "normal",
-            204,
-            None,
-            [("x-amz-delete-marker", "true")],
-        )];
-        let b = vec![Observation::new("DeleteObject", "normal", 204, None, [])];
-        let differences = compare(&a, &b);
-        assert_eq!(fields(&differences), ["header.x-amz-delete-marker"]);
-        assert_eq!((differences[0].a.as_str(), differences[0].b.as_str()), ("true", ABSENT));
-    }
-
-    #[test]
-    fn every_server_side_encryption_header_is_compared() {
-        let header = "x-amz-server-side-encryption-customer-key-md5";
-        let a = vec![Observation::new("PutObject", "sse-c", 200, None, [(header, "one")])];
-        let b = vec![Observation::new("PutObject", "sse-c", 200, None, [(header, "two")])];
-        assert_eq!(fields(&compare(&a, &b)), [format!("header.{header}")]);
-    }
-
-    #[test]
-    fn headers_outside_the_allowlist_are_not_compared() {
-        let a = vec![Observation::new(
-            "ListBuckets",
-            "normal",
-            200,
-            None,
-            [
-                ("date", "Mon"),
-                ("x-amz-request-id", "1"),
-                ("server", "a"),
-                ("content-length", "10"),
-            ],
-        )];
-        let b = vec![Observation::new(
-            "ListBuckets",
-            "normal",
-            200,
-            None,
-            [
-                ("date", "Tue"),
-                ("x-amz-request-id", "2"),
-                ("server", "b"),
-                ("content-length", "11"),
-            ],
-        )];
-        assert_eq!(compare(&a, &b), Vec::new());
-        assert!(a[0].headers.is_empty(), "non-allowlisted headers were kept: {:?}", a[0].headers);
-    }
-
-    #[test]
-    fn distinct_version_ids_are_not_a_difference() {
-        let a = vec![Observation::new(
-            "PutObject",
-            "normal",
-            200,
-            None,
-            [("x-amz-version-id", "0b1c")],
-        )];
-        let b = vec![Observation::new(
-            "PutObject",
-            "normal",
-            200,
-            None,
-            [("x-amz-version-id", "9f8e")],
-        )];
-        assert_eq!(compare(&a, &b), Vec::new());
-    }
-
-    #[test]
-    fn a_null_version_id_against_a_real_one_is_reported() {
-        let a = vec![Observation::new(
-            "PutObject",
-            "normal",
-            200,
-            None,
-            [("x-amz-version-id", "null")],
-        )];
-        let b = vec![Observation::new(
-            "PutObject",
-            "normal",
-            200,
-            None,
-            [("x-amz-version-id", "9f8e")],
-        )];
-        assert_eq!(fields(&compare(&a, &b)), ["header.x-amz-version-id"]);
-    }
-
-    #[test]
-    fn a_body_field_difference_is_reported() {
-        let a = vec![ok("GetBucketLocation").with_field("location_constraint", "us-east-1")];
-        let b = vec![ok("GetBucketLocation").with_field("location_constraint", "eu-west-1")];
-        let differences = compare(&a, &b);
-        assert_eq!(fields(&differences), ["body.location_constraint"]);
-        assert_eq!((differences[0].a.as_str(), differences[0].b.as_str()), ("us-east-1", "eu-west-1"));
-    }
-
-    #[test]
-    fn a_body_field_on_one_side_only_is_reported() {
-        let a = vec![ok("ListObjectsV2").with_field("keys", "[]")];
-        let b = vec![ok("ListObjectsV2")];
-        assert_eq!(fields(&compare(&a, &b)), ["body.keys"]);
-    }
-
-    #[test]
-    fn error_messages_are_never_compared() {
-        let mut a = Observation::new("GetObject", "negative", 404, Some("NoSuchKey".into()), []);
-        let mut b = a.clone();
-        a.message = Some("The specified key does not exist.".into());
-        b.message = Some("Object not found".into());
-        assert_eq!(compare(&[a], &[b]), Vec::new());
-    }
-
-    #[test]
-    fn a_case_observed_on_one_side_only_is_reported() {
-        let a = vec![ok("HeadBucket"), ok("ListBuckets")];
-        let b = vec![ok("HeadBucket")];
-        let differences = compare(&a, &b);
-        assert_eq!(fields(&differences), ["observation"]);
-        assert_eq!(differences[0].op, "ListBuckets");
-        assert_eq!(compared_ops(&a, &b).into_iter().collect::<Vec<_>>(), ["HeadBucket"]);
-    }
-
-    #[test]
-    fn a_case_recorded_twice_is_reported() {
-        let a = vec![ok("HeadBucket"), ok("HeadBucket")];
-        let b = vec![ok("HeadBucket")];
-        assert_eq!(fields(&compare(&a, &b)), ["duplicate_case"]);
-    }
-
-    #[test]
-    fn compared_ops_counts_operations_not_cases() {
-        let negative = Observation::new("HeadBucket", "negative", 404, Some("NotFound".into()), []);
-        let a = vec![ok("HeadBucket"), negative, ok("ListBuckets")];
-        let b = a.clone();
-        assert_eq!(compared_ops(&a, &b).len(), 2);
-    }
-
-    #[test]
-    fn the_report_lists_each_difference_as_op_field_a_b() {
-        let directory = std::env::temp_dir().join(format!("dual-stack-report-{}", uuid::Uuid::new_v4()));
-        let report = Report {
-            schema: 1,
-            run: "unit".into(),
-            stack_a: "legacy".into(),
-            stack_b: "legacy".into(),
-            region_a: None,
-            region_b: Some("eu-west-1".into()),
-            compared_ops: 1,
-            compared_cases: 1,
-            differences: compare(
-                &[ok("GetBucketLocation").with_field("location_constraint", "us-east-1")],
-                &[ok("GetBucketLocation").with_field("location_constraint", "eu-west-1")],
-            ),
-        };
-        let path = report.write_to(&directory).expect("write the report");
-        let json: serde_json::Value = serde_json::from_slice(&std::fs::read(&path).expect("read the report")).expect("parse");
-        let _ = std::fs::remove_dir_all(&directory);
-        assert_eq!(path.file_name().and_then(|name| name.to_str()), Some("unit.json"));
-        assert_eq!(json["compared_ops"], 1);
-        assert_eq!(
-            json["differences"],
-            serde_json::json!([{
-                "op": "GetBucketLocation",
-                "case": "normal",
-                "field": "body.location_constraint",
-                "a": "us-east-1",
-                "b": "eu-west-1",
-            }])
-        );
-    }
-}

crates/e2e_test/src/dual_stack_differential/mod.rs

@@ -1,199 +0,0 @@
-// Copyright 2024 RustFS Team
-//
-// Licensed under the Apache License, Version 2.0 (the "License");
-// you may not use this file except in compliance with the License.
-// You may obtain a copy of the License at
-//
-//     http://www.apache.org/licenses/LICENSE-2.0
-//
-// Unless required by applicable law or agreed to in writing, software
-// distributed under the License is distributed on an "AS IS" BASIS,
-// WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
-// See the License for the specific language governing permissions and
-// limitations under the License.
-
-//! Dual-stack differential: one `rustfs` binary, two processes, one SDK script (rustfs/backlog#2740).
-//!
-//! Stack A always runs `RUSTFS_S3_STACK=legacy`. Stack B runs the value of `RUSTFS_E2E_STACK_B`
-//! (default `legacy` until the gateway stack is assembled), and `RUSTFS_E2E_STACK_B_REGION`, when
-//! set, starts B in another region so that a difference can be injected by hand. Both servers
-//! answer the script in `scenarios`, which covers every operation of `impl S3 for FS`; `compare`
-//! reports each `(op, field, a, b)` on which the answers differ, and every run writes its report
-//! to `<target>/dual-stack/<run>.json`. Two processes, not two embedded servers: bucket metadata
-//! is process-global, so two stacks in one process would read each other's buckets.
-
-mod compare;
-mod scenarios;
-
-use crate::common::{RustFSTestEnvironment, init_logging, workspace_root};
-use compare::{Difference, Observation, Report, compare, compared_cases, compared_ops, render};
-use std::path::PathBuf;
-use std::time::{SystemTime, UNIX_EPOCH};
-
-type BoxError = Box<dyn std::error::Error + Send + Sync>;
-
-const STACK_A: &str = "legacy";
-const ENV_STACK_B: &str = "RUSTFS_E2E_STACK_B";
-const ENV_STACK_B_REGION: &str = "RUSTFS_E2E_STACK_B_REGION";
-/// Becomes `gateway` once rustfs/backlog#2734 T2.5 assembles the gateway stack.
-const DEFAULT_STACK_B: &str = "legacy";
-/// The region the negative control gives stack A explicitly, so an inherited `RUSTFS_REGION`
-/// cannot erase the injected difference. Not `us-east-1`: that region answers GetBucketLocation
-/// with an empty LocationConstraint and re-creating an owned bucket with 200, so it would not
-/// report its own name and would add CreateBucket differences to the control.
-const CONTROL_REGION_A: &str = "us-west-2";
-const CONTROL_REGION_B: &str = "eu-west-1";
-/// `normal` cases the legacy stack must answer with a 2xx. The comparison alone cannot tell a
-/// script that exercises real behavior from one in which both stacks fail the same way. Five of
-/// the 73 fail on legacy by design of this script: PutBucketReplication (no registered remote
-/// target), GetBucketReplication (so no configuration), PutBucketNotificationConfiguration (no
-/// notification target configured), RestoreObject (the object is not archived) and
-/// GetObjectTorrent (legacy answers `NoSuchKey`).
-const MIN_NORMAL_SUCCESSES: usize = 68;
-
-struct PairConfig {
-    stack_b: String,
-    region_a: Option<String>,
-    region_b: Option<String>,
-}
-
-impl PairConfig {
-    fn from_env() -> Self {
-        Self {
-            stack_b: std::env::var(ENV_STACK_B).unwrap_or_else(|_| DEFAULT_STACK_B.to_owned()),
-            region_a: None,
-            region_b: std::env::var(ENV_STACK_B_REGION).ok(),
-        }
-    }
-}
-
-/// Starts one server on `stack`, overriding its region only when asked.
-async fn start_stack(stack: &str, region: Option<&str>) -> Result<RustFSTestEnvironment, BoxError> {
-    let mut env = RustFSTestEnvironment::new().await?;
-    let mut vars = vec![("RUSTFS_S3_STACK", stack)];
-    if let Some(region) = region {
-        vars.push(("RUSTFS_REGION", region));
-    }
-    env.start_rustfs_server_without_cleanup_with_env(&vars).await?;
-    Ok(env)
-}
-
-/// Starts stack A and stack B side by side, each with its own port and data directory.
-async fn spawn_pair(config: &PairConfig) -> Result<(RustFSTestEnvironment, RustFSTestEnvironment), BoxError> {
-    tokio::try_join!(
-        start_stack(STACK_A, config.region_a.as_deref()),
-        start_stack(&config.stack_b, config.region_b.as_deref())
-    )
-}
-
-fn report_directory() -> PathBuf {
-    let target = std::env::var_os("CARGO_TARGET_DIR").map_or_else(|| PathBuf::from("target"), PathBuf::from);
-    let target = if target.is_absolute() {
-        target
-    } else {
-        workspace_root().join(target)
-    };
-    target.join("dual-stack")
-}
-
-struct Run {
-    report: Report,
-    path: PathBuf,
-    a: Vec<Observation>,
-}
-
-/// Runs the script against both stacks concurrently and writes the report.
-async fn run_differential(label: &str, config: &PairConfig) -> Run {
-    let (server_a, server_b) = spawn_pair(config).await.expect("start stack A and stack B");
-    let (a, b) = tokio::join!(Box::pin(scenarios::run(&server_a)), Box::pin(scenarios::run(&server_b)));
-    let millis = SystemTime::now()
-        .duration_since(UNIX_EPOCH)
-        .map_or(0, |elapsed| elapsed.as_millis());
-    let report = Report {
-        schema: 1,
-        run: format!("{label}-{millis}-{}", std::process::id()),
-        stack_a: STACK_A.to_owned(),
-        stack_b: config.stack_b.clone(),
-        region_a: config.region_a.clone(),
-        region_b: config.region_b.clone(),
-        compared_ops: compared_ops(&a, &b).len(),
-        compared_cases: compared_cases(&a, &b),
-        differences: compare(&a, &b),
-    };
-    let path = report.write_to(&report_directory()).expect("write the dual-stack report");
-    Run { report, path, a }
-}
-
-#[cfg(test)]
-mod tests {
-    use super::*;
-
-    #[tokio::test]
-    async fn both_stacks_answer_every_operation_identically() {
-        init_logging();
-        let config = PairConfig::from_env();
-        let run = run_differential("self-comparison", &config).await;
-        let report = &run.report;
-
-        let mut observed: Vec<_> = run.a.iter().map(|observation| observation.op).collect();
-        observed.sort_unstable();
-        observed.dedup();
-        assert_eq!(
-            observed,
-            scenarios::OPERATIONS,
-            "the script must drive exactly the operations of impl S3 for FS"
-        );
-        assert_eq!(report.compared_ops, scenarios::OPERATIONS.len(), "report: {}", run.path.display());
-        assert!(
-            report.differences.is_empty(),
-            "stack {} and stack {} differ on {} of {} cases (report: {}):\n{}",
-            report.stack_a,
-            report.stack_b,
-            report.differences.len(),
-            report.compared_cases,
-            run.path.display(),
-            render(&report.differences)
-        );
-
-        let not_4xx: Vec<_> = run
-            .a
-            .iter()
-            .filter(|observation| observation.case == "negative" && !(400..500).contains(&observation.status))
-            .collect();
-        assert!(not_4xx.is_empty(), "negative cases must fail with a 4xx on {STACK_A}: {not_4xx:#?}");
-        let successes = run
-            .a
-            .iter()
-            .filter(|observation| observation.case == "normal" && observation.is_success())
-            .count();
-        assert!(
-            successes >= MIN_NORMAL_SUCCESSES,
-            "only {successes} normal cases succeeded on {STACK_A}: {:#?}",
-            run.a
-        );
-    }
-
-    #[tokio::test]
-    async fn differential_detects_an_injected_difference() {
-        init_logging();
-        let config = PairConfig {
-            stack_b: STACK_A.to_owned(),
-            region_a: Some(CONTROL_REGION_A.to_owned()),
-            region_b: Some(CONTROL_REGION_B.to_owned()),
-        };
-        let run = run_differential("injected-region", &config).await;
-        let expected = Difference {
-            op: "GetBucketLocation".to_owned(),
-            case: "normal".to_owned(),
-            field: "body.location_constraint".to_owned(),
-            a: CONTROL_REGION_A.to_owned(),
-            b: CONTROL_REGION_B.to_owned(),
-        };
-        assert!(
-            run.report.differences.contains(&expected),
-            "B started in {CONTROL_REGION_B} but GetBucketLocation was not reported (report: {}):\n{}",
-            run.path.display(),
-            render(&run.report.differences)
-        );
-    }
-}

crates/e2e_test/src/lib.rs

@@ -31,12 +31,6 @@ pub mod chaos;
 #[cfg(test)]
 pub mod fake_s3_target;
 
-// Dual-stack differential (rustfs/backlog#2740): one binary, two processes on
-// different RUSTFS_S3_STACK values, the same SDK script for every S3 operation,
-// compared on status, error code, allowlisted headers and typed body fields.
-#[cfg(test)]
-mod dual_stack_differential;
-
 // On-demand migration (backlog#2147): shared two-server environment, admin
 // wrappers, and the harness self-test (backlog#2151). Behavior scenarios are
 // added by later ODM tasks.

crates/e2e_test/src/protocols/webdav_core.rs

@@ -32,9 +32,7 @@
 //! Advisory: <https://github.com/rustfs/rustfs/security/advisories/GHSA-3p3x-734c-h5vx>
 
 use crate::common::rustfs_binary_path_with_features;
-use crate::common::{
-    AdminTransport, admin_add_canned_policy_via, admin_attach_user_policy_via, admin_create_user_via, signed_s3_request,
-};
+use crate::common::{AdminTransport, admin_add_canned_policy_via, admin_attach_user_policy_via, admin_create_user_via};
 use crate::common::{FAST_DATA_USAGE_SCANNER_ENV, admin_request};
 use crate::protocols::test_env::{DEFAULT_ACCESS_KEY, DEFAULT_SECRET_KEY, ProtocolTestEnvironment};
 use anyhow::Result;
@@ -776,81 +774,6 @@ async fn test_webdav_core_operations_direct() -> Result<()> {
     test_webdav_core_operations().await
 }
 
-#[tokio::test]
-async fn test_webdav_content_type_directory_marker() -> Result<()> {
-    let env = ProtocolTestEnvironment::new().map_err(|e| anyhow::anyhow!("{}", e))?;
-    let admin_base_url = format!("http://{}", S3_TEST_ADDRESS);
-    let binary_path = rustfs_binary_path_with_features(Some("webdav"));
-    let mut server_process = Command::new(&binary_path)
-        .arg("--address")
-        .arg(S3_TEST_ADDRESS)
-        .env("RUSTFS_CONSOLE_ENABLE", "false")
-        .env("RUSTFS_WEBDAV_ENABLE", "true")
-        .env("RUSTFS_WEBDAV_ADDRESS", WEBDAV_ADDRESS)
-        .env("RUSTFS_WEBDAV_TLS_ENABLED", "false")
-        .arg(&env.temp_dir)
-        .spawn()?;
-
-    let result = async {
-        ProtocolTestEnvironment::wait_for_port_ready(WEBDAV_PORT, 30)
-            .await
-            .map_err(|e| anyhow::anyhow!("{}", e))?;
-
-        let client = create_client();
-        let bucket_name = "webdav-content-type-bucket";
-        let directory_name = "content-type-directory";
-        let webdav_base_url = format!("http://{}", WEBDAV_ADDRESS);
-        let mkcol_bucket = client
-            .request(
-                reqwest::Method::from_bytes(b"MKCOL").expect("MKCOL is a valid HTTP method"),
-                format!("{}/{}", webdav_base_url, bucket_name),
-            )
-            .header("Authorization", basic_auth_header())
-            .send()
-            .await?;
-        anyhow::ensure!(
-            mkcol_bucket.status().is_success() || mkcol_bucket.status().as_u16() == 201,
-            "MKCOL bucket should succeed, got: {}",
-            mkcol_bucket.status()
-        );
-
-        let mkcol_directory = client
-            .request(
-                reqwest::Method::from_bytes(b"MKCOL").expect("MKCOL is a valid HTTP method"),
-                format!("{}/{}/{}", webdav_base_url, bucket_name, directory_name),
-            )
-            .header("Authorization", basic_auth_header())
-            .send()
-            .await?;
-        anyhow::ensure!(
-            mkcol_directory.status().is_success() || mkcol_directory.status().as_u16() == 201,
-            "MKCOL directory should succeed, got: {}",
-            mkcol_directory.status()
-        );
-
-        let head_url = format!("{}/{}/{}/", admin_base_url, bucket_name, directory_name);
-        let head = signed_s3_request(http::Method::HEAD, &head_url, None, None, DEFAULT_ACCESS_KEY, DEFAULT_SECRET_KEY)
-            .await
-            .map_err(|error| anyhow::anyhow!("{error}"))?;
-        anyhow::ensure!(head.status().is_success(), "HEAD directory marker should succeed, got: {}", head.status());
-        let content_type = head
-            .headers()
-            .get("content-type")
-            .and_then(|value| value.to_str().ok())
-            .unwrap_or_default();
-        anyhow::ensure!(
-            content_type == "application/x-directory",
-            "WebDAV directory marker must preserve its content type, got: {content_type}"
-        );
-        Ok(())
-    }
-    .await;
-
-    let _ = server_process.kill().await;
-    let _ = server_process.wait().await;
-    result
-}
-
 /// Extract the numeric value of the first occurrence of a DAV property in a
 /// PROPFIND multistatus body. Namespace-prefix agnostic; a self-closing
 /// (unreported) property yields None.

crates/e2e_test/src/rpc_fault_proxy.rs

@@ -27,8 +27,8 @@ use hyper_util::{
 };
 use prost::Message as _;
 use rustfs_protos::proto_gen::node_service::{
-    BatchGenerallyLockRequest, DeleteRequest, GenerallyLockRequest, ReadVersionRequest, RecoverMultipartSourceRequest,
-    RenameDataResponse,
+    BatchGenerallyLockRequest, DeleteRequest, GenerallyLockRequest, PreparePartTransactionResponse, ReadVersionRequest,
+    RecoverMultipartSourceRequest, RenameDataResponse, SettlePartTransactionRequest, SettlePartTransactionResponse,
 };
 use std::{
     collections::HashSet,
@@ -56,6 +56,35 @@ enum Disposition {
     Discard,
 }
 
+#[derive(Clone, Copy)]
+enum MutationReply {
+    None,
+    Rename,
+    PreparePart,
+    SettlePart,
+}
+
+impl MutationReply {
+    fn outcome(self, bytes: &[u8]) -> io::Result<Option<(bool, bool)>> {
+        let result = match self {
+            Self::None => return Ok(None),
+            Self::Rename => {
+                let reply = RenameDataResponse::decode(unary_payload(bytes)?).map_err(io::Error::other)?;
+                (reply.success, reply.error.is_some())
+            }
+            Self::PreparePart => {
+                let reply = PreparePartTransactionResponse::decode(unary_payload(bytes)?).map_err(io::Error::other)?;
+                (reply.success, reply.error.is_some())
+            }
+            Self::SettlePart => {
+                let reply = SettlePartTransactionResponse::decode(unary_payload(bytes)?).map_err(io::Error::other)?;
+                (reply.success, reply.error.is_some())
+            }
+        };
+        Ok(Some(result))
+    }
+}
+
 struct PauseState {
     methods: Vec<&'static str>,
     scope: Option<Vec<u8>>,
@@ -64,6 +93,9 @@ struct PauseState {
     limit: usize,
     retain_after_disconnect: bool,
     refused_mutations: AtomicUsize,
+    successful_mutation_replies: AtomicUsize,
+    settle_commits: AtomicUsize,
+    settle_rollbacks: AtomicUsize,
     responses: AtomicUsize,
     matched: AtomicUsize,
     blocked: AtomicUsize,
@@ -122,6 +154,17 @@ impl RpcPause {
         self.state.refused_mutations.load(Ordering::SeqCst)
     }
 
+    pub(crate) fn successful_mutation_replies(&self) -> usize {
+        self.state.successful_mutation_replies.load(Ordering::SeqCst)
+    }
+
+    pub(crate) fn retained_settle_actions(&self) -> (usize, usize) {
+        (
+            self.state.settle_commits.load(Ordering::SeqCst),
+            self.state.settle_rollbacks.load(Ordering::SeqCst),
+        )
+    }
+
     pub(crate) fn unsupported_source_inspection_routes(&self) -> HashSet<String> {
         self.state
             .unsupported_inspection_routes
@@ -394,6 +437,9 @@ impl RpcFaultProxy {
             limit,
             retain_after_disconnect,
             refused_mutations: AtomicUsize::new(0),
+            successful_mutation_replies: AtomicUsize::new(0),
+            settle_commits: AtomicUsize::new(0),
+            settle_rollbacks: AtomicUsize::new(0),
             responses: AtomicUsize::new(0),
             matched: AtomicUsize::new(0),
             blocked: AtomicUsize::new(0),
@@ -509,6 +555,7 @@ async fn forward_request(
         if let Some(method) = method.filter(|name| pause.methods.contains(name)) {
             let mut route = None;
             let mut lock_owners = Vec::new();
+            let mut settle_action = None;
             let matches = if let Some(scope) = pause.scope.as_deref() {
                 // Inspect bounded control routing while preserving the exact
                 // signed bytes and trailers forwarded to the backend.
@@ -522,12 +569,19 @@ async fn forward_request(
                     inspection = Some((pause.clone(), disk));
                 }
                 let mut matches = scope_matches(method, &bytes, scope)?;
+                if matches && method == "SettlePartTransaction" {
+                    settle_action = Some(
+                        SettlePartTransactionRequest::decode(unary_payload(&bytes)?)
+                            .map_err(io::Error::other)?
+                            .rollback,
+                    );
+                }
                 if matches && (pause.record_lock_owners || pause.lock_owner.is_some()) {
                     lock_owners = scoped_lock_owners(method, &bytes, scope)?;
                     if let Some(owner) = &pause.lock_owner {
                         matches = lock_owners.iter().any(|candidate| candidate == owner);
                         if !matches && pause.record_lock_owners {
-                            observed = Some((pause.clone(), None, std::mem::take(&mut lock_owners), false));
+                            observed = Some((pause.clone(), None, std::mem::take(&mut lock_owners), MutationReply::None));
                         }
                     }
                 }
@@ -549,6 +603,14 @@ async fn forward_request(
                 true
             };
             if matches && pause.matched.fetch_add(1, Ordering::SeqCst) < pause.limit {
+                if let Some(rollback) = settle_action {
+                    let counter = if rollback {
+                        &pause.settle_rollbacks
+                    } else {
+                        &pause.settle_commits
+                    };
+                    counter.fetch_add(1, Ordering::SeqCst);
+                }
                 let mut released = pause.release.subscribe();
                 pause.blocked.fetch_add(1, Ordering::SeqCst);
                 pause.reached.notify_one();
@@ -568,10 +630,14 @@ async fn forward_request(
                     pause,
                     route,
                     lock_owners,
-                    matches!(
-                        method,
-                        "RenameData" | "RenameMultipartData" | "RenameDataAtIncarnation" | "RenameDataWithQuotaFence"
-                    ),
+                    match method {
+                        "RenameData" | "RenameMultipartData" | "RenameDataAtIncarnation" | "RenameDataWithQuotaFence" => {
+                            MutationReply::Rename
+                        }
+                        "PreparePartTransaction" => MutationReply::PreparePart,
+                        "SettlePartTransaction" => MutationReply::SettlePart,
+                        _ => MutationReply::None,
+                    },
                 ));
             }
         }
@@ -604,7 +670,7 @@ async fn forward_request(
             .expect("inspection response routes")
             .insert(disk);
     }
-    if let Some((pause, route, lock_owners, rename)) = observed {
+    if let Some((pause, route, lock_owners, mutation_reply)) = observed {
         pause.responses.fetch_add(1, Ordering::SeqCst);
         if response.status().is_success() && response.headers().get("grpc-status").is_none_or(|status| status == "0") {
             pause.forwarded.fetch_add(1, Ordering::SeqCst);
@@ -626,10 +692,14 @@ async fn forward_request(
                 .headers
                 .get("grpc-status")
                 .or_else(|| trailers.as_ref()?.get("grpc-status"));
-            if rename && parts.status.is_success() && status.is_none_or(|status| status == "0") {
-                let result = RenameDataResponse::decode(unary_payload(&bytes)?).map_err(io::Error::other)?;
-                if !result.success && result.error.is_some() {
+            if parts.status.is_success()
+                && status.is_some_and(|status| status == "0")
+                && let Some((success, has_error)) = mutation_reply.outcome(&bytes)?
+            {
+                if !success && has_error {
                     pause.refused_mutations.fetch_add(1, Ordering::SeqCst);
+                } else if success && !has_error {
+                    pause.successful_mutation_replies.fetch_add(1, Ordering::SeqCst);
                 }
             }
             let body = Full::new(bytes)
@@ -676,6 +746,10 @@ mod tests {
     }
 
     async fn backend_with_status(status: Option<&'static str>) -> io::Result<Backend> {
+        backend_with_reply(status, None).await
+    }
+
+    async fn backend_with_reply(status: Option<&'static str>, reply: Option<Bytes>) -> io::Result<Backend> {
         let listener = TcpListener::bind("127.0.0.1:0").await?;
         let address = listener.local_addr()?;
         let received = Arc::new(AtomicUsize::new(0));
@@ -687,9 +761,11 @@ mod tests {
                     accepted = listener.accept() => {
                         let Ok((socket, _)) = accepted else { break };
                         let received = received_task.clone();
+                        let reply = reply.clone();
                         children.spawn(async move {
                             let service = service_fn(move |request: Request<Incoming>| {
                                 let received = received.clone();
+                                let reply = reply.clone();
                                 async move {
                                 received.fetch_add(1, Ordering::SeqCst);
                                 let authority = request.headers().get(http::header::HOST).cloned().or_else(|| {
@@ -698,6 +774,7 @@ mod tests {
                                     })
                                 });
                                 let body = request.into_body().collect().await?.to_bytes();
+                                let body = reply.unwrap_or(body);
                                 let mut trailers = http::HeaderMap::new();
                                 trailers.insert("grpc-status", http::HeaderValue::from_static(status.unwrap_or("0")));
                                 trailers.insert("x-proof", http::HeaderValue::from_static("unchanged-proof"));
@@ -1434,6 +1511,93 @@ mod tests {
         .await
         .expect("scoped discard deadline");
         assert_late_packet_ownership().await;
+        assert_late_part_reply_refusal().await;
+    }
+
+    async fn assert_late_part_reply_refusal() {
+        use rustfs_protos::proto_gen::node_service::{Error, PreparePartTransactionRequest};
+        tokio::time::timeout(Duration::from_secs(10), async {
+            // method, success, error, gRPC status, rollback, expected successes/refusals
+            for (method, success, has_error, status, rollback, expected) in [
+                ("PreparePartTransaction", false, true, None, false, (0, 1)),
+                ("PreparePartTransaction", true, false, None, false, (1, 0)),
+                ("PreparePartTransaction", false, false, None, false, (0, 0)),
+                ("SettlePartTransaction", true, false, None, false, (1, 0)),
+                ("SettlePartTransaction", false, true, None, true, (0, 1)),
+                ("SettlePartTransaction", true, true, None, false, (0, 0)),
+                ("SettlePartTransaction", false, true, Some("16"), false, (0, 0)),
+                ("PreparePartTransaction", false, true, Some("16"), false, (0, 0)),
+                ("RenameMultipartData", false, true, None, false, (0, 1)),
+            ] {
+                let error = has_error.then(|| Error {
+                    code: 1,
+                    error_info: "retired source".to_owned(),
+                });
+                let reply = match method {
+                    "PreparePartTransaction" => unary_frame(&PreparePartTransactionResponse { success, error }),
+                    "SettlePartTransaction" => unary_frame(&SettlePartTransactionResponse { success, error }),
+                    _ => unary_frame(&RenameDataResponse {
+                        success,
+                        error,
+                        ..Default::default()
+                    }),
+                };
+                // An unknown field must survive typed inspection byte-for-byte.
+                let mut framed = reply.to_vec();
+                framed.extend_from_slice(&[0xfa, 0x03, 0x03, b'x', b'y', b'z']);
+                let payload_len = u32::try_from(framed.len() - 5).expect("small reply");
+                framed[1..5].copy_from_slice(&payload_len.to_be_bytes());
+                let reply = Bytes::from(framed);
+                let backend = backend_with_reply(status, Some(reply.clone()))
+                    .await
+                    .expect("typed reply backend");
+                let proxy = RpcFaultProxy::start_on("127.0.0.1:0".parse().expect("loopback"), backend.address)
+                    .await
+                    .expect("proxy");
+                let pause = proxy.pause_late(&[method], 1, "late-upload").expect("part gate");
+                let request = match method {
+                    "PreparePartTransaction" => unary_frame(&PreparePartTransactionRequest {
+                        disk: "disk".to_owned(),
+                        dst_path: "late-upload/part.1".to_owned(),
+                        ..Default::default()
+                    }),
+                    "SettlePartTransaction" => unary_frame(&SettlePartTransactionRequest {
+                        disk: "disk".to_owned(),
+                        path: "late-upload/part.1".to_owned(),
+                        rollback,
+                        ..Default::default()
+                    }),
+                    _ => Bytes::from_static(b"late-upload"),
+                };
+                let address = proxy.local_addr();
+                let caller = tokio::spawn(async move {
+                    let client: ProxyClient = Client::builder(TokioExecutor::new()).http2_only(true).build_http();
+                    let response = call_body(&client, address, method, request).await.expect("retained reply");
+                    response.into_body().collect().await.expect("typed response frames")
+                });
+                pause.wait_for_hits(1).await.expect("actual retained request");
+                pause.release();
+                let response = caller.await.expect("caller task");
+                let trailers = response.trailers().expect("reply trailers");
+                assert_eq!(trailers["x-proof"], "unchanged-proof");
+                assert_eq!(trailers["grpc-status"], status.unwrap_or("0"));
+                assert_eq!(response.to_bytes(), reply);
+                assert_eq!(pause.responses(), 1);
+                assert_eq!(
+                    (pause.successful_mutation_replies(), pause.refused_mutations()),
+                    expected,
+                    "typed {method} outcome"
+                );
+                let actions = if method == "SettlePartTransaction" {
+                    if rollback { (0, 1) } else { (1, 0) }
+                } else {
+                    (0, 0)
+                };
+                assert_eq!(pause.retained_settle_actions(), actions);
+            }
+        })
+        .await
+        .expect("part reply deadline");
     }
     async fn assert_late_packet_ownership() {
         tokio::time::timeout(Duration::from_secs(10), async {

crates/ecstore/AGENTS.md

@@ -12,6 +12,9 @@ Applies to `crates/ecstore/`.
 
 - Be careful with allocations and locking in hot paths.
 - Keep network and disk operations async-friendly; avoid introducing unnecessary blocking.
+- Coalesce related `open`, `read`/`write`, `fsync`, and `close` filesystem steps into a single per-disk blocking closure rather than chaining multiple async or blocking dispatches, and route any closure that performs file or directory sync to the disk's dedicated fsync lane so stalled syncs cannot starve foreground metadata reads.
+- Pin a parent directory once (`openat2(BENEATH | NO_SYMLINKS)` on Linux) and reuse the pinned handle across sibling reads or same-directory renames instead of re-walking ancestor components per file.
+- When the file size is already known from `fstat` metadata, pre-size the destination buffer (`len + 1`) before reading to EOF so small metadata reads complete in a single `read()` + EOF check rather than exponential-growth reallocations and syscalls.
 - Benchmark-sensitive changes should include measurable rationale.
 
 ## Allocation Discipline in Hot Paths

crates/ecstore/Cargo.toml

@@ -137,7 +137,6 @@ rustfs-lifecycle.workspace = true
 rustfs-s3-client = { workspace = true }
 rustfs-s3-types = { workspace = true }
 rustfs-data-usage.workspace = true
-rustfs-gateway-types.workspace = true
 rustfs-object-capacity.workspace = true
 rustfs-object-data-cache = { workspace = true, features = ["runtime-memory"] }
 arc-swap.workspace = true
@@ -156,6 +155,7 @@ serde = { workspace = true, features = ["derive"] }
 time = { workspace = true, features = ["parsing", "formatting", "macros", "serde"] }
 bytesize.workspace = true
 serde_json = { workspace = true, features = ["raw_value"] }
+s3s = { workspace = true, features = ["minio"] }
 http.workspace = true
 opentelemetry.workspace = true
 http-body = { workspace = true }
@@ -244,7 +244,6 @@ windows-sys = { workspace = true, features = [
 windows-sys = { workspace = true, features = ["Win32_System_Ioctl"] }
 
 [dev-dependencies]
-s3s = { workspace = true, features = ["minio"] }
 aws-smithy-async.workspace = true
 rustfs-s3-ops.workspace = true
 tokio = { workspace = true, features = ["rt-multi-thread", "macros", "test-util", "fs"] }

crates/ecstore/src/api/mod.rs

@@ -62,8 +62,6 @@ pub mod bucket {
         }
 
         pub mod bucket_lifecycle_ops {
-            pub use rustfs_lifecycle::{RestoreOutputLocation, RestoreRequest, RestoreS3Location, RestoreTag, RestoreTagging};
-
             pub use crate::bucket::lifecycle::bucket_lifecycle_ops::{
                 ExpiryState, LifecycleOps, ManualTransitionCancelCheck, ManualTransitionProgressSink,
                 ManualTransitionQueueSnapshot, ManualTransitionRunExecution, ManualTransitionRunOptions,
@@ -135,7 +133,7 @@ pub mod bucket {
             pub use crate::bucket::lifecycle::lifecycle::{
                 Event, ExpirationOptions, IlmAction, LIFECYCLE_MALFORMED_XML_ERROR_KIND, Lifecycle, LifecycleCalculate,
                 ObjectOpts, RuleValidate, TRANSITION_COMPLETE, TRANSITION_PENDING, TransitionOptions, expected_expiry_time,
-                format_lifecycle_date, object_opts_from_object_info, parse_lifecycle_date,
+                object_opts_from_object_info,
             };
         }
 
@@ -227,7 +225,7 @@ pub mod bucket {
     }
 
     pub mod object_lock {
-        pub use crate::bucket::object_lock::ObjectLockApi;
+        pub use crate::bucket::object_lock::{ObjectLockApi, ObjectLockStatusExt};
 
         pub mod types {
             pub use crate::bucket::object_lock::types::{
@@ -322,9 +320,9 @@ pub mod bucket {
 
     pub mod utils {
         pub use crate::bucket::utils::{
-            BucketConfigXml, BucketConfigXmlError, check_bucket_and_object_names, check_list_objs_args,
-            check_object_name_for_length_and_slash, check_valid_bucket_name_strict, deserialize, has_bad_path_component,
-            is_meta_bucketname, is_valid_object_prefix, serialize,
+            check_bucket_and_object_names, check_list_objs_args, check_object_name_for_length_and_slash,
+            check_valid_bucket_name_strict, deserialize, has_bad_path_component, is_meta_bucketname, is_valid_object_prefix,
+            serialize,
         };
     }
 
@@ -343,15 +341,15 @@ pub mod cache {
 
 pub mod capacity {
     pub use crate::core::pools::{
-        DecommissionUnresolvedEntry, PoolDecommissionInfo, PoolMeta, PoolStatus, get_total_usable_capacity,
-        get_total_usable_capacity_free, is_pool_activation_fleet_proof_error, path2_bucket_object,
-        path2_bucket_object_with_base_path,
+        DecommissionUnresolvedEntry, PoolAdminDiagnostic, PoolAvailability, PoolCapacityStatus, PoolDecommissionInfo,
+        PoolDiagnosticStatus, PoolMeta, PoolStatus, get_total_usable_capacity, get_total_usable_capacity_free,
+        is_pool_activation_fleet_proof_error, path2_bucket_object, path2_bucket_object_with_base_path,
     };
     pub use crate::store::utils::is_reserved_or_invalid_bucket;
 }
 
 pub mod object_api_utils {
-    pub use crate::object_api::object_api_utils::{ObjReaderFn, PutObjReader, get_raw_etag, new_getobjectreader};
+    pub use crate::object_api::object_api_utils::{ObjReaderFn, PutObjReader, get_raw_etag, new_getobjectreader, to_s3s_etag};
 }
 
 pub mod cluster {

crates/ecstore/src/bucket/lifecycle/bucket_lifecycle_ops.rs

@@ -87,14 +87,12 @@ use rustfs_filemeta::{
     FileInfo, FileInfoOpts, NULL_VERSION_ID, RestoreStatus, RestoreStatusOps, TRANSITION_COMPLETE, get_file_info,
     is_restored_object_on_disk,
 };
-use rustfs_gateway_types::persistence::PersistedLifecycleConfiguration;
-use rustfs_gateway_types::persistence::PersistedObjectLockConfiguration as ObjectLockConfiguration;
-use rustfs_lifecycle::{RestoreRequest, RestoreS3Location};
 use rustfs_lock::{NamespaceLockGuard, error::LockError};
 use rustfs_scanner_metrics::metrics::{
     IlmAction, Metrics, ScannerLifecycleExpiryStateUpdate, ScannerLifecycleTransitionStateUpdate, global_metrics,
 };
 use rustfs_utils::{get_env_i64, get_env_usize, path::encode_dir_object, string::parse_bool};
+use s3s::dto::{BucketLifecycleConfiguration, ExpirationStatus, ObjectLockConfiguration, RestoreRequest, RestoreRequestType};
 use sha2::{Digest, Sha256};
 use std::any::Any;
 use std::collections::{BTreeMap, HashMap, HashSet};
@@ -300,7 +298,7 @@ impl LifecycleSys {
         Arc::new(Self)
     }
 
-    pub async fn get(&self, bucket: &str) -> Option<PersistedLifecycleConfiguration> {
+    pub async fn get(&self, bucket: &str) -> Option<BucketLifecycleConfiguration> {
         match metadata_boundary::get_lifecycle_config(bucket).await {
             Ok((lc, _)) => Some(lc),
             Err(Error::ConfigNotFound) => None,
@@ -1088,8 +1086,9 @@ impl ExpiryState {
         }))
     }
 
-    #[cfg(test)]
-    fn new_with_unconsumed_worker_channel(capacity: usize) -> Arc<RwLock<Self>> {
+    #[cfg(any(test, feature = "test-util"))]
+    #[doc(hidden)]
+    pub fn new_with_unconsumed_worker_channel(capacity: usize) -> Arc<RwLock<Self>> {
         let (tx, rx) = mpsc::channel(capacity);
         Arc::new(RwLock::new(Self {
             tasks_tx: vec![tx],
@@ -3936,7 +3935,7 @@ pub fn init_background_stale_multipart_upload_cleanup(api: Arc<ECStore>) {
     });
 }
 
-pub async fn validate_transition_tier(lc: &PersistedLifecycleConfiguration) -> Result<(), std::io::Error> {
+pub async fn validate_transition_tier(lc: &BucketLifecycleConfiguration) -> Result<(), std::io::Error> {
     for rule in &lc.rules {
         if let Some(transitions) = &rule.transitions {
             for transition in transitions {
@@ -4053,9 +4052,8 @@ pub(crate) async fn enqueue_immediate_expiry(api: Arc<ECStore>, oi: &ObjectInfo,
         .map(lifecycle::object_opts_from_object_info)
         .collect::<Vec<ObjectOpts>>();
     let lock_config = configs.object_lock;
-    let evaluator_lock_config = lock_config.clone();
     let events = match Evaluator::new(lifecycle)
-        .with_lock_retention(evaluator_lock_config)
+        .with_lock_retention(lock_config.clone())
         .eval(&object_opts)
         .await
     {
@@ -4535,26 +4533,19 @@ pub async fn enqueue_transition_for_existing_objects_scoped(
     }
 }
 
-fn lifecycle_rule_has_date_expiration(lc: &PersistedLifecycleConfiguration, rule_id: &str) -> bool {
+fn lifecycle_rule_has_date_expiration(lc: &BucketLifecycleConfiguration, rule_id: &str) -> bool {
     lc.rules.iter().any(|rule| {
-        rule.status == "Enabled"
+        rule.status == ExpirationStatus::from_static(ExpirationStatus::ENABLED)
             && rule.id.as_deref() == Some(rule_id)
             && rule.expiration.as_ref().is_some_and(|expiration| expiration.date.is_some())
     })
 }
 
-fn should_defer_date_expiry_for_recent_config_update(lc: &PersistedLifecycleConfiguration, now: OffsetDateTime) -> bool {
-    lc.expiry_updated_at
-        .as_ref()
-        .is_some_and(|updated_at| match rustfs_lifecycle::parse_lifecycle_date(updated_at) {
-            Ok(updated_at) => {
-                now.unix_timestamp().saturating_sub(updated_at.unix_timestamp()) < DATE_EXPIRY_EXISTING_OBJECTS_GRACE_SECS
-            }
-            Err(error) => {
-                warn!(error = %error, "Deferring date expiry with an unreadable update timestamp");
-                true
-            }
-        })
+fn should_defer_date_expiry_for_recent_config_update(lc: &BucketLifecycleConfiguration, now: OffsetDateTime) -> bool {
+    lc.expiry_updated_at.as_ref().is_some_and(|updated_at| {
+        let updated_at = OffsetDateTime::from(updated_at.clone());
+        now.unix_timestamp().saturating_sub(updated_at.unix_timestamp()) < DATE_EXPIRY_EXISTING_OBJECTS_GRACE_SECS
+    })
 }
 
 async fn apply_existing_object_expiry(
@@ -4574,7 +4565,7 @@ async fn apply_existing_object_expiry(
 struct ExistingObjectExpiryContext<'a> {
     api: Arc<ECStore>,
     bucket: &'a str,
-    lc: Arc<PersistedLifecycleConfiguration>,
+    lc: Arc<BucketLifecycleConfiguration>,
     lock_config: Option<Arc<ObjectLockConfiguration>>,
     bucket_incarnation_id: Uuid,
     src: &'a LcEventSrc,
@@ -4594,9 +4585,8 @@ async fn enqueue_expiry_for_existing_object_group(
         .iter()
         .map(lifecycle::object_opts_from_object_info)
         .collect::<Vec<ObjectOpts>>();
-    let evaluator_lock_config = context.lock_config.clone();
     let events = match Evaluator::new(context.lc.clone())
-        .with_lock_retention(evaluator_lock_config)
+        .with_lock_retention(context.lock_config.clone())
         .eval(&object_opts)
         .await
     {
@@ -4790,7 +4780,7 @@ fn manual_transition_version_marker(oi: &ObjectInfo) -> String {
 async fn enqueue_transition_with_lifecycle_report(
     api: Option<Arc<ECStore>>,
     oi: &ObjectInfo,
-    lc: &PersistedLifecycleConfiguration,
+    lc: &BucketLifecycleConfiguration,
     src: &LcEventSrc,
     options: &ManualTransitionRunOptions,
     report: &mut ManualTransitionRunReport,
@@ -4856,7 +4846,7 @@ async fn enqueue_transition_with_lifecycle_report(
     false
 }
 
-async fn enqueue_transition_with_lifecycle(oi: &ObjectInfo, lc: &PersistedLifecycleConfiguration, src: &LcEventSrc) -> bool {
+async fn enqueue_transition_with_lifecycle(oi: &ObjectInfo, lc: &BucketLifecycleConfiguration, src: &LcEventSrc) -> bool {
     let event = lc.eval(&oi.to_lifecycle_opts()).await;
     match event.action {
         IlmAction::TransitionAction | IlmAction::TransitionVersionAction => {
@@ -5370,8 +5360,12 @@ pub async fn post_restore_opts(version_id: &str, bucket: &str, object: &str) ->
     })
 }
 
-fn select_restore_s3_location(rreq: &RestoreRequest) -> Result<Option<&RestoreS3Location>, std::io::Error> {
-    if rreq.type_.as_ref().is_none_or(|type_| type_.as_str() != "SELECT") {
+fn select_restore_s3_location(rreq: &RestoreRequest) -> Result<Option<&s3s::dto::S3Location>, std::io::Error> {
+    if rreq
+        .type_
+        .as_ref()
+        .is_none_or(|type_| type_.as_str() != RestoreRequestType::SELECT)
+    {
         return Ok(None);
     }
     let output_location = rreq
@@ -5413,25 +5407,27 @@ pub async fn put_restore_opts(
     // the OutputLocation metadata instead of a write to `OutputLocation.S3`
     // (backlog#1341). RestoreObject rejects SELECT at the API boundary; this
     // is the fail-closed backstop for any other caller.
-    if rreq.type_.as_ref().is_some_and(|type_| type_.as_str() == "SELECT") {
+    if rreq
+        .type_
+        .as_ref()
+        .is_some_and(|type_| type_.as_str() == RestoreRequestType::SELECT)
+    {
         return Err(std::io::Error::other("SELECT restore requests are not supported"));
     }
     for (k, v) in oi.user_defined.iter() {
         meta.insert(k.to_string(), v.clone());
     }
-    // The fleet capability gate guarantees physical copy-back support. Keep
-    // the completed transition identity for orphan recovery and remote cleanup.
     rustfs_utils::http::metadata_compat::remove_str(&mut meta, rustfs_utils::http::metadata_compat::SUFFIX_RESTORE_OPERATION_ID);
+    rustfs_utils::http::metadata_compat::remove_str(&mut meta, rustfs_utils::http::metadata_compat::SUFFIX_RESTORE_WORKER_LOCK);
     if !oi.user_tags.is_empty() {
         meta.insert(AMZ_OBJECT_TAGGING.to_string(), (*oi.user_tags).clone());
     }
-    // Finalization publishes completion and its expiry under the object lock.
-    // The local copy-back commit must not make unfinished work eligible for expiry.
+    let restore_expiry = lifecycle::expected_expiry_time(OffsetDateTime::now_utc(), rreq.days.unwrap_or(1));
     meta.insert(
         metadata_keys::RESTORE.to_string(),
         RestoreStatus {
-            is_restore_in_progress: Some(true),
-            restore_expiry_date: None,
+            is_restore_in_progress: Some(false),
+            restore_expiry_date: Some(restore_expiry),
         }
         .to_string(),
     );
@@ -5445,7 +5441,6 @@ pub async fn put_restore_opts(
         // computed MD5 is not the object's public plaintext ETag.
         preserve_etag: oi.etag.clone(),
         shard_integrity_write_mode: Some(oi.shard_integrity_write_mode()),
-        restore_copyback: true,
         //expires:           oi.expires,
         ..Default::default()
     })
@@ -5465,18 +5460,6 @@ impl LifecycleOps for ObjectInfo {
         if self.transitioned_object.status != lifecycle::TRANSITION_COMPLETE {
             return false;
         }
-        if rustfs_utils::http::metadata_compat::get_consistent_str(
-            &self.user_defined,
-            rustfs_utils::http::metadata_compat::SUFFIX_RESTORE_COPYBACK,
-        ) == Some(rustfs_utils::http::metadata_compat::RESTORE_COPYBACK_PROTOCOL_V1)
-            && self
-                .user_defined
-                .get(metadata_keys::RESTORE)
-                .and_then(|value| rustfs_filemeta::parse_restore_obj_status(value).ok())
-                .is_some_and(|status| status.is_restore_in_progress == Some(true))
-        {
-            return false;
-        }
         !is_restored_object_on_disk(&self.user_defined)
     }
 }
@@ -5488,25 +5471,25 @@ pub trait RestoreRequestOps {
 impl RestoreRequestOps for RestoreRequest {
     fn validate(&self, _api: Arc<ECStore>) -> Result<(), std::io::Error> {
         // SELECT type requires select_parameters, and vice versa
-        if self.type_.as_ref().is_none_or(|t| t.as_str() != "SELECT") && self.select_parameters.is_some() {
+        if self.type_.as_ref().is_none_or(|t| t.as_str() != RestoreRequestType::SELECT) && self.select_parameters.is_some() {
             return Err(std::io::Error::other("Select parameters can only be specified with SELECT request type"));
         }
         if let Some(type_) = &self.type_
-            && type_.as_str() == "SELECT"
+            && type_.as_str() == RestoreRequestType::SELECT
             && self.select_parameters.is_none()
         {
             return Err(std::io::Error::other("SELECT restore request requires select parameters to be specified"));
         }
 
         // OutputLocation is only valid for SELECT requests
-        if self.type_.as_ref().is_none_or(|t| t.as_str() != "SELECT") && self.output_location.is_some() {
+        if self.type_.as_ref().is_none_or(|t| t.as_str() != RestoreRequestType::SELECT) && self.output_location.is_some() {
             return Err(std::io::Error::other("OutputLocation can only be specified with SELECT request type"));
         }
         select_restore_s3_location(self)?;
 
         // Days must not be specified with SELECT requests
         if let Some(type_) = &self.type_
-            && type_.as_str() == "SELECT"
+            && type_.as_str() == RestoreRequestType::SELECT
             && self.days.is_some_and(|d| d > 0)
         {
             return Err(std::io::Error::other("Days cannot be specified with SELECT restore request"));
@@ -5548,7 +5531,7 @@ pub fn lifecycle_version_delete_target(oi: &ObjectInfo) -> Option<ObjectToDelete
 }
 
 pub async fn eval_action_from_lifecycle(
-    lc: &PersistedLifecycleConfiguration,
+    lc: &BucketLifecycleConfiguration,
     lock_config: Option<&ObjectLockConfiguration>,
     oi: &ObjectInfo,
 ) -> lifecycle::Event {
@@ -6253,19 +6236,14 @@ mod tests {
     use rustfs_data_usage::TierStats;
     use rustfs_filemeta::metadata_keys;
     use rustfs_filemeta::{FileInfo, FileMeta};
-    use rustfs_gateway_types::dto::MetadataEntry;
-    use rustfs_gateway_types::persistence::{
-        PersistedDefaultRetention as DefaultRetention, PersistedObjectLockConfiguration as ObjectLockConfiguration,
-        PersistedObjectLockRule as ObjectLockRule,
-    };
-    use rustfs_gateway_types::persistence::{
-        PersistedLifecycleConfiguration, PersistedLifecycleExpiration, PersistedLifecycleRule,
-        PersistedNoncurrentVersionExpiration, PersistedTransition,
-    };
-    use rustfs_lifecycle::{RestoreOutputLocation as OutputLocation, RestoreRequest, RestoreS3Location as S3Location};
     #[cfg(feature = "test-util")]
     use rustfs_s3_client::transition_api::ReaderImpl;
     use rustfs_scanner_metrics::metrics::{IlmAction, global_metrics};
+    use s3s::dto::{
+        BucketLifecycleConfiguration, DefaultRetention, ExpirationStatus, LifecycleExpiration, LifecycleRule, MetadataEntry,
+        NoncurrentVersionExpiration, ObjectLockConfiguration, ObjectLockEnabled, ObjectLockRetentionMode, ObjectLockRule,
+        OutputLocation, RestoreRequest, RestoreRequestType, S3Location, Timestamp, Transition, TransitionStorageClass,
+    };
     use serial_test::serial;
     use sha2::{Digest, Sha256};
     use std::collections::HashMap;
@@ -9272,7 +9250,7 @@ mod tests {
             output_location,
             select_parameters: None,
             tier: None,
-            type_: Some("SELECT".to_owned()),
+            type_: Some(RestoreRequestType::from_static(RestoreRequestType::SELECT)),
         }
     }
 
@@ -9589,12 +9567,12 @@ mod tests {
 
     #[test]
     fn lifecycle_rule_has_date_expiration_detects_enabled_date_rule() {
-        let lc = PersistedLifecycleConfiguration {
+        let lc = BucketLifecycleConfiguration {
             expiry_updated_at: None,
-            rules: vec![PersistedLifecycleRule {
-                status: "Enabled".to_owned(),
-                expiration: Some(PersistedLifecycleExpiration {
-                    date: Some(lifecycle::format_lifecycle_date(OffsetDateTime::now_utc()).unwrap()),
+            rules: vec![LifecycleRule {
+                status: ExpirationStatus::from_static(ExpirationStatus::ENABLED),
+                expiration: Some(LifecycleExpiration {
+                    date: Some(Timestamp::from(OffsetDateTime::now_utc())),
                     ..Default::default()
                 }),
                 id: Some("rule-date".to_string()),
@@ -10111,15 +10089,14 @@ mod tests {
     #[test]
     fn should_defer_date_expiry_for_recent_config_update_respects_grace_window() {
         let now = OffsetDateTime::now_utc();
-        let recent = PersistedLifecycleConfiguration {
-            expiry_updated_at: Some(lifecycle::format_lifecycle_date(now - time::Duration::seconds(1)).unwrap()),
+        let recent = BucketLifecycleConfiguration {
+            expiry_updated_at: Some(Timestamp::from(now - time::Duration::seconds(1))),
             rules: Vec::new(),
         };
-        let stale = PersistedLifecycleConfiguration {
-            expiry_updated_at: Some(
-                lifecycle::format_lifecycle_date(now - time::Duration::seconds(DATE_EXPIRY_EXISTING_OBJECTS_GRACE_SECS + 1))
-                    .unwrap(),
-            ),
+        let stale = BucketLifecycleConfiguration {
+            expiry_updated_at: Some(Timestamp::from(
+                now - time::Duration::seconds(DATE_EXPIRY_EXISTING_OBJECTS_GRACE_SECS + 1),
+            )),
             rules: Vec::new(),
         };
 
@@ -10247,12 +10224,12 @@ mod tests {
         assert_eq!(deleted.version_id, None);
     }
 
-    fn expired_delete_marker_lifecycle() -> PersistedLifecycleConfiguration {
-        PersistedLifecycleConfiguration {
+    fn expired_delete_marker_lifecycle() -> BucketLifecycleConfiguration {
+        BucketLifecycleConfiguration {
             expiry_updated_at: None,
-            rules: vec![PersistedLifecycleRule {
-                status: "Enabled".to_owned(),
-                expiration: Some(PersistedLifecycleExpiration {
+            rules: vec![LifecycleRule {
+                status: ExpirationStatus::from_static(ExpirationStatus::ENABLED),
+                expiration: Some(LifecycleExpiration {
                     expired_object_delete_marker: Some(true),
                     ..Default::default()
                 }),
@@ -10268,12 +10245,12 @@ mod tests {
         }
     }
 
-    fn latest_expiration_lifecycle() -> PersistedLifecycleConfiguration {
-        PersistedLifecycleConfiguration {
+    fn latest_expiration_lifecycle() -> BucketLifecycleConfiguration {
+        BucketLifecycleConfiguration {
             expiry_updated_at: None,
-            rules: vec![PersistedLifecycleRule {
-                status: "Enabled".to_owned(),
-                expiration: Some(PersistedLifecycleExpiration {
+            rules: vec![LifecycleRule {
+                status: ExpirationStatus::from_static(ExpirationStatus::ENABLED),
+                expiration: Some(LifecycleExpiration {
                     days: Some(1),
                     ..Default::default()
                 }),
@@ -10289,12 +10266,12 @@ mod tests {
         }
     }
 
-    fn current_and_noncurrent_expiration_lifecycle() -> PersistedLifecycleConfiguration {
-        PersistedLifecycleConfiguration {
+    fn current_and_noncurrent_expiration_lifecycle() -> BucketLifecycleConfiguration {
+        BucketLifecycleConfiguration {
             expiry_updated_at: None,
-            rules: vec![PersistedLifecycleRule {
-                status: "Enabled".to_owned(),
-                expiration: Some(PersistedLifecycleExpiration {
+            rules: vec![LifecycleRule {
+                status: ExpirationStatus::from_static(ExpirationStatus::ENABLED),
+                expiration: Some(LifecycleExpiration {
                     days: Some(30),
                     ..Default::default()
                 }),
@@ -10302,7 +10279,7 @@ mod tests {
                 del_marker_expiration: None,
                 filter: None,
                 id: Some("expire-current-and-noncurrent".to_string()),
-                noncurrent_version_expiration: Some(PersistedNoncurrentVersionExpiration {
+                noncurrent_version_expiration: Some(NoncurrentVersionExpiration {
                     noncurrent_days: Some(1),
                     newer_noncurrent_versions: None,
                 }),
@@ -10313,12 +10290,12 @@ mod tests {
         }
     }
 
-    fn all_versions_expiration_lifecycle() -> PersistedLifecycleConfiguration {
-        PersistedLifecycleConfiguration {
+    fn all_versions_expiration_lifecycle() -> BucketLifecycleConfiguration {
+        BucketLifecycleConfiguration {
             expiry_updated_at: None,
-            rules: vec![PersistedLifecycleRule {
-                status: "Enabled".to_owned(),
-                expiration: Some(PersistedLifecycleExpiration {
+            rules: vec![LifecycleRule {
+                status: ExpirationStatus::from_static(ExpirationStatus::ENABLED),
+                expiration: Some(LifecycleExpiration {
                     days: Some(1),
                     expired_object_all_versions: Some(true),
                     ..Default::default()
@@ -10337,29 +10314,29 @@ mod tests {
 
     fn lock_enabled_without_default_retention() -> ObjectLockConfiguration {
         ObjectLockConfiguration {
-            object_lock_enabled: Some("Enabled".to_owned()),
+            object_lock_enabled: Some(ObjectLockEnabled::from_static(ObjectLockEnabled::ENABLED)),
             rule: None,
         }
     }
 
     fn lock_enabled_with_default_retention() -> ObjectLockConfiguration {
         ObjectLockConfiguration {
-            object_lock_enabled: Some("Enabled".to_owned()),
+            object_lock_enabled: Some(ObjectLockEnabled::from_static(ObjectLockEnabled::ENABLED)),
             rule: Some(ObjectLockRule {
                 default_retention: Some(DefaultRetention {
                     days: Some(30),
-                    mode: Some("COMPLIANCE".to_owned()),
+                    mode: Some(ObjectLockRetentionMode::from_static(ObjectLockRetentionMode::COMPLIANCE)),
                     years: None,
                 }),
             }),
         }
     }
 
-    fn latest_transition_lifecycle() -> PersistedLifecycleConfiguration {
-        PersistedLifecycleConfiguration {
+    fn latest_transition_lifecycle() -> BucketLifecycleConfiguration {
+        BucketLifecycleConfiguration {
             expiry_updated_at: None,
-            rules: vec![PersistedLifecycleRule {
-                status: "Enabled".to_owned(),
+            rules: vec![LifecycleRule {
+                status: ExpirationStatus::from_static(ExpirationStatus::ENABLED),
                 expiration: None,
                 abort_incomplete_multipart_upload: None,
                 del_marker_expiration: None,
@@ -10368,10 +10345,10 @@ mod tests {
                 noncurrent_version_expiration: None,
                 noncurrent_version_transitions: None,
                 prefix: None,
-                transitions: Some(vec![PersistedTransition {
+                transitions: Some(vec![Transition {
                     days: Some(1),
                     date: None,
-                    storage_class: Some("WARM".to_owned()),
+                    storage_class: Some(TransitionStorageClass::from_static("WARM")),
                 }]),
             }],
         }
@@ -12908,9 +12885,8 @@ mod tests {
 
         for (name, object, lock_config, expected) in cases {
             let object_opts = lifecycle::object_opts_from_object_info(&object);
-            let evaluator_lock_config = lock_config.clone().map(Arc::new);
             let batch_event = Evaluator::new(Arc::new(lifecycle.clone()))
-                .with_lock_retention(evaluator_lock_config)
+                .with_lock_retention(lock_config.clone().map(Arc::new))
                 .eval(&[object_opts])
                 .await
                 .expect("batch lifecycle evaluation should succeed")
@@ -14167,7 +14143,10 @@ mod tests {
         let object = current_object_with_metadata(
             ReplicationStatusType::Completed,
             HashMap::from([
-                (metadata_keys::OBJECT_LOCK_MODE.to_string(), "COMPLIANCE".to_string()),
+                (
+                    metadata_keys::OBJECT_LOCK_MODE.to_string(),
+                    s3s::dto::ObjectLockRetentionMode::COMPLIANCE.to_string(),
+                ),
                 (metadata_keys::OBJECT_LOCK_RETAIN_UNTIL_DATE.to_string(), retain_until),
             ]),
         );
@@ -14179,7 +14158,7 @@ mod tests {
 
     #[tokio::test]
     async fn restored_copy_expiry_is_not_blocked_by_retention() {
-        let lifecycle = PersistedLifecycleConfiguration {
+        let lifecycle = BucketLifecycleConfiguration {
             expiry_updated_at: None,
             rules: Vec::new(),
         };
@@ -14189,7 +14168,10 @@ mod tests {
         let mut object = current_object_with_metadata(
             ReplicationStatusType::Completed,
             HashMap::from([
-                (metadata_keys::OBJECT_LOCK_MODE.to_string(), "COMPLIANCE".to_string()),
+                (
+                    metadata_keys::OBJECT_LOCK_MODE.to_string(),
+                    ObjectLockRetentionMode::COMPLIANCE.to_string(),
+                ),
                 (metadata_keys::OBJECT_LOCK_RETAIN_UNTIL_DATE.to_string(), retain_until),
             ]),
         );
@@ -14406,7 +14388,7 @@ mod tests {
                 "removed" => {
                     metadata_sys::delete(&bucket, BUCKET_LIFECYCLE_CONFIG).await.unwrap();
                 }
-                "disabled" => changed.rules[0].status = "Disabled".to_owned(),
+                "disabled" => changed.rules[0].status = ExpirationStatus::from_static(ExpirationStatus::DISABLED),
                 "longer-days" => {
                     changed.rules[0]
                         .noncurrent_version_expiration
@@ -14522,7 +14504,7 @@ mod tests {
         );
     }
 
-    async fn install_test_expiry_policy(ecstore: &Arc<ECStore>, bucket: &str) -> Arc<PersistedLifecycleConfiguration> {
+    async fn install_test_expiry_policy(ecstore: &Arc<ECStore>, bucket: &str) -> Arc<BucketLifecycleConfiguration> {
         let xml = br#"<LifecycleConfiguration><Rule><ID>expire</ID><Status>Enabled</Status><Filter><Prefix></Prefix></Filter><Expiration><Days>0</Days></Expiration><NoncurrentVersionExpiration><NoncurrentDays>0</NoncurrentDays></NoncurrentVersionExpiration></Rule></LifecycleConfiguration>"#;
         metadata_sys::update_in(&ecstore.ctx, bucket, BUCKET_LIFECYCLE_CONFIG, xml.to_vec())
             .await
@@ -14995,11 +14977,7 @@ mod tests {
     #[tokio::test]
     #[serial]
     async fn tier_free_version_recovery_production_entrypoint_enqueues_seeded_item() {
-        let (disk_paths, ecstore) =
-            temp_env::async_with_vars([(super::ENV_TIER_FREE_VERSION_RECOVERY_ENABLED, Some("false"))], async {
-                setup_test_env().await
-            })
-            .await;
+        let (disk_paths, ecstore) = setup_test_env().await;
         let tier = format!("RECOVERY{}", Uuid::new_v4().simple()).to_uppercase();
         crate::services::tier::test_util::register_mock_tier(&ecstore.tier_config_mgr(), &tier).await;
         let runtime_state = install_unconsumed_runtime_expiry_worker(&ecstore, 1).await;

crates/ecstore/src/bucket/lifecycle/core.rs

@@ -17,7 +17,7 @@ use crate::object_api::ObjectInfo;
 pub use rustfs_lifecycle::{
     Event, ExpirationOptions, IlmAction, LIFECYCLE_MALFORMED_XML_ERROR_KIND, Lifecycle, LifecycleCalculate, ObjectOpts,
     RuleValidate, TRANSITION_COMPLETE, TRANSITION_PENDING, TransitionOptions, abort_incomplete_multipart_upload_due,
-    expected_expiry_time, expiration_action_has_valid_target, format_lifecycle_date, parse_lifecycle_date,
+    expected_expiry_time, expiration_action_has_valid_target,
 };
 
 pub fn object_opts_from_object_info(oi: &ObjectInfo) -> ObjectOpts {

crates/ecstore/src/bucket/lifecycle/metadata_boundary.rs

@@ -14,8 +14,7 @@
 
 use std::sync::Arc;
 
-use rustfs_gateway_types::persistence::PersistedLifecycleConfiguration;
-use rustfs_gateway_types::persistence::PersistedObjectLockConfiguration as ObjectLockConfiguration;
+use s3s::dto::{BucketLifecycleConfiguration, ObjectLockConfiguration};
 use time::OffsetDateTime;
 use uuid::Uuid;
 
@@ -25,7 +24,7 @@ use crate::error::{Error, Result};
 
 #[derive(Debug)]
 pub(crate) struct LifecycleExpiryConfigs {
-    pub(crate) lifecycle: Option<Arc<PersistedLifecycleConfiguration>>,
+    pub(crate) lifecycle: Option<Arc<BucketLifecycleConfiguration>>,
     pub(crate) object_lock: Option<Arc<ObjectLockConfiguration>>,
     pub(crate) bucket_incarnation_id: Uuid,
     pub(crate) table_bucket_enabled: bool,
@@ -108,7 +107,7 @@ pub(crate) async fn get_expiry_configs_for_options(
     })
 }
 
-pub(crate) async fn get_lifecycle_config(bucket: &str) -> Result<(PersistedLifecycleConfiguration, OffsetDateTime)> {
+pub(crate) async fn get_lifecycle_config(bucket: &str) -> Result<(BucketLifecycleConfiguration, OffsetDateTime)> {
     metadata_sys::get_lifecycle_config(bucket).await
 }
 
@@ -118,15 +117,15 @@ mod tests {
     use crate::bucket::metadata::BucketMetadata;
     use crate::bucket::metadata_sys::{self, test_support::isolated_store_over_temp_disks};
     use crate::storage_api_contracts::bucket::MakeBucketOptions;
-    use rustfs_gateway_types::persistence::{PersistedLifecycleExpiration, PersistedLifecycleRule};
+    use s3s::dto::{ExpirationStatus, LifecycleExpiration, LifecycleRule};
     use serial_test::serial;
 
-    fn lifecycle_config() -> PersistedLifecycleConfiguration {
-        PersistedLifecycleConfiguration {
+    fn lifecycle_config() -> BucketLifecycleConfiguration {
+        BucketLifecycleConfiguration {
             expiry_updated_at: None,
-            rules: vec![PersistedLifecycleRule {
-                status: "Enabled".to_owned(),
-                expiration: Some(PersistedLifecycleExpiration {
+            rules: vec![LifecycleRule {
+                status: ExpirationStatus::from_static(ExpirationStatus::ENABLED),
+                expiration: Some(LifecycleExpiration {
                     days: Some(1),
                     ..Default::default()
                 }),

crates/ecstore/src/bucket/lifecycle/object_lock_boundary.rs

@@ -22,7 +22,7 @@ pub(crate) fn is_object_locked_by_metadata(user_defined: &HashMap<String, String
 }
 
 pub(crate) fn check_object_lock_for_deletion_with_config(
-    config: Option<&rustfs_gateway_types::persistence::PersistedObjectLockConfiguration>,
+    config: Option<&s3s::dto::ObjectLockConfiguration>,
     obj_info: &ObjectInfo,
     bypass_governance: bool,
 ) -> crate::error::Result<Option<ObjectLockBlockReason>> {

crates/ecstore/src/bucket/lifecycle/replication_sink.rs

@@ -82,13 +82,12 @@ mod tests {
     use crate::bucket::replication::{DeleteReplicationConfigSnapshot, ReplicationObjectBridge};
     use crate::object_api::{ObjectInfo, ObjectOptions};
     use crate::storage_api_contracts::object::ObjectToDelete;
-    use rustfs_gateway_types::dto::Status;
-    use rustfs_gateway_types::persistence::PersistedVersioningConfiguration as VersioningConfiguration;
-    use rustfs_gateway_types::persistence::{
-        PersistedOptionalReplicationStatus, PersistedReplicationConfiguration, PersistedReplicationDestination,
-        PersistedReplicationRule, PersistedReplicationStatus,
-    };
     use rustfs_scanner_metrics::metrics::IlmAction;
+    use s3s::dto::{
+        BucketVersioningStatus, DeleteMarkerReplication, DeleteMarkerReplicationStatus, DeleteReplication,
+        DeleteReplicationStatus, Destination, ReplicationConfiguration, ReplicationRule, ReplicationRuleStatus,
+        VersioningConfiguration,
+    };
     use uuid::Uuid;
 
     use super::*;
@@ -161,27 +160,27 @@ mod tests {
             for purge_enabled in [false, true] {
                 let snapshot = DeleteReplicationConfigSnapshot::from_configs_for_test(
                     VersioningConfiguration {
-                        status: Some("Enabled".to_owned()),
+                        status: Some(BucketVersioningStatus::from_static(BucketVersioningStatus::ENABLED)),
                         ..Default::default()
                     },
-                    Some(PersistedReplicationConfiguration {
+                    Some(ReplicationConfiguration {
                         role: String::new(),
-                        rules: vec![PersistedReplicationRule {
-                            delete_marker_replication: Some(PersistedOptionalReplicationStatus {
+                        rules: vec![ReplicationRule {
+                            delete_marker_replication: Some(DeleteMarkerReplication {
                                 status: Some(if marker_enabled {
-                                    Status::ENABLED.to_string()
+                                    DeleteMarkerReplicationStatus::from_static(DeleteMarkerReplicationStatus::ENABLED)
                                 } else {
-                                    Status::DISABLED.to_string()
+                                    DeleteMarkerReplicationStatus::from_static(DeleteMarkerReplicationStatus::DISABLED)
                                 }),
                             }),
-                            delete_replication: Some(PersistedReplicationStatus {
+                            delete_replication: Some(DeleteReplication {
                                 status: if purge_enabled {
-                                    Status::ENABLED.to_string()
+                                    DeleteReplicationStatus::from_static(DeleteReplicationStatus::ENABLED)
                                 } else {
-                                    Status::DISABLED.to_string()
+                                    DeleteReplicationStatus::from_static(DeleteReplicationStatus::DISABLED)
                                 },
                             }),
-                            destination: PersistedReplicationDestination {
+                            destination: Destination {
                                 bucket: "arn:rustfs:replication:target".to_string(),
                                 ..Default::default()
                             },
@@ -191,7 +190,7 @@ mod tests {
                             prefix: Some(String::new()),
                             priority: Some(1),
                             source_selection_criteria: None,
-                            status: Status::ENABLED.to_string(),
+                            status: ReplicationRuleStatus::from_static(ReplicationRuleStatus::ENABLED),
                         }],
                     }),
                 );

crates/ecstore/src/bucket/lifecycle/runtime_boundary.rs

@@ -14,7 +14,7 @@
 
 use std::sync::Arc;
 
-use rustfs_gateway_types::persistence::PersistedLifecycleConfiguration;
+use s3s::dto::BucketLifecycleConfiguration;
 use tokio::sync::RwLock;
 use tokio_util::sync::CancellationToken;
 
@@ -52,7 +52,7 @@ pub(crate) fn deployment_id() -> Option<String> {
     sources::deployment_id()
 }
 
-pub(crate) async fn bucket_lifecycle_config(bucket: &str) -> Option<PersistedLifecycleConfiguration> {
+pub(crate) async fn bucket_lifecycle_config(bucket: &str) -> Option<BucketLifecycleConfiguration> {
     sources::bucket_lifecycle_config(bucket).await
 }
 

crates/ecstore/src/bucket/metadata.rs

@@ -13,7 +13,7 @@
 // limitations under the License.
 
 use super::msgp_decode::{read_msgp_ext8_time, skip_msgp_value, write_msgp_time};
-use super::object_lock::ObjectLockApi;
+use super::object_lock::{ObjectLockApi, ObjectLockStatusExt};
 use super::versioning::VersioningApi;
 use super::{quota::BucketQuota, target::BucketTargets};
 use crate::bucket::replication::invalid_replication_config_status_field;
@@ -25,17 +25,12 @@ use crate::object_api::WriteCompletion;
 use crate::runtime::sources as runtime_sources;
 use crate::store::ECStore;
 use byteorder::{BigEndian, ByteOrder, LittleEndian};
-use rustfs_gateway_types::cors_tagging::{PersistedCorsConfiguration as CORSConfiguration, PersistedTagging as Tagging};
-use rustfs_gateway_types::persistence::{
-    PersistedAccelerateConfiguration as AccelerateConfiguration,
-    PersistedBucketEncryptionConfiguration as ServerSideEncryptionConfiguration,
-    PersistedBucketLoggingStatus as BucketLoggingStatus, PersistedLifecycleConfiguration,
-    PersistedNotificationConfiguration as NotificationConfiguration, PersistedObjectLockConfiguration as ObjectLockConfiguration,
-    PersistedPublicAccessBlockConfiguration as PublicAccessBlockConfiguration, PersistedReplicationConfiguration,
-    PersistedRequestPaymentConfiguration as RequestPaymentConfiguration,
-    PersistedVersioningConfiguration as VersioningConfiguration, PersistedWebsiteConfiguration as WebsiteConfiguration,
-};
 use rustfs_policy::policy::BucketPolicy;
+use s3s::dto::{
+    AccelerateConfiguration, BucketLifecycleConfiguration, BucketLoggingStatus, BucketVersioningStatus, CORSConfiguration,
+    NotificationConfiguration, ObjectLockConfiguration, PublicAccessBlockConfiguration, ReplicationConfiguration,
+    RequestPaymentConfiguration, ServerSideEncryptionConfiguration, Tagging, VersioningConfiguration, WebsiteConfiguration,
+};
 use serde::Serializer;
 use sha2::{Digest, Sha256};
 use std::collections::HashMap;
@@ -45,11 +40,23 @@ use time::{Date, OffsetDateTime, PrimitiveDateTime, Time as CivilTime, UtcOffset
 use tracing::error;
 use uuid::Uuid;
 
-// The persisted Object Lock configuration answers the storage-level trait
-// here because this module owns the persisted `ObjectLockConfiguration`.
+// The serving-layer DTO impls for the storage-level Object Lock traits live
+// here because this module owns the persisted `ObjectLockConfiguration`
+// during the s3s ratchet migration (rustfs/backlog#1842).
 impl ObjectLockApi for ObjectLockConfiguration {
     fn enabled(&self) -> bool {
-        self.object_lock_enabled.as_deref() == Some("Enabled")
+        self.object_lock_enabled
+            .as_ref()
+            .is_some_and(|v| v.as_str() == s3s::dto::ObjectLockEnabled::ENABLED)
+    }
+}
+
+impl ObjectLockStatusExt for s3s::dto::ObjectLockLegalHoldStatus {
+    fn valid(&self) -> bool {
+        matches!(
+            self.as_str(),
+            s3s::dto::ObjectLockLegalHoldStatus::ON | s3s::dto::ObjectLockLegalHoldStatus::OFF
+        )
     }
 }
 
@@ -432,8 +439,6 @@ pub struct BucketMetadata {
     pub bucket_incarnation_id: Uuid,
     pub(crate) bucket_incarnation_sidecar: bool,
     pub(crate) bucket_creation_committed: bool,
-    // A legacy default is not proof of an interrupted creation.
-    pub(crate) bucket_creation_commit_record_present: bool,
     pub policy_config_json: Vec<u8>,
     pub notification_config_xml: Vec<u8>,
     pub lifecycle_config_xml: Vec<u8>,
@@ -482,13 +487,13 @@ pub struct BucketMetadata {
 
     pub policy_config: Option<BucketPolicy>,
     pub notification_config: Option<NotificationConfiguration>,
-    pub lifecycle_config: Option<PersistedLifecycleConfiguration>,
+    pub lifecycle_config: Option<BucketLifecycleConfiguration>,
     pub object_lock_config: Option<ObjectLockConfiguration>,
     pub versioning_config: Option<VersioningConfiguration>,
     pub sse_config: Option<ServerSideEncryptionConfiguration>,
     pub tagging_config: Option<Tagging>,
     pub quota_config: Option<BucketQuota>,
-    pub replication_config: Option<PersistedReplicationConfiguration>,
+    pub replication_config: Option<ReplicationConfiguration>,
     pub bucket_target_config: Option<BucketTargets>,
     pub bucket_target_config_meta: Option<HashMap<String, String>>,
     pub cors_config: Option<CORSConfiguration>,
@@ -509,7 +514,6 @@ impl Default for BucketMetadata {
             bucket_incarnation_id: Uuid::nil(),
             bucket_incarnation_sidecar: false,
             bucket_creation_committed: false,
-            bucket_creation_commit_record_present: false,
             policy_config_json: Default::default(),
             notification_config_xml: Default::default(),
             lifecycle_config_xml: Default::default(),
@@ -748,10 +752,7 @@ impl BucketMetadata {
                 "TableBucketConfigUpdatedAt" => self.table_bucket_config_updated_at = read_msgp_time_value(rd)?,
                 "DurabilityConfigUpdatedAt" => self.durability_config_updated_at = read_msgp_time_value(rd)?,
                 "OnDemandMigrationConfigUpdatedAt" => self.on_demand_migration_config_updated_at = read_msgp_time_value(rd)?,
-                "BucketCreationCommitted" => {
-                    self.bucket_creation_committed = read_msgp_bool(rd)?;
-                    self.bucket_creation_commit_record_present = true;
-                }
+                "BucketCreationCommitted" => self.bucket_creation_committed = read_msgp_bool(rd)?,
                 other => {
                     tracing::debug!(field = %other, "BucketMetadata decode_from: skipping unknown field");
                     skip_msgp_value(rd)?;
@@ -764,10 +765,8 @@ impl BucketMetadata {
 
     /// Encode to msgp bytes. Field order follows MinIO BucketMetadata for compatibility.
     pub fn encode_to<W: Write>(&self, wr: &mut W) -> Result<()> {
-        // Preserve the absence of a legacy commit record across config rewrites.
-        let commit_record_present = self.bucket_creation_commit_record_present || self.bucket_creation_committed;
-        // Map size: MinIO fields (25) + RustFS extensions (21) + optional commit record.
-        let map_len: u32 = 46 + u32::from(commit_record_present);
+        // Map size: MinIO fields (25) + RustFS extensions (22)
+        let map_len: u32 = 47;
         rmp::encode::write_map_len(wr, map_len)?;
 
         // MinIO field order (same as Go struct)
@@ -848,10 +847,8 @@ impl BucketMetadata {
         write_msgp_time(wr, self.durability_config_updated_at)?;
         rmp::encode::write_str(wr, "OnDemandMigrationConfigUpdatedAt")?;
         write_msgp_time(wr, self.on_demand_migration_config_updated_at)?;
-        if commit_record_present {
-            rmp::encode::write_str(wr, "BucketCreationCommitted")?;
-            rmp::encode::write_bool(wr, self.bucket_creation_committed)?;
-        }
+        rmp::encode::write_str(wr, "BucketCreationCommitted")?;
+        rmp::encode::write_bool(wr, self.bucket_creation_committed)?;
 
         Ok(())
     }
@@ -1021,11 +1018,9 @@ impl BucketMetadata {
                     None
                 } else {
                     let config = deserialize::<VersioningConfiguration>(&data)?;
-                    if config
-                        .status
-                        .as_ref()
-                        .is_some_and(|status| !matches!(status.as_str(), "Enabled" | "Suspended"))
-                    {
+                    if config.status.as_ref().is_some_and(|status| {
+                        !matches!(status.as_str(), BucketVersioningStatus::ENABLED | BucketVersioningStatus::SUSPENDED)
+                    }) {
                         return Err(Error::other("bucket versioning configuration has an invalid status"));
                     }
                     Some(config)
@@ -1038,7 +1033,7 @@ impl BucketMetadata {
                 let config = if data.is_empty() {
                     None
                 } else {
-                    let config = deserialize::<PersistedReplicationConfiguration>(&data)?;
+                    let config = deserialize::<ReplicationConfiguration>(&data)?;
                     if let Some(field) = invalid_replication_config_status_field(&config) {
                         return Err(Error::other(format!("replication field {field} has an invalid status")));
                     }
@@ -1232,8 +1227,8 @@ impl BucketMetadata {
         }
         self.lifecycle_config = None;
         if !self.lifecycle_config_xml.is_empty()
-            && let Err(e) = deserialize::<PersistedLifecycleConfiguration>(&self.lifecycle_config_xml)
-                .map(|c| self.lifecycle_config = Some(c))
+            && let Err(e) =
+                deserialize::<BucketLifecycleConfiguration>(&self.lifecycle_config_xml).map(|c| self.lifecycle_config = Some(c))
         {
             tracing::warn!(
                 event = "bucket_metadata_parse_failed",
@@ -1319,8 +1314,8 @@ impl BucketMetadata {
         }
         self.replication_config = None;
         if !self.replication_config_xml.is_empty()
-            && let Err(e) = deserialize::<PersistedReplicationConfiguration>(&self.replication_config_xml)
-                .map(|c| self.replication_config = Some(c))
+            && let Err(e) =
+                deserialize::<ReplicationConfiguration>(&self.replication_config_xml).map(|c| self.replication_config = Some(c))
         {
             tracing::warn!(
                 event = "bucket_metadata_parse_failed",
@@ -1696,13 +1691,6 @@ mod test {
         let mut bm = BucketMetadata::unmarshal(&body[4..]).expect("unmarshal recovered blob");
         assert_eq!(bm.name, "interop");
         assert!(!bm.bucket_creation_committed, "legacy metadata must default to uncommitted");
-        assert!(!bm.bucket_creation_commit_record_present, "legacy metadata has no creation-commit record");
-        let rewritten = BucketMetadata::unmarshal(&bm.marshal_msg().expect("rewrite legacy metadata"))
-            .expect("read rewritten legacy metadata");
-        assert!(
-            !rewritten.bucket_creation_commit_record_present,
-            "rewriting legacy metadata must not fabricate a creation intent"
-        );
         bm.parse_all_configs().expect("parse recovered configs");
         assert!(bm.lifecycle_config.is_some());
     }
@@ -1720,7 +1708,6 @@ mod test {
 
         assert_eq!(bm.name, new.name);
         assert!(new.bucket_creation_committed);
-        assert!(new.bucket_creation_commit_record_present);
         assert!(!bm.bucket_incarnation_id.is_nil());
         assert_eq!(bm.bucket_incarnation_id, new.bucket_incarnation_id);
     }
@@ -1825,15 +1812,17 @@ mod test {
 
     #[test]
     fn object_locking_requires_lock_metadata_not_plain_versioning() {
+        use s3s::dto::ObjectLockEnabled;
+
         let mut bm = BucketMetadata::new("test-bucket");
         bm.versioning_config = Some(VersioningConfiguration {
-            status: Some("Enabled".to_owned()),
+            status: Some(s3s::dto::BucketVersioningStatus::from_static("Enabled")),
             ..Default::default()
         });
         assert!(!bm.object_locking());
 
         bm.object_lock_config = Some(ObjectLockConfiguration {
-            object_lock_enabled: Some("Enabled".to_owned()),
+            object_lock_enabled: Some(ObjectLockEnabled::from_static(ObjectLockEnabled::ENABLED)),
             ..Default::default()
         });
         assert!(bm.object_locking());

crates/ecstore/src/bucket/metadata_sys.rs

@@ -30,18 +30,14 @@ use crate::storage_api_contracts::heal::HealOperations as _;
 use crate::storage_api_contracts::namespace::NamespaceLocking as _;
 use crate::store::{ECStore, await_bucket_namespace_operation};
 use futures::future::join_all;
-use rustfs_gateway_types::cors_tagging::{PersistedCorsConfiguration as CORSConfiguration, PersistedTagging as Tagging};
-use rustfs_gateway_types::persistence::{
-    PersistedAccelerateConfiguration as AccelerateConfiguration,
-    PersistedBucketEncryptionConfiguration as ServerSideEncryptionConfiguration,
-    PersistedBucketLoggingStatus as BucketLoggingStatus, PersistedLifecycleConfiguration,
-    PersistedNotificationConfiguration as NotificationConfiguration, PersistedObjectLockConfiguration as ObjectLockConfiguration,
-    PersistedPublicAccessBlockConfiguration as PublicAccessBlockConfiguration, PersistedReplicationConfiguration,
-    PersistedRequestPaymentConfiguration as RequestPaymentConfiguration,
-    PersistedVersioningConfiguration as VersioningConfiguration, PersistedWebsiteConfiguration as WebsiteConfiguration,
-};
 use rustfs_heal_contracts::heal_channel::HealOpts;
 use rustfs_policy::policy::BucketPolicy;
+use s3s::dto::ReplicationConfiguration;
+use s3s::dto::{
+    AccelerateConfiguration, BucketLifecycleConfiguration, BucketLoggingStatus, CORSConfiguration, NotificationConfiguration,
+    ObjectLockConfiguration, ObjectLockEnabled, ObjectLockRetentionMode, PublicAccessBlockConfiguration,
+    RequestPaymentConfiguration, ServerSideEncryptionConfiguration, Tagging, VersioningConfiguration, WebsiteConfiguration,
+};
 use std::collections::HashSet;
 use std::time::Duration;
 use std::{
@@ -172,7 +168,7 @@ pub(crate) fn object_lock_config_state_from_authoritative_metadata(bm: &BucketMe
     if bm.lock_enabled {
         return Ok(ObjectLockConfigState::Configured {
             config: ObjectLockConfiguration {
-                object_lock_enabled: Some("Enabled".to_owned()),
+                object_lock_enabled: Some(ObjectLockEnabled::from_static(ObjectLockEnabled::ENABLED)),
                 rule: None,
             },
             updated_at: bm.object_lock_config_updated_at,
@@ -216,10 +212,17 @@ pub(crate) fn configured_object_lock_state_for_tests(
 ) -> ObjectLockConfigState {
     ObjectLockConfigState::Configured {
         config: ObjectLockConfiguration {
-            object_lock_enabled: Some("Enabled".to_owned()),
-            rule: Some(rustfs_gateway_types::persistence::PersistedObjectLockRule {
-                default_retention: Some(rustfs_gateway_types::persistence::PersistedDefaultRetention {
-                    mode: Some(mode.as_str().to_owned()),
+            object_lock_enabled: Some(ObjectLockEnabled::from_static(ObjectLockEnabled::ENABLED)),
+            rule: Some(s3s::dto::ObjectLockRule {
+                default_retention: Some(s3s::dto::DefaultRetention {
+                    mode: Some(s3s::dto::ObjectLockRetentionMode::from_static(match mode {
+                        crate::bucket::object_lock::types::RetentionMode::Governance => {
+                            s3s::dto::ObjectLockRetentionMode::GOVERNANCE
+                        }
+                        crate::bucket::object_lock::types::RetentionMode::Compliance => {
+                            s3s::dto::ObjectLockRetentionMode::COMPLIANCE
+                        }
+                    })),
                     days: Some(days),
                     years: None,
                 }),
@@ -230,7 +233,7 @@ pub(crate) fn configured_object_lock_state_for_tests(
 }
 
 fn validate_authoritative_object_lock_config(config: &ObjectLockConfiguration) -> Result<()> {
-    if config.object_lock_enabled.as_deref() != Some("Enabled") {
+    if config.object_lock_enabled.as_ref().map(ObjectLockEnabled::as_str) != Some(ObjectLockEnabled::ENABLED) {
         return Err(Error::other("persisted bucket Object Lock enabled state is invalid"));
     }
 
@@ -243,7 +246,7 @@ fn validate_authoritative_object_lock_config(config: &ObjectLockConfiguration) -
     if !retention
         .mode
         .as_ref()
-        .is_some_and(|mode| matches!(mode.as_str(), "COMPLIANCE" | "GOVERNANCE"))
+        .is_some_and(|mode| matches!(mode.as_str(), ObjectLockRetentionMode::COMPLIANCE | ObjectLockRetentionMode::GOVERNANCE))
     {
         return Err(Error::other("persisted bucket Object Lock retention mode is invalid"));
     }
@@ -1365,7 +1368,7 @@ pub async fn get_public_access_block_config(bucket: &str) -> Result<(PublicAcces
     bucket_meta_sys.get_public_access_block_config(bucket).await
 }
 
-pub async fn get_lifecycle_config(bucket: &str) -> Result<(PersistedLifecycleConfiguration, OffsetDateTime)> {
+pub async fn get_lifecycle_config(bucket: &str) -> Result<(BucketLifecycleConfiguration, OffsetDateTime)> {
     let bucket_meta_sys_lock = get_bucket_metadata_sys()?;
     let bucket_meta_sys = bucket_meta_sys_lock.read().await;
 
@@ -1584,7 +1587,7 @@ pub(crate) async fn get_cached_quota_config_and_incarnation_in(
     bucket_meta_sys.cached_quota_config_and_incarnation(bucket).await
 }
 
-pub async fn get_replication_config(bucket: &str) -> Result<(PersistedReplicationConfiguration, OffsetDateTime)> {
+pub async fn get_replication_config(bucket: &str) -> Result<(ReplicationConfiguration, OffsetDateTime)> {
     let bucket_meta_sys_lock = get_bucket_metadata_sys()?;
     let bucket_meta_sys = bucket_meta_sys_lock.read().await;
 
@@ -1594,7 +1597,7 @@ pub async fn get_replication_config(bucket: &str) -> Result<(PersistedReplicatio
 pub(crate) async fn get_replication_config_in(
     ctx: &crate::runtime::instance::InstanceContext,
     bucket: &str,
-) -> Result<(PersistedReplicationConfiguration, OffsetDateTime)> {
+) -> Result<(ReplicationConfiguration, OffsetDateTime)> {
     let bucket_meta_sys_lock = bucket_metadata_sys_of(ctx)?;
     let bucket_meta_sys = bucket_meta_sys_lock.read().await;
 
@@ -1720,27 +1723,9 @@ struct MetadataPublishGuard {
     _guard: tokio::sync::OwnedMutexGuard<MetadataPublishLockState>,
 }
 
-#[derive(Debug, Clone)]
-pub(crate) struct PendingBucketCreationProof {
-    pub(crate) generation: Uuid,
-    pub(crate) intent: Vec<u8>,
-    pub(crate) versioning_requested: bool,
-    pub(crate) witnesses: Vec<crate::disk::os::NativeBucketCreationWitness>,
-}
-
-impl PendingBucketCreationProof {
-    pub(crate) fn matches_metadata(&self, metadata: &BucketMetadata) -> Result<bool> {
-        Ok(!metadata.bucket_creation_committed
-            && metadata.bucket_incarnation_sidecar
-            && metadata.bucket_incarnation_id == self.generation
-            && metadata.marshal_msg()? == self.intent)
-    }
-}
-
 #[derive(Debug, Clone)]
 pub struct BucketMetadataSys {
     metadata_map: Arc<RwLock<HashMap<String, Arc<BucketMetadata>>>>,
-    pending_bucket_creations: Arc<RwLock<HashMap<String, PendingBucketCreationProof>>>,
     /// Serializes metadata-map commits and their derived cache updates for one
     /// bucket. Namespace locks, when present, are acquired before this lock.
     metadata_publish_locks: Arc<MetadataPublishLockRegistry>,
@@ -1784,7 +1769,6 @@ impl BucketMetadataSys {
     pub fn new(api: Arc<ECStore>) -> Self {
         Self {
             metadata_map: Arc::new(RwLock::new(HashMap::new())),
-            pending_bucket_creations: Arc::new(RwLock::new(HashMap::new())),
             metadata_publish_locks: Arc::new(MetadataPublishLockRegistry {
                 locks: StdMutex::new(HashMap::new()),
             }),
@@ -1815,81 +1799,6 @@ impl BucketMetadataSys {
             .expect("bucket metadata object store should still be live")
     }
 
-    pub(crate) async fn pending_bucket_creation(&self, bucket: &str) -> Option<PendingBucketCreationProof> {
-        self.pending_bucket_creations.read().await.get(bucket).cloned()
-    }
-
-    pub(crate) async fn grant_pending_bucket_creation(&self, bucket: &str, proof: PendingBucketCreationProof) {
-        self.pending_bucket_creations.write().await.insert(bucket.to_owned(), proof);
-    }
-
-    pub(crate) async fn revoke_pending_bucket_creation(&self, bucket: &str) {
-        self.pending_bucket_creations.write().await.remove(bucket);
-    }
-
-    pub(crate) async fn consume_pending_bucket_creation(
-        &self,
-        bucket: &str,
-        metadata: &BucketMetadata,
-        witnesses: &[crate::disk::os::NativeBucketCreationWitness],
-    ) -> Result<bool> {
-        let mut pending = self.pending_bucket_creations.write().await;
-        let Some(proof) = pending.get(bucket) else {
-            return Ok(false);
-        };
-        if !proof.matches_metadata(metadata)?
-            || proof.witnesses.len() != witnesses.len()
-            || !proof
-                .witnesses
-                .iter()
-                .zip(witnesses)
-                .all(|(old, current)| old.matches(current))
-            || !crate::disk::os::consume_native_creation_witnesses(witnesses)
-        {
-            return Ok(false);
-        }
-        pending.remove(bucket);
-        Ok(true)
-    }
-
-    #[cfg(test)]
-    pub(crate) async fn refresh_bucket_for_test(&self, bucket: &str) -> Result<()> {
-        let expected = self.metadata_map.read().await.get(bucket).cloned();
-        let lock = self.object_store().new_ns_lock(bucket, bucket).await?;
-        let guard = lock.get_read_lock(crate::set_disk::get_lock_acquire_timeout()).await?;
-        self.load_bucket_under_namespace(bucket, MetadataLoadMode::Refresh, expected.as_ref(), &guard)
-            .await
-    }
-
-    #[cfg(test)]
-    pub(crate) async fn publish_persisted_metadata_for_test(&self, bucket: &str) -> Result<()> {
-        let expected = self.metadata_map.read().await.get(bucket).cloned();
-        let lock = self.object_store().new_ns_lock(bucket, bucket).await?;
-        let guard = lock.get_read_lock(crate::set_disk::get_lock_acquire_timeout()).await?;
-        let (metadata, persisted) = await_bucket_namespace_operation(
-            Some(&guard),
-            bucket,
-            "test cached metadata load",
-            load_bucket_metadata_parse_with_presence(self.object_store(), bucket, true),
-        )
-        .await?;
-        self.publish_if_unchanged(bucket, expected.as_ref(), metadata, persisted, &guard)
-            .await
-    }
-
-    #[cfg(test)]
-    pub(crate) async fn migrate_bucket_creation_commit_for_test(&self, bucket: &str) -> Result<()> {
-        self.migrate_bucket_creation_commit(bucket).await.map(|_| ())
-    }
-
-    #[cfg(test)]
-    pub(crate) async fn legacy_migration_reports_missing_bucket_for_test(&self, bucket: &str) -> Result<bool> {
-        Ok(matches!(
-            self.migrate_legacy_metadata(bucket).await?,
-            BucketMetadataAuthority::MissingBucket
-        ))
-    }
-
     fn object_store_if_live(&self) -> Option<Arc<ECStore>> {
         self.api.upgrade()
     }
@@ -2080,7 +1989,6 @@ impl BucketMetadataSys {
         expected: Option<&Arc<BucketMetadata>>,
         namespace_guard: &rustfs_lock::NamespaceLockGuard,
     ) -> Result<()> {
-        self.revoke_pending_bucket_creation(bucket).await;
         if !await_bucket_namespace_operation(
             Some(namespace_guard),
             bucket,
@@ -2164,7 +2072,6 @@ impl BucketMetadataSys {
         persisted: bool,
         namespace_guard: &rustfs_lock::NamespaceLockGuard,
     ) -> Result<()> {
-        self.revoke_pending_bucket_creation(bucket).await;
         if !persisted {
             let _publish_guard = self
                 .lock_metadata_publish(bucket, namespace_guard, "refreshed bucket metadata absence publish")
@@ -2226,7 +2133,6 @@ impl BucketMetadataSys {
     }
 
     pub async fn set(&self, bucket: String, bm: Arc<BucketMetadata>) {
-        self.revoke_pending_bucket_creation(&bucket).await;
         if !is_meta_bucketname(&bucket) {
             let publish_lock = self.metadata_publish_lock(&bucket);
             let _publish_guard = publish_lock.lock().await;
@@ -2248,7 +2154,6 @@ impl BucketMetadataSys {
         if is_meta_bucketname(bucket) {
             return false;
         }
-        self.revoke_pending_bucket_creation(bucket).await;
         let publish_lock = self.metadata_publish_lock(bucket);
         let _publish_guard = publish_lock.lock().await;
         let mut map = self.metadata_map.write().await;
@@ -2266,7 +2171,6 @@ impl BucketMetadataSys {
     }
 
     async fn _reset(&mut self) {
-        self.pending_bucket_creations.write().await.clear();
         let mut map = self.metadata_map.write().await;
         map.clear();
         drop(map);
@@ -2405,7 +2309,6 @@ impl BucketMetadataSys {
     /// server's metadata never leaks into the ambient (first) instance.
     pub(crate) async fn persist_and_set(&self, bm: BucketMetadata) -> Result<()> {
         let mut bm = bm;
-        self.revoke_pending_bucket_creation(&bm.name).await;
         bm.save_with_store(self.object_store()).await?;
 
         self.set(bm.name.clone(), Arc::new(bm)).await;
@@ -2414,7 +2317,6 @@ impl BucketMetadataSys {
     }
 
     async fn persist_new_and_set(&self, mut bm: BucketMetadata) -> Result<()> {
-        self.revoke_pending_bucket_creation(&bm.name).await;
         bm.bucket_creation_committed = true;
         bm.save_with_store_committed(self.object_store()).await?;
         save_bucket_incarnation(self.object_store(), &bm.name, bm.bucket_incarnation_id).await?;
@@ -2424,9 +2326,7 @@ impl BucketMetadataSys {
     }
 
     async fn persist_new_bucket_metadata_intent(&self, mut bm: BucketMetadata) -> Result<()> {
-        self.revoke_pending_bucket_creation(&bm.name).await;
         bm.bucket_creation_committed = false;
-        bm.bucket_creation_commit_record_present = true;
         bm.save_with_store(self.object_store()).await?;
         save_bucket_incarnation(self.object_store(), &bm.name, bm.bucket_incarnation_id).await?;
         bm.bucket_incarnation_sidecar = true;
@@ -2436,7 +2336,6 @@ impl BucketMetadataSys {
     }
 
     async fn commit_bucket_metadata(&self, mut bm: BucketMetadata) -> Result<()> {
-        self.revoke_pending_bucket_creation(&bm.name).await;
         bm.parse_all_configs()?;
         bm.bucket_incarnation_sidecar = true;
         // The caller holds the full bucket creation fence (lifecycle, metadata
@@ -2485,7 +2384,6 @@ impl BucketMetadataSys {
         bucket: &str,
         namespace_guard: &rustfs_lock::NamespaceLockGuard,
     ) -> Result<()> {
-        self.revoke_pending_bucket_creation(bucket).await;
         let expected = self.metadata_map.read().await.get(bucket).cloned();
         if !self
             .bucket_exists(bucket, namespace_guard, "peer bucket metadata existence check")
@@ -2542,7 +2440,6 @@ impl BucketMetadataSys {
             #[cfg(test)]
             self.lazy_disk_loads.fetch_add(1, std::sync::atomic::Ordering::Relaxed);
 
-            self.revoke_pending_bucket_creation(bucket).await;
             let lock = self.object_store().new_ns_lock(bucket, bucket).await?;
             let guard = lock.get_read_lock(crate::set_disk::get_lock_acquire_timeout()).await?;
             #[cfg(test)]
@@ -2868,7 +2765,6 @@ impl BucketMetadataSys {
     /// revalidated at write quorum, so a stale snapshot can neither revert an
     /// acknowledged configuration update nor outlive a delete/recreate.
     async fn migrate_bucket_creation_commit(&self, bucket: &str) -> Result<BucketMetadataAuthority> {
-        self.revoke_pending_bucket_creation(bucket).await;
         let transaction_lock = self
             .object_store()
             .new_ns_lock(RUSTFS_META_BUCKET, &bucket_metadata_transaction_lock_key(bucket))
@@ -2946,7 +2842,6 @@ impl BucketMetadataSys {
         &self,
         bucket: &str,
     ) -> Result<BucketMetadataAuthority> {
-        self.revoke_pending_bucket_creation(bucket).await;
         #[cfg(test)]
         if self.object_lock_disk_read_errors.write().await.remove(bucket) {
             return Err(Error::other(format!("injected Object Lock metadata disk read failure: {bucket}")));
@@ -3122,7 +3017,7 @@ impl BucketMetadataSys {
         }
     }
 
-    pub async fn get_lifecycle_config(&self, bucket: &str) -> Result<(PersistedLifecycleConfiguration, OffsetDateTime)> {
+    pub async fn get_lifecycle_config(&self, bucket: &str) -> Result<(BucketLifecycleConfiguration, OffsetDateTime)> {
         let (bm, _) = self.get_config(bucket).await?;
 
         match ConfigState::of(&bm.lifecycle_config_xml, &bm.lifecycle_config).require(bucket, BUCKET_LIFECYCLE_CONFIG)? {
@@ -3245,7 +3140,7 @@ impl BucketMetadataSys {
         }))
     }
 
-    pub async fn get_replication_config(&self, bucket: &str) -> Result<(PersistedReplicationConfiguration, OffsetDateTime)> {
+    pub async fn get_replication_config(&self, bucket: &str) -> Result<(ReplicationConfiguration, OffsetDateTime)> {
         let (bm, _) = self.get_config(bucket).await?;
 
         if !bm.replication_config_xml.is_empty() && bm.replication_config.is_none() {
@@ -3339,7 +3234,6 @@ mod tests {
     };
     use crate::bucket::target::{BucketTarget, BucketTargetType, Credentials};
     use crate::config::com::read_config;
-    use crate::disk::{DiskAPI as _, RUSTFS_META_TMP_BUCKET};
     use crate::storage_api_contracts::bucket::{BucketOperations as _, DeleteBucketOptions, MakeBucketOptions};
     use byteorder::{ByteOrder as _, LittleEndian};
     use serial_test::serial;
@@ -3394,180 +3288,6 @@ mod tests {
         ),
     ];
 
-    #[tokio::test]
-    #[serial(storage_class_env)]
-    async fn persistence_entry_failures_revoke_pending_creation_before_cache_publish() {
-        let (dirs, store) = isolated_store_over_temp_disks().await;
-        init_bucket_metadata_sys(store.clone(), Vec::new()).await;
-        let sys = bucket_metadata_sys_of(&store.ctx).expect("instance metadata system");
-        let sys = sys.read().await.clone();
-        let options = MakeBucketOptions {
-            lock_enabled: true,
-            ..Default::default()
-        };
-        for entry in ["save", "persist", "persist-new", "commit"] {
-            let bucket = format!("revoke-{entry}-{}", Uuid::new_v4());
-            store.fail_next_bucket_creation_after_intent_for_test(&bucket);
-            store
-                .make_bucket(&bucket, &options)
-                .await
-                .expect_err("stop the real producer before physical publication");
-            sys.pending_bucket_creation(&bucket)
-                .await
-                .expect("the producer must leave a real live creation permission");
-            let (intent, persisted) = get_config_from_disk_with_presence_in(&store.ctx, &bucket)
-                .await
-                .expect("read the original producer intent");
-            assert!(persisted, "{entry}: the complete original intent must be persisted");
-            let original_intent = intent.marshal_msg().expect("encode the original intent");
-
-            // Keep the creation fence order used by the real producer.
-            let lifecycle_guard = store
-                .acquire_bucket_lifecycle_write_lock(&bucket)
-                .await
-                .expect("hold the bucket lifecycle fence");
-            let transaction_guard = acquire_bucket_metadata_transaction_lock_in(&store.ctx, &bucket)
-                .await
-                .expect("hold the metadata transaction fence");
-            let namespace_lock = store
-                .new_ns_lock(&bucket, &bucket)
-                .await
-                .expect("create the exact bucket namespace lock");
-            let namespace_guard = namespace_lock
-                .get_write_lock(crate::set_disk::get_lock_acquire_timeout())
-                .await
-                .expect("hold the exact bucket namespace fence");
-
-            // A file at each native staging root causes actual IO to fail before
-            // metadata can reach the cache-publishing set call. Preserve the
-            // original staging directories so no fixture data is discarded.
-            for dir in &dirs {
-                let staging = dir.path().join(RUSTFS_META_TMP_BUCKET);
-                let saved = dir.path().join(".rustfs.sys/revocation-saved-tmp");
-                std::fs::rename(&staging, &saved).expect("preserve the native staging directory");
-                std::fs::write(&staging, b"block metadata staging with a non-directory")
-                    .expect("install an actual non-directory IO obstruction");
-            }
-            let result = match entry {
-                "save" => sys.save(intent).await,
-                "persist" => sys.persist_and_set(intent).await,
-                "persist-new" => sys.persist_new_and_set(intent).await,
-                "commit" => sys.commit_bucket_metadata(intent).await,
-                _ => unreachable!(),
-            };
-            for dir in &dirs {
-                let staging = dir.path().join(RUSTFS_META_TMP_BUCKET);
-                let saved = dir.path().join(".rustfs.sys/revocation-saved-tmp");
-                std::fs::remove_file(&staging).expect("remove only the fixture's staging obstruction");
-                std::fs::rename(&saved, &staging).expect("restore the complete original staging directory");
-            }
-            drop((namespace_guard, transaction_guard, lifecycle_guard));
-            result.expect_err("native persistence must fail before its downstream cache set");
-            assert!(
-                sys.pending_bucket_creation(&bucket).await.is_none(),
-                "{entry}: the entry revoker must retire live permission even when persistence fails"
-            );
-            let (after_failure, present) = get_config_from_disk_with_presence_in(&store.ctx, &bucket)
-                .await
-                .expect("read the intent after the failed persistence entry");
-            assert!(present, "{entry}: a failed persistence entry must retain the original metadata");
-            assert_eq!(
-                after_failure
-                    .marshal_msg()
-                    .expect("encode the original intent after persistence failure"),
-                original_intent,
-                "{entry}: native IO failure must not change the original generation or complete intent"
-            );
-            let error = store
-                .make_bucket(&bucket, &options)
-                .await
-                .expect_err("a failed ordinary persistence entry cannot preserve creation permission");
-            assert!(error.to_string().contains("administrator recovery"), "{entry}: {error}");
-            let (after_retry, _) = get_config_from_disk_with_presence_in(&store.ctx, &bucket)
-                .await
-                .expect("read the original intent after the rejected retry");
-            assert_eq!(
-                after_retry.marshal_msg().expect("encode the intent after the rejected retry"),
-                original_intent,
-                "{entry}: the rejected retry must preserve the complete original intent"
-            );
-            for disk in store.pools[0].disk_set[0].disks.read().await.iter() {
-                assert!(
-                    matches!(
-                        disk.as_ref().expect("native disk").stat_volume(&bucket).await,
-                        Err(crate::disk::error::DiskError::VolumeNotFound)
-                    ),
-                    "{entry}: failed persistence and the rejected retry must not publish a physical volume"
-                );
-            }
-        }
-    }
-
-    #[tokio::test]
-    #[serial(storage_class_env)]
-    async fn metadata_reset_revokes_real_pending_creation_without_changing_persisted_intent() {
-        let (_dirs, store) = isolated_store_over_temp_disks().await;
-        init_bucket_metadata_sys(store.clone(), Vec::new()).await;
-        let bucket = format!("revoke-reset-{}", Uuid::new_v4());
-        let options = MakeBucketOptions {
-            lock_enabled: true,
-            ..Default::default()
-        };
-        store.fail_next_bucket_creation_after_intent_for_test(&bucket);
-        store
-            .make_bucket(&bucket, &options)
-            .await
-            .expect_err("stop the real producer before physical publication");
-        let sys = bucket_metadata_sys_of(&store.ctx).expect("instance metadata system");
-        let mut sys = sys.read().await.clone();
-        sys.pending_bucket_creation(&bucket)
-            .await
-            .expect("the real producer must leave live creation permission before reset");
-        let (intent, persisted) = get_config_from_disk_with_presence_in(&store.ctx, &bucket)
-            .await
-            .expect("read the original intent before cache reset");
-        assert!(persisted, "cache reset starts from a complete real persisted intent");
-        let original_intent = intent.marshal_msg().expect("encode the original reset intent");
-        sys._reset().await;
-        assert!(
-            sys.pending_bucket_creation(&bucket).await.is_none(),
-            "reset must revoke the producer-issued permission shared with the instance metadata system"
-        );
-        let (after_reset, present) = get_config_from_disk_with_presence_in(&store.ctx, &bucket)
-            .await
-            .expect("read the original intent after cache reset");
-        assert!(present, "cache reset must retain the persisted intent");
-        assert_eq!(
-            after_reset.marshal_msg().expect("encode the intent after cache reset"),
-            original_intent,
-            "cache reset must leave the original generation and complete persisted intent unchanged"
-        );
-        let error = store
-            .make_bucket(&bucket, &options)
-            .await
-            .expect_err("reset cannot regrant the producer's old creation permission");
-        assert!(error.to_string().contains("administrator recovery"), "{error}");
-        let (after_retry, _) = get_config_from_disk_with_presence_in(&store.ctx, &bucket)
-            .await
-            .expect("read the intent after the reset-induced retry rejection");
-        assert_eq!(
-            after_retry
-                .marshal_msg()
-                .expect("encode the intent after the rejected reset retry"),
-            original_intent,
-            "the reset-induced rejection must preserve the original complete intent"
-        );
-        for disk in store.pools[0].disk_set[0].disks.read().await.iter() {
-            assert!(
-                matches!(
-                    disk.as_ref().expect("native disk").stat_volume(&bucket).await,
-                    Err(crate::disk::error::DiskError::VolumeNotFound)
-                ),
-                "cache reset and the rejected retry must not publish a physical volume"
-            );
-        }
-    }
-
     #[tokio::test]
     async fn g_d3_003_new_writer_replication_loads_without_fail_closed_state() {
         let (dirs, store) = isolated_store_over_temp_disks().await;
@@ -3756,7 +3476,7 @@ mod tests {
         let mut versioned = BucketMetadata::new("authoritative-versioned");
         versioned.versioning_config_xml = b"<VersioningConfiguration><Status>Enabled</Status></VersioningConfiguration>".to_vec();
         versioned.versioning_config = Some(VersioningConfiguration {
-            status: Some("Enabled".to_owned()),
+            status: Some(s3s::dto::BucketVersioningStatus::from_static(s3s::dto::BucketVersioningStatus::ENABLED)),
             ..Default::default()
         });
         sys.set("authoritative-versioned".to_string(), Arc::new(versioned)).await;
@@ -3879,24 +3599,25 @@ mod tests {
         else {
             panic!("legacy lock-enabled metadata must remain configured");
         };
-        assert_eq!(config.object_lock_enabled.as_deref(), Some("Enabled"));
+        assert_eq!(
+            config.object_lock_enabled.as_ref().map(|value| value.as_str()),
+            Some(ObjectLockEnabled::ENABLED)
+        );
         assert!(config.rule.is_none());
     }
 
     #[test]
     fn authoritative_object_lock_state_rejects_invalid_retention_periods() {
-        use rustfs_gateway_types::persistence::{
-            PersistedDefaultRetention as DefaultRetention, PersistedObjectLockRule as ObjectLockRule,
-        };
+        use s3s::dto::{DefaultRetention, ObjectLockRule};
 
         for (days, years) in [(Some(0), None), (Some(-1), None), (Some(1), Some(1)), (None, None)] {
             let mut metadata = BucketMetadata::new("invalid-object-lock");
             metadata.object_lock_config = Some(ObjectLockConfiguration {
-                object_lock_enabled: Some("Enabled".to_owned()),
+                object_lock_enabled: Some(ObjectLockEnabled::from_static(ObjectLockEnabled::ENABLED)),
                 rule: Some(ObjectLockRule {
                     default_retention: Some(DefaultRetention {
                         days,
-                        mode: Some("COMPLIANCE".to_owned()),
+                        mode: Some(ObjectLockRetentionMode::from_static(ObjectLockRetentionMode::COMPLIANCE)),
                         years,
                     }),
                 }),
@@ -3911,11 +3632,9 @@ mod tests {
 
     #[test]
     fn authoritative_object_lock_state_rejects_invalid_enabled_mode_and_rule() {
-        use rustfs_gateway_types::persistence::{
-            PersistedDefaultRetention as DefaultRetention, PersistedObjectLockRule as ObjectLockRule,
-        };
+        use s3s::dto::{DefaultRetention, ObjectLockRule};
 
-        for enabled in [None, Some("Disabled".to_string())] {
+        for enabled in [None, Some(ObjectLockEnabled::from("Disabled".to_string()))] {
             let mut metadata = BucketMetadata::new("invalid-object-lock-enabled");
             metadata.lock_enabled = true;
             metadata.object_lock_config = Some(ObjectLockConfiguration {
@@ -3927,11 +3646,11 @@ mod tests {
 
         let mut invalid_mode = BucketMetadata::new("invalid-object-lock-mode");
         invalid_mode.object_lock_config = Some(ObjectLockConfiguration {
-            object_lock_enabled: Some("Enabled".to_owned()),
+            object_lock_enabled: Some(ObjectLockEnabled::from_static(ObjectLockEnabled::ENABLED)),
             rule: Some(ObjectLockRule {
                 default_retention: Some(DefaultRetention {
                     days: Some(1),
-                    mode: Some("invalid".to_string()),
+                    mode: Some(ObjectLockRetentionMode::from("invalid".to_string())),
                     years: None,
                 }),
             }),
@@ -3940,7 +3659,7 @@ mod tests {
 
         let mut missing_retention = BucketMetadata::new("missing-object-lock-retention");
         missing_retention.object_lock_config = Some(ObjectLockConfiguration {
-            object_lock_enabled: Some("Enabled".to_owned()),
+            object_lock_enabled: Some(ObjectLockEnabled::from_static(ObjectLockEnabled::ENABLED)),
             rule: Some(ObjectLockRule { default_retention: None }),
         });
         assert!(object_lock_config_state_from_authoritative_metadata(&missing_retention).is_err());
@@ -3950,7 +3669,7 @@ mod tests {
     fn object_lock_ever_enabled_marker_survives_config_removal() {
         let mut metadata = BucketMetadata::new("lock-config-removed");
         let config = ObjectLockConfiguration {
-            object_lock_enabled: Some("Enabled".to_owned()),
+            object_lock_enabled: Some(ObjectLockEnabled::from_static(ObjectLockEnabled::ENABLED)),
             rule: None,
         };
         metadata
@@ -3968,7 +3687,10 @@ mod tests {
         else {
             panic!("an ever-enabled bucket must remain Object Lock configured");
         };
-        assert_eq!(config.object_lock_enabled.as_deref(), Some("Enabled"));
+        assert_eq!(
+            config.object_lock_enabled.as_ref().map(ObjectLockEnabled::as_str),
+            Some(ObjectLockEnabled::ENABLED)
+        );
     }
 
     #[tokio::test]
@@ -3984,7 +3706,7 @@ mod tests {
 
         let node = Arc::new(RwLock::new(BucketMetadataSys::new(ecstore.clone())));
         let config = ObjectLockConfiguration {
-            object_lock_enabled: Some("Enabled".to_owned()),
+            object_lock_enabled: Some(ObjectLockEnabled::from_static(ObjectLockEnabled::ENABLED)),
             rule: None,
         };
         update_with_sys(
@@ -4965,7 +4687,7 @@ mod tests {
     #[tokio::test]
     async fn update_config_with_persists_tagging_rewrite_across_disk_reload() {
         use crate::bucket::metadata::BUCKET_TAGGING_CONFIG;
-        use rustfs_gateway_types::cors_tagging::PersistedTag as Tag;
+        use s3s::dto::Tag;
 
         let (dirs, ecstore) = isolated_store_over_temp_disks().await;
 
@@ -5129,7 +4851,7 @@ mod tests {
     #[tokio::test]
     async fn concurrent_update_config_with_calls_do_not_lose_writes() {
         use crate::bucket::metadata::BUCKET_TAGGING_CONFIG;
-        use rustfs_gateway_types::cors_tagging::PersistedTag as Tag;
+        use s3s::dto::Tag;
 
         let (dirs, ecstore) = isolated_store_over_temp_disks().await;
         let sys = Arc::new(RwLock::new(BucketMetadataSys::new(ecstore)));
@@ -5229,7 +4951,7 @@ mod tests {
         let mut persisted = BucketMetadata::new("reload-bucket");
         persisted.policy_config_json = b"persisted-marker".to_vec();
         let lock_config = ObjectLockConfiguration {
-            object_lock_enabled: Some("Enabled".to_owned()),
+            object_lock_enabled: Some(ObjectLockEnabled::from_static(ObjectLockEnabled::ENABLED)),
             rule: None,
         };
         persisted
@@ -5508,7 +5230,7 @@ mod tests {
     #[serial]
     async fn bucket_delete_waits_for_config_mutation_fence() {
         use crate::bucket::metadata::BUCKET_TAGGING_CONFIG;
-        use rustfs_gateway_types::cors_tagging::PersistedTag as Tag;
+        use s3s::dto::Tag;
 
         let (_dirs, store) = isolated_store_over_temp_disks().await;
         init_bucket_metadata_sys(store.clone(), Vec::new()).await;

crates/ecstore/src/bucket/object_lock/mod.rs

@@ -17,8 +17,15 @@ pub mod objectlock_sys;
 pub mod types;
 
 /// Whether a bucket Object Lock configuration has locking enabled. The
-/// persisted `ObjectLockConfiguration` implements this in the bucket-metadata
-/// module, which owns the persisted configuration type.
+/// serving-layer `ObjectLockConfiguration` DTO implements this in
+/// the bucket-metadata module, which owns the persisted configuration type
+/// during the s3s ratchet migration (rustfs/backlog#1842).
 pub trait ObjectLockApi {
     fn enabled(&self) -> bool;
 }
+
+/// Whether a legal-hold status value is one of the two valid wire values.
+/// Implemented for the serving-layer DTO in the bucket-metadata module.
+pub trait ObjectLockStatusExt {
+    fn valid(&self) -> bool;
+}

crates/ecstore/src/bucket/replication/replication_config_boundary.rs

@@ -12,7 +12,6 @@
 // See the License for the specific language governing permissions and
 // limitations under the License.
 
-pub use rustfs_replication::config_cache_serde;
 pub use rustfs_replication::{
     ObjectOpts, OperatorRuleContract, REMOTE_TARGET_CAPABILITY_CONTRACT_VERSION, REMOTE_TARGET_READ_ONLY_HISTORICAL_FIELDS,
     REMOTE_TARGET_UNSUPPORTED_FIELDS, REMOTE_TARGET_WRITABLE_FIELDS, REPLICATION_CAPABILITY_CONTRACT_VERSION,

crates/ecstore/src/bucket/replication/replication_lifecycle_bridge.rs

@@ -12,7 +12,7 @@
 // See the License for the specific language governing permissions and
 // limitations under the License.
 
-use rustfs_gateway_types::persistence::PersistedReplicationConfiguration;
+use s3s::dto::ReplicationConfiguration;
 
 use super::replication_config_boundary::ReplicationConfigurationExt as _;
 use super::replication_filemeta_boundary::{
@@ -42,7 +42,7 @@ impl ReplicationLifecycleBridge {
         dead_code,
         reason = "declared boundary surface for the ECStore replication split plan; no caller in this port (backlog#1823)"
     )]
-    pub(crate) fn new_config(config: PersistedReplicationConfiguration) -> ReplicationLifecycleConfig {
+    pub(crate) fn new_config(config: ReplicationConfiguration) -> ReplicationLifecycleConfig {
         ReplicationConfig::new(Some(config), None)
     }
 
@@ -118,17 +118,16 @@ impl ReplicationLifecycleBridge {
 
 #[cfg(test)]
 mod tests {
-    use rustfs_gateway_types::dto::Status;
-    use rustfs_gateway_types::persistence::{PersistedReplicationDestination, PersistedReplicationRule};
+    use s3s::dto::{Destination, ReplicationRule, ReplicationRuleStatus};
 
     use super::super::replication_filemeta_boundary::{ReplicateTargetDecision, VersionPurgeStatusType};
     use super::*;
 
-    fn replication_rule() -> PersistedReplicationRule {
-        PersistedReplicationRule {
+    fn replication_rule() -> ReplicationRule {
+        ReplicationRule {
             delete_marker_replication: None,
             delete_replication: None,
-            destination: PersistedReplicationDestination {
+            destination: Destination {
                 bucket: "arn:aws:s3:::target-bucket".to_string(),
                 ..Default::default()
             },
@@ -138,13 +137,13 @@ mod tests {
             prefix: Some(String::new()),
             priority: Some(1),
             source_selection_criteria: None,
-            status: Status::ENABLED.to_string(),
+            status: ReplicationRuleStatus::from_static(ReplicationRuleStatus::ENABLED),
         }
     }
 
     #[test]
     fn has_pending_version_purge_preserves_replication_active_rule_behavior() {
-        let config = ReplicationLifecycleBridge::new_config(PersistedReplicationConfiguration {
+        let config = ReplicationLifecycleBridge::new_config(ReplicationConfiguration {
             role: String::new(),
             rules: vec![replication_rule()],
         });

crates/ecstore/src/bucket/replication/replication_metadata_boundary.rs

@@ -17,8 +17,8 @@ use std::sync::Arc;
 use crate::bucket::{metadata::BucketMetadata, metadata_sys};
 use crate::disk::{BUCKET_META_PREFIX, RUSTFS_META_BUCKET};
 use crate::runtime::instance::InstanceContext;
-use rustfs_gateway_types::persistence::PersistedReplicationConfiguration;
 use rustfs_utils::path::path_join_buf;
+use s3s::dto::ReplicationConfiguration;
 use time::OffsetDateTime;
 
 use super::replication_error_boundary::{Error, Result};
@@ -36,11 +36,11 @@ impl ReplicationMetadataStore {
     pub(crate) const FORCE_DELETE_REPLICATION_FILE: &'static str = "config/replication/force-delete.bin";
     pub(crate) const FORCE_DELETE_REPLICATION_TRANSACTION_LOCK: &'static str = "config/replication/force-delete.bin.transaction";
 
-    pub(crate) async fn replication_config(bucket: &str) -> Result<(PersistedReplicationConfiguration, OffsetDateTime)> {
+    pub(crate) async fn replication_config(bucket: &str) -> Result<(ReplicationConfiguration, OffsetDateTime)> {
         metadata_sys::get_replication_config(bucket).await
     }
 
-    pub(crate) async fn optional_replication_config(bucket: &str) -> Result<Option<PersistedReplicationConfiguration>> {
+    pub(crate) async fn optional_replication_config(bucket: &str) -> Result<Option<ReplicationConfiguration>> {
         let config = match Self::replication_config(bucket).await {
             Ok((config, _)) => Some(config),
             Err(err) => {

crates/ecstore/src/bucket/replication/replication_object_config.rs

@@ -16,9 +16,7 @@ use std::{collections::HashMap, fmt, sync::Arc};
 
 use super::replication_filemeta_boundary::metadata_keys;
 use crate::bucket::metadata::BucketMetadata;
-use rustfs_gateway_types::dto::Status;
-use rustfs_gateway_types::persistence::PersistedReplicationConfiguration;
-use rustfs_gateway_types::persistence::PersistedVersioningConfiguration as VersioningConfiguration;
+use s3s::dto::{BucketVersioningStatus, ReplicationConfiguration, ReplicationRuleStatus, VersioningConfiguration};
 use serde::{Deserialize, Serialize};
 use time::OffsetDateTime;
 use tracing::error;
@@ -41,7 +39,7 @@ use super::replication_target_boundary::{BucketTargets, ReplicationTargetStore};
 use super::replication_versioning_boundary::ReplicationVersioningStore;
 use super::runtime_boundary as runtime_sources;
 
-pub(crate) async fn get_replication_config(bucket: &str) -> Result<Option<PersistedReplicationConfiguration>> {
+pub(crate) async fn get_replication_config(bucket: &str) -> Result<Option<ReplicationConfiguration>> {
     let config = ReplicationMetadataStore::optional_replication_config(bucket).await?;
     validate_delete_replication_config(&VersioningConfiguration::default(), config.as_ref())?;
     Ok(config)
@@ -66,7 +64,7 @@ impl DeleteReplicationConfigSnapshot {
     #[cfg(test)]
     pub(crate) fn from_configs_for_test(
         versioning: VersioningConfiguration,
-        replication: Option<PersistedReplicationConfiguration>,
+        replication: Option<ReplicationConfiguration>,
     ) -> Self {
         let metadata = replication.map(|config| {
             let mut metadata = BucketMetadata::new("test-bucket");
@@ -80,7 +78,7 @@ impl DeleteReplicationConfigSnapshot {
         &self.versioning
     }
 
-    pub fn replication_config(&self) -> Option<&PersistedReplicationConfiguration> {
+    pub fn replication_config(&self) -> Option<&ReplicationConfiguration> {
         self.metadata
             .as_ref()
             .and_then(|metadata| metadata.replication_config.as_ref())
@@ -103,7 +101,7 @@ impl DeleteReplicationConfigSnapshot {
     pub(crate) fn active_delete_marker_rules_require_tags(&self, object: &str) -> bool {
         self.replication_config().is_some_and(|config| {
             config.rules.iter().any(|rule| {
-                if rule.status == Status::DISABLED.as_str() {
+                if rule.status == ReplicationRuleStatus::from_static(ReplicationRuleStatus::DISABLED) {
                     return false;
                 }
                 if !object.starts_with(rule.prefix()) {
@@ -124,12 +122,12 @@ impl DeleteReplicationConfigSnapshot {
 
 fn validate_delete_replication_config(
     versioning: &VersioningConfiguration,
-    config: Option<&PersistedReplicationConfiguration>,
+    config: Option<&ReplicationConfiguration>,
 ) -> Result<()> {
     if versioning
         .status
         .as_ref()
-        .is_some_and(|status| !matches!(status.as_str(), "Enabled" | "Suspended"))
+        .is_some_and(|status| !matches!(status.as_str(), BucketVersioningStatus::ENABLED | BucketVersioningStatus::SUSPENDED))
     {
         return Err(super::replication_error_boundary::Error::other(
             "bucket versioning configuration has an invalid status",
@@ -146,7 +144,7 @@ fn validate_delete_replication_config(
         let role = config.role.trim();
         let mut role_destination = None;
         for rule in &config.rules {
-            if rule.status.as_str() == Status::ENABLED.as_str() {
+            if rule.status.as_str() == ReplicationRuleStatus::ENABLED {
                 let destination = rule.destination.bucket.trim();
                 if role.is_empty() && destination.is_empty() {
                     return Err(super::replication_error_boundary::Error::other(
@@ -171,7 +169,7 @@ fn validate_delete_replication_config(
     Ok(())
 }
 
-fn replication_config_from_metadata(metadata: &BucketMetadata) -> Result<Option<&PersistedReplicationConfiguration>> {
+fn replication_config_from_metadata(metadata: &BucketMetadata) -> Result<Option<&ReplicationConfiguration>> {
     if !metadata.replication_config_xml.is_empty() && metadata.replication_config.is_none() {
         return Err(super::replication_error_boundary::Error::other(
             "persisted bucket replication configuration is invalid",
@@ -252,15 +250,12 @@ pub(crate) async fn load_delete_replication_config_in(
 
 #[derive(Debug, Clone, Default, Serialize, Deserialize)]
 pub struct ReplicationConfig {
-    /// Carried in the scanner's persisted data usage cache, so its serde form
-    /// is the one the s3s DTO wrote there (see `config_cache_serde` in the replication crate).
-    #[serde(with = "super::replication_config_boundary::config_cache_serde")]
-    pub config: Option<PersistedReplicationConfiguration>,
+    pub config: Option<ReplicationConfiguration>,
     pub remotes: Option<BucketTargets>,
 }
 
 impl ReplicationConfig {
-    pub fn new(config: Option<PersistedReplicationConfiguration>, remotes: Option<BucketTargets>) -> Self {
+    pub fn new(config: Option<ReplicationConfiguration>, remotes: Option<BucketTargets>) -> Self {
         Self { config, remotes }
     }
 
@@ -473,7 +468,7 @@ fn check_replicate_delete_with_config(
     oi: &ObjectInfo,
     del_opts: &ObjectOptions,
     source_error: bool,
-    config: Option<&PersistedReplicationConfiguration>,
+    config: Option<&ReplicationConfiguration>,
     existing_delete_marker: bool,
     trust_persisted_replica_status: bool,
 ) -> ReplicateDecision {
@@ -592,7 +587,7 @@ pub(crate) async fn must_replicate(bucket: &str, object: &str, mopts: MustReplic
 }
 
 fn metadata_target_should_replicate(
-    cfg: &PersistedReplicationConfiguration,
+    cfg: &ReplicationConfiguration,
     opts: &ObjectOpts,
     mopts: &MustReplicateOptions,
     arn: &str,
@@ -602,21 +597,20 @@ fn metadata_target_should_replicate(
 
 #[cfg(test)]
 mod tests {
-    use rustfs_gateway_types::dto::Status;
-    use rustfs_gateway_types::persistence::{
-        PersistedOptionalReplicationStatus, PersistedReplicationDestination, PersistedReplicationFilter,
-        PersistedReplicationRule, PersistedReplicationStatus, PersistedReplicationTag, PersistedSourceSelectionCriteria,
+    use s3s::dto::{
+        DeleteMarkerReplication, DeleteMarkerReplicationStatus, DeleteReplication, DeleteReplicationStatus, Destination,
+        ReplicaModifications, ReplicationRule, ReplicationRuleFilter, ReplicationRuleStatus, SourceSelectionCriteria, Tag,
     };
 
     use super::super::replication_filemeta_boundary::VersionPurgeStatusType;
     use super::super::replication_target_boundary::BucketTarget;
     use super::*;
 
-    fn replication_rule() -> PersistedReplicationRule {
-        PersistedReplicationRule {
+    fn replication_rule() -> ReplicationRule {
+        ReplicationRule {
             delete_marker_replication: None,
             delete_replication: None,
-            destination: PersistedReplicationDestination {
+            destination: Destination {
                 bucket: "arn:aws:s3:::target-bucket".to_string(),
                 ..Default::default()
             },
@@ -626,7 +620,7 @@ mod tests {
             prefix: Some(String::new()),
             priority: Some(1),
             source_selection_criteria: None,
-            status: Status::ENABLED.to_string(),
+            status: ReplicationRuleStatus::from_static(ReplicationRuleStatus::ENABLED),
         }
     }
 
@@ -635,11 +629,11 @@ mod tests {
         let arn = "arn:rustfs:replication:us-east-1:target:bucket";
         let mut rule = replication_rule();
         rule.destination.bucket = arn.to_string();
-        rule.filter = Some(PersistedReplicationFilter {
+        rule.filter = Some(ReplicationRuleFilter {
             prefix: Some("admitted/".to_string()),
             ..Default::default()
         });
-        let cfg = PersistedReplicationConfiguration {
+        let cfg = ReplicationConfiguration {
             role: String::new(),
             rules: vec![rule],
         };
@@ -677,7 +671,7 @@ mod tests {
         assert!(!empty.replicate(&ObjectOpts::default()));
 
         let config = ReplicationConfig::new(
-            Some(PersistedReplicationConfiguration {
+            Some(ReplicationConfiguration {
                 role: String::new(),
                 rules: vec![replication_rule()],
             }),
@@ -731,7 +725,7 @@ mod tests {
     fn delete_snapshot_rejects_enabled_rules_without_a_destination() {
         let mut rule = replication_rule();
         rule.destination.bucket.clear();
-        let config = PersistedReplicationConfiguration {
+        let config = ReplicationConfiguration {
             role: String::new(),
             rules: vec![rule],
         };
@@ -747,7 +741,7 @@ mod tests {
         let first = replication_rule();
         let mut second = replication_rule();
         second.destination.bucket = "arn:aws:s3:::other-target".to_string();
-        let config = PersistedReplicationConfiguration {
+        let config = ReplicationConfiguration {
             role: "arn:aws:s3:::role-target".to_string(),
             rules: vec![first, second],
         };
@@ -758,47 +752,47 @@ mod tests {
     #[test]
     fn delete_snapshot_rejects_unknown_status_values() {
         let invalid_versioning = VersioningConfiguration {
-            status: Some("Enabld".to_string()),
+            status: Some("Enabld".to_string().into()),
             ..Default::default()
         };
         assert!(validate_delete_replication_config(&invalid_versioning, None).is_err());
 
         let mut invalid_rule = replication_rule();
-        invalid_rule.status = "Enabld".to_string();
-        let config = PersistedReplicationConfiguration {
+        invalid_rule.status = "Enabld".to_string().into();
+        let config = ReplicationConfiguration {
             role: String::new(),
             rules: vec![invalid_rule],
         };
         assert!(validate_delete_replication_config(&VersioningConfiguration::default(), Some(&config)).is_err());
 
         let mut invalid_delete = replication_rule();
-        invalid_delete.delete_replication = Some(PersistedReplicationStatus {
-            status: "Enabld".to_string(),
+        invalid_delete.delete_replication = Some(DeleteReplication {
+            status: "Enabld".to_string().into(),
         });
-        let config = PersistedReplicationConfiguration {
+        let config = ReplicationConfiguration {
             role: String::new(),
             rules: vec![invalid_delete],
         };
         assert!(validate_delete_replication_config(&VersioningConfiguration::default(), Some(&config)).is_err());
 
         let mut invalid_delete_marker = replication_rule();
-        invalid_delete_marker.delete_marker_replication = Some(PersistedOptionalReplicationStatus {
-            status: Some("Enabld".to_string()),
+        invalid_delete_marker.delete_marker_replication = Some(DeleteMarkerReplication {
+            status: Some("Enabld".to_string().into()),
         });
-        let config = PersistedReplicationConfiguration {
+        let config = ReplicationConfiguration {
             role: String::new(),
             rules: vec![invalid_delete_marker],
         };
         assert!(validate_delete_replication_config(&VersioningConfiguration::default(), Some(&config)).is_err());
 
         let mut invalid_replica_modifications = replication_rule();
-        invalid_replica_modifications.source_selection_criteria = Some(PersistedSourceSelectionCriteria {
-            replica_modifications: Some(PersistedReplicationStatus {
-                status: "Enabld".to_string(),
+        invalid_replica_modifications.source_selection_criteria = Some(SourceSelectionCriteria {
+            replica_modifications: Some(ReplicaModifications {
+                status: "Enabld".to_string().into(),
             }),
             sse_kms_encrypted_objects: None,
         });
-        let config = PersistedReplicationConfiguration {
+        let config = ReplicationConfiguration {
             role: String::new(),
             rules: vec![invalid_replica_modifications],
         };
@@ -810,11 +804,11 @@ mod tests {
         let mut metadata = BucketMetadata::new("bucket");
         metadata.versioning_config_xml = b"configured".to_vec();
         metadata.versioning_config = Some(VersioningConfiguration {
-            status: Some("Enabled".to_owned()),
+            status: Some(BucketVersioningStatus::from_static(BucketVersioningStatus::ENABLED)),
             ..Default::default()
         });
         metadata.replication_config_xml = b"configured".to_vec();
-        metadata.replication_config = Some(PersistedReplicationConfiguration {
+        metadata.replication_config = Some(ReplicationConfiguration {
             role: String::new(),
             rules: vec![replication_rule()],
         });
@@ -854,7 +848,7 @@ mod tests {
         );
 
         let mut inconsistent = BucketMetadata::new("bucket");
-        inconsistent.replication_config = Some(PersistedReplicationConfiguration {
+        inconsistent.replication_config = Some(ReplicationConfiguration {
             role: String::new(),
             rules: vec![replication_rule()],
         });
@@ -869,11 +863,11 @@ mod tests {
         let arn = "arn:rustfs:replication:us-east-1:target:bucket";
         let mut rule = replication_rule();
         rule.destination.bucket = arn.to_string();
-        rule.delete_marker_replication = Some(PersistedOptionalReplicationStatus {
-            status: Some(Status::ENABLED.to_string()),
+        rule.delete_marker_replication = Some(DeleteMarkerReplication {
+            status: Some(DeleteMarkerReplicationStatus::from_static(DeleteMarkerReplicationStatus::ENABLED)),
         });
         let mut metadata = BucketMetadata::new("bucket");
-        metadata.replication_config = Some(PersistedReplicationConfiguration {
+        metadata.replication_config = Some(ReplicationConfiguration {
             role: String::new(),
             rules: vec![rule],
         });
@@ -904,27 +898,27 @@ mod tests {
     fn delete_marker_source_read_is_required_only_for_tag_filtered_rules() {
         let mut prefix_rule = replication_rule();
         prefix_rule.prefix = Some("logs/".to_string());
-        prefix_rule.delete_marker_replication = Some(PersistedOptionalReplicationStatus {
-            status: Some(Status::ENABLED.to_string()),
+        prefix_rule.delete_marker_replication = Some(DeleteMarkerReplication {
+            status: Some(DeleteMarkerReplicationStatus::from_static(DeleteMarkerReplicationStatus::ENABLED)),
         });
 
         let prefix_snapshot = DeleteReplicationConfigSnapshot::from_configs_for_test(
             VersioningConfiguration {
-                status: Some("Enabled".to_owned()),
+                status: Some(BucketVersioningStatus::from_static(BucketVersioningStatus::ENABLED)),
                 ..Default::default()
             },
-            Some(PersistedReplicationConfiguration {
+            Some(ReplicationConfiguration {
                 role: String::new(),
                 rules: vec![prefix_rule],
             }),
         );
 
         let mut tag_rule = replication_rule();
-        tag_rule.delete_marker_replication = Some(PersistedOptionalReplicationStatus {
-            status: Some(Status::ENABLED.to_string()),
+        tag_rule.delete_marker_replication = Some(DeleteMarkerReplication {
+            status: Some(DeleteMarkerReplicationStatus::from_static(DeleteMarkerReplicationStatus::ENABLED)),
         });
-        tag_rule.filter = Some(PersistedReplicationFilter {
-            tag: Some(PersistedReplicationTag {
+        tag_rule.filter = Some(ReplicationRuleFilter {
+            tag: Some(Tag {
                 key: Some("class".to_string()),
                 value: Some("audit".to_string()),
             }),
@@ -932,10 +926,10 @@ mod tests {
         });
         let tag_snapshot = DeleteReplicationConfigSnapshot::from_configs_for_test(
             VersioningConfiguration {
-                status: Some("Enabled".to_owned()),
+                status: Some(BucketVersioningStatus::from_static(BucketVersioningStatus::ENABLED)),
                 ..Default::default()
             },
-            Some(PersistedReplicationConfiguration {
+            Some(ReplicationConfiguration {
                 role: String::new(),
                 rules: vec![tag_rule],
             }),
@@ -950,14 +944,14 @@ mod tests {
         let arn = "arn:rustfs:replication:us-east-1:target:bucket";
         let mut rule = replication_rule();
         rule.destination.bucket = arn.to_string();
-        rule.delete_replication = Some(PersistedReplicationStatus {
-            status: Status::ENABLED.to_string(),
+        rule.delete_replication = Some(DeleteReplication {
+            status: DeleteReplicationStatus::from_static(DeleteReplicationStatus::ENABLED),
         });
-        rule.delete_marker_replication = Some(PersistedOptionalReplicationStatus {
-            status: Some(Status::DISABLED.to_string()),
+        rule.delete_marker_replication = Some(DeleteMarkerReplication {
+            status: Some(DeleteMarkerReplicationStatus::from_static(DeleteMarkerReplicationStatus::DISABLED)),
         });
         let config = ReplicationConfig::new(
-            Some(PersistedReplicationConfiguration {
+            Some(ReplicationConfiguration {
                 role: String::new(),
                 rules: vec![rule],
             }),
@@ -991,22 +985,22 @@ mod tests {
     #[test]
     fn live_delete_does_not_trust_persisted_replica_status() {
         let mut rule = replication_rule();
-        rule.delete_replication = Some(PersistedReplicationStatus {
-            status: Status::ENABLED.to_string(),
+        rule.delete_replication = Some(DeleteReplication {
+            status: DeleteReplicationStatus::from_static(DeleteReplicationStatus::ENABLED),
         });
-        rule.source_selection_criteria = Some(PersistedSourceSelectionCriteria {
-            replica_modifications: Some(PersistedReplicationStatus {
-                status: Status::DISABLED.to_string(),
+        rule.source_selection_criteria = Some(SourceSelectionCriteria {
+            replica_modifications: Some(ReplicaModifications {
+                status: s3s::dto::ReplicaModificationsStatus::from_static(s3s::dto::ReplicaModificationsStatus::DISABLED),
             }),
             sse_kms_encrypted_objects: None,
         });
-        let replication = PersistedReplicationConfiguration {
+        let replication = ReplicationConfiguration {
             role: String::new(),
             rules: vec![rule],
         };
         let snapshot = DeleteReplicationConfigSnapshot::from_configs_for_test(
             VersioningConfiguration {
-                status: Some("Enabled".to_owned()),
+                status: Some(BucketVersioningStatus::from_static(BucketVersioningStatus::ENABLED)),
                 ..Default::default()
             },
             Some(replication.clone()),
@@ -1042,14 +1036,14 @@ mod tests {
         let arn = "arn:rustfs:replication:us-east-1:target:bucket";
         let mut rule = replication_rule();
         rule.destination.bucket = arn.to_string();
-        rule.delete_replication = Some(PersistedReplicationStatus {
-            status: Status::ENABLED.to_string(),
+        rule.delete_replication = Some(DeleteReplication {
+            status: DeleteReplicationStatus::from_static(DeleteReplicationStatus::ENABLED),
         });
-        rule.delete_marker_replication = Some(PersistedOptionalReplicationStatus {
-            status: Some(Status::DISABLED.to_string()),
+        rule.delete_marker_replication = Some(DeleteMarkerReplication {
+            status: Some(DeleteMarkerReplicationStatus::from_static(DeleteMarkerReplicationStatus::DISABLED)),
         });
         let config = ReplicationConfig::new(
-            Some(PersistedReplicationConfiguration {
+            Some(ReplicationConfiguration {
                 role: String::new(),
                 rules: vec![rule],
             }),

crates/ecstore/src/bucket/replication/replication_pool.rs

@@ -4794,10 +4794,9 @@ mod tests {
     #[tokio::test]
     async fn heal_queue_marks_missing_versioning_state_as_missed() {
         use super::super::replication_target_boundary::BucketTargets;
-        use rustfs_gateway_types::dto::Status;
-        use rustfs_gateway_types::persistence::{
-            PersistedReplicationConfiguration, PersistedReplicationDestination, PersistedReplicationRule,
-            PersistedReplicationStatus,
+        use s3s::dto::{
+            DeleteReplication, DeleteReplicationStatus, Destination, ReplicationConfiguration, ReplicationRule,
+            ReplicationRuleStatus,
         };
 
         let arn = "arn:rustfs:replication:us-east-1:target:bucket";
@@ -4812,14 +4811,14 @@ mod tests {
                 ..Default::default()
             },
             ReplicationConfig::new(
-                Some(PersistedReplicationConfiguration {
+                Some(ReplicationConfiguration {
                     role: String::new(),
-                    rules: vec![PersistedReplicationRule {
+                    rules: vec![ReplicationRule {
                         delete_marker_replication: None,
-                        delete_replication: Some(PersistedReplicationStatus {
-                            status: Status::ENABLED.to_string(),
+                        delete_replication: Some(DeleteReplication {
+                            status: DeleteReplicationStatus::from_static(DeleteReplicationStatus::ENABLED),
                         }),
-                        destination: PersistedReplicationDestination {
+                        destination: Destination {
                             bucket: arn.to_string(),
                             ..Default::default()
                         },
@@ -4829,7 +4828,7 @@ mod tests {
                         prefix: Some(String::new()),
                         priority: Some(1),
                         source_selection_criteria: None,
-                        status: Status::ENABLED.to_string(),
+                        status: ReplicationRuleStatus::from_static(ReplicationRuleStatus::ENABLED),
                     }],
                 }),
                 Some(BucketTargets::default()),

crates/ecstore/src/bucket/replication/replication_resyncer.rs

@@ -85,8 +85,6 @@ use http_body_util::StreamBody;
 use metrics::counter;
 #[cfg(test)]
 use rmp_serde;
-#[cfg(test)]
-use rustfs_gateway_types::persistence::PersistedReplicationConfiguration;
 use rustfs_s3_types::EventName;
 use rustfs_utils::http::{
     AMZ_OBJECT_LOCK_LEGAL_HOLD, AMZ_OBJECT_LOCK_MODE, AMZ_OBJECT_LOCK_RETAIN_UNTIL_DATE, AMZ_TAGGING_DIRECTIVE,
@@ -94,6 +92,8 @@ use rustfs_utils::http::{
     insert_str, replication_target_versions,
 };
 use rustfs_utils::{DEFAULT_SIP_HASH_KEY, get_env_usize, sip_hash};
+#[cfg(test)]
+use s3s::dto::ReplicationConfiguration;
 use std::collections::{HashMap, HashSet};
 use std::fmt::Display;
 use std::pin::Pin;
@@ -6500,10 +6500,9 @@ mod tests {
 
     use super::super::replication_target_boundary::{BucketTarget, BucketTargets};
     use super::*;
-    use rustfs_gateway_types::dto::Status;
-    use rustfs_gateway_types::persistence::PersistedVersioningConfiguration as VersioningConfiguration;
-    use rustfs_gateway_types::persistence::{
-        PersistedReplicationDestination, PersistedReplicationRule, PersistedReplicationStatus,
+    use s3s::dto::{
+        BucketVersioningStatus, DeleteReplication, DeleteReplicationStatus, Destination, ExcludedPrefix, ReplicationRule,
+        ReplicationRuleStatus, VersioningConfiguration,
     };
     use std::collections::HashMap;
     use time::OffsetDateTime;
@@ -8569,7 +8568,7 @@ mod tests {
             ..Default::default()
         };
         let rcfg = ReplicationConfig::new(
-            Some(PersistedReplicationConfiguration {
+            Some(ReplicationConfiguration {
                 role: role.to_string(),
                 rules: vec![],
             }),
@@ -8595,7 +8594,7 @@ mod tests {
         ReplicationVersioningStore::install_prefix_state_test_config(
             &bucket,
             VersioningConfiguration {
-                status: Some("Enabled".to_owned()),
+                status: Some(BucketVersioningStatus::from_static(BucketVersioningStatus::ENABLED)),
                 ..Default::default()
             },
         );
@@ -8628,20 +8627,22 @@ mod tests {
         ReplicationVersioningStore::install_prefix_state_test_config(
             &bucket,
             VersioningConfiguration {
-                status: Some("Enabled".to_owned()),
-                excluded_prefixes: Some(vec![Some("archive/".to_string())]),
+                status: Some(BucketVersioningStatus::from_static(BucketVersioningStatus::ENABLED)),
+                excluded_prefixes: Some(vec![ExcludedPrefix {
+                    prefix: Some("archive/".to_string()),
+                }]),
                 ..Default::default()
             },
         );
         let rcfg = ReplicationConfig::new(
-            Some(PersistedReplicationConfiguration {
+            Some(ReplicationConfiguration {
                 role: String::new(),
-                rules: vec![PersistedReplicationRule {
+                rules: vec![ReplicationRule {
                     delete_marker_replication: None,
-                    delete_replication: Some(PersistedReplicationStatus {
-                        status: Status::ENABLED.to_string(),
+                    delete_replication: Some(DeleteReplication {
+                        status: DeleteReplicationStatus::from_static(DeleteReplicationStatus::ENABLED),
                     }),
-                    destination: PersistedReplicationDestination {
+                    destination: Destination {
                         bucket: arn.to_string(),
                         ..Default::default()
                     },
@@ -8651,7 +8652,7 @@ mod tests {
                     prefix: Some(String::new()),
                     priority: Some(1),
                     source_selection_criteria: None,
-                    status: Status::ENABLED.to_string(),
+                    status: ReplicationRuleStatus::from_static(ReplicationRuleStatus::ENABLED),
                 }],
             }),
             Some(BucketTargets {
@@ -8687,14 +8688,14 @@ mod tests {
         let admitted_arn = "arn:rustfs:replication:us-east-1:target:admitted";
         let current_role = "arn:rustfs:replication:us-east-1:target:current";
         let rcfg = ReplicationConfig::new(
-            Some(PersistedReplicationConfiguration {
+            Some(ReplicationConfiguration {
                 role: current_role.to_string(),
-                rules: vec![PersistedReplicationRule {
+                rules: vec![ReplicationRule {
                     delete_marker_replication: None,
-                    delete_replication: Some(PersistedReplicationStatus {
-                        status: Status::DISABLED.to_string(),
+                    delete_replication: Some(DeleteReplication {
+                        status: DeleteReplicationStatus::from_static(DeleteReplicationStatus::DISABLED),
                     }),
-                    destination: PersistedReplicationDestination {
+                    destination: Destination {
                         bucket: current_role.to_string(),
                         ..Default::default()
                     },
@@ -8704,7 +8705,7 @@ mod tests {
                     prefix: Some(String::new()),
                     priority: Some(1),
                     source_selection_criteria: None,
-                    status: Status::ENABLED.to_string(),
+                    status: ReplicationRuleStatus::from_static(ReplicationRuleStatus::ENABLED),
                 }],
             }),
             Some(BucketTargets {
@@ -8752,12 +8753,12 @@ mod tests {
     #[tokio::test]
     async fn heal_rejects_semantically_invalid_replication_config() {
         let rcfg = ReplicationConfig::new(
-            Some(PersistedReplicationConfiguration {
+            Some(ReplicationConfiguration {
                 role: String::new(),
-                rules: vec![PersistedReplicationRule {
+                rules: vec![ReplicationRule {
                     delete_marker_replication: None,
                     delete_replication: None,
-                    destination: PersistedReplicationDestination {
+                    destination: Destination {
                         bucket: "arn:rustfs:replication:us-east-1:target:bucket".to_string(),
                         ..Default::default()
                     },
@@ -8767,7 +8768,7 @@ mod tests {
                     prefix: Some(String::new()),
                     priority: Some(1),
                     source_selection_criteria: None,
-                    status: "Enabld".to_owned(),
+                    status: ReplicationRuleStatus::from_static("Enabld"),
                 }],
             }),
             Some(BucketTargets::default()),

crates/ecstore/src/bucket/replication/replication_target_config_bridge.rs

@@ -12,7 +12,7 @@
 // See the License for the specific language governing permissions and
 // limitations under the License.
 
-use rustfs_gateway_types::persistence::PersistedReplicationConfiguration;
+use s3s::dto::ReplicationConfiguration;
 
 use super::replication_config_boundary::{ObjectOpts, ReplicationConfigurationExt};
 use super::replication_filemeta_boundary::ReplicationType;
@@ -20,7 +20,7 @@ use super::replication_filemeta_boundary::ReplicationType;
 pub(crate) struct ReplicationTargetConfigBridge;
 
 impl ReplicationTargetConfigBridge {
-    pub(crate) fn target_is_used_by_rules(config: &PersistedReplicationConfiguration, arn: &str) -> bool {
+    pub(crate) fn target_is_used_by_rules(config: &ReplicationConfiguration, arn: &str) -> bool {
         config
             .filter_target_arns(&ObjectOpts {
                 op_type: ReplicationType::All,
@@ -37,7 +37,7 @@ mod tests {
 
     #[test]
     fn target_config_bridge_matches_role_target() {
-        let config = PersistedReplicationConfiguration {
+        let config = ReplicationConfiguration {
             role: " arn:target ".to_string(),
             rules: Vec::new(),
         };

crates/ecstore/src/bucket/replication/replication_versioning_boundary.rs

@@ -15,7 +15,7 @@
 use super::replication_error_boundary::Result;
 use crate::bucket::{versioning::VersioningApi as _, versioning_sys::BucketVersioningSys};
 #[cfg(test)]
-use rustfs_gateway_types::persistence::PersistedVersioningConfiguration as VersioningConfiguration;
+use s3s::dto::VersioningConfiguration;
 #[cfg(test)]
 use std::{collections::HashMap, sync::LazyLock, sync::Mutex};
 

crates/ecstore/src/bucket/tagging/mod.rs

@@ -12,7 +12,7 @@
 // See the License for the specific language governing permissions and
 // limitations under the License.
 
-use rustfs_gateway_types::cors_tagging::PersistedTag as Tag;
+use s3s::dto::Tag;
 use std::cmp::Ordering;
 use std::collections::HashMap;
 use url::form_urlencoded;

crates/ecstore/src/bucket/utils.rs

@@ -15,23 +15,9 @@
 use crate::disk::{MIGRATING_META_BUCKET, RUSTFS_META_BUCKET};
 use crate::error::{Error, Result, StorageError};
 use regex::Regex;
-use rustfs_gateway_types::cors_tagging::{
-    CorsTaggingCodecError, PersistedCorsConfiguration, PersistedTagging, parse_cors, parse_tagging, serialize_cors,
-    serialize_tagging,
-};
-use rustfs_gateway_types::persistence::{
-    PersistedAccelerateConfiguration, PersistedBucketEncryptionConfiguration, PersistedBucketLoggingStatus,
-    PersistedLifecycleConfiguration, PersistedNotificationConfiguration, PersistedObjectLockConfiguration,
-    PersistedPublicAccessBlockConfiguration, PersistedReplicationConfiguration, PersistedRequestPaymentConfiguration,
-    PersistedVersioningConfiguration, PersistedWebsiteConfiguration, PersistenceCodecError, parse_accelerate,
-    parse_bucket_encryption, parse_bucket_logging, parse_lifecycle_legacy_dates, parse_notification, parse_object_lock,
-    parse_public_access_block, parse_replication, parse_request_payment, parse_versioning, parse_website, serialize_accelerate,
-    serialize_bucket_encryption, serialize_bucket_logging, serialize_lifecycle_legacy_dates, serialize_notification,
-    serialize_object_lock, serialize_public_access_block, serialize_replication, serialize_request_payment, serialize_versioning,
-    serialize_website,
-};
 use rustfs_utils::WindowsFileNameError;
 use rustfs_utils::path::SLASH_SEPARATOR;
+use s3s::xml;
 use tracing::instrument;
 
 pub fn is_meta_bucketname(name: &str) -> bool {
@@ -88,173 +74,37 @@ pub fn check_valid_bucket_name_strict(bucket_name: &str) -> Result<()> {
     check_bucket_name_common(bucket_name, true)
 }
 
-/// A bucket configuration persisted as XML in the bucket metadata, together
-/// with the codec that reads and writes its stored bytes.
-///
-/// Every family reads and writes its stored bytes through the gateway
-/// persistence codecs (`rustfs_gateway_types::persistence` and
-/// `rustfs_gateway_types::cors_tagging`, rustfs/backlog#2745). The codecs keep
-/// the persisted bytes the `bucket_config_goldens` pin, so a document the old
-/// s3s codec wrote reads back and re-encodes unchanged.
-pub trait BucketConfigXml: Sized {
-    /// Decodes the stored bytes of one configuration document.
-    fn decode_config_xml(input: &[u8]) -> std::result::Result<Self, BucketConfigXmlError>;
-
-    /// Encodes the value as RustFS stores it.
-    fn encode_config_xml(&self) -> std::result::Result<Vec<u8>, BucketConfigXmlError>;
+// RUSTFS_COMPAT_TODO(s3gate-metadata-xml): the s3s codec reads persisted XML during migration. Remove after every supported writer uses the gateway codec and every retained metadata object and backup archive is verified or rewritten.
+pub fn deserialize<T>(input: &[u8]) -> xml::DeResult<T>
+where
+    T: for<'xml> xml::Deserialize<'xml>,
+{
+    let mut d = xml::Deserializer::new(input);
+    let ans = T::deserialize(&mut d)?;
+    d.expect_eof()?;
+    Ok(ans)
 }
 
-/// Why a persisted bucket configuration could not be decoded or encoded.
-#[derive(Debug, thiserror::Error)]
-pub enum BucketConfigXmlError {
-    /// A gateway persistence codec refused the bytes or the value.
-    #[error(transparent)]
-    Persisted(#[from] PersistenceCodecError),
-    /// The gateway CORS or Tagging codec refused the bytes or the value.
-    #[error(transparent)]
-    CorsTagging(#[from] CorsTaggingCodecError),
+#[allow(
+    dead_code,
+    reason = "xml serialize helper with no caller in this port; the live sibling is deserialize (backlog#1823)"
+)]
+pub fn serialize_content<T: xml::SerializeContent>(val: &T) -> xml::SerResult<String> {
+    let mut buf = Vec::with_capacity(256);
+    {
+        let mut ser = xml::Serializer::new(&mut buf);
+        val.serialize_content(&mut ser)?;
+    }
+    Ok(String::from_utf8(buf).unwrap())
 }
 
-/// Decodes one persisted bucket configuration document.
-pub fn deserialize<T: BucketConfigXml>(input: &[u8]) -> std::result::Result<T, BucketConfigXmlError> {
-    T::decode_config_xml(input)
-}
-
-/// Encodes one bucket configuration as RustFS persists it.
-pub fn serialize<T: BucketConfigXml>(val: &T) -> std::result::Result<Vec<u8>, BucketConfigXmlError> {
-    val.encode_config_xml()
-}
-
-impl BucketConfigXml for PersistedReplicationConfiguration {
-    fn decode_config_xml(input: &[u8]) -> std::result::Result<Self, BucketConfigXmlError> {
-        Ok(parse_replication(input)?)
-    }
-
-    fn encode_config_xml(&self) -> std::result::Result<Vec<u8>, BucketConfigXmlError> {
-        Ok(serialize_replication(self)?)
-    }
-}
-
-impl BucketConfigXml for PersistedLifecycleConfiguration {
-    fn decode_config_xml(input: &[u8]) -> std::result::Result<Self, BucketConfigXmlError> {
-        Ok(parse_lifecycle_legacy_dates(input)?)
-    }
-
-    fn encode_config_xml(&self) -> std::result::Result<Vec<u8>, BucketConfigXmlError> {
-        Ok(serialize_lifecycle_legacy_dates(self)?)
-    }
-}
-
-// The gateway writers of these families are infallible: a value that exists is
-// always encodable, so only the readers can refuse bytes.
-impl BucketConfigXml for PersistedAccelerateConfiguration {
-    fn decode_config_xml(input: &[u8]) -> std::result::Result<Self, BucketConfigXmlError> {
-        Ok(parse_accelerate(input)?)
-    }
-
-    fn encode_config_xml(&self) -> std::result::Result<Vec<u8>, BucketConfigXmlError> {
-        Ok(serialize_accelerate(self))
-    }
-}
-
-impl BucketConfigXml for PersistedBucketLoggingStatus {
-    fn decode_config_xml(input: &[u8]) -> std::result::Result<Self, BucketConfigXmlError> {
-        Ok(parse_bucket_logging(input)?)
-    }
-
-    fn encode_config_xml(&self) -> std::result::Result<Vec<u8>, BucketConfigXmlError> {
-        Ok(serialize_bucket_logging(self))
-    }
-}
-
-impl BucketConfigXml for PersistedCorsConfiguration {
-    fn decode_config_xml(input: &[u8]) -> std::result::Result<Self, BucketConfigXmlError> {
-        Ok(parse_cors(input)?)
-    }
-
-    fn encode_config_xml(&self) -> std::result::Result<Vec<u8>, BucketConfigXmlError> {
-        Ok(serialize_cors(self))
-    }
-}
-
-impl BucketConfigXml for PersistedNotificationConfiguration {
-    fn decode_config_xml(input: &[u8]) -> std::result::Result<Self, BucketConfigXmlError> {
-        Ok(parse_notification(input)?)
-    }
-
-    fn encode_config_xml(&self) -> std::result::Result<Vec<u8>, BucketConfigXmlError> {
-        Ok(serialize_notification(self))
-    }
-}
-
-impl BucketConfigXml for PersistedObjectLockConfiguration {
-    fn decode_config_xml(input: &[u8]) -> std::result::Result<Self, BucketConfigXmlError> {
-        Ok(parse_object_lock(input)?)
-    }
-
-    fn encode_config_xml(&self) -> std::result::Result<Vec<u8>, BucketConfigXmlError> {
-        Ok(serialize_object_lock(self))
-    }
-}
-
-impl BucketConfigXml for PersistedPublicAccessBlockConfiguration {
-    fn decode_config_xml(input: &[u8]) -> std::result::Result<Self, BucketConfigXmlError> {
-        Ok(parse_public_access_block(input)?)
-    }
-
-    fn encode_config_xml(&self) -> std::result::Result<Vec<u8>, BucketConfigXmlError> {
-        Ok(serialize_public_access_block(self))
-    }
-}
-
-impl BucketConfigXml for PersistedRequestPaymentConfiguration {
-    fn decode_config_xml(input: &[u8]) -> std::result::Result<Self, BucketConfigXmlError> {
-        Ok(parse_request_payment(input)?)
-    }
-
-    fn encode_config_xml(&self) -> std::result::Result<Vec<u8>, BucketConfigXmlError> {
-        Ok(serialize_request_payment(self))
-    }
-}
-
-impl BucketConfigXml for PersistedBucketEncryptionConfiguration {
-    fn decode_config_xml(input: &[u8]) -> std::result::Result<Self, BucketConfigXmlError> {
-        Ok(parse_bucket_encryption(input)?)
-    }
-
-    fn encode_config_xml(&self) -> std::result::Result<Vec<u8>, BucketConfigXmlError> {
-        Ok(serialize_bucket_encryption(self))
-    }
-}
-
-impl BucketConfigXml for PersistedTagging {
-    fn decode_config_xml(input: &[u8]) -> std::result::Result<Self, BucketConfigXmlError> {
-        Ok(parse_tagging(input)?)
-    }
-
-    fn encode_config_xml(&self) -> std::result::Result<Vec<u8>, BucketConfigXmlError> {
-        Ok(serialize_tagging(self))
-    }
-}
-
-impl BucketConfigXml for PersistedVersioningConfiguration {
-    fn decode_config_xml(input: &[u8]) -> std::result::Result<Self, BucketConfigXmlError> {
-        Ok(parse_versioning(input)?)
-    }
-
-    fn encode_config_xml(&self) -> std::result::Result<Vec<u8>, BucketConfigXmlError> {
-        Ok(serialize_versioning(self))
-    }
-}
-
-impl BucketConfigXml for PersistedWebsiteConfiguration {
-    fn decode_config_xml(input: &[u8]) -> std::result::Result<Self, BucketConfigXmlError> {
-        Ok(parse_website(input)?)
-    }
-
-    fn encode_config_xml(&self) -> std::result::Result<Vec<u8>, BucketConfigXmlError> {
-        Ok(serialize_website(self))
+pub fn serialize<T: xml::Serialize>(val: &T) -> xml::SerResult<Vec<u8>> {
+    let mut buf = Vec::with_capacity(256);
+    {
+        let mut ser = xml::Serializer::new(&mut buf);
+        val.serialize(&mut ser)?;
     }
+    Ok(buf)
 }
 
 pub fn has_bad_path_component(path: &str) -> bool {
@@ -584,40 +434,6 @@ mod tests {
     use super::*;
     use proptest::prelude::*;
 
-    /// Every bucket-configuration family reads and writes through its own
-    /// gateway codec (rustfs/backlog#2745): the family's golden document
-    /// decodes and re-encodes to the same bytes, and a document of another
-    /// family is refused instead of decoding into an empty configuration.
-    #[test]
-    fn every_bucket_config_family_uses_its_own_gateway_codec() {
-        fn assert_family<T: BucketConfigXml>(family: &str, own: &[u8], foreign: &[u8]) {
-            let value = deserialize::<T>(own).unwrap_or_else(|err| panic!("{family}: own document must decode: {err}"));
-            let encoded = serialize(&value).unwrap_or_else(|err| panic!("{family}: value must encode: {err}"));
-            assert_eq!(encoded, own, "{family}: re-encoding must keep the persisted bytes");
-            assert!(deserialize::<T>(foreign).is_err(), "{family}: a foreign root element must be refused");
-        }
-
-        macro_rules! golden {
-            ($family:literal) => {
-                include_bytes!(concat!("../../tests/fixtures/bucket-config-goldens/", $family, "/minio.out.xml"))
-            };
-        }
-        let tagging = golden!("tagging");
-        let versioning = golden!("versioning");
-
-        assert_family::<PersistedAccelerateConfiguration>("accelerate", golden!("accelerate"), tagging);
-        assert_family::<PersistedBucketLoggingStatus>("logging", golden!("logging"), tagging);
-        assert_family::<PersistedCorsConfiguration>("cors", golden!("cors"), tagging);
-        assert_family::<PersistedNotificationConfiguration>("notification", golden!("notification"), tagging);
-        assert_family::<PersistedObjectLockConfiguration>("object_lock", golden!("object_lock"), tagging);
-        assert_family::<PersistedPublicAccessBlockConfiguration>("public_access_block", golden!("public_access_block"), tagging);
-        assert_family::<PersistedRequestPaymentConfiguration>("request_payment", golden!("request_payment"), tagging);
-        assert_family::<PersistedBucketEncryptionConfiguration>("encryption", golden!("encryption"), tagging);
-        assert_family::<PersistedTagging>("tagging", tagging, versioning);
-        assert_family::<PersistedVersioningConfiguration>("versioning", versioning, tagging);
-        assert_family::<PersistedWebsiteConfiguration>("website", golden!("website"), tagging);
-    }
-
     /// rustfs#5785: keys whose path segments exceed the on-disk NAME_MAX
     /// budget must be rejected up front as ObjectNameInvalid (4xx), not leak
     /// ENAMETOOLONG as InternalError 500 from the disk layer.

crates/ecstore/src/bucket/versioning/mod.rs

@@ -12,8 +12,8 @@
 // See the License for the specific language governing permissions and
 // limitations under the License.
 
-use rustfs_gateway_types::persistence::PersistedVersioningConfiguration as VersioningConfiguration;
 use rustfs_utils::string::match_simple;
+use s3s::dto::{BucketVersioningStatus, VersioningConfiguration};
 
 pub trait VersioningApi {
     fn enabled(&self) -> bool;
@@ -28,10 +28,10 @@ pub trait VersioningApi {
 
 impl VersioningApi for VersioningConfiguration {
     fn enabled(&self) -> bool {
-        self.status.as_deref() == Some("Enabled")
+        self.status == Some(BucketVersioningStatus::from_static(BucketVersioningStatus::ENABLED))
     }
     fn prefix_enabled(&self, prefix: &str) -> bool {
-        if self.status.as_deref() != Some("Enabled") {
+        if self.status != Some(BucketVersioningStatus::from_static(BucketVersioningStatus::ENABLED)) {
             return false;
         }
 
@@ -44,10 +44,12 @@ impl VersioningApi for VersioningConfiguration {
         }
 
         if let Some(ref excluded_prefixes) = self.excluded_prefixes {
-            for sprefix in excluded_prefixes.iter().flatten() {
-                let pattern = format!("{sprefix}*");
-                if match_simple(&pattern, prefix) {
-                    return false;
+            for p in excluded_prefixes.iter() {
+                if let Some(ref sprefix) = p.prefix {
+                    let pattern = format!("{sprefix}*");
+                    if match_simple(&pattern, prefix) {
+                        return false;
+                    }
                 }
             }
         }
@@ -56,11 +58,11 @@ impl VersioningApi for VersioningConfiguration {
     }
 
     fn prefix_suspended(&self, prefix: &str) -> bool {
-        if self.status.as_deref() == Some("Suspended") {
+        if self.status == Some(BucketVersioningStatus::from_static(BucketVersioningStatus::SUSPENDED)) {
             return true;
         }
 
-        if self.status.as_deref() == Some("Enabled") {
+        if self.status == Some(BucketVersioningStatus::from_static(BucketVersioningStatus::ENABLED)) {
             if prefix.is_empty() {
                 return false;
             }
@@ -73,10 +75,12 @@ impl VersioningApi for VersioningConfiguration {
             }
 
             if let Some(ref excluded_prefixes) = self.excluded_prefixes {
-                for sprefix in excluded_prefixes.iter().flatten() {
-                    let pattern = format!("{sprefix}*");
-                    if match_simple(&pattern, prefix) {
-                        return true;
+                for p in excluded_prefixes.iter() {
+                    if let Some(ref sprefix) = p.prefix {
+                        let pattern = format!("{sprefix}*");
+                        if match_simple(&pattern, prefix) {
+                            return true;
+                        }
                     }
                 }
             }
@@ -88,12 +92,14 @@ impl VersioningApi for VersioningConfiguration {
         self.prefix_enabled(prefix) || self.prefix_suspended(prefix)
     }
     fn suspended(&self) -> bool {
-        self.status.as_deref() == Some("Suspended")
+        self.status == Some(BucketVersioningStatus::from_static(BucketVersioningStatus::SUSPENDED))
     }
 }
 
 #[cfg(test)]
 mod tests {
+    use s3s::dto::{BucketVersioningStatus, ExcludedPrefix};
+
     use super::*;
 
     struct LegacyVersioning;
@@ -128,8 +134,10 @@ mod tests {
     #[test]
     fn delete_state_treats_excluded_prefixes_as_unversioned() {
         let config = VersioningConfiguration {
-            status: Some("Enabled".to_owned()),
-            excluded_prefixes: Some(vec![Some("archive/".to_string())]),
+            status: Some(BucketVersioningStatus::from_static(BucketVersioningStatus::ENABLED)),
+            excluded_prefixes: Some(vec![ExcludedPrefix {
+                prefix: Some("archive/".to_string()),
+            }]),
             ..Default::default()
         };
 
@@ -138,7 +146,7 @@ mod tests {
         assert_eq!(config.delete_state("live/object"), (true, false));
 
         let suspended = VersioningConfiguration {
-            status: Some("Suspended".to_owned()),
+            status: Some(BucketVersioningStatus::from_static(BucketVersioningStatus::SUSPENDED)),
             ..Default::default()
         };
         assert_eq!(suspended.delete_state("archive/object"), (false, true));

crates/ecstore/src/bucket/versioning_sys.rs

@@ -17,7 +17,7 @@ use super::metadata::unreadable_config_refusal;
 use super::{metadata_sys::get_bucket_metadata_sys, versioning::VersioningApi};
 use crate::disk::RUSTFS_META_BUCKET;
 use crate::error::Result;
-use rustfs_gateway_types::persistence::PersistedVersioningConfiguration as VersioningConfiguration;
+use s3s::dto::VersioningConfiguration;
 use tracing::{error, warn};
 
 pub struct BucketVersioningSys {}
@@ -163,7 +163,7 @@ mod tests {
 
     fn enabled() -> VersioningConfiguration {
         VersioningConfiguration {
-            status: Some("Enabled".to_owned()),
+            status: Some(s3s::dto::BucketVersioningStatus::from_static(s3s::dto::BucketVersioningStatus::ENABLED)),
             ..Default::default()
         }
     }

crates/ecstore/src/cluster/rpc/peer_rest_client.rs

@@ -1965,15 +1965,6 @@ impl PeerRestClient {
         Ok((self.topology_member.clone(), epoch))
     }
 
-    pub async fn probe_restore_copyback_capability(&self, topology_fingerprint: String) -> Result<(String, Uuid)> {
-        let probe = rustfs_protos::restore_copyback_capability_probe(Uuid::new_v4().as_bytes());
-        let result = self
-            .heal_control(rustfs_protos::HEAL_CONTROL_PROTOCOL_VERSION, topology_fingerprint, probe)
-            .await?;
-        let epoch = decode_remote_version_state_capability(&self.topology_member, &result)?;
-        Ok((self.topology_member.clone(), epoch))
-    }
-
     pub async fn probe_cross_pool_fence(&self, topology_fingerprint: String) -> Result<(String, u32, Uuid)> {
         let mut probe = rustfs_protos::CROSS_POOL_FENCE_CAPABILITY_PROBE_PREFIX.to_vec();
         probe.extend_from_slice(Uuid::new_v4().as_bytes());

crates/ecstore/src/cluster/rpc/remote_locker.rs

@@ -41,6 +41,10 @@ use tonic::service::interceptor::InterceptedService;
 use tonic::{Request, Response};
 use tracing::{debug, info, warn};
 
+const EVENT_NAMESPACE_LOCK_RELEASE: &str = "namespace_lock_release";
+const LOG_COMPONENT_ECSTORE: &str = "ecstore";
+const LOG_SUBSYSTEM_REMOTE_LOCK: &str = "remote_lock";
+
 fn attach_lock_mutation_body_digest<T: rustfs_protos::CanonicalMutationBody>(request: &mut Request<T>) -> std::io::Result<()> {
     set_tonic_rolling_mutation_body_digest(request)
 }
@@ -233,6 +237,20 @@ pub struct RemoteClient {
 impl RemoteClient {
     const ONLINE_CHECK_RESOURCE: &'static str = "health-lock-online";
 
+    fn record_release_ack(&self, lock_id: &LockId) {
+        tracing::trace!(
+            target: "rustfs_ecstore::object_lock_diag",
+            event = EVENT_NAMESPACE_LOCK_RELEASE,
+            component = LOG_COMPONENT_ECSTORE,
+            subsystem = LOG_SUBSYSTEM_REMOTE_LOCK,
+            state = "acknowledged",
+            peer_kind = "remote",
+            lock_id = %lock_id,
+            peer = %self.addr,
+            "Remote namespace lock release acknowledged"
+        );
+    }
+
     pub fn new(endpoint: String) -> Self {
         Self { addr: endpoint }
     }
@@ -1204,8 +1222,6 @@ impl LockClient for RemoteClient {
     }
 
     async fn release(&self, lock_id: &LockId) -> Result<bool> {
-        info!("remote release for {}", lock_id);
-
         let unlock_request = Self::create_unlock_request(lock_id);
         let request_string = serde_json::to_string(&unlock_request)
             .map_err(|e| LockError::internal(format!("Failed to serialize request: {e}")))?;
@@ -1226,6 +1242,9 @@ impl LockClient for RemoteClient {
         if let Some(error_info) = resp.error_info {
             return Err(LockError::internal(error_info));
         }
+        if resp.success {
+            self.record_release_ack(lock_id);
+        }
         Ok(resp.success)
     }
 
@@ -1261,7 +1280,13 @@ impl LockClient for RemoteClient {
         Ok(lock_ids
             .iter()
             .enumerate()
-            .map(|(idx, _)| resp.results.get(idx).map(|result| result.success).unwrap_or(false))
+            .map(|(idx, lock_id)| {
+                let result = resp.results.get(idx);
+                if result.is_some_and(|result| result.success && result.error_info.is_none()) {
+                    self.record_release_ack(lock_id);
+                }
+                result.map(|result| result.success).unwrap_or(false)
+            })
             .collect())
     }
 

crates/ecstore/src/core/pools.rs

@@ -84,13 +84,12 @@ use rand::RngExt as _;
 use rmp_serde::Deserializer;
 use rmp_serde::Serializer;
 use rustfs_filemeta::{FileInfo, FileInfoVersions, MetaCacheEntries, MetaCacheEntry, MetadataResolutionParams};
-use rustfs_gateway_types::persistence::PersistedObjectLockConfiguration as ObjectLockConfiguration;
-use rustfs_gateway_types::persistence::{PersistedLifecycleConfiguration, PersistedReplicationConfiguration};
 use rustfs_heal_contracts::heal_channel::HealOpts;
 use rustfs_utils::crypto::{hex_sha256, is_sha256_checksum};
 use rustfs_utils::path::{
     decode_dir_object, encode_dir_object, path_join, path_to_bucket_object, path_to_bucket_object_with_base_path,
 };
+use s3s::dto::{BucketLifecycleConfiguration, ObjectLockConfiguration, ReplicationConfiguration};
 use serde::{Deserialize, Serialize};
 use sha2::{Digest, Sha256};
 use std::collections::{HashMap, HashSet};
@@ -3721,6 +3720,46 @@ fn is_decommission_copy_quorum_deferral(err: &Error) -> bool {
     data_movement::data_movement_stage_source_as::<DecommissionCopyQuorumFailure>(err).is_some()
 }
 
+/// The storage error of a version copy that kept timing out on a namespace
+/// lock through its in-place retries, kept typed so the entry failure policy
+/// defers it.
+#[derive(Debug, thiserror::Error)]
+#[error("{0}")]
+struct DecommissionCopyLockTimeout(#[source] Error);
+
+fn is_decommission_copy_lock_timeout_error(err: &Error) -> bool {
+    let err = data_movement::data_movement_stage_source(err).unwrap_or(err);
+    matches!(err, Error::Lock(rustfs_lock::LockError::Timeout { .. }))
+}
+
+/// A version copy that timed out acquiring a namespace lock leaves the source
+/// in place; the next attempt copies the version again or finds an equivalent
+/// committed target. The lock is held by another mutation of the key or, when
+/// this node restarted mid-copy, by its previous incarnation until that lock's
+/// lease expires on the peers. Either clears by waiting, so the entry is
+/// deferred instead of being counted as a permanent per-object failure.
+fn decommission_copy_lock_timeout_deferral(stage: &str, bucket: &str, object: &str, err: Error) -> Error {
+    data_movement::data_movement_context_error(
+        format!("decommission {stage} timed out on a namespace lock for {bucket}/{object}: {err}"),
+        DecommissionCopyLockTimeout(err),
+    )
+}
+
+fn is_decommission_copy_lock_timeout_deferral(err: &Error) -> bool {
+    data_movement::data_movement_stage_source_as::<DecommissionCopyLockTimeout>(err).is_some()
+}
+
+/// The deferral of a version copy whose in-place retries ended on `error`, if
+/// that failure belongs to the pools at that moment rather than to the object.
+fn take_decommission_copy_deferral(stage: &str, bucket: &str, object: &str, error: &mut Option<Error>) -> Option<Error> {
+    if let Some(err) = error.take_if(|err| is_decommission_copy_quorum_error(err)) {
+        return Some(decommission_copy_quorum_deferral(stage, bucket, object, err));
+    }
+    error
+        .take_if(|err| is_decommission_copy_lock_timeout_error(err))
+        .map(|err| decommission_copy_lock_timeout_deferral(stage, bucket, object, err))
+}
+
 fn resolve_decommission_entry_reload_result(result: Result<()>, bucket: &str, object_name: &str) -> Result<()> {
     result.map_err(|err| Error::other(format!("decommission reload_pool_meta failed for {bucket}/{object_name}: {err}")))
 }
@@ -5783,11 +5822,14 @@ enum DecommissionEntryOutcome {
 /// entry is done and must not be counted as a failure or escalated. A quorum
 /// failure of the source cleanup is deferred too: the copied source is still
 /// in place and the next attempt cleans it up again. So is a version copy that
-/// lost drive quorum through its in-place retries: the source still holds the
-/// version and the next attempt copies it again. Every other failure stays
-/// terminal.
+/// lost drive quorum or timed out on a namespace lock through its in-place
+/// retries: the source still holds the version and the next attempt copies it
+/// again. Every other failure stays terminal.
 fn resolve_decommission_entry_failure(err: Error) -> Result<DecommissionEntryOutcome> {
-    if is_decommission_source_cleanup_retryable_error(&err) || is_decommission_copy_quorum_deferral(&err) {
+    if is_decommission_source_cleanup_retryable_error(&err)
+        || is_decommission_copy_quorum_deferral(&err)
+        || is_decommission_copy_lock_timeout_deferral(&err)
+    {
         return Ok(DecommissionEntryOutcome::Deferred(err));
     }
     match classify_decommission_capacity_failure(&err) {
@@ -11703,6 +11745,61 @@ pub struct PoolSpaceInfo {
     pub used: usize,
 }
 
+/// Read-only observations used by the admin pool diagnostic surface. The HTTP
+/// layer converts these stable values to strings without changing the pool
+/// lifecycle status contract.
+#[derive(Debug, Clone, Copy, PartialEq, Eq)]
+pub enum PoolAvailability {
+    Online,
+    Degraded,
+    Offline,
+    Unknown,
+}
+
+impl PoolAvailability {
+    pub fn as_str(self) -> &'static str {
+        match self {
+            Self::Online => "online",
+            Self::Degraded => "degraded",
+            Self::Offline => "offline",
+            Self::Unknown => "unknown",
+        }
+    }
+}
+
+#[derive(Debug, Clone, Copy, PartialEq, Eq)]
+pub enum PoolCapacityStatus {
+    Live,
+    Cached,
+    Partial,
+    Unknown,
+}
+
+impl PoolCapacityStatus {
+    pub fn as_str(self) -> &'static str {
+        match self {
+            Self::Live => "live",
+            Self::Cached => "cached",
+            Self::Partial => "partial",
+            Self::Unknown => "unknown",
+        }
+    }
+}
+
+#[derive(Debug, Clone, Copy, PartialEq, Eq)]
+pub struct PoolDiagnosticStatus {
+    pub availability: PoolAvailability,
+    pub capacity_status: PoolCapacityStatus,
+}
+
+#[derive(Debug, Clone)]
+pub struct PoolAdminDiagnostic {
+    pub id: usize,
+    pub cmd_line: String,
+    pub pool: Option<PoolStatus>,
+    pub diagnostic: PoolDiagnosticStatus,
+}
+
 #[derive(Debug, Clone, Copy, PartialEq, Eq)]
 pub(crate) struct DecommissionPoolCapacityInfo {
     pool_index: usize,
@@ -11711,6 +11808,7 @@ pub(crate) struct DecommissionPoolCapacityInfo {
     physical_free: usize,
     physical_total: usize,
     physical_used: usize,
+    diagnostic: PoolDiagnosticStatus,
 }
 
 impl DecommissionPoolCapacityInfo {
@@ -11733,6 +11831,10 @@ impl DecommissionPoolCapacityInfo {
             physical_free,
             physical_total,
             physical_used,
+            diagnostic: PoolDiagnosticStatus {
+                availability: PoolAvailability::Unknown,
+                capacity_status: PoolCapacityStatus::Unknown,
+            },
         }
     }
 }
@@ -11754,6 +11856,121 @@ where
     Ok(capacities)
 }
 
+fn pool_diagnostic_status(
+    disks: &[rustfs_madmin::Disk],
+    pool_index: usize,
+    set_count: usize,
+    set_drive_count: usize,
+    read_quorum: usize,
+) -> PoolDiagnosticStatus {
+    if set_count == 0 || set_drive_count == 0 || read_quorum == 0 || read_quorum > set_drive_count {
+        return PoolDiagnosticStatus {
+            availability: PoolAvailability::Unknown,
+            capacity_status: PoolCapacityStatus::Unknown,
+        };
+    }
+
+    let mut live_capacity = true;
+    let mut cached_capacity = false;
+    let mut partial_capacity = false;
+    let mut online_by_set = vec![0usize; set_count];
+    let mut observed_disks = 0usize;
+    let mut seen_slots = HashSet::new();
+
+    for disk in disks.iter().filter(|disk| disk.pool_index == pool_index as i32) {
+        observed_disks = observed_disks.saturating_add(1);
+        match disk.capacity_observation_source.as_deref() {
+            Some("live_probe") => {}
+            Some("snapshot") => {
+                live_capacity = false;
+                cached_capacity = true;
+            }
+            Some("missing") | None => {
+                live_capacity = false;
+                partial_capacity = true;
+            }
+            Some(_) => {
+                live_capacity = false;
+                partial_capacity = true;
+            }
+        }
+
+        let Some(set_index) = usize::try_from(disk.set_index).ok() else {
+            continue;
+        };
+        let Some(disk_index) = usize::try_from(disk.disk_index).ok() else {
+            continue;
+        };
+        if set_index >= set_count
+            || disk_index >= set_drive_count
+            || !seen_slots.insert((set_index, disk_index))
+            || !disk_is_available_for_pool_diagnostic(disk)
+        {
+            continue;
+        }
+        online_by_set[set_index] = online_by_set[set_index].saturating_add(1);
+    }
+
+    let availability = if online_by_set.iter().any(|&online| online < read_quorum) {
+        PoolAvailability::Offline
+    } else if online_by_set.iter().all(|&online| online == set_drive_count) {
+        PoolAvailability::Online
+    } else {
+        PoolAvailability::Degraded
+    };
+    let capacity_status = if observed_disks == 0 {
+        PoolCapacityStatus::Unknown
+    } else if partial_capacity {
+        PoolCapacityStatus::Partial
+    } else if cached_capacity {
+        PoolCapacityStatus::Cached
+    } else if live_capacity {
+        PoolCapacityStatus::Live
+    } else {
+        PoolCapacityStatus::Unknown
+    };
+
+    PoolDiagnosticStatus {
+        availability,
+        capacity_status,
+    }
+}
+
+fn disk_is_available_for_pool_diagnostic(disk: &rustfs_madmin::Disk) -> bool {
+    let state_ok = disk.state.eq_ignore_ascii_case("ok") || disk.state.eq_ignore_ascii_case(rustfs_madmin::ITEM_ONLINE);
+    let runtime_ok = disk
+        .runtime_state
+        .as_deref()
+        .map(|state| state.eq_ignore_ascii_case(rustfs_madmin::ITEM_ONLINE))
+        .unwrap_or(state_ok);
+    state_ok && runtime_ok
+}
+
+fn pool_diagnostic_metadata_status(err: &Error) -> &'static str {
+    if let Some(failure) = err.pool_metadata_failure() {
+        return match failure.kind {
+            crate::error::PoolMetadataFailure::ReadUnavailable => "stale",
+            crate::error::PoolMetadataFailure::RecoveryRequired
+            | crate::error::PoolMetadataFailure::TransactionUnknown
+            | crate::error::PoolMetadataFailure::FenceLost => "recovery_required",
+        };
+    }
+
+    match err {
+        Error::Timeout
+        | Error::Lock(_)
+        | Error::DiskNotFound
+        | Error::FaultyDisk
+        | Error::FaultyRemoteDisk
+        | Error::RemoteClientUnavailable(_)
+        | Error::RemoteNotInitialized
+        | Error::ErasureReadQuorum
+        | Error::InsufficientReadQuorum(_, _)
+        | Error::NamespaceLockQuorumUnavailable { .. } => "stale",
+        _ => "recovery_required",
+    }
+}
+
 #[cfg(test)]
 static DECOMMISSION_START_GATE_WAIT_OVERRIDES: std::sync::OnceLock<std::sync::Mutex<HashMap<uuid::Uuid, std::time::Duration>>> =
     std::sync::OnceLock::new();
@@ -12666,7 +12883,7 @@ pub(crate) async fn should_skip_lifecycle_for_data_movement(
     store: Arc<ECStore>,
     bucket: &str,
     version: &rustfs_filemeta::FileInfo,
-    lifecycle_config: Option<&PersistedLifecycleConfiguration>,
+    lifecycle_config: Option<&BucketLifecycleConfiguration>,
     object_lock_config: Option<&ObjectLockConfiguration>,
     apply_actions: bool,
     event_source: &LcEventSrc,
@@ -15126,6 +15343,70 @@ impl ECStore {
         Ok(snapshot)
     }
 
+    /// Admin-only diagnostic snapshot. A transient metadata read failure is
+    /// reported as stale without publishing the fallback or relaxing any
+    /// mutation/recovery path.
+    pub async fn admin_pool_diagnostic_snapshot(
+        &self,
+        target_pool_index: Option<usize>,
+    ) -> (Vec<PoolAdminDiagnostic>, &'static str) {
+        let fallback = self.pool_meta.read().await.clone();
+        let (pool_meta, mut metadata_status) = match self.pool_status_meta_snapshot().await {
+            Ok(pool_meta) => (pool_meta, "fresh"),
+            Err(err) => (fallback, pool_diagnostic_metadata_status(&err)),
+        };
+        if metadata_status == "stale" && pool_meta.pools.len() != self.pools.len() {
+            metadata_status = "unknown";
+        }
+
+        let backend = rustfs_madmin::StorageInfo {
+            backend: StorageAdminApi::backend_info(self).await,
+            ..Default::default()
+        };
+        let indices = target_pool_index.map_or_else(|| (0..self.pools.len()).collect(), |idx| vec![idx]);
+        let backend = &backend;
+        let samples = futures::future::join_all(indices.into_iter().map(|idx| async move {
+            (
+                idx,
+                self.get_decommission_pool_capacity_info_with_backend(idx, false, backend)
+                    .await,
+            )
+        }))
+        .await;
+
+        let mut diagnostics = Vec::with_capacity(samples.len());
+        for (idx, sample) in samples {
+            let Some(sets) = self.pools.get(idx) else {
+                continue;
+            };
+            let (pool, diagnostic) = match sample {
+                Ok(capacity) => (
+                    pool_meta
+                        .pools
+                        .get(idx)
+                        .cloned()
+                        .map(|pool| apply_decommission_status_space_info(pool, capacity.space)),
+                    capacity.diagnostic,
+                ),
+                Err(_) => (
+                    pool_meta.pools.get(idx).cloned(),
+                    PoolDiagnosticStatus {
+                        availability: PoolAvailability::Unknown,
+                        capacity_status: PoolCapacityStatus::Unknown,
+                    },
+                ),
+            };
+            diagnostics.push(PoolAdminDiagnostic {
+                id: idx,
+                cmd_line: sets.endpoints.cmd_line.clone(),
+                pool,
+                diagnostic,
+            });
+        }
+
+        (diagnostics, metadata_status)
+    }
+
     /// Load and publish under the movement writer, so no movement read sees
     /// the transition mid-operation and no transition interleaves the load.
     async fn publish_pool_status_refresh(&self, _movement_guard: tokio::sync::OwnedRwLockWriteGuard<()>) -> Result<()> {
@@ -15202,6 +15483,7 @@ impl ECStore {
             };
             let (physical_total, physical_free, physical_used) =
                 decommission_physical_pool_capacity(&info.disks, idx, layout, space, owned_physical_used.as_ref());
+            let diagnostic = pool_diagnostic_status(&info.disks, idx, sets.set_count, sets.set_drive_count, layout.data);
 
             Ok(DecommissionPoolCapacityInfo {
                 pool_index: idx,
@@ -15210,6 +15492,7 @@ impl ECStore {
                 physical_free,
                 physical_total,
                 physical_used,
+                diagnostic,
             })
         } else {
             Err(invalid_decommission_pool_index_error(self.pools.len(), idx))
@@ -16529,9 +16812,9 @@ impl ECStore {
         generation: OffsetDateTime,
         bucket: String,
         set: Arc<SetDisks>,
-        lifecycle_config: Option<PersistedLifecycleConfiguration>,
+        lifecycle_config: Option<BucketLifecycleConfiguration>,
         object_lock_config: Option<ObjectLockConfiguration>,
-        replication_config: Option<(PersistedReplicationConfiguration, OffsetDateTime)>,
+        replication_config: Option<(ReplicationConfiguration, OffsetDateTime)>,
         expected_bucket_incarnation_id: Option<uuid::Uuid>,
         source_changed_exhaustions: Arc<AtomicUsize>,
         entry_budget: Arc<Semaphore>,
@@ -16703,9 +16986,9 @@ impl ECStore {
         mut deferred_entry: DeferredDecommissionEntry,
         bucket: String,
         set: Arc<SetDisks>,
-        lifecycle_config: Option<PersistedLifecycleConfiguration>,
+        lifecycle_config: Option<BucketLifecycleConfiguration>,
         object_lock_config: Option<ObjectLockConfiguration>,
-        replication_config: Option<(PersistedReplicationConfiguration, OffsetDateTime)>,
+        replication_config: Option<(ReplicationConfiguration, OffsetDateTime)>,
         expected_bucket_incarnation_id: Option<uuid::Uuid>,
         source_changed_exhaustions: Arc<AtomicUsize>,
         entry_budget: Arc<Semaphore>,
@@ -16812,9 +17095,9 @@ impl ECStore {
         generation: OffsetDateTime,
         set: Arc<SetDisks>,
         bi: DecomBucketInfo,
-        lifecycle_config: Option<PersistedLifecycleConfiguration>,
+        lifecycle_config: Option<BucketLifecycleConfiguration>,
         object_lock_config: Option<ObjectLockConfiguration>,
-        replication_config: Option<(PersistedReplicationConfiguration, OffsetDateTime)>,
+        replication_config: Option<(ReplicationConfiguration, OffsetDateTime)>,
         expected_bucket_incarnation_id: Option<uuid::Uuid>,
         source_changed_exhaustions: Arc<AtomicUsize>,
         entry_budget: Arc<Semaphore>,
@@ -17489,9 +17772,9 @@ impl ECStore {
         entry: MetaCacheEntry,
         bucket: String,
         set: Arc<SetDisks>,
-        lifecycle_config: Option<PersistedLifecycleConfiguration>,
+        lifecycle_config: Option<BucketLifecycleConfiguration>,
         object_lock_config: Option<ObjectLockConfiguration>,
-        replication_config: Option<(PersistedReplicationConfiguration, OffsetDateTime)>,
+        replication_config: Option<(ReplicationConfiguration, OffsetDateTime)>,
         expected_bucket_incarnation_id: Option<uuid::Uuid>,
         source_changed_exhaustions: Arc<AtomicUsize>,
         attempts: &mut usize,
@@ -17675,9 +17958,9 @@ impl ECStore {
         entry: MetaCacheEntry,
         bucket: String,
         set: Arc<SetDisks>,
-        lifecycle_config: Option<PersistedLifecycleConfiguration>,
+        lifecycle_config: Option<BucketLifecycleConfiguration>,
         object_lock_config: Option<ObjectLockConfiguration>,
-        replication_config: Option<(PersistedReplicationConfiguration, OffsetDateTime)>,
+        replication_config: Option<(ReplicationConfiguration, OffsetDateTime)>,
         expected_bucket_incarnation_id: Option<uuid::Uuid>,
         entry_attempt: usize,
         scanner_backlog_handoff_attempt: usize,
@@ -18151,13 +18434,11 @@ impl ECStore {
                     }
                 }
 
-                if failure && let Some(err) = error.take_if(|err| is_decommission_copy_quorum_error(err)) {
-                    return Err(decommission_copy_quorum_deferral(
-                        "delete_marker_copy",
-                        bucket.as_str(),
-                        version.name.as_str(),
-                        err,
-                    ));
+                if failure
+                    && let Some(err) =
+                        take_decommission_copy_deferral("delete_marker_copy", bucket.as_str(), version.name.as_str(), &mut error)
+                {
+                    return Err(err);
                 }
 
                 if ignore {
@@ -18543,14 +18824,14 @@ impl ECStore {
 
             if failure
                 && !source_read_failed
-                && let Some(err) = error.take_if(|err| is_decommission_copy_quorum_error(err))
-            {
-                return Err(decommission_copy_quorum_deferral(
+                && let Some(err) = take_decommission_copy_deferral(
                     DECOMMISSION_STAGE_MIGRATE_OBJECT,
                     bucket.as_str(),
                     version.name.as_str(),
-                    err,
-                ));
+                    &mut error,
+                )
+            {
+                return Err(err);
             }
 
             if ignore {
@@ -18965,7 +19246,7 @@ impl ECStore {
         entry: MetaCacheEntry,
         bucket: String,
         set: Arc<SetDisks>,
-        lifecycle_config: Option<PersistedLifecycleConfiguration>,
+        lifecycle_config: Option<BucketLifecycleConfiguration>,
     ) -> Result<()> {
         let expected_bucket_incarnation_id = if is_meta_bucketname(&bucket) {
             None
@@ -21804,6 +22085,85 @@ mod tests {
     use crate::storage_api_contracts::multipart::MultipartOperations as _;
     use serde::Serialize;
 
+    fn diagnostic_disk(set_index: i32, disk_index: i32, online: bool) -> rustfs_madmin::Disk {
+        rustfs_madmin::Disk {
+            pool_index: 0,
+            set_index,
+            disk_index,
+            state: if online { "ok" } else { "offline" }.to_string(),
+            runtime_state: Some(if online { "online" } else { "offline" }.to_string()),
+            capacity_observation_source: Some("live_probe".to_string()),
+            ..Default::default()
+        }
+    }
+
+    #[test]
+    fn pool_diagnostic_status_tracks_read_quorum_and_capacity_source() {
+        let online = vec![
+            diagnostic_disk(0, 0, true),
+            diagnostic_disk(0, 1, true),
+            diagnostic_disk(1, 0, true),
+            diagnostic_disk(1, 1, true),
+        ];
+        assert_eq!(
+            pool_diagnostic_status(&online, 0, 2, 2, 1),
+            PoolDiagnosticStatus {
+                availability: PoolAvailability::Online,
+                capacity_status: PoolCapacityStatus::Live,
+            }
+        );
+
+        let mut degraded = online.clone();
+        degraded[0] = diagnostic_disk(0, 0, false);
+        degraded[1].capacity_observation_source = Some("snapshot".to_string());
+        assert_eq!(
+            pool_diagnostic_status(&degraded, 0, 2, 2, 1),
+            PoolDiagnosticStatus {
+                availability: PoolAvailability::Degraded,
+                capacity_status: PoolCapacityStatus::Cached,
+            }
+        );
+
+        degraded[1] = diagnostic_disk(0, 1, false);
+        assert_eq!(pool_diagnostic_status(&degraded, 0, 2, 2, 1).availability, PoolAvailability::Offline);
+        assert_eq!(
+            pool_diagnostic_status(&[], 0, 2, 2, 1),
+            PoolDiagnosticStatus {
+                availability: PoolAvailability::Offline,
+                capacity_status: PoolCapacityStatus::Unknown,
+            }
+        );
+        assert_eq!(pool_diagnostic_status(&online, 0, 0, 2, 1).availability, PoolAvailability::Unknown);
+    }
+
+    #[tokio::test]
+    #[serial_test::serial]
+    async fn admin_pool_diagnostic_snapshot_lists_an_unavailable_nonfirst_pool() {
+        let (_temp_dirs, store, _other_store) = Box::pin(crate::services::rebalance::test_two_pool_stores(None)).await;
+        for set in &store.pools[1].disk_set {
+            let mut disks = set.disks.write().await;
+            let disk_count = disks.len();
+            *disks = vec![None; disk_count];
+        }
+
+        let (snapshot, metadata_status) =
+            tokio::time::timeout(std::time::Duration::from_secs(10), store.admin_pool_diagnostic_snapshot(None))
+                .await
+                .expect("pool diagnostics should remain bounded");
+
+        assert_eq!(metadata_status, "stale");
+        assert_eq!(snapshot.len(), 2, "configured topology must remain visible");
+        assert_eq!(snapshot[0].id, 0);
+        assert_eq!(snapshot[1].id, 1);
+        assert_eq!(snapshot[1].diagnostic.availability, PoolAvailability::Offline);
+        store
+            .pool_meta_save_gate
+            .lock()
+            .await
+            .ensure_write_safe("ordinary object write")
+            .expect("a failed diagnostic read must not latch the writer gate");
+    }
+
     #[test]
     fn decommission_set_local_usage_cache_classification_is_exact() {
         for object in [
@@ -28297,6 +28657,61 @@ mod tests {
         ));
     }
 
+    /// A restarted node's previous incarnation keeps its namespace locks on the
+    /// peers until their lease expires, so a resumed copy of a key that was in
+    /// flight at the crash times out acquiring the lock. That copy never
+    /// committed: it is deferred, never counted as a failed object.
+    #[test]
+    fn decommission_copy_lock_timeout_is_deferred_not_failed() {
+        let timeout = || {
+            Error::Lock(rustfs_lock::LockError::timeout(
+                "bucket-a/object-a@latest",
+                std::time::Duration::from_secs(5),
+            ))
+        };
+        let prepare = data_movement::data_movement_stage_error_for_test(
+            "decommission_object",
+            "prepare_put_object",
+            "bucket-a",
+            "object-a",
+            timeout(),
+        );
+        for err in [timeout(), prepare] {
+            assert!(is_decommission_copy_lock_timeout_error(&err), "{err}");
+            assert!(!is_decommission_copy_quorum_error(&err), "{err}");
+            let mut error = Some(err);
+            let deferred = take_decommission_copy_deferral("migrate_object", "bucket-a", "object-a", &mut error)
+                .expect("a lock timeout copy must be deferred");
+            assert!(error.is_none());
+            assert!(is_decommission_copy_lock_timeout_deferral(&deferred));
+            assert!(!is_decommission_copy_quorum_deferral(&deferred));
+            let failure = data_movement::data_movement_stage_source_as::<DecommissionCopyLockTimeout>(&deferred)
+                .expect("the deferred error must keep its lock-timeout origin");
+            assert!(is_decommission_copy_lock_timeout_error(&failure.0));
+            assert!(matches!(
+                resolve_decommission_entry_failure(deferred),
+                Ok(DecommissionEntryOutcome::Deferred(_))
+            ));
+        }
+
+        let mut quorum = Some(Error::ErasureWriteQuorum);
+        let deferred = take_decommission_copy_deferral("migrate_object", "bucket-a", "object-a", &mut quorum)
+            .expect("a lost drive quorum copy stays deferred");
+        assert!(is_decommission_copy_quorum_deferral(&deferred));
+
+        for other in [
+            Error::Lock(rustfs_lock::LockError::internal("lock backend failed")),
+            Error::other("Lock acquisition timeout for resource 'bucket-a/object-a@latest' after 5s"),
+        ] {
+            let mut error = Some(other);
+            assert!(
+                take_decommission_copy_deferral("migrate_object", "bucket-a", "object-a", &mut error).is_none(),
+                "{error:?}"
+            );
+            assert!(error.is_some(), "a per-object failure keeps its error");
+        }
+    }
+
     #[test]
     fn decommission_target_capacity_error_accepts_wrapped_capacity_errors() {
         let disk_full = Error::other(format!("decommission_object: put_object failed for bucket/object: {}", Error::DiskFull));

crates/ecstore/src/core/pools_test.rs

@@ -2480,6 +2480,93 @@ mod decommission_lock_order_tests {
         });
     }
 
+    /// A node killed mid-copy leaves its namespace lock on the peers until the
+    /// lock lease expires, so the resumed decommission times out acquiring the
+    /// lock of the key it was copying through every in-place retry. The entry
+    /// is deferred and copied once the lease is gone; it is never counted as a
+    /// failed object, which would fail the whole decommission.
+    #[test]
+    #[serial_test::serial]
+    fn decommission_defers_a_copy_that_timed_out_on_a_stale_namespace_lock() {
+        run_large_stack_async_test("decommission-copy-lock-timeout", || async {
+            let (_temp_dirs, store, _other_store) = test_three_pool_stores_with_isolated_node_contexts(None).await;
+            let bucket = test_bucket("copy-lock-timeout");
+            let object = "external/005.bin";
+            let payload = vec![0x3c; 2048];
+            store
+                .make_bucket(&bucket, &MakeBucketOptions::default())
+                .await
+                .expect("create the bucket");
+            put_source_object(&store, &bucket, object, payload.clone()).await;
+
+            // The stale lease outlives every in-place attempt of the first
+            // pass and has expired by the deferred replay.
+            let stale_lease_calls = crate::core::pools::DECOMMISSION_VERSION_COPY_ATTEMPTS;
+            let copy_calls = Arc::new(AtomicUsize::new(0));
+            let _fault = DecommissionTestFaultGuard::install_with_error(
+                Arc::new({
+                    let copy_calls = Arc::clone(&copy_calls);
+                    let bucket = bucket.clone();
+                    move |stage, entry_bucket, entry_object, _, _| {
+                        stage == "before_target_commit"
+                            && entry_bucket == bucket
+                            && entry_object == object
+                            && copy_calls.fetch_add(1, Ordering::AcqRel) < stale_lease_calls
+                    }
+                }),
+                Some(Arc::new(|_, bucket, object| {
+                    // The shape the target write reports when the lock domain
+                    // still holds the previous incarnation's lock.
+                    data_movement::data_movement_stage_error_for_test(
+                        "decommission_object",
+                        "prepare_put_object",
+                        bucket,
+                        object,
+                        crate::error::Error::Lock(rustfs_lock::LockError::timeout(
+                            format!("{bucket}/{object}@latest"),
+                            Duration::from_secs(5),
+                        )),
+                    )
+                })),
+            );
+
+            start_transitioned_entry_decommission(&store, &bucket).await;
+            spawn_decommission_routine(&store, 1)
+                .await
+                .expect("the decommission routine task should join")
+                .expect("the decommission routine should return");
+
+            assert_eq!(
+                copy_calls.load(Ordering::Acquire),
+                stale_lease_calls + 1,
+                "the first pass exhausts its in-place retries and the replay copies the object once"
+            );
+            let local = store.pool_meta.read().await;
+            let info = local.pools[0]
+                .decommission
+                .as_ref()
+                .expect("the decommission state should remain present");
+            assert_eq!(
+                info.items_decommission_failed, 0,
+                "a lock held until its lease expires is not a permanent failure: {info:?}"
+            );
+            assert!(
+                info.complete && !info.failed && !info.canceled && info.capacity_blocked_reason.is_none(),
+                "the replayed entry must complete the decommission: {info:?}"
+            );
+            drop(local);
+
+            assert!(exact_versions(&store, 0, &bucket, object).await.is_empty(), "the source is cleaned up");
+            let mut reader = store
+                .get_object_reader(&bucket, object, None, HeaderMap::new(), &ObjectOptions::default())
+                .await
+                .expect("the migrated object stays readable");
+            let mut body = Vec::new();
+            reader.stream.read_to_end(&mut body).await.expect("read the migrated object");
+            assert_eq!(body, payload);
+        });
+    }
+
     /// Seed `object` in pool 0 with three 2 KiB data versions and a newest
     /// delete marker; returns the data versions oldest first.
     async fn seed_versioned_entry_with_marker(

crates/ecstore/src/core/sets.rs

@@ -65,38 +65,6 @@ use uuid::Uuid;
 
 const LIST_MULTIPART_SETS_CONCURRENCY: usize = 4;
 
-#[cfg(test)]
-type TestSetLockers = HashMap<usize, Vec<Arc<dyn rustfs_lock::LockClient>>>;
-
-#[cfg(test)]
-static TEST_SET_LOCKERS: std::sync::LazyLock<std::sync::Mutex<TestSetLockers>> =
-    std::sync::LazyLock::new(|| std::sync::Mutex::new(HashMap::new()));
-
-/// Keep a fixture's real locker domain scoped to its owning instance.
-#[cfg(test)]
-pub(crate) struct ScopedSetLockers(Arc<InstanceContext>);
-
-#[cfg(test)]
-impl ScopedSetLockers {
-    pub(crate) fn install(ctx: Arc<InstanceContext>, lockers: Vec<Arc<dyn rustfs_lock::LockClient>>) -> Self {
-        let key = Arc::as_ptr(&ctx) as usize;
-        let mut domains = TEST_SET_LOCKERS.lock().expect("test set lockers mutex");
-        assert!(!domains.contains_key(&key), "the instance already owns a test locker domain");
-        domains.insert(key, lockers);
-        Self(ctx)
-    }
-}
-
-#[cfg(test)]
-impl Drop for ScopedSetLockers {
-    fn drop(&mut self) {
-        TEST_SET_LOCKERS
-            .lock()
-            .expect("test set lockers mutex")
-            .remove(&(Arc::as_ptr(&self.0) as usize));
-    }
-}
-
 fn is_idempotent_delete_prefix_error(err: &Error) -> bool {
     is_err_object_not_found(err) || is_err_strict_volume_not_found(err)
 }
@@ -298,14 +266,6 @@ impl Sets {
                 }
             }
 
-            #[cfg(test)]
-            let pool_lockers = TEST_SET_LOCKERS
-                .lock()
-                .expect("test set lockers mutex")
-                .get(&(Arc::as_ptr(&instance_ctx) as usize))
-                .cloned()
-                .unwrap_or_else(|| pool_lockers.clone());
-
             let set_disks = SetDisks::new_with_instance_ctx(
                 runtime_sources::local_node_name().await,
                 Arc::new(RwLock::new(set_drive)),

crates/ecstore/src/data_usage/mod.rs

@@ -28,7 +28,7 @@ use crate::{
     error::{Error, classify_system_path_failure_reason},
     object_api::{ObjectInfo, ObjectOptions, PutObjReader},
     runtime::sources as runtime_sources,
-    store::{ECStore, list_objects::list_marker_key},
+    store::ECStore,
 };
 pub use local_snapshot::{LocalUsageSnapshot, read_snapshot as read_local_snapshot, snapshot_path};
 use rustfs_data_usage::{
@@ -1966,8 +1966,8 @@ fn advance_version_listing_cursor(
     let next_version_marker = next_version_marker
         .filter(|next_version_marker| !next_version_marker.is_empty())
         .ok_or_else(|| Error::other(format!("bucket {bucket} version listing was truncated without a version marker")))?;
-    let current_key_marker = marker.as_deref().map(list_marker_key);
-    let next_key_marker = list_marker_key(&next_marker);
+    let current_key_marker = marker.as_deref();
+    let next_key_marker = next_marker.as_str();
     if current_key_marker == Some(next_key_marker) && version_marker.as_deref() == Some(next_version_marker.as_str()) {
         return Err(Error::other(format!(
             "bucket {bucket} version listing returned a repeated continuation marker"
@@ -5167,20 +5167,20 @@ mod tests {
         );
         assert!(ensure_truncated_version_page_has_entries("bucket-a", 0).is_err());
         ensure_truncated_version_page_has_entries("bucket-a", 1).expect("a truncated page with an entry can advance");
+    }
 
-        let mut tagged_marker = Some("object-a[rustfs_cache:v2,id:old]".to_string());
-        let mut tagged_version_marker = Some("version-a".to_string());
-        assert!(
-            advance_version_listing_cursor(
-                "bucket-a",
-                &mut tagged_marker,
-                &mut tagged_version_marker,
-                Some("object-a[rustfs_cache:v2,id:new]".to_string()),
-                Some("version-a".to_string()),
-            )
-            .is_err(),
-            "different cache tags for the same logical marker must not count as progress"
-        );
+    #[test]
+    fn version_listing_cursor_compares_literal_cache_shaped_keys() {
+        let mut marker = Some("object-a".to_string());
+        let mut version_marker = Some("version-a".to_string());
+        advance_version_listing_cursor(
+            "bucket-a",
+            &mut marker,
+            &mut version_marker,
+            Some("object-a[rustfs_cache:v2,id:old]".to_string()),
+            Some("version-a".to_string()),
+        )
+        .expect("a distinct literal key advances the cursor");
     }
 
     #[test]

crates/ecstore/src/diagnostics/object_lock.rs

@@ -19,6 +19,7 @@ use tracing::Span;
 use uuid::Uuid;
 
 const EVENT_OBJECT_NAMESPACE_LOCK: &str = "object_namespace_lock";
+const EVENT_OBJECT_NAMESPACE_LOCK_PEER: &str = "object_namespace_lock_peer";
 const LOG_COMPONENT_ECSTORE: &str = "ecstore";
 const LOG_SUBSYSTEM_OBJECT_LOCK: &str = "object_lock";
 
@@ -56,6 +57,7 @@ struct AttemptContext {
     requested_version_id: Option<String>,
     namespace: String,
     owner: String,
+    lock_id: Option<String>,
     mode: &'static str,
     timeout: Duration,
     started_at: Instant,
@@ -85,6 +87,7 @@ impl ObjectLockAttempt {
             requested_version_id: requested_version_id.map(str::to_owned),
             namespace: lock.namespace().to_owned(),
             owner: lock.owner().to_owned(),
+            lock_id: None,
             mode,
             timeout,
             started_at: Instant::now(),
@@ -105,9 +108,29 @@ impl ObjectLockAttempt {
                 context.finished = true;
                 context.trace("disabled");
             }
-            Ok(_) => {
+            Ok(guard) => {
+                context.lock_id = guard.lock_id().map(ToString::to_string);
                 context.acquired_at = Some(Instant::now());
                 context.trace("acquired");
+                if let NamespaceLockGuard::Standard(guard) = guard {
+                    for (peer_index, peer_lock_id) in guard.granted_lock_ids().enumerate() {
+                        context.parent.in_scope(|| {
+                            tracing::trace!(
+                                target: "rustfs_ecstore::object_lock_diag",
+                                parent: &context.parent,
+                                event = EVENT_OBJECT_NAMESPACE_LOCK_PEER,
+                                component = LOG_COMPONENT_ECSTORE,
+                                subsystem = LOG_SUBSYSTEM_OBJECT_LOCK,
+                                state = "granted",
+                                lock_attempt_id = %context.id,
+                                lock_id = context.lock_id.as_deref(),
+                                peer_lock_id = %peer_lock_id,
+                                peer_index,
+                                "Object namespace peer owner granted"
+                            );
+                        });
+                    }
+                }
             }
             Err(error) => {
                 context.finished = true;
@@ -171,6 +194,7 @@ impl AttemptContext {
                 mode = self.mode,
                 owner = %self.owner,
                 lock_attempt_id = %self.id,
+                lock_id = self.lock_id.as_deref(),
                 acquire_ms = duration_ms(acquired_at.duration_since(self.started_at)),
                 hold_ms = duration_ms(now.duration_since(acquired_at)),
                 timeout_ms = duration_ms(self.timeout),
@@ -235,15 +259,238 @@ pub(crate) mod tests {
         }
 
         pub(crate) fn rows(&self) -> Vec<serde_json::Value> {
+            self.all_rows()
+                .into_iter()
+                .filter(|row| row["fields"]["event"] == EVENT_OBJECT_NAMESPACE_LOCK)
+                .collect()
+        }
+
+        fn all_rows(&self) -> Vec<serde_json::Value> {
             let bytes = self.0.lock().expect("lock diagnostic capture mutex");
             let text = std::str::from_utf8(&bytes).expect("UTF-8 diagnostic events");
             text.lines()
                 .map(|line| serde_json::from_str::<serde_json::Value>(line).expect("JSON diagnostic event"))
-                .filter(|row| row["fields"]["event"] == EVENT_OBJECT_NAMESPACE_LOCK)
                 .collect()
         }
     }
 
+    #[test]
+    #[serial_test::serial]
+    fn object_lock_diagnostics_join_guard_identity_to_physical_release() {
+        use rustfs_lock::{
+            DisabledLockManager, FastObjectLockManager, GlobalLockManager, LocalClient, LockClient, LockRequest, LockType,
+            NamespaceLock, ObjectKey,
+        };
+
+        temp_env::with_vars([(rustfs_config::ENV_OBJECT_LOCK_DIAG_ENABLE, Some("true"))], || {
+            let capture = CapturedLockEvents::default();
+            let runtime = tokio::runtime::Builder::new_current_thread()
+                .enable_all()
+                .build()
+                .expect("diagnostic release runtime");
+            tracing::subscriber::with_default(capture.subscriber(), || {
+                runtime.block_on(async {
+                    let clients = (0..4)
+                        .map(|_| Arc::new(LocalClient::with_manager(Arc::new(GlobalLockManager::new()))) as Arc<dyn LockClient>)
+                        .collect::<Vec<_>>();
+                    let lock = NamespaceLockWrapper::new(
+                        NamespaceLock::with_clients_and_quorum("diagnostics".to_owned(), clients.clone(), 4),
+                        ObjectKey::new("bucket", "object"),
+                        "coordinator".to_owned(),
+                    );
+                    let request = tracing::info_span!("request", request_id = "original-multipart-request");
+                    let mut attempt = request.in_scope(|| {
+                        ObjectLockAttempt::start(
+                            "abort_multipart_upload",
+                            "bucket",
+                            "object",
+                            None,
+                            &lock,
+                            "write",
+                            Duration::from_secs(5),
+                        )
+                    });
+                    let result = lock.get_write_lock(Duration::from_secs(5)).await;
+                    attempt.observe(&result);
+                    let guard = result.expect("four real namespace owners");
+                    let id = guard.lock_id().expect("distributed guard identity").clone();
+                    let peer_ids = match &guard {
+                        NamespaceLockGuard::Standard(guard) => {
+                            assert_eq!(guard.granted_lock_ids().count(), 4);
+                            guard.granted_lock_ids().cloned().collect::<std::collections::HashSet<_>>()
+                        }
+                        NamespaceLockGuard::Fast(_) => panic!("real distributed namespace required"),
+                    };
+                    let rows = capture.rows();
+                    assert_eq!(rows.last().expect("acquired event")["fields"]["lock_id"], id.to_string());
+                    let rows = capture.all_rows();
+                    let mappings = rows
+                        .iter()
+                        .filter(|row| row["fields"]["event"] == EVENT_OBJECT_NAMESPACE_LOCK_PEER)
+                        .collect::<Vec<_>>();
+                    assert_eq!(mappings.len(), 4, "every granted peer must map to its logical guard");
+                    assert!(mappings.iter().all(|row| {
+                        row["fields"]["lock_id"] == id.to_string()
+                            && peer_ids
+                                .iter()
+                                .any(|peer_id| row["fields"]["peer_lock_id"] == peer_id.to_string())
+                    }));
+                    for client in &clients {
+                        let mut owned = 0;
+                        for peer_id in &peer_ids {
+                            owned += usize::from(
+                                client
+                                    .check_status(peer_id)
+                                    .await
+                                    .expect("actual peer owner status")
+                                    .is_some(),
+                            );
+                        }
+                        assert_eq!(owned, 1, "each client must hold exactly one actual granted identity");
+                    }
+                    drop((guard, attempt));
+                    assert!(
+                        capture.all_rows().iter().all(|row| row["fields"]["state"] != "released"),
+                        "enqueue is not release acknowledgement"
+                    );
+                    tokio::time::timeout(Duration::from_secs(5), async {
+                        while capture
+                            .all_rows()
+                            .iter()
+                            .filter(|row| {
+                                row["fields"]["event"] == "namespace_lock_release" && row["fields"]["state"] == "released"
+                            })
+                            .count()
+                            < 4
+                        {
+                            tokio::task::yield_now().await;
+                        }
+                    })
+                    .await
+                    .expect("all physical release callbacks finish");
+                    for client in &clients {
+                        for peer_id in &peer_ids {
+                            assert!(
+                                client
+                                    .check_status(peer_id)
+                                    .await
+                                    .expect("released peer owner status")
+                                    .is_none()
+                            );
+                            assert!(!client.release(peer_id).await.expect("duplicate release is absent"));
+                        }
+                    }
+                    let disabled = LocalClient::with_manager(Arc::new(GlobalLockManager::Disabled(DisabledLockManager::new())));
+                    let disabled_request =
+                        LockRequest::new(ObjectKey::new("bucket", "disabled"), LockType::Exclusive, "disabled-owner".to_owned());
+                    assert!(
+                        disabled
+                            .acquire_lock(&disabled_request)
+                            .await
+                            .expect("disabled acquire bypass")
+                            .success
+                    );
+                    assert!(
+                        disabled
+                            .release(&disabled_request.lock_id)
+                            .await
+                            .expect("disabled release bypass")
+                    );
+                    let rows = capture.all_rows();
+                    let releases = rows
+                        .iter()
+                        .filter(|row| row["fields"]["event"] == "namespace_lock_release" && row["fields"]["state"] == "released")
+                        .collect::<Vec<_>>();
+                    assert_eq!(
+                        releases.len(),
+                        4,
+                        "duplicates and disabled guards cannot acknowledge another physical release"
+                    );
+                    let physical_mappings = rows
+                        .iter()
+                        .filter(|row| row["fields"]["event"] == "namespace_lock_release" && row["fields"]["state"] == "mapped")
+                        .collect::<Vec<_>>();
+                    assert_eq!(physical_mappings.len(), 4);
+                    assert!(releases.iter().all(|row| physical_mappings.iter().any(|mapped| {
+                        mapped["fields"]["physical_guard_id"] == row["fields"]["physical_guard_id"]
+                            && mapped["fields"]["process_id"] == row["fields"]["process_id"]
+                            && peer_ids
+                                .iter()
+                                .any(|peer_id| mapped["fields"]["lock_id"] == peer_id.to_string())
+                    })));
+                    let dropped = capture.rows().pop().expect("guard drop event");
+                    assert_eq!(dropped["fields"]["lock_id"], id.to_string());
+                    assert!(
+                        dropped["spans"]
+                            .as_array()
+                            .expect("retained request span")
+                            .iter()
+                            .any(|span| span["request_id"] == "original-multipart-request")
+                    );
+                    let manager = Arc::new(FastObjectLockManager::new());
+                    let stale_client = LocalClient::with_manager(Arc::new(GlobalLockManager::Enabled(manager.clone())));
+                    let stale_request = LockRequest::new(
+                        ObjectKey::new("bucket", "externally-released"),
+                        LockType::Exclusive,
+                        "stale-owner".to_owned(),
+                    );
+                    assert!(
+                        stale_client
+                            .acquire_lock(&stale_request)
+                            .await
+                            .expect("real stale owner")
+                            .success
+                    );
+                    assert_eq!(manager.force_unlock(&stale_request.resource), 1);
+                    let mut successor = manager
+                        .acquire_write_lock(stale_request.resource.clone(), "successor-owner")
+                        .await
+                        .expect("real successor guard");
+                    assert!(
+                        stale_client
+                            .release(&stale_request.lock_id)
+                            .await
+                            .expect("retire stale client record")
+                    );
+                    assert_eq!(
+                        manager
+                            .get_lock_info(&stale_request.resource)
+                            .expect("successor remains locked")
+                            .owner
+                            .as_ref(),
+                        "successor-owner"
+                    );
+                    assert_eq!(
+                        capture
+                            .all_rows()
+                            .iter()
+                            .filter(
+                                |row| row["fields"]["event"] == "namespace_lock_release" && row["fields"]["state"] == "released"
+                            )
+                            .count(),
+                        4,
+                        "a stale guard's failed physical release must not produce an ACK"
+                    );
+                    let successor_id = successor.guard_id();
+                    assert!(successor.release(), "the actual successor release succeeds");
+                    assert!(!successor.release(), "a duplicate explicit release cannot complete again");
+                    drop(successor);
+                    assert_eq!(
+                        capture
+                            .all_rows()
+                            .iter()
+                            .filter(|row| row["fields"]["event"] == "namespace_lock_release"
+                                && row["fields"]["state"] == "released"
+                                && row["fields"]["physical_guard_id"] == successor_id)
+                            .count(),
+                        1,
+                        "explicit release followed by Drop records one physical completion"
+                    );
+                });
+            });
+        });
+    }
+
     fn namespace() -> NamespaceLockWrapper {
         let manager = rustfs_lock::GlobalLockManager::Enabled(Arc::new(rustfs_lock::fast_lock::FastObjectLockManager::new()));
         NamespaceLockWrapper::new(

crates/ecstore/src/disk/disk_store.rs

@@ -2776,15 +2776,6 @@ impl DiskAPI for LocalDiskWrapper {
         })
     }
 
-    async fn bucket_creation_witness(&self, volume: &str) -> Result<Option<super::os::NativeBucketCreationWitness>> {
-        self.track_disk_health_with_op(
-            "bucket_creation_witness",
-            || async { self.disk.bucket_creation_witness(volume).await },
-            Duration::ZERO,
-        )
-        .await
-    }
-
     async fn make_volume(&self, volume: &str) -> Result<()> {
         // Scoped heal must drain directory creation before releasing its lifecycle owner.
         let timeout = if crate::store::bucket_heal_scope(volume).is_some() {

crates/ecstore/src/disk/fs.rs

@@ -117,9 +117,6 @@ pub const O_APPEND: FileMode = 0x00400;
 //     create_new: bool,
 
 pub async fn open_file(path: impl AsRef<Path>, mode: FileMode) -> io::Result<File> {
-    if mode & O_CREATE != 0 {
-        super::os::record_native_publication(path.as_ref());
-    }
     let base_opts = match mode & (O_RDONLY | O_WRONLY | O_RDWR) {
         O_RDONLY => get_readonly_options(),
         O_WRONLY => get_writeonly_options(),
@@ -172,7 +169,6 @@ pub fn create_dir_all_std(path: &Path) -> io::Result<()> {
 }
 
 pub async fn make_dir_all(path: impl AsRef<Path>) -> io::Result<()> {
-    super::os::record_native_publication(path.as_ref());
     let path = path.as_ref().to_path_buf();
     tokio::task::spawn_blocking(move || create_dir_all_std(&path)).await?
 }
@@ -226,18 +222,15 @@ pub fn remove_all_std(path: impl AsRef<Path>) -> io::Result<()> {
 
 #[cfg(test)]
 pub async fn mkdir(path: impl AsRef<Path>) -> io::Result<()> {
-    super::os::record_native_publication(path.as_ref());
     fs::create_dir(path.as_ref()).await
 }
 
 pub async fn rename(from: impl AsRef<Path>, to: impl AsRef<Path>) -> io::Result<()> {
-    super::os::record_native_publication(to.as_ref());
     fs::rename(from, to).await
 }
 
 #[cfg(any(not(windows), test))]
 pub fn rename_std(from: impl AsRef<Path>, to: impl AsRef<Path>) -> io::Result<()> {
-    super::os::record_native_publication(to.as_ref());
     std::fs::rename(from, to)
 }
 

crates/ecstore/src/disk/local/commit.rs

@@ -642,20 +642,8 @@ impl LocalDisk {
         };
         let dst_file_path = self.io_get_object_open_path(dst_volume, format!("{}/{}", dst_path, STORAGE_FORMAT_FILE).as_str())?;
 
-        // Multipart restore completion has validated staged local parts,
-        // while its public status stays pending until finalization.
-        let pending_multipart_restore = src_volume == crate::disk::RUSTFS_META_MULTIPART_BUCKET
-            && fi
-                .metadata
-                .get(rustfs_filemeta::metadata_keys::RESTORE)
-                .and_then(|value| rustfs_filemeta::parse_restore_obj_status(value).ok())
-                .is_some_and(|status| status.is_restore_in_progress == Some(true));
-        if pending_multipart_restore && fi.data_dir.is_none_or(|dir| dir.is_nil()) {
-            return Err(DiskError::FileCorrupt);
-        }
-
         // data_dir path (only used by the non-inline commit branch)
-        let has_data_dir_path = if no_inline && (!fi.is_remote() || pending_multipart_restore) {
+        let has_data_dir_path = if no_inline && !fi.is_remote() {
             if let Some(data_dir) = fi
                 .data_dir
                 .map(|dir| rustfs_utils::path::retain_slash(dir.to_string().as_str()))

crates/ecstore/src/disk/local/multipart_source_abort.rs

@@ -6,6 +6,7 @@ use super::{
     LocalDisk,
     multipart_source_state::{
         LoadedSource, SourceHead, SourceState, SourceStore, decode, encode, load_source, raw_sha, read_optional,
+        read_optional_in_pinned,
     },
 };
 use crate::disk::{
@@ -127,20 +128,6 @@ pub(super) fn validate_retired_head(
 }
 
 impl SourceStore {
-    pub async fn root_missing(&self) -> Result<bool> {
-        let source = self.source.clone();
-        let base = self.base.clone();
-        let root = self.publication_root.clone();
-        os::run_blocking_namespace_operation(self.lease.clone(), move || {
-            match os::pin_multipart_source_directory(&source, &base, &root) {
-                Ok(_) => Ok(false),
-                Err(error) if error.kind() == ErrorKind::NotFound => Ok(true),
-                Err(error) => Err(error),
-            }
-        })
-        .await
-        .map_err(DiskError::from)
-    }
     fn verify_abort_cohort(&self, certificate: &MultipartAbortCertificate) -> Result<()> {
         certificate.verify_with_authority(&self.authority)?;
         if certificate.disk_ids != self.disk_ids
@@ -265,7 +252,7 @@ impl SourceStore {
         let certificate = self.read_abort_certificate(&loaded).await?;
         Ok(MultipartSourceInspection {
             disk_id: self.disk_id,
-            root_missing: self.root_missing().await?,
+            root_missing: loaded.root_missing,
             abort_round: loaded
                 .state
                 .as_ref()
@@ -561,11 +548,18 @@ impl AbortDeletePermit {
 pub(super) struct MultipartDeleteOwner {
     owner: Option<Arc<dyn Send + Sync>>,
     source_key: Option<PathBuf>,
+    admission_validated: bool,
 }
 impl MultipartDeleteOwner {
     pub(super) fn namespace_owner(&self) -> Option<Arc<dyn Send + Sync>> {
         self.owner.clone()
     }
+    pub(super) fn mark_validated(&mut self) {
+        self.admission_validated = true;
+    }
+    pub(super) fn is_validated(&self) -> bool {
+        self.admission_validated
+    }
     pub(super) fn chain(self, owner: Option<Arc<dyn Send + Sync>>) -> Self {
         let owner = match (self.owner, owner) {
             (None, owner) | (owner, None) => owner,
@@ -575,6 +569,7 @@ impl MultipartDeleteOwner {
         Self {
             owner,
             source_key: self.source_key,
+            admission_validated: self.admission_validated,
         }
     }
 }
@@ -595,7 +590,11 @@ impl LocalDisk {
     ) -> Result<MultipartDeleteOwner> {
         let source_key = self.multipart_delete_owner_key(volume, path)?;
         let owner = self.multipart_source_owner(volume, path, true, true, owner).await?;
-        Ok(MultipartDeleteOwner { owner, source_key })
+        Ok(MultipartDeleteOwner {
+            owner,
+            source_key,
+            admission_validated: false,
+        })
     }
     pub(super) fn validate_multipart_delete_owner(
         &self,
@@ -620,12 +619,17 @@ pub(super) struct OrdinaryMultipartDeleteCheck {
 }
 impl OrdinaryMultipartDeleteCheck {
     pub(super) fn validate(&self) -> Result<()> {
-        if let Some(raw) = read_optional(&self.source.join(STATE), &self.base, &self.root)? {
+        let pinned = match os::pin_multipart_source_directory(&self.source, &self.base, &self.root) {
+            Ok(pinned) => pinned,
+            Err(error) if error.kind() == ErrorKind::NotFound => return Ok(()),
+            Err(error) => return Err(error.into()),
+        };
+        if let Some(raw) = read_optional_in_pinned(&pinned, &self.source, STATE, &self.base, &self.root)? {
             let state: SourceState = decode(&raw)?;
             if state.abort.is_some() || state.phase == MultipartSourcePhase::Aborting {
                 return Err(DiskError::ConditionalDeleteConflict);
             }
-        } else if os::multipart_source_has_native_controls(&self.source, &self.base, &self.root)? {
+        } else if os::multipart_source_has_native_controls_in_pinned(&self.source, &pinned)? {
             return Err(DiskError::ConditionalDeleteConflict);
         }
         Ok(())

crates/ecstore/src/disk/local/multipart_source_recovery.rs

@@ -170,7 +170,7 @@ impl LocalDisk {
             recovery: loaded.state.as_ref().and_then(|state| state.recovery.clone()),
             negative_certificate,
             ready_certificate,
-            root_missing: store.root_missing().await?,
+            root_missing: loaded.root_missing,
             abort_round: loaded
                 .state
                 .as_ref()
@@ -552,25 +552,33 @@ impl LocalDisk {
         let target = self.get_object_path(&publication.lineage.bucket, &publication.lineage.object)?;
         let _target_owner = os::acquire_metadata_mutation_lease(&target, store.owner.clone()).await;
         if require_target {
-            let raw = store
-                .read_target(publication)
-                .await?
-                .ok_or(DiskError::ConditionalDeleteConflict)?;
-            if !store.target_matches(publication, &raw)? {
-                return Err(DiskError::ConditionalDeleteConflict);
-            }
-            store.sync_target(publication).await?;
+            let expected_publication = publication.clone();
+            let physical_slot = store.physical_slot;
+            let disk_count = store.disk_ids.len();
+            store
+                .read_and_sync_target(publication, move |raw| {
+                    if !super::multipart_source_state::SourceStore::target_matches_for_slot(
+                        &expected_publication,
+                        raw,
+                        physical_slot,
+                        disk_count,
+                    )? {
+                        return Err(DiskError::ConditionalDeleteConflict);
+                    }
+                    Ok(())
+                })
+                .await?;
         }
-        if !matches!(
+        let snapshot = if !matches!(
             state.phase,
             MultipartSourcePhase::LocalTerminalSeal | MultipartSourcePhase::GlobalTerminalCertified
         ) {
             state.capability_sha256 = None;
             state.advance(MultipartSourcePhase::LocalTerminalSeal)?;
-            store.persist(&state).await?;
-        }
-        let current = store.load().await?;
-        let snapshot = current.snapshot.ok_or(DiskError::ConditionalDeleteConflict)?;
+            store.persist_with_source_sync(&state, false).await?
+        } else {
+            loaded.snapshot.ok_or(DiskError::ConditionalDeleteConflict)?
+        };
         if snapshot.head_sha256.is_some()
             || snapshot.capability_sha256.is_some()
             || snapshot.publication.as_ref() != Some(publication)

crates/ecstore/src/disk/local/multipart_source_state.rs

@@ -106,6 +106,7 @@ impl SourceHead {
 }
 
 pub(super) struct LoadedSource {
+    pub root_missing: bool,
     pub state: Option<SourceState>,
     pub head: Option<SourceHead>,
     pub original: Option<Bytes>,
@@ -224,11 +225,31 @@ pub(super) fn read_optional(path: &Path, base: &Path, root: &os::PublicationRoot
         Err(error) => Err(error.into()),
     }
 }
+pub(super) fn read_optional_in_pinned(
+    pinned: &os::ExistingBaseDirectoryGuard,
+    parent: &Path,
+    name: &str,
+    base: &Path,
+    root: &os::PublicationRoot,
+) -> Result<Option<Bytes>> {
+    match os::read_multipart_source_metadata_in_pinned(pinned, parent, std::ffi::OsStr::new(name), base, root) {
+        Ok(bytes) => Ok(Some(Bytes::from(bytes))),
+        Err(error) if error.kind() == std::io::ErrorKind::NotFound => Ok(None),
+        Err(error) => Err(error.into()),
+    }
+}
 /// Canonical source mutations are admitted under SourceRW at the syscall,
 /// not on the earlier async caller that may have timed out.
 pub(super) fn validate_source_mutation(source: &Path, base: &Path, root: &os::PublicationRoot, initial: bool) -> Result<()> {
-    let state = read_optional(&source.join(STATE), base, root)?;
-    let head = read_optional(&source.join(STORAGE_FORMAT_FILE), base, root)?;
+    let pinned = match os::pin_multipart_source_directory(source, base, root) {
+        Ok(pinned) => pinned,
+        Err(error) if error.kind() == std::io::ErrorKind::NotFound => {
+            return if initial { Ok(()) } else { Err(DiskError::FileNotFound) };
+        }
+        Err(error) => return Err(error.into()),
+    };
+    let state = read_optional_in_pinned(&pinned, source, STATE, base, root)?;
+    let head = read_optional_in_pinned(&pinned, source, STORAGE_FORMAT_FILE, base, root)?;
     let Some(head) = head else {
         return if initial && state.is_none() {
             Ok(())
@@ -314,6 +335,48 @@ fn target_backup_path(source: &Path, owner: Uuid) -> PathBuf {
     source.join(".mpu-completions").join(owner.to_string()).join("target.xl.meta")
 }
 
+fn terminal_record_path(source: &Path, state: &SourceState) -> Option<PathBuf> {
+    matches!(
+        state.phase,
+        MultipartSourcePhase::LocalTerminalSeal | MultipartSourcePhase::GlobalTerminalCertified
+    )
+    .then(|| {
+        source
+            .join(".mpu-completions")
+            .join(state.backup_owner.to_string())
+            .join(format!("terminal-{}.msgpack", state.revision))
+    })
+}
+
+fn snapshot_for_state(
+    disk_id: Uuid,
+    state: &SourceState,
+    state_raw: &[u8],
+    head_sha256: Option<[u8; 32]>,
+) -> Result<MultipartSourceSnapshot> {
+    let state_sha256 = raw_sha(state_raw);
+    let inventory = raw_sha(&encode(&(
+        Some(state_sha256),
+        head_sha256,
+        state.original_sha256,
+        state.negative_certificate_sha256,
+        state.ready_certificate_sha256,
+    ))?);
+    Ok(MultipartSourceSnapshot {
+        disk_id,
+        physical_slot: state.physical_slot,
+        lineage: state.lineage.clone(),
+        generation: state.generation,
+        revision: state.revision,
+        phase: state.phase,
+        state_sha256,
+        head_sha256,
+        inventory_sha256: inventory,
+        capability_sha256: state.capability_sha256,
+        publication: state.publication.clone(),
+    })
+}
+
 impl SourceStore {
     pub fn validate_certificate_topology(&self, certificate: &MultipartCertificate) -> Result<()> {
         if certificate.disk_ids != self.disk_ids || certificate.disk_ids.get(self.physical_slot) != Some(&self.disk_id) {
@@ -348,31 +411,72 @@ impl SourceStore {
         .map_err(DiskError::from)?
     }
     pub async fn persist(&self, state: &SourceState) -> Result<()> {
+        self.persist_with_source_sync(state, false).await.map(|_| ())
+    }
+    pub(super) async fn persist_with_source_sync(
+        &self,
+        state: &SourceState,
+        sync_source_before: bool,
+    ) -> Result<MultipartSourceSnapshot> {
+        state.validate(self.disk_id, &self.path)?;
+        let raw = encode(state)?;
+        let snapshot = snapshot_for_state(self.disk_id, state, &raw, None)?;
+        let source = self.source.clone();
+        let base = self.base.clone();
+        let root = self.publication_root.clone();
+        let terminal = terminal_record_path(&source, state);
+        os::run_blocking_namespace_operation_with_sync(self.lease.clone(), true, move || {
+            Self::persist_state_std(&source, &base, &root, &raw, terminal.as_deref(), sync_source_before)
+        })
+        .await
+        .map_err(DiskError::from)?;
+        Ok(snapshot)
+    }
+    fn persist_state_std(
+        source: &Path,
+        base: &Path,
+        root: &os::PublicationRoot,
+        raw: &[u8],
+        terminal: Option<&Path>,
+        sync_source_before: bool,
+    ) -> std::io::Result<()> {
+        #[cfg(unix)]
+        let mut pinned = os::pin_multipart_source_directory(source, base, root)?;
+        #[cfg(not(unix))]
+        let _pinned = os::pin_multipart_source_directory(source, base, root)?;
+        if sync_source_before {
+            #[cfg(unix)]
+            {
+                let opened = std::fs::File::from(pinned.pop().ok_or_else(|| IoError::other("empty source directory guard"))?);
+                os::fsync_directory_handle_std(source, Some(&opened))?;
+            }
+            #[cfg(not(unix))]
+            os::fsync_dir_std(source)?;
+        }
+        if let Some(path) = terminal {
+            os::replace_multipart_source_metadata(path, raw, base, root)?;
+        }
+        os::replace_multipart_source_metadata(&source.join(STATE), raw, base, root)
+    }
+    async fn write_owned_backup_and_persist(&self, owner: Uuid, target: bool, bytes: &[u8], state: &SourceState) -> Result<()> {
+        if owner.is_nil() {
+            return Err(DiskError::FileCorrupt);
+        }
         state.validate(self.disk_id, &self.path)?;
         let raw = encode(state)?;
         let source = self.source.clone();
         let base = self.base.clone();
         let root = self.publication_root.clone();
-        let terminal = matches!(
-            state.phase,
-            MultipartSourcePhase::LocalTerminalSeal | MultipartSourcePhase::GlobalTerminalCertified
-        )
-        .then(|| {
-            source
-                .join(".mpu-completions")
-                .join(state.backup_owner.to_string())
-                .join(format!("terminal-{}.msgpack", state.revision))
-        });
-        os::run_blocking_namespace_operation(self.lease.clone(), move || {
-            let _pin = os::pin_multipart_source_directory(&source, &base, &root)?;
-            if let Some(path) = terminal {
-                os::replace_multipart_source_metadata(&path, &raw, &base, &root)?;
-            }
-            os::replace_multipart_source_metadata(&source.join(STATE), &raw, &base, &root)
+        let terminal = terminal_record_path(&source, state);
+        let bytes = bytes.to_vec();
+        os::run_blocking_namespace_operation_with_sync(self.lease.clone(), true, move || {
+            Self::write_owned_backup_std(&source, &base, &root, owner, target, &bytes)?;
+            Self::persist_state_std(&source, &base, &root, &raw, terminal.as_deref(), false)
         })
         .await
         .map_err(DiskError::from)
     }
+    #[cfg(test)]
     pub async fn original(&self, state: &SourceState) -> Result<Bytes> {
         let path = original_path(&self.source, state.backup_owner);
         let base = self.base.clone();
@@ -421,54 +525,65 @@ impl SourceStore {
             restore_from: None,
             abort: None,
         };
-        self.write_owned_backup(state.backup_owner, false, original).await?;
-        self.persist(&state).await?;
+        self.write_owned_backup_and_persist(state.backup_owner, false, original, &state)
+            .await?;
         Ok(state)
     }
+    #[cfg(test)]
     pub async fn write_owned_backup(&self, owner: Uuid, target: bool, bytes: &[u8]) -> Result<()> {
         if owner.is_nil() {
             return Err(DiskError::FileCorrupt);
         }
-        let path = if target {
-            target_backup_path(&self.source, owner)
-        } else {
-            original_path(&self.source, owner)
-        };
         let source = self.source.clone();
         let base = self.base.clone();
         let root = self.publication_root.clone();
         let bytes = bytes.to_vec();
         os::run_blocking_namespace_operation(self.lease.clone(), move || {
-            let _pin = os::pin_multipart_source_directory(&source, &base, &root)?;
-            let parent = path
-                .parent()
-                .ok_or_else(|| std::io::Error::other("source backup has no parent"))?;
-            let _parents = os::mkdir_all_below_existing_base_std(parent, &base, &root)?;
-            let existed = if target {
-                match os::observe_multipart_source_metadata(&path, &base, &root, bytes.len() as u64) {
-                    Ok((length, sha)) if length == bytes.len() as u64 && sha == raw_sha(&bytes) => true,
-                    Ok(_) => {
-                        return Err(IoError::new(std::io::ErrorKind::AlreadyExists, "immutable target backup changed"));
-                    }
-                    Err(error) if error.kind() == std::io::ErrorKind::NotFound => false,
-                    Err(error) => return Err(error),
-                }
-            } else if let Some(previous) = read_optional(&path, &base, &root).map_err(IoError::other)? {
-                if previous.as_ref() != bytes.as_slice() {
-                    return Err(IoError::new(std::io::ErrorKind::AlreadyExists, "immutable source backup changed"));
-                }
-                true
-            } else {
-                false
-            };
-            if !existed {
-                os::replace_multipart_source_metadata(&path, &bytes, &base, &root)?;
-            }
-            os::sync_multipart_owned_record_parents(&source, parent, existed, &base, &root)
+            Self::write_owned_backup_std(&source, &base, &root, owner, target, &bytes)
         })
         .await
         .map_err(DiskError::from)
     }
+    fn write_owned_backup_std(
+        source: &Path,
+        base: &Path,
+        root: &os::PublicationRoot,
+        owner: Uuid,
+        target: bool,
+        bytes: &[u8],
+    ) -> std::io::Result<()> {
+        let path = if target {
+            target_backup_path(source, owner)
+        } else {
+            original_path(source, owner)
+        };
+        let _pin = os::pin_multipart_source_directory(source, base, root)?;
+        let parent = path
+            .parent()
+            .ok_or_else(|| std::io::Error::other("source backup has no parent"))?;
+        let _parents = os::mkdir_all_below_existing_base_std(parent, base, root)?;
+        let existed = if target {
+            match os::observe_multipart_source_metadata(&path, base, root, bytes.len() as u64) {
+                Ok((length, sha)) if length == bytes.len() as u64 && sha == raw_sha(bytes) => true,
+                Ok(_) => {
+                    return Err(IoError::new(std::io::ErrorKind::AlreadyExists, "immutable target backup changed"));
+                }
+                Err(error) if error.kind() == std::io::ErrorKind::NotFound => false,
+                Err(error) => return Err(error),
+            }
+        } else if let Some(previous) = read_optional(&path, base, root).map_err(IoError::other)? {
+            if previous.as_ref() != bytes {
+                return Err(IoError::new(std::io::ErrorKind::AlreadyExists, "immutable source backup changed"));
+            }
+            true
+        } else {
+            false
+        };
+        if !existed {
+            os::replace_multipart_source_metadata(&path, bytes, base, root)?;
+        }
+        os::sync_multipart_owned_record_parents(source, parent, existed, base, root)
+    }
     pub async fn replace_head(&self, bytes: &[u8]) -> Result<()> {
         let path = self.source.join(STORAGE_FORMAT_FILE);
         let base = self.base.clone();
@@ -490,7 +605,7 @@ impl SourceStore {
         if encode(actual)? != encode(state)? {
             return Err(DiskError::ConditionalDeleteConflict);
         }
-        let raw = self.original(state).await?;
+        let raw = loaded.original.as_ref().ok_or(DiskError::FileCorrupt)?;
         let allowed = match loaded.head.as_ref() {
             None => {
                 state.phase == MultipartSourcePhase::Recovering
@@ -551,7 +666,7 @@ impl SourceStore {
         if !allowed {
             return Err(DiskError::ConditionalDeleteConflict);
         }
-        let mut info = file_info(&raw, RUSTFS_META_MULTIPART_BUCKET, &self.path, state.lineage.original_version_id)?;
+        let mut info = file_info(raw, RUSTFS_META_MULTIPART_BUCKET, &self.path, state.lineage.original_version_id)?;
         strip_native_controls(&mut info.metadata);
         mark_native_source(&mut info);
         set_source_status(
@@ -649,6 +764,14 @@ impl SourceStore {
         Ok(metadata.map(|_| bytes))
     }
     pub fn target_matches(&self, publication: &PublicationIdentity, raw: &[u8]) -> Result<bool> {
+        Self::target_matches_for_slot(publication, raw, self.physical_slot, self.disk_ids.len())
+    }
+    pub(super) fn target_matches_for_slot(
+        publication: &PublicationIdentity,
+        raw: &[u8],
+        physical_slot: usize,
+        disk_count: usize,
+    ) -> Result<bool> {
         let info = file_info(raw, &publication.lineage.bucket, &publication.lineage.object, publication.version_id)?;
         let slot = publication
             .lineage
@@ -670,8 +793,8 @@ impl SourceStore {
             && info.erasure.parity_blocks == publication.lineage.geometry.parity_blocks
             && info.erasure.block_size == publication.lineage.geometry.block_size
             && info.erasure.distribution == publication.lineage.geometry.distribution
-            && slot == self.physical_slot
-            && publication.lineage.geometry.disks() == self.disk_ids.len())
+            && slot == physical_slot
+            && publication.lineage.geometry.disks() == disk_count)
     }
     pub async fn restore_exact_target(&self, state: &SourceState, owner: Option<Arc<dyn Send + Sync>>) -> Result<()> {
         let publication = state.publication.as_ref().ok_or(DiskError::FileCorrupt)?;
@@ -772,13 +895,32 @@ pub(super) fn load_source(
     physical_slot: usize,
     disk_ids: &[Uuid],
 ) -> Result<LoadedSource> {
-    let state_raw = read_optional(&source.join(STATE), base, root)?;
+    let pinned = match os::pin_multipart_source_directory(source, base, root) {
+        Ok(pinned) => pinned,
+        Err(error) if error.kind() == std::io::ErrorKind::NotFound => {
+            return Ok(LoadedSource {
+                root_missing: true,
+                state: None,
+                head: None,
+                original: None,
+                info: None,
+                snapshot: None,
+                head_matches_state: false,
+                head_matches_activation: false,
+            });
+        }
+        Err(error) => return Err(error.into()),
+    };
+    let state_raw = read_optional_in_pinned(&pinned, source, STATE, base, root)?;
     let state: Option<SourceState> = state_raw.as_deref().map(decode).transpose()?;
-    if state.is_none() && os::multipart_source_has_native_controls(source, base, root)? {
+    if state.is_none() && os::multipart_source_has_native_controls_in_pinned(source, &pinned)? {
         return Err(DiskError::ConditionalDeleteConflict);
     }
-    let head_path = source.join(STORAGE_FORMAT_FILE);
-    let head = if let Some(state) = state.as_ref() {
+    let head = if let Some(state) = state
+        .as_ref()
+        .filter(|state| state.prepared_length.is_some_and(|len| len > 64 * 1024))
+    {
+        let head_path = source.join(STORAGE_FORMAT_FILE);
         let maximum = state
             .prepared_length
             .unwrap_or(0)
@@ -789,7 +931,8 @@ pub(super) fn load_source(
                 let raw = if is_proxy {
                     None
                 } else {
-                    let raw = read_optional(&head_path, base, root)?.ok_or(DiskError::FileCorrupt)?;
+                    let raw = read_optional_in_pinned(&pinned, source, STORAGE_FORMAT_FILE, base, root)?
+                        .ok_or(DiskError::FileCorrupt)?;
                     if raw_sha(&raw) != sha256 {
                         return Err(DiskError::FileCorrupt);
                     }
@@ -801,10 +944,17 @@ pub(super) fn load_source(
             Err(error) => return Err(error.into()),
         }
     } else {
-        read_optional(&head_path, base, root)?.map(|raw| SourceHead {
-            sha256: raw_sha(&raw),
-            length: raw.len() as u64,
-            raw: Some(raw),
+        read_optional_in_pinned(&pinned, source, STORAGE_FORMAT_FILE, base, root)?.map(|raw| {
+            let sha256 = raw_sha(&raw);
+            let length = raw.len() as u64;
+            let is_proxy = state
+                .as_ref()
+                .is_some_and(|state| Some(sha256) == state.prepared_sha256 && Some(length) == state.prepared_length);
+            SourceHead {
+                sha256,
+                length,
+                raw: (!is_proxy).then_some(raw),
+            }
         })
     };
     let mut original = None;
@@ -870,26 +1020,12 @@ pub(super) fn load_source(
             }
         }
         let head_sha = head.as_ref().map(|head| head.sha256);
-        let inventory = raw_sha(&encode(&(
-            state_raw.as_ref().map(|bytes| raw_sha(bytes)),
-            head_sha,
-            state.original_sha256,
-            state.negative_certificate_sha256,
-            state.ready_certificate_sha256,
-        ))?);
-        snapshot = Some(MultipartSourceSnapshot {
+        snapshot = Some(snapshot_for_state(
             disk_id,
-            physical_slot: state.physical_slot,
-            lineage: state.lineage.clone(),
-            generation: state.generation,
-            revision: state.revision,
-            phase: state.phase,
-            state_sha256: raw_sha(state_raw.as_ref().ok_or(DiskError::FileCorrupt)?),
-            head_sha256: head_sha,
-            inventory_sha256: inventory,
-            capability_sha256: state.capability_sha256,
-            publication: state.publication.clone(),
-        });
+            state,
+            state_raw.as_ref().ok_or(DiskError::FileCorrupt)?,
+            head_sha,
+        )?);
     } else if let Some(observed) = &head {
         let raw = observed.raw().ok_or(DiskError::FileCorrupt)?;
         let current = get_file_info(
@@ -950,6 +1086,7 @@ pub(super) fn load_source(
         head_matches_state = true;
     }
     Ok(LoadedSource {
+        root_missing: false,
         state,
         head,
         original,
@@ -1195,11 +1332,7 @@ impl SourceStore {
         directory: &Path,
         lease: Arc<os::NamespaceMutationLease>,
     ) -> Result<()> {
-        let loaded = self.load().await?;
-        let current = loaded.state.as_ref().ok_or(DiskError::ConditionalDeleteConflict)?;
-        if encode(current)? != encode(state)?
-            || !loaded.head_matches_state
-            || state.lineage != publication.lineage
+        if state.lineage != publication.lineage
             || state.generation != publication.source_generation
             || state.disk_id != self.disk_id
             || state.physical_slot != self.physical_slot
@@ -1436,8 +1569,7 @@ impl LocalDisk {
         };
         // Copy the small source FI to an independent immutable inode. Its
         // canonical inode is never truncated to build the destination proxy.
-        store.write_owned_backup(owner, false, original).await?;
-        store.persist(&state).await?;
+        store.write_owned_backup_and_persist(owner, false, original, &state).await?;
         let prepared_path = store.source.join(format!(".mpu-prepared-{owner}"));
         let context = NativeMultipartPublication {
             store,
@@ -1480,14 +1612,19 @@ impl NativeMultipartPublication {
             length: new_metadata.len() as u64,
         };
         if let Some(previous) = old_metadata {
-            self.store.write_owned_backup(state.backup_owner, true, previous).await?;
             state.previous_target_sha256 = Some(raw_sha(previous));
             state.previous_target_length = Some(previous.len() as u64);
         }
         state.prepared_sha256 = Some(validated.sha256);
         state.prepared_length = Some(validated.length);
         state.advance(MultipartSourcePhase::Prepared)?;
-        self.store.persist(&state).await?;
+        if let Some(previous) = old_metadata {
+            self.store
+                .write_owned_backup_and_persist(state.backup_owner, true, previous, &state)
+                .await?;
+        } else {
+            self.store.persist(&state).await?;
+        }
         *self.state.lock() = state;
         *self.validated_prepared.lock() = Some(validated);
         Ok(())
@@ -1569,31 +1706,28 @@ impl NativeMultipartPublication {
         if current.head.is_some() || encode(current_state)? != encode(&state)? {
             return Err(DiskError::ConditionalDeleteConflict);
         }
-        let target = self
-            .store
-            .read_target(&self.publication)
-            .await?
-            .ok_or(DiskError::ConditionalDeleteConflict)?;
-        {
+        let (expected_sha256, expected_length) = {
             let guard = self.validated_prepared.lock();
             let validated = guard.as_ref().ok_or(DiskError::ConditionalDeleteConflict)?;
             if validated.publication != self.publication
                 || state.prepared_sha256 != Some(validated.sha256)
                 || state.prepared_length != Some(validated.length)
-                || target.len() as u64 != validated.length
-                || raw_sha(&target) != validated.sha256
             {
                 return Err(DiskError::ConditionalDeleteConflict);
             }
-        }
-        self.store.sync_target_with_lease(&self.publication, target_lease).await?;
-        let source = self.store.source.clone();
-        os::run_blocking_namespace_operation(self.store.lease.clone(), move || os::fsync_dir_std(&source)).await?;
+            (validated.sha256, validated.length)
+        };
+        self.store
+            .read_and_sync_target_with_lease(&self.publication, target_lease, move |target| {
+                if target.len() as u64 != expected_length || raw_sha(target) != expected_sha256 {
+                    return Err(DiskError::ConditionalDeleteConflict);
+                }
+                Ok(())
+            })
+            .await?;
         state.advance(MultipartSourcePhase::ConsumedUnsealed)?;
-        self.store.persist(&state).await?;
+        let snapshot = self.store.persist_with_source_sync(&state, true).await?;
         *self.state.lock() = state;
-        let loaded = self.store.load().await?;
-        let snapshot = loaded.snapshot.ok_or(DiskError::FileCorrupt)?;
         let proof = self
             .store
             .issue_proof(MultipartProofKind::Published, self.publication.attempt, None, snapshot)?;
@@ -1632,6 +1766,46 @@ impl SourceStore {
         .await
         .map_err(DiskError::from)
     }
+    pub async fn read_and_sync_target(
+        &self,
+        publication: &PublicationIdentity,
+        validate: impl FnOnce(&[u8]) -> Result<()> + Send + 'static,
+    ) -> Result<()> {
+        let io = self.target_path(publication)?;
+        let logical = self
+            .root
+            .join(io.strip_prefix(&self.base).map_err(|_| DiskError::InvalidPath)?);
+        let lease = os::acquire_namespace_mutation_lease_with_owner(&logical, Some(self.lease.clone())).await;
+        self.read_and_sync_target_with_lease(publication, lease, validate).await
+    }
+    pub async fn read_and_sync_target_with_lease(
+        &self,
+        publication: &PublicationIdentity,
+        lease: Arc<os::NamespaceMutationLease>,
+        validate: impl FnOnce(&[u8]) -> Result<()> + Send + 'static,
+    ) -> Result<()> {
+        let directory = self.target_path(publication)?;
+        let base = self.base.clone();
+        let root = self.publication_root.clone();
+        os::run_blocking_namespace_operation_with_sync(lease, true, move || {
+            os::read_and_sync_multipart_target_metadata(
+                &directory,
+                std::ffi::OsStr::new(STORAGE_FORMAT_FILE),
+                &base,
+                &root,
+                |bytes| validate(bytes).map_err(IoError::other),
+            )
+            .map_err(|error| {
+                if error.kind() == std::io::ErrorKind::NotFound {
+                    IoError::other(DiskError::ConditionalDeleteConflict)
+                } else {
+                    error
+                }
+            })
+        })
+        .await
+        .map_err(DiskError::from)
+    }
     async fn sync_source_absence(&self) -> Result<()> {
         let source = self.source.clone();
         let base = self.base.clone();

crates/ecstore/src/disk/local/multipart_staging.rs

@@ -411,15 +411,23 @@ impl LocalDisk {
             }
             let _attempt = os::mkdir_all_below_existing_base_std(&private_object, &base, &publication_root)?;
             if incoming.data.is_none() && !incoming.is_remote() {
+                let source_data_dir = source_object.join(source_dir.to_string());
+                let pinned_source_data = (!incoming.parts.is_empty())
+                    .then(|| os::pin_multipart_source_directory(&source_data_dir, &base, &publication_root))
+                    .transpose()?;
                 let mut names = Vec::with_capacity(incoming.parts.len().saturating_mul(3));
                 for part in &incoming.parts {
                     if cancelled() {
                         return Err(std::io::Error::new(std::io::ErrorKind::Interrupted, "multipart staging cancelled"));
                     }
-                    let path = source_object
-                        .join(source_dir.to_string())
-                        .join(format!("part.{}.meta", part.number));
-                    let bytes = os::read_multipart_source_metadata(&path, &base, &publication_root)?;
+                    let meta_name = format!("part.{}.meta", part.number);
+                    let bytes = os::read_multipart_source_metadata_in_pinned(
+                        pinned_source_data.as_ref().expect("pinned source data directory"),
+                        &source_data_dir,
+                        std::ffi::OsStr::new(&meta_name),
+                        &base,
+                        &publication_root,
+                    )?;
                     let actual = rustfs_filemeta::ObjectPartInfo::unmarshal(&bytes).map_err(std::io::Error::other)?;
                     let normalized = i64::try_from(actual.size)
                         .map_err(std::io::Error::other)?
@@ -441,13 +449,13 @@ impl LocalDisk {
                     // Preserve the native completed-MPU directory format. This
                     // immutable, validated generation record is distinct from
                     // the upload's mutable top-level xl.meta.
-                    names.push(format!("part.{}.meta", part.number));
+                    names.push(meta_name);
                     if let Some(integrity) = &part.integrity {
                         names.push(integrity.file_name());
                     }
                 }
                 os::hardlink_multipart_source_files(
-                    &source_object.join(source_dir.to_string()),
+                    &source_data_dir,
                     &private_object.join(final_dir.to_string()),
                     &names,
                     &base,

crates/ecstore/src/disk/mod.rs

@@ -613,13 +613,6 @@ impl DiskAPI for Disk {
         }
     }
 
-    async fn bucket_creation_witness(&self, volume: &str) -> Result<Option<os::NativeBucketCreationWitness>> {
-        match self {
-            Disk::Local(local_disk) => local_disk.bucket_creation_witness(volume).await,
-            Disk::Remote(_) => Ok(None),
-        }
-    }
-
     #[tracing::instrument(level = "trace", skip_all)]
     async fn make_volume(&self, volume: &str) -> Result<()> {
         match self {
@@ -1477,11 +1470,6 @@ pub trait DiskAPI: Debug + Send + Sync + 'static {
     // NSScanner
 
     // Volume operations.
-    /// Unknown or remote backends cannot attest process-local publication history.
-    async fn bucket_creation_witness(&self, _volume: &str) -> Result<Option<os::NativeBucketCreationWitness>> {
-        Ok(None)
-    }
-
     async fn make_volume(&self, volume: &str) -> Result<()>;
     async fn make_volumes(&self, volume: Vec<&str>) -> Result<()>;
     async fn list_volumes(&self) -> Result<Vec<VolumeInfo>>;
@@ -2257,7 +2245,7 @@ impl DeleteRecoveryJournal {
             }
             let id = request.version_id.unwrap_or_default();
             let Some(original) = originals.get(&id) else {
-                return !id.is_nil() && !locals.contains_key(&id);
+                return false;
             };
             match locals.get(&id) {
                 Some(local) => local == original,
@@ -2307,10 +2295,7 @@ impl DeleteRecoveryJournal {
             };
             match (local, remote) {
                 (Some(local), Some(remote)) => local == remote,
-                (None, None) => {
-                    (!(request.deleted && request.mark_deleted) && request.version_id.is_some_and(|id| !id.is_nil()))
-                        || (self.original_digest.is_none() && authority.original_digest.is_none())
-                }
+                (None, None) => self.original_digest.is_none() && authority.original_digest.is_none(),
                 _ => false,
             }
         })
@@ -3635,46 +3620,6 @@ mod delete_recovery_adoption_tests {
         };
         assert!(journal.matches_adoption_source(&[]), "certified non-null removal can already be absent");
         assert!(journal.matches_adoption_source(std::slice::from_ref(&source)));
-        let missing = Uuid::new_v4();
-        journal.requests.push(FileInfo {
-            version_id: Some(missing),
-            ..Default::default()
-        });
-        journal.original_digest = Some([3; 32]);
-        assert!(
-            journal.matches_prepared_authority(&journal),
-            "an explicit UUID absent from both prepared sources remains a no-op with original metadata"
-        );
-        assert!(journal.matches_adoption_source(&[]), "both non-null versions can remain absent");
-        assert!(
-            journal.matches_adoption_source(std::slice::from_ref(&source)),
-            "a missing non-null version in the batch remains a no-op"
-        );
-        let mut unexpected = source.clone();
-        unexpected.version_id = Some(missing);
-        let mut foreign_preparation = journal.clone();
-        foreign_preparation.original_versions.push(unexpected.clone());
-        assert!(
-            !journal.matches_prepared_authority(&foreign_preparation)
-                && !foreign_preparation.matches_prepared_authority(&journal),
-            "absence from only one prepared source cannot establish a generation"
-        );
-        assert!(
-            !journal.matches_adoption_source(&[source.clone(), unexpected]),
-            "a version absent from the authority cannot be deleted locally"
-        );
-        for version_id in [None, Some(Uuid::nil())] {
-            journal.requests.last_mut().expect("missing-version request").version_id = version_id;
-            assert!(
-                !journal.matches_prepared_authority(&journal),
-                "a missing null version with original metadata cannot establish its generation"
-            );
-            assert!(
-                !journal.matches_adoption_source(std::slice::from_ref(&source)),
-                "a missing null version cannot establish its generation"
-            );
-        }
-        journal.requests.pop();
         let mut foreign = source.clone();
         foreign.metadata_digest = [2; 32];
         assert!(!journal.matches_adoption_source(std::slice::from_ref(&foreign)));
@@ -3690,25 +3635,6 @@ mod delete_recovery_adoption_tests {
         assert!(journal.matches_adoption_source(std::slice::from_ref(&journal.original_versions[0])));
         journal.requests.clear();
         assert!(!journal.matches_adoption_source(&journal.original_versions));
-
-        let mut marker = journal.clone();
-        marker.requests = vec![FileInfo {
-            version_id: Some(id),
-            deleted: true,
-            mark_deleted: true,
-            ..Default::default()
-        }];
-        marker.original_versions.clear();
-        marker.original_digest = Some([3; 32]);
-        assert!(
-            !marker.matches_prepared_authority(&marker),
-            "a marker with original metadata requires a source generation"
-        );
-        marker.original_digest = None;
-        assert!(
-            marker.matches_prepared_authority(&marker),
-            "marker preparation without original metadata retains the existing allowance"
-        );
     }
 }
 

crates/ecstore/src/error/mod.rs

@@ -17,8 +17,8 @@
 use crate::bucket::error::BucketMetadataError;
 use crate::disk::error::DiskError;
 use crate::storage_api_contracts::{error::StorageErrorCode, range::HTTPRangeError};
-use rustfs_s3_types::S3ErrorCode;
 use rustfs_utils::path::decode_dir_object;
+use s3s::S3ErrorCode;
 
 pub type Error = StorageError;
 pub type Result<T> = core::result::Result<T, Error>;
@@ -1086,8 +1086,14 @@ impl From<rmp_serde::decode::Error> for Error {
     }
 }
 
-impl From<crate::bucket::utils::BucketConfigXmlError> for Error {
-    fn from(e: crate::bucket::utils::BucketConfigXmlError) -> Self {
+impl From<s3s::xml::SerError> for Error {
+    fn from(e: s3s::xml::SerError) -> Self {
+        Error::other(e)
+    }
+}
+
+impl From<s3s::xml::DeError> for Error {
+    fn from(e: s3s::xml::DeError) -> Self {
         Error::other(e)
     }
 }

crates/ecstore/src/object_api/object_api_utils.rs

@@ -20,6 +20,7 @@
 #![allow(clippy::all)]
 
 use http::HeaderMap;
+use s3s::dto::ETag;
 use std::{collections::HashMap, io::Cursor, sync::Arc};
 use tokio::io::BufReader;
 
@@ -28,7 +29,7 @@ use crate::object_api::{GetObjectReader, ObjectInfo, ObjectOptions};
 use crate::storage_api_contracts::range::HTTPRangeSpec;
 use rustfs_filemeta::ObjectPartInfo;
 use rustfs_rio::HashReader;
-use rustfs_s3_types::S3ErrorCode;
+use s3s::S3ErrorCode;
 
 //#[derive(Clone)]
 pub struct PutObjReader {
@@ -149,6 +150,24 @@ pub fn new_getobjectreader<'a>(
     })
 }
 
+/// Convert a raw stored ETag into the strongly-typed `s3s::dto::ETag`.
+///
+/// Supports already quoted (`"abc"`), weak (`W/"abc"`), or plain (`abc`) values.
+pub fn to_s3s_etag(etag: &str) -> ETag {
+    if let Some(rest) = etag.strip_prefix("W/\"") {
+        if let Some(body) = rest.strip_suffix('"') {
+            return ETag::Weak(body.to_string());
+        }
+        return ETag::Weak(rest.to_string());
+    }
+
+    if let Some(body) = etag.strip_prefix('"').and_then(|rest| rest.strip_suffix('"')) {
+        return ETag::Strong(body.to_string());
+    }
+
+    ETag::Strong(etag.to_string())
+}
+
 pub fn get_raw_etag(metadata: &HashMap<String, String>) -> String {
     metadata
         .get("etag")
@@ -213,42 +232,28 @@ mod tests {
     }
 
     #[test]
-    fn test_new_getobjectreader_rejects_a_range_past_the_object_as_invalid_range() {
-        let oi = multipart_object_info();
-        let past_the_end = Some(HTTPRangeSpec {
-            is_suffix_length: false,
-            start: oi.size,
-            end: oi.size + 10,
-        });
+    fn test_to_s3s_etag() {
+        // Test unquoted ETag - should become strong etag
+        assert_eq!(
+            to_s3s_etag("6af8d12c0c74b78094884349f3c8a079"),
+            ETag::Strong("6af8d12c0c74b78094884349f3c8a079".to_string())
+        );
 
-        let err = new_getobjectreader(&past_the_end, &oi, &ObjectOptions::default(), &HeaderMap::new())
-            .err()
-            .expect("a range starting at the object size must be rejected");
+        assert_eq!(
+            to_s3s_etag("\"6af8d12c0c74b78094884349f3c8a079\""),
+            ETag::Strong("6af8d12c0c74b78094884349f3c8a079".to_string())
+        );
 
-        assert_eq!(err.code, S3ErrorCode::InvalidRange);
-        assert_eq!(err.code.as_str(), "InvalidRange");
-        assert_eq!(err.key, None);
-        assert_eq!(err.bucket_name, None);
-    }
+        assert_eq!(
+            to_s3s_etag("W/\"6af8d12c0c74b78094884349f3c8a079\""),
+            ETag::Weak("6af8d12c0c74b78094884349f3c8a079".to_string())
+        );
 
-    #[test]
-    fn test_new_getobjectreader_rejects_an_unknown_size_without_a_range_as_invalid_range() {
-        let oi = ObjectInfo {
-            bucket: "bucket".to_string(),
-            name: "object".to_string(),
-            size: -1,
-            ..Default::default()
-        };
+        assert_eq!(to_s3s_etag(""), ETag::Strong(String::new()));
 
-        let err = new_getobjectreader(&None, &oi, &ObjectOptions::default(), &HeaderMap::new())
-            .err()
-            .expect("an object of unknown size cannot be read without a range");
+        assert_eq!(to_s3s_etag("\"incomplete"), ETag::Strong("\"incomplete".to_string()));
 
-        assert_eq!(err.code, S3ErrorCode::InvalidRange);
-        assert_eq!(err.code.as_str(), "InvalidRange");
-        assert_eq!(err.message, "Invalid range");
-        assert_eq!(err.key.as_deref(), Some("object"));
-        assert_eq!(err.bucket_name.as_deref(), Some("bucket"));
+        assert_eq!(to_s3s_etag("incomplete\""), ETag::Strong("incomplete\"".to_string()));
     }
 
     #[test]

crates/ecstore/src/object_api/types.rs

@@ -1013,10 +1013,6 @@ pub struct ObjectOptions {
     /// `ecstore` must leave it unset.
     #[doc(hidden)]
     pub data_movement_transferred_tier_versions: Option<crate::data_movement::TransferredTierVersions>,
-    /// Marks the physical copy-back write of a RestoreObject operation. This
-    /// is in-memory request context and must not be copied into object metadata.
-    #[doc(hidden)]
-    pub restore_copyback: bool,
     /// Internal, in-memory protection context. Never populated from S3 metadata.
     /// None selects the rollout default only for a new write, not for a rewrite.
     #[doc(hidden)]

crates/ecstore/src/runtime/global.rs

@@ -275,16 +275,16 @@ pub(crate) type TypeLocalDiskSetDrives = Vec<Vec<Vec<Option<DiskStore>>>>;
 ///
 /// # Returns
 /// * None
-pub fn set_global_region(region: rustfs_s3_types::Region) {
+pub fn set_global_region(region: s3s::region::Region) {
     current_ctx().set_region(region);
 }
 
 /// Get the global region
 ///
 /// # Returns
-/// * `Option<rustfs_s3_types::Region>` - The global region, if set
+/// * `Option<s3s::region::Region>` - The global region, if set
 ///
-pub fn get_global_region() -> Option<rustfs_s3_types::Region> {
+pub fn get_global_region() -> Option<s3s::region::Region> {
     current_ctx().region()
 }
 

crates/ecstore/src/runtime/instance.rs

@@ -52,7 +52,7 @@ use crate::services::event_notification::EventNotifier;
 use crate::services::tier::tier::TierConfigMgr;
 use crate::store::delete_recovery_runtime::DeleteRecoveryHints;
 use rustfs_lock::{GlobalLockManager, get_global_lock_manager};
-use rustfs_s3_types::Region;
+use s3s::region::Region;
 use std::collections::{HashMap, HashSet};
 use std::sync::{
     Arc, OnceLock,

crates/ecstore/src/runtime/sources.rs

@@ -44,10 +44,10 @@ use crate::{
 };
 use rustfs_concurrency::WorkloadAdmissionSnapshotProvider;
 use rustfs_config::server_config::{Config, get_global_server_config, set_global_server_config};
-use rustfs_gateway_types::persistence::PersistedLifecycleConfiguration;
 use rustfs_io_metrics::internode_metrics::global_internode_metrics;
 use rustfs_lock::client::LockClient;
-use rustfs_s3_types::Region;
+use s3s::dto::BucketLifecycleConfiguration;
+use s3s::region::Region;
 use tokio::sync::{OwnedRwLockReadGuard, RwLock};
 use tokio_util::sync::CancellationToken;
 #[cfg(test)]
@@ -371,7 +371,7 @@ pub fn global_tier_config_mgr() -> Arc<RwLock<TierConfigMgr>> {
     get_global_tier_config_mgr()
 }
 
-pub(crate) async fn bucket_lifecycle_config(bucket: &str) -> Option<PersistedLifecycleConfiguration> {
+pub(crate) async fn bucket_lifecycle_config(bucket: &str) -> Option<BucketLifecycleConfiguration> {
     GLOBAL_LIFECYCLE_SYS.get(bucket).await
 }
 

crates/ecstore/src/services/notification_sys.rs

@@ -703,28 +703,6 @@ pub async fn acquire_legacy_transition_state_reconcile_fleet_proof() -> Option<L
     .then_some(proof)
 }
 
-pub async fn require_restore_copyback_fleet_capability() -> Result<()> {
-    let Some(topology) = REMOTE_VERSION_STATE_PROBE_TOPOLOGY.get() else {
-        #[cfg(any(test, feature = "test-util"))]
-        return Ok(());
-        #[cfg(not(any(test, feature = "test-util")))]
-        return Err(Error::other("restore copy-back capability topology is unavailable"));
-    };
-    let Some(notification_sys) = get_global_notification_sys() else {
-        #[cfg(any(test, feature = "test-util"))]
-        return Ok(());
-        #[cfg(not(any(test, feature = "test-util")))]
-        return Err(Error::other("restore copy-back capability probe is unavailable"));
-    };
-    timeout(
-        REMOTE_VERSION_STATE_PROBE_TIMEOUT,
-        notification_sys.probe_restore_copyback_fleet(topology),
-    )
-    .await
-    .map_err(|_| Error::other("restore copy-back capability fleet probe timed out"))??;
-    Ok(())
-}
-
 fn acquire_legacy_transition_state_reconcile_fleet_proof_from(
     state: &FleetCapabilityProofState,
     expected_topology: &str,
@@ -1782,30 +1760,6 @@ impl NotificationSys {
         Ok(peer_epochs)
     }
 
-    async fn probe_restore_copyback_fleet(&self, topology_fingerprint: &str) -> Result<BTreeMap<String, Uuid>> {
-        if self.peer_clients.len() != self.peer_topology_hosts.len() {
-            return Err(Error::other("restore copy-back capability fleet membership is incomplete"));
-        }
-        let probes = self.peer_clients.iter().map(|client| async {
-            let client = client
-                .as_ref()
-                .ok_or_else(|| Error::other("restore copy-back capability peer is unreachable"))?;
-            client
-                .probe_restore_copyback_capability(topology_fingerprint.to_string())
-                .await
-        });
-        let mut peer_epochs = BTreeMap::new();
-        for result in join_all(probes).await {
-            let (peer, epoch) = result?;
-            insert_remote_version_state_peer(&mut peer_epochs, peer, epoch)?;
-        }
-        let expected = self.peer_topology_hosts.iter().cloned().collect::<BTreeSet<_>>();
-        if expected.len() != self.peer_topology_hosts.len() || peer_epochs.keys().ne(expected.iter()) {
-            return Err(Error::other("restore copy-back capability fleet membership does not match topology"));
-        }
-        Ok(peer_epochs)
-    }
-
     async fn probe_transition_transaction_compaction_fleet(&self, topology_fingerprint: &str) -> Result<BTreeMap<String, Uuid>> {
         if self.peer_clients.len() != self.peer_topology_hosts.len() {
             return Err(Error::other(

crates/ecstore/src/services/rebalance/control.rs

@@ -1,8 +1,8 @@
 use super::meta::{
-    RebalanceMetaMergeOutcome, clone_first_arc, clone_rebalance_pool_stats, defer_bucket_in_rebalance_queue,
-    ensure_valid_rebalance_pool_index, invalid_rebalance_pool_index_error, is_rebalance_actively_running,
-    is_rebalance_conflicting_with_decommission, mark_rebalance_bucket_done, merge_rebalance_meta, percent_free_ratio,
-    rebalance_metadata_not_initialized_error, record_rebalance_cleanup_warning_in_meta,
+    RebalanceMetaMergeOutcome, RebalanceMetadataLoadError, clone_first_arc, clone_rebalance_pool_stats,
+    defer_bucket_in_rebalance_queue, ensure_valid_rebalance_pool_index, invalid_rebalance_pool_index_error,
+    is_rebalance_actively_running, is_rebalance_conflicting_with_decommission, mark_rebalance_bucket_done, merge_rebalance_meta,
+    percent_free_ratio, rebalance_metadata_not_initialized_error, record_rebalance_cleanup_warning_in_meta,
     record_rebalance_stop_propagation_snapshot, resolve_next_rebalance_bucket, rollback_rebalance_start_meta_snapshot_for_id,
     should_accept_rebalance_stats_update, should_pool_participate, stop_rebalance_meta_snapshot_for_id,
     validate_init_rebalance_state,
@@ -853,6 +853,26 @@ impl ECStore {
         Ok(snapshot)
     }
 
+    /// Admin-only rebalance status snapshot. Diagnostic reads may use the last
+    /// published state when the persisted record is temporarily unavailable;
+    /// mutation paths continue to use the authoritative loaders above.
+    pub async fn rebalance_admin_diagnostic_snapshot(&self) -> (Option<RebalanceMeta>, &'static str) {
+        let fallback = self.rebalance_meta.read().await.clone();
+        match self.rebalance_status_meta_snapshot().await {
+            Ok(snapshot) => (snapshot, "fresh"),
+            Err(err) => {
+                let status = if error_chain_has_rebalance_metadata_load_error(&err) {
+                    "recovery_required"
+                } else if fallback.is_some() {
+                    "stale"
+                } else {
+                    "unknown"
+                };
+                (fallback, status)
+            }
+        }
+    }
+
     /// Persisted rebalance metadata, or `None` when no rebalance was recorded.
     async fn load_rebalance_status_meta(&self) -> Result<Option<RebalanceMeta>> {
         let pool = clone_first_arc(&self.pools, "refresh_rebalance_status_meta: no pools available")?;
@@ -860,7 +880,7 @@ impl ECStore {
         match persisted.load(pool).await {
             Ok(()) => Ok(Some(persisted)),
             Err(Error::ConfigNotFound) => Ok(None),
-            Err(err) => Err(Error::other(format!("rebalance metadata refresh failed during pool status: {err}"))),
+            Err(err) => Err(Error::other_with_context("rebalance metadata refresh failed during pool status", err)),
         }
     }
 
@@ -1447,6 +1467,21 @@ impl ECStore {
     }
 }
 
+fn error_chain_has_rebalance_metadata_load_error(err: &(dyn std::error::Error + 'static)) -> bool {
+    let mut current = Some(err);
+    while let Some(error) = current {
+        if error.downcast_ref::<RebalanceMetadataLoadError>().is_some() {
+            return true;
+        }
+        current = if let Some(io) = error.downcast_ref::<std::io::Error>() {
+            io.get_ref().map(|source| source as &(dyn std::error::Error + 'static))
+        } else {
+            error.source()
+        };
+    }
+    false
+}
+
 #[cfg(test)]
 mod tests {
     use super::*;

crates/ecstore/src/services/rebalance/entry.rs

@@ -1032,9 +1032,8 @@ mod tests {
     use crate::storage_api_contracts::object::ObjectIO as _;
     use http::HeaderMap;
     use rustfs_filemeta::{FileInfo, FileMeta, ObjectPartInfo, TransitionVersionState};
-    use rustfs_gateway_types::persistence::{
-        PersistedLifecycleConfiguration, PersistedLifecycleExpiration, PersistedLifecycleRule,
-        PersistedNoncurrentVersionExpiration,
+    use s3s::dto::{
+        BucketLifecycleConfiguration, ExpirationStatus, LifecycleExpiration, LifecycleRule, NoncurrentVersionExpiration,
     };
     use std::time::Duration as StdDuration;
     use time::OffsetDateTime;
@@ -1078,12 +1077,12 @@ mod tests {
         }
     }
 
-    fn expired_delete_marker_lifecycle() -> PersistedLifecycleConfiguration {
-        PersistedLifecycleConfiguration {
+    fn expired_delete_marker_lifecycle() -> BucketLifecycleConfiguration {
+        BucketLifecycleConfiguration {
             expiry_updated_at: None,
-            rules: vec![PersistedLifecycleRule {
-                status: "Enabled".to_owned(),
-                expiration: Some(PersistedLifecycleExpiration {
+            rules: vec![LifecycleRule {
+                status: ExpirationStatus::from_static(ExpirationStatus::ENABLED),
+                expiration: Some(LifecycleExpiration {
                     expired_object_delete_marker: Some(true),
                     ..Default::default()
                 }),
@@ -1832,11 +1831,11 @@ mod tests {
         ))
         .await
         .expect("seed the expired source version");
-        let lifecycle = PersistedLifecycleConfiguration {
+        let lifecycle = BucketLifecycleConfiguration {
             expiry_updated_at: None,
-            rules: vec![PersistedLifecycleRule {
-                status: "Enabled".to_owned(),
-                expiration: Some(PersistedLifecycleExpiration {
+            rules: vec![LifecycleRule {
+                status: ExpirationStatus::from_static(ExpirationStatus::ENABLED),
+                expiration: Some(LifecycleExpiration {
                     days: Some(1),
                     expired_object_all_versions: Some(true),
                     ..Default::default()
@@ -2085,7 +2084,7 @@ mod tests {
                         versioned,
                         version_id,
                         transition: crate::bucket::lifecycle::lifecycle::TransitionOptions {
-                            restore_request: rustfs_lifecycle::RestoreRequest {
+                            restore_request: s3s::dto::RestoreRequest {
                                 days: Some(1),
                                 ..Default::default()
                             },
@@ -2165,7 +2164,7 @@ mod tests {
         async fn run_entry_with(
             &self,
             entry: MetaCacheEntry,
-            lifecycle_config: Option<PersistedLifecycleConfiguration>,
+            lifecycle_config: Option<BucketLifecycleConfiguration>,
         ) -> RebalanceEntryOutcome {
             Box::pin(Arc::clone(&self.store).rebalance_entry(
                 RebalanceEntryTarget {
@@ -2501,17 +2500,17 @@ mod tests {
     }
 
     /// A rule that expires every noncurrent version at once.
-    fn noncurrent_expire_all_lifecycle() -> PersistedLifecycleConfiguration {
-        PersistedLifecycleConfiguration {
+    fn noncurrent_expire_all_lifecycle() -> BucketLifecycleConfiguration {
+        BucketLifecycleConfiguration {
             expiry_updated_at: None,
-            rules: vec![PersistedLifecycleRule {
-                status: "Enabled".to_owned(),
+            rules: vec![LifecycleRule {
+                status: ExpirationStatus::from_static(ExpirationStatus::ENABLED),
                 expiration: None,
                 abort_incomplete_multipart_upload: None,
                 del_marker_expiration: None,
                 filter: None,
                 id: Some("noncurrent-expire-all".to_string()),
-                noncurrent_version_expiration: Some(PersistedNoncurrentVersionExpiration {
+                noncurrent_version_expiration: Some(NoncurrentVersionExpiration {
                     noncurrent_days: Some(0),
                     newer_noncurrent_versions: None,
                 }),

crates/ecstore/src/services/rebalance/meta.rs

@@ -154,7 +154,8 @@ impl RebalanceMeta {
             ver => return Err(rebalance_meta_load_unknown_version_error(ver)),
         }
 
-        let meta: Self = rmp_serde::from_read(Cursor::new(&data[4..]))?;
+        let meta: Self = rmp_serde::from_read(Cursor::new(&data[4..]))
+            .map_err(|source| Error::other(RebalanceMetadataLoadError::Decode(source)))?;
         *self = meta;
 
         self.last_refreshed_at = Some(OffsetDateTime::now_utc());
@@ -259,16 +260,28 @@ pub(super) fn first_rebalance_bucket(pool_stat: &RebalanceStats) -> Option<Strin
     pool_stat.buckets.first().cloned()
 }
 
+#[derive(Debug, thiserror::Error)]
+pub(super) enum RebalanceMetadataLoadError {
+    #[error("rebalance metadata load failed: metadata payload is too short")]
+    PayloadTooShort,
+    #[error("rebalance metadata load failed: unknown format {0}")]
+    UnknownFormat(u16),
+    #[error("rebalance metadata load failed: unknown version {0}")]
+    UnknownVersion(u16),
+    #[error("rebalance metadata load failed: payload decode failed: {0}")]
+    Decode(#[source] rmp_serde::decode::Error),
+}
+
 pub(super) fn rebalance_meta_load_no_data_error() -> Error {
-    Error::other("rebalance metadata load failed: metadata payload is too short")
+    Error::other(RebalanceMetadataLoadError::PayloadTooShort)
 }
 
 pub(super) fn rebalance_meta_load_unknown_format_error(fmt: u16) -> Error {
-    Error::other(format!("rebalance metadata load failed: unknown format {fmt}"))
+    Error::other(RebalanceMetadataLoadError::UnknownFormat(fmt))
 }
 
 pub(super) fn rebalance_meta_load_unknown_version_error(ver: u16) -> Error {
-    Error::other(format!("rebalance metadata load failed: unknown version {ver}"))
+    Error::other(RebalanceMetadataLoadError::UnknownVersion(ver))
 }
 
 pub fn encode_rebalance_stop_propagation_record(record: &RebalanceStopPropagationRecord) -> String {

crates/ecstore/src/services/rebalance/rebalance_unit_tests.rs

@@ -67,8 +67,8 @@ use crate::error::{Error, Result};
 use crate::storage_api_contracts::range::HTTPRangeSpec;
 use rustfs_filemeta::TRANSITION_COMPLETE;
 use rustfs_filemeta::{FileInfo, MetaCacheEntry};
-use rustfs_gateway_types::persistence::PersistedReplicationConfiguration;
 use rustfs_rio::Index;
+use s3s::dto::ReplicationConfiguration;
 use serde::Serialize;
 use std::io::Cursor;
 use std::sync::Arc;
@@ -2676,7 +2676,7 @@ fn test_resolve_rebalance_optional_bucket_config_result_passthrough() {
     let result = resolve_rebalance_optional_bucket_config_result(
         "bucket-a",
         "replication",
-        Ok((PersistedReplicationConfiguration::default(), OffsetDateTime::UNIX_EPOCH)),
+        Ok((ReplicationConfiguration::default(), OffsetDateTime::UNIX_EPOCH)),
     )
     .expect("bucket config should pass through");
     assert!(result.is_some());

crates/ecstore/src/services/rebalance/types.rs

@@ -34,9 +34,9 @@ pub struct RebalanceStats {
 #[derive(Debug, Default)]
 pub(super) struct RebalanceBucketConfigs {
     pub(super) bucket_incarnation_id: Option<uuid::Uuid>,
-    pub(super) lifecycle_config: Option<rustfs_gateway_types::persistence::PersistedLifecycleConfiguration>,
-    pub(super) object_lock_config: Option<rustfs_gateway_types::persistence::PersistedObjectLockConfiguration>,
-    pub(super) replication_config: Option<(rustfs_gateway_types::persistence::PersistedReplicationConfiguration, OffsetDateTime)>,
+    pub(super) lifecycle_config: Option<s3s::dto::BucketLifecycleConfiguration>,
+    pub(super) object_lock_config: Option<s3s::dto::ObjectLockConfiguration>,
+    pub(super) replication_config: Option<(s3s::dto::ReplicationConfiguration, OffsetDateTime)>,
 }
 
 #[derive(Debug, Clone, PartialEq, Eq)]

crates/ecstore/src/services/tier/tier.rs

@@ -80,11 +80,10 @@ use crate::{
     store::ECStore,
 };
 use rustfs_filemeta::FileInfo;
-use rustfs_gateway_types::persistence::PersistedLifecycleConfiguration;
 use rustfs_rio::HashReader;
 use rustfs_s3_client::{admin_handler_utils::AdminError, provider_versions::ProviderVersionCapabilities};
-use rustfs_s3_types::S3ErrorCode;
 use rustfs_utils::path::{SLASH_SEPARATOR, path_join};
+use s3s::{S3ErrorCode, dto::BucketLifecycleConfiguration};
 
 use super::{
     tier_handlers::{ERR_TIER_BUCKET_NOT_FOUND, ERR_TIER_CONNECT_ERR, ERR_TIER_INVALID_CREDENTIALS, ERR_TIER_PERM_ERR},
@@ -1037,7 +1036,7 @@ trait TierReferenceProofStore:
         .await
     }
 
-    async fn lifecycle_config_for_reference_proof(&self, bucket: &str) -> Result<Option<PersistedLifecycleConfiguration>>;
+    async fn lifecycle_config_for_reference_proof(&self, bucket: &str) -> Result<Option<BucketLifecycleConfiguration>>;
 
     async fn authoritative_bucket_tier_reference(
         self: Arc<Self>,
@@ -1077,7 +1076,7 @@ impl TierReferenceProofStore for ECStore {
         Ok(buckets.into_values().collect())
     }
 
-    async fn lifecycle_config_for_reference_proof(&self, bucket: &str) -> Result<Option<PersistedLifecycleConfiguration>> {
+    async fn lifecycle_config_for_reference_proof(&self, bucket: &str) -> Result<Option<BucketLifecycleConfiguration>> {
         match get_lifecycle_config(bucket).await {
             Ok((config, _updated_at)) => Ok(Some(config)),
             Err(Error::ConfigNotFound) => Ok(None),
@@ -1339,7 +1338,7 @@ struct TierLifecycleReference<'a> {
 }
 
 fn lifecycle_config_target_reference<'a>(
-    config: &'a PersistedLifecycleConfiguration,
+    config: &'a BucketLifecycleConfiguration,
     targets: &[TierMutationIntentTarget],
 ) -> Option<TierLifecycleReference<'a>> {
     for rule in &config.rules {
@@ -6928,8 +6927,9 @@ mod tests {
         endpoints::{Endpoints, PoolEndpoints, SetupType},
     };
     use crate::services::tier::tier_mutation_intent::TIER_MUTATION_INTENT_RECORD_PREFIX;
-    use rustfs_gateway_types::persistence::{
-        PersistedLifecycleConfiguration, PersistedLifecycleRule, PersistedNoncurrentVersionTransition, PersistedTransition,
+    use s3s::dto::{
+        BucketLifecycleConfiguration, ExpirationStatus, LifecycleRule, NoncurrentVersionTransition, Transition,
+        TransitionStorageClass,
     };
 
     #[test]
@@ -7756,7 +7756,7 @@ mod tests {
 
     #[async_trait::async_trait]
     impl TierReferenceProofStore for LockingTierConfigStore {
-        async fn lifecycle_config_for_reference_proof(&self, _bucket: &str) -> Result<Option<PersistedLifecycleConfiguration>> {
+        async fn lifecycle_config_for_reference_proof(&self, _bucket: &str) -> Result<Option<BucketLifecycleConfiguration>> {
             Ok(None)
         }
     }
@@ -10396,42 +10396,6 @@ mod tests {
         assert_eq!(observed.committed_config_etag.as_deref(), Some("etag-new"));
     }
 
-    #[tokio::test]
-    async fn intent_advance_retries_cas_conflict_reload_lock_timeout() {
-        let store = Arc::new(CasConfigStore::default());
-        let mutation_id = uuid::Uuid::from_u128(39);
-        let prepared = prepared_remove_intent("COLD-A", mutation_id);
-        crate::services::tier::tier_mutation_intent::save_tier_mutation_intent_record(store.clone(), &prepared)
-            .await
-            .expect("prepared intent fixture should persist");
-
-        let committed = committed_remove_intent("COLD-A", mutation_id, "etag-new");
-        let committed_bytes = committed.encode().expect("committed intent fixture should encode");
-        let object = crate::services::tier::tier_mutation_intent::tier_mutation_intent_record_object_name(mutation_id)
-            .expect("record object should build");
-        let winning_etag = format!("etag-{}", store.next_etag.load(Ordering::SeqCst) + 1);
-        store.rewrite_on_next_if_match(object.clone(), committed_bytes.clone()).await;
-        *store.read_timeout_after_if_match_race.lock().await = Some(object.clone());
-
-        let (observed, applied) = crate::services::tier::tier_mutation_intent::advance_tier_mutation_intent_record_idempotent(
-            store.clone(),
-            mutation_id,
-            TierMutationIntentState::Committed,
-            Some("etag-new".to_string()),
-        )
-        .await
-        .expect("CAS conflict reload should retry its typed lock timeout");
-
-        assert!(!applied);
-        assert_eq!(observed, committed);
-        assert!(store.read_timeout_after_if_match_race.lock().await.is_none());
-        assert_eq!(
-            store.objects.lock().await.get(&object).cloned(),
-            Some((committed_bytes, winning_etag)),
-            "retry must preserve the complete winning record and its object ETag"
-        );
-    }
-
     #[tokio::test]
     async fn intent_advance_reconciles_matching_commit_after_final_cas_race() {
         let store = Arc::new(CasConfigStore::default());
@@ -14903,7 +14867,6 @@ mod tests {
         objects: tokio::sync::Mutex<HashMap<String, (Vec<u8>, String)>>,
         legacy_state: tokio::sync::Mutex<Option<Vec<u8>>>,
         if_match_race_rewrite: tokio::sync::Mutex<std::collections::VecDeque<(String, Vec<u8>)>>,
-        read_timeout_after_if_match_race: tokio::sync::Mutex<Option<String>>,
         next_etag: AtomicUsize,
         content_derived_etags: AtomicBool,
         fail_put: AtomicBool,
@@ -14918,7 +14881,7 @@ mod tests {
         lock_manager: Arc<rustfs_lock::GlobalLockManager>,
         lock_requests: Mutex<Vec<(String, String)>>,
         listed_versions: Mutex<Vec<ObjectInfo>>,
-        lifecycle_configs: Mutex<HashMap<String, PersistedLifecycleConfiguration>>,
+        lifecycle_configs: Mutex<HashMap<String, BucketLifecycleConfiguration>>,
     }
 
     impl Default for CasConfigStore {
@@ -14929,7 +14892,6 @@ mod tests {
                 objects: tokio::sync::Mutex::new(HashMap::new()),
                 legacy_state: tokio::sync::Mutex::new(None),
                 if_match_race_rewrite: tokio::sync::Mutex::new(std::collections::VecDeque::new()),
-                read_timeout_after_if_match_race: tokio::sync::Mutex::new(None),
                 next_etag: AtomicUsize::new(0),
                 content_derived_etags: AtomicBool::new(false),
                 fail_put: AtomicBool::new(false),
@@ -14978,7 +14940,7 @@ mod tests {
                 .retain(|version| version.bucket != bucket || version.name != object);
         }
 
-        fn add_lifecycle_config(&self, bucket: &str, config: PersistedLifecycleConfiguration) {
+        fn add_lifecycle_config(&self, bucket: &str, config: BucketLifecycleConfiguration) {
             self.lifecycle_configs
                 .lock()
                 .expect("tier reference fixture should not poison")
@@ -15087,19 +15049,6 @@ mod tests {
                 crate::api::diagnostic::current_read_version_entry_source(),
                 rustfs_io_metrics::s3_http_metrics::current_s3_http_operation_name(),
             ));
-            {
-                // The queued rewrite keeps the initial read successful. Only
-                // the first target read after that CAS race consumes the fault.
-                let race_completed = self.if_match_race_rewrite.lock().await.is_empty();
-                let mut timeout = self.read_timeout_after_if_match_race.lock().await;
-                if race_completed && timeout.as_deref() == Some(object) {
-                    timeout.take();
-                    return Err(Error::Lock(rustfs_lock::LockError::Timeout {
-                        resource: format!("{bucket}/{object}@latest"),
-                        timeout: std::time::Duration::from_secs(5),
-                    }));
-                }
-            }
             let legacy_config_path = tier_config_path(TIER_CONFIG_LEGACY_FILE);
             let (data, etag) = if object == legacy_config_path {
                 let state = self.legacy_state.lock().await;
@@ -15681,7 +15630,7 @@ mod tests {
 
     #[async_trait::async_trait]
     impl TierReferenceProofStore for CasConfigStore {
-        async fn lifecycle_config_for_reference_proof(&self, bucket: &str) -> Result<Option<PersistedLifecycleConfiguration>> {
+        async fn lifecycle_config_for_reference_proof(&self, bucket: &str) -> Result<Option<BucketLifecycleConfiguration>> {
             Ok(self
                 .lifecycle_configs
                 .lock()
@@ -15813,10 +15762,10 @@ mod tests {
         });
         store.add_lifecycle_config(
             "photos",
-            PersistedLifecycleConfiguration {
+            BucketLifecycleConfiguration {
                 expiry_updated_at: None,
-                rules: vec![PersistedLifecycleRule {
-                    status: "Enabled".to_owned(),
+                rules: vec![LifecycleRule {
+                    status: ExpirationStatus::from_static(ExpirationStatus::ENABLED),
                     expiration: None,
                     abort_incomplete_multipart_upload: None,
                     del_marker_expiration: None,
@@ -15825,10 +15774,10 @@ mod tests {
                     noncurrent_version_expiration: None,
                     noncurrent_version_transitions: None,
                     prefix: None,
-                    transitions: Some(vec![PersistedTransition {
+                    transitions: Some(vec![Transition {
                         days: Some(1),
                         date: None,
-                        storage_class: Some("COLD-A".to_owned()),
+                        storage_class: Some(TransitionStorageClass::from_static("COLD-A")),
                     }]),
                 }],
             },
@@ -16481,10 +16430,10 @@ mod tests {
             .save_tiering_config_if_current(store.clone(), None)
             .await
             .expect("reference proof fixture should persist");
-        let config = PersistedLifecycleConfiguration {
+        let config = BucketLifecycleConfiguration {
             expiry_updated_at: None,
-            rules: vec![PersistedLifecycleRule {
-                status: "Enabled".to_owned(),
+            rules: vec![LifecycleRule {
+                status: ExpirationStatus::from_static(ExpirationStatus::ENABLED),
                 expiration: None,
                 abort_incomplete_multipart_upload: None,
                 del_marker_expiration: None,
@@ -16493,10 +16442,10 @@ mod tests {
                 noncurrent_version_expiration: None,
                 noncurrent_version_transitions: None,
                 prefix: None,
-                transitions: Some(vec![PersistedTransition {
+                transitions: Some(vec![Transition {
                     days: Some(1),
                     date: None,
-                    storage_class: Some("COLD-A".to_owned()),
+                    storage_class: Some(TransitionStorageClass::from_static("COLD-A")),
                 }]),
             }],
         };
@@ -16653,10 +16602,10 @@ mod tests {
             old_backend_identity: Some(current_identity),
             new_backend_identity: None,
         };
-        let config = PersistedLifecycleConfiguration {
+        let config = BucketLifecycleConfiguration {
             expiry_updated_at: None,
-            rules: vec![PersistedLifecycleRule {
-                status: "Enabled".to_owned(),
+            rules: vec![LifecycleRule {
+                status: ExpirationStatus::from_static(ExpirationStatus::ENABLED),
                 expiration: None,
                 abort_incomplete_multipart_upload: None,
                 del_marker_expiration: None,
@@ -16665,10 +16614,10 @@ mod tests {
                 noncurrent_version_expiration: None,
                 noncurrent_version_transitions: None,
                 prefix: None,
-                transitions: Some(vec![PersistedTransition {
+                transitions: Some(vec![Transition {
                     days: Some(1),
                     date: None,
-                    storage_class: Some("COLD-A".to_owned()),
+                    storage_class: Some(TransitionStorageClass::from_static("COLD-A")),
                 }]),
             }],
         };
@@ -16698,20 +16647,20 @@ mod tests {
             old_backend_identity: Some(current_identity),
             new_backend_identity: None,
         };
-        let config = PersistedLifecycleConfiguration {
+        let config = BucketLifecycleConfiguration {
             expiry_updated_at: None,
-            rules: vec![PersistedLifecycleRule {
-                status: "Enabled".to_owned(),
+            rules: vec![LifecycleRule {
+                status: ExpirationStatus::from_static(ExpirationStatus::ENABLED),
                 expiration: None,
                 abort_incomplete_multipart_upload: None,
                 del_marker_expiration: None,
                 filter: None,
                 id: Some("move-noncurrent".to_string()),
                 noncurrent_version_expiration: None,
-                noncurrent_version_transitions: Some(vec![PersistedNoncurrentVersionTransition {
+                noncurrent_version_transitions: Some(vec![NoncurrentVersionTransition {
                     noncurrent_days: Some(1),
                     newer_noncurrent_versions: None,
-                    storage_class: Some("COLD-A".to_owned()),
+                    storage_class: Some(TransitionStorageClass::from_static("COLD-A")),
                 }]),
                 prefix: None,
                 transitions: None,
@@ -16747,10 +16696,10 @@ mod tests {
             old_backend_identity: Some(current_identity),
             new_backend_identity: None,
         };
-        let config = PersistedLifecycleConfiguration {
+        let config = BucketLifecycleConfiguration {
             expiry_updated_at: None,
-            rules: vec![PersistedLifecycleRule {
-                status: "Enabled".to_owned(),
+            rules: vec![LifecycleRule {
+                status: ExpirationStatus::from_static(ExpirationStatus::ENABLED),
                 expiration: None,
                 abort_incomplete_multipart_upload: None,
                 del_marker_expiration: None,
@@ -16759,10 +16708,10 @@ mod tests {
                 noncurrent_version_expiration: None,
                 noncurrent_version_transitions: None,
                 prefix: None,
-                transitions: Some(vec![PersistedTransition {
+                transitions: Some(vec![Transition {
                     days: Some(1),
                     date: None,
-                    storage_class: Some("COLD-A".to_owned()),
+                    storage_class: Some(TransitionStorageClass::from_static("COLD-A")),
                 }]),
             }],
         };

crates/ecstore/src/services/tier/tier_mutation_intent.rs

@@ -700,25 +700,9 @@ async fn advance_tier_mutation_intent_record_idempotent_at_prefix<S>(
 where
     S: EcstoreObjectIO,
 {
-    let mut original_intent: Option<TierMutationIntent> = None;
     for attempt in 0..TIER_MUTATION_INTENT_ADVANCE_CAS_ATTEMPTS {
         let (mut intent, current_etag) =
-            match load_tier_mutation_intent_record_with_etag_at_prefix(api.clone(), prefix, mutation_id).await {
-                Ok(current) => current,
-                Err(Error::Lock(rustfs_lock::LockError::Timeout { .. }))
-                    if attempt + 1 < TIER_MUTATION_INTENT_ADVANCE_CAS_ATTEMPTS =>
-                {
-                    continue;
-                }
-                Err(err) => return Err(err),
-            };
-        if let Some(original) = &original_intent {
-            if !intent.same_identity_as(original) {
-                return Err(Error::PreconditionFailed);
-            }
-        } else {
-            original_intent = Some(intent.clone());
-        }
+            load_tier_mutation_intent_record_with_etag_at_prefix(api.clone(), prefix, mutation_id).await?;
         let advanced = intent
             .advance_idempotent(next, committed_config_etag.clone())
             .map_err(tier_mutation_intent_store_error)?;
@@ -729,15 +713,7 @@ where
             Ok(()) => return Ok((intent, true)),
             Err(Error::PreconditionFailed) => {
                 let (mut current, _) =
-                    match load_tier_mutation_intent_record_with_etag_at_prefix(api.clone(), prefix, mutation_id).await {
-                        Ok(current) => current,
-                        Err(Error::Lock(rustfs_lock::LockError::Timeout { .. }))
-                            if attempt + 1 < TIER_MUTATION_INTENT_ADVANCE_CAS_ATTEMPTS =>
-                        {
-                            continue;
-                        }
-                        Err(err) => return Err(err),
-                    };
+                    load_tier_mutation_intent_record_with_etag_at_prefix(api.clone(), prefix, mutation_id).await?;
                 if !current.same_identity_as(&intent) {
                     return Err(Error::PreconditionFailed);
                 }
@@ -752,11 +728,6 @@ where
                 }
                 return Ok((current, false));
             }
-            Err(Error::Lock(rustfs_lock::LockError::Timeout { .. }))
-                if attempt + 1 < TIER_MUTATION_INTENT_ADVANCE_CAS_ATTEMPTS =>
-            {
-                continue;
-            }
             Err(err) => return Err(err),
         }
     }

crates/ecstore/src/services/tier/warm_backend.rs

@@ -36,9 +36,7 @@ use crate::services::tier::{
 use bytes::Bytes;
 use http::StatusCode;
 use rustfs_filemeta::metadata_keys;
-use rustfs_gateway_types::dto::{ObjectLockLegalHoldStatus, ObjectLockMode, ReplicationStatus};
 use rustfs_s3_client::credentials::{Credentials, SignatureType, Static, Value};
-use rustfs_s3_client::remote_error_code::RemoteErrorCode;
 use rustfs_s3_client::transition_api::{BucketLookupType, Options, TransitionClient, TransitionClientTimeouts, TransitionCore};
 use rustfs_s3_client::{
     admin_handler_utils::AdminError,
@@ -51,6 +49,10 @@ use rustfs_utils::egress::validate_outbound_url;
 use rustfs_utils::http::headers::{
     CACHE_CONTROL, CONTENT_DISPOSITION, CONTENT_ENCODING, CONTENT_LANGUAGE, CONTENT_TYPE, EXPIRES, HeaderExt as _,
 };
+use s3s::{
+    S3ErrorCode,
+    dto::{ObjectLockLegalHoldStatus, ObjectLockRetentionMode, ReplicationStatus},
+};
 use std::collections::HashMap;
 use std::sync::Arc;
 use std::time::Duration;
@@ -191,15 +193,12 @@ pub trait WarmBackend {
             .await
         {
             Ok(_) => Ok(TransitionCandidateProbe::VersionedPresent(remote_version_id.to_string())),
-            Err(err) if matches!(to_error_response(&err).code, RemoteErrorCode::INVALID_RANGE) => {
+            Err(err) if matches!(to_error_response(&err).code, S3ErrorCode::InvalidRange) => {
                 Ok(TransitionCandidateProbe::VersionedPresent(remote_version_id.to_string()))
             }
             Err(err)
                 if err.kind() == std::io::ErrorKind::NotFound
-                    || matches!(
-                        to_error_response(&err).code,
-                        RemoteErrorCode::NO_SUCH_KEY | RemoteErrorCode::NO_SUCH_VERSION
-                    ) =>
+                    || matches!(to_error_response(&err).code, S3ErrorCode::NoSuchKey | S3ErrorCode::NoSuchVersion) =>
             {
                 Ok(TransitionCandidateProbe::Missing)
             }
@@ -219,9 +218,9 @@ pub fn build_transition_put_options(storage_class: String, mut metadata: HashMap
     let mut opts = PutObjectOptions {
         storage_class,
         send_content_md5: true,
-        legalhold: ObjectLockLegalHoldStatus::custom(""),
+        legalhold: ObjectLockLegalHoldStatus::from_static(""),
         internal: AdvancedPutOptions {
-            replication_status: ReplicationStatus::custom(""),
+            replication_status: ReplicationStatus::from_static(""),
             ..Default::default()
         },
         ..Default::default()
@@ -252,7 +251,7 @@ pub fn build_transition_put_options(storage_class: String, mut metadata: HashMap
     }
 
     if let Some(mode) = metadata.lookup(metadata_keys::OBJECT_LOCK_MODE) {
-        opts.mode = ObjectLockMode::custom(mode.to_ascii_uppercase());
+        opts.mode = ObjectLockRetentionMode::from(mode.to_ascii_uppercase());
     }
 
     if let Some(retain_until_date) = metadata
@@ -263,7 +262,7 @@ pub fn build_transition_put_options(storage_class: String, mut metadata: HashMap
     }
 
     if let Some(legalhold) = metadata.lookup(metadata_keys::OBJECT_LOCK_LEGAL_HOLD) {
-        opts.legalhold = ObjectLockLegalHoldStatus::custom(legalhold.to_ascii_uppercase());
+        opts.legalhold = ObjectLockLegalHoldStatus::from(legalhold.to_ascii_uppercase());
     }
 
     // Promoted keys are read above through `lookup`, which accepts more than
@@ -680,7 +679,7 @@ fn is_remote_tier_auth_rejection(err: &std::io::Error) -> bool {
     matches!(response.status_code, StatusCode::UNAUTHORIZED | StatusCode::FORBIDDEN)
         || matches!(
             response.code,
-            RemoteErrorCode::ACCESS_DENIED | RemoteErrorCode::INVALID_ACCESS_KEY_ID | RemoteErrorCode::SIGNATURE_DOES_NOT_MATCH
+            S3ErrorCode::AccessDenied | S3ErrorCode::InvalidAccessKeyId | S3ErrorCode::SignatureDoesNotMatch
         )
 }
 
@@ -1305,7 +1304,7 @@ mod tests {
 
     struct AuthRejectingProbePutBackend {
         status_code: StatusCode,
-        code: RemoteErrorCode,
+        code: S3ErrorCode,
         probes: Arc<AtomicUsize>,
         removes: Arc<AtomicUsize>,
     }
@@ -1768,10 +1767,10 @@ mod tests {
     #[tokio::test]
     async fn check_warm_backend_reports_auth_rejected_probe_put_as_invalid_credentials() {
         for (status_code, code) in [
-            (StatusCode::FORBIDDEN, RemoteErrorCode::ACCESS_DENIED),
-            (StatusCode::UNAUTHORIZED, RemoteErrorCode::custom("")),
-            (StatusCode::OK, RemoteErrorCode::INVALID_ACCESS_KEY_ID),
-            (StatusCode::OK, RemoteErrorCode::SIGNATURE_DOES_NOT_MATCH),
+            (StatusCode::FORBIDDEN, S3ErrorCode::AccessDenied),
+            (StatusCode::UNAUTHORIZED, S3ErrorCode::Custom("".into())),
+            (StatusCode::OK, S3ErrorCode::InvalidAccessKeyId),
+            (StatusCode::OK, S3ErrorCode::SignatureDoesNotMatch),
         ] {
             let probes = Arc::new(AtomicUsize::new(0));
             let removes = Arc::new(AtomicUsize::new(0));
@@ -2166,31 +2165,18 @@ mod tests {
             metadata_keys::OBJECT_LOCK_LEGAL_HOLD.to_string(),
             ObjectLockLegalHoldStatus::ON.to_string(),
         );
-        metadata.insert(metadata_keys::OBJECT_LOCK_MODE.to_string(), ObjectLockMode::GOVERNANCE.to_string());
+        metadata.insert(
+            metadata_keys::OBJECT_LOCK_MODE.to_string(),
+            ObjectLockRetentionMode::GOVERNANCE.to_string(),
+        );
 
         let opts = build_transition_put_options("COLD".to_string(), metadata);
 
-        assert_eq!(opts.mode, ObjectLockMode::GOVERNANCE);
-        assert_eq!(opts.legalhold, ObjectLockLegalHoldStatus::ON);
+        assert_eq!(opts.mode.as_str(), ObjectLockRetentionMode::GOVERNANCE);
+        assert_eq!(opts.legalhold.as_str(), ObjectLockLegalHoldStatus::ON);
         assert_ne!(opts.retain_until_date, OffsetDateTime::UNIX_EPOCH);
     }
 
-    #[test]
-    fn build_transition_put_options_sends_no_object_lock_or_replication_headers_by_default() {
-        let opts = build_transition_put_options("COLD".to_string(), HashMap::new());
-        assert_eq!(opts.mode.as_str(), "");
-        assert_eq!(opts.legalhold.as_str(), "");
-        assert_eq!(opts.internal.replication_status.as_str(), "");
-        let header = opts.header();
-        for name in [
-            "x-amz-object-lock-mode",
-            "x-amz-object-lock-legal-hold",
-            "x-amz-replication-status",
-        ] {
-            assert!(header.get(name).is_none(), "{name} must not be sent: {header:?}");
-        }
-    }
-
     #[test]
     fn build_transition_put_options_filters_promoted_headers_from_user_metadata() {
         let mut metadata = HashMap::new();

crates/ecstore/src/services/tier/warm_backend_s3.rs

@@ -28,8 +28,6 @@ use crate::services::tier::{
     },
 };
 use http::HeaderMap;
-use rustfs_gateway_types::dto::Status as BucketVersioningStatus;
-use rustfs_s3_client::remote_error_code::RemoteErrorCode;
 use rustfs_s3_client::{
     api_error_response::to_error_response,
     api_get_options::GetObjectOptions,
@@ -44,6 +42,7 @@ use rustfs_s3_client::{
 };
 use rustfs_utils::egress::validate_outbound_url_allow_private;
 use rustfs_utils::path::SLASH_SEPARATOR;
+use s3s::{S3ErrorCode, dto::BucketVersioningStatus};
 
 pub struct WarmBackendS3 {
     pub client: Arc<TransitionClient>,
@@ -62,8 +61,8 @@ enum RemoteBucketVersioning {
 
 fn remote_bucket_versioning_from_status(status: Option<&str>) -> Result<RemoteBucketVersioning, std::io::Error> {
     Ok(match status {
-        Some(status) if status == BucketVersioningStatus::ENABLED.as_str() => RemoteBucketVersioning::Enabled,
-        Some(status) if status == BucketVersioningStatus::SUSPENDED.as_str() => RemoteBucketVersioning::Suspended,
+        Some(BucketVersioningStatus::ENABLED) => RemoteBucketVersioning::Enabled,
+        Some(BucketVersioningStatus::SUSPENDED) => RemoteBucketVersioning::Suspended,
         Some(status) => {
             return Err(std::io::Error::new(
                 std::io::ErrorKind::InvalidData,
@@ -238,7 +237,7 @@ impl WarmBackendS3 {
             }
             Err(err) => {
                 let response = to_error_response(&err);
-                if response.code == RemoteErrorCode::NO_SUCH_KEY {
+                if response.code == S3ErrorCode::NoSuchKey {
                     Ok(TransitionCandidateProbe::Missing)
                 } else {
                     Err(err)
@@ -325,7 +324,7 @@ impl WarmBackendS3 {
             }
             Err(err) => {
                 let response = to_error_response(&err);
-                if response.code == RemoteErrorCode::NO_SUCH_KEY {
+                if response.code == S3ErrorCode::NoSuchKey {
                     Ok(TransitionCandidateProbe::Missing)
                 } else {
                     Err(err)
@@ -689,7 +688,7 @@ mod tests {
             .probe_transition_candidate("forbidden-probe")
             .await
             .expect_err("an authorization failure must not be mistaken for a missing key");
-        assert_eq!(to_error_response(&err).code, RemoteErrorCode::ACCESS_DENIED);
+        assert_eq!(to_error_response(&err).code, S3ErrorCode::AccessDenied);
         assert_eq!(
             backend
                 .probe_transition_candidate("delete-marker-hidden")
@@ -1067,13 +1066,11 @@ mod tests {
             RemoteBucketVersioning::Disabled
         );
         assert_eq!(
-            remote_bucket_versioning_from_status(Some(BucketVersioningStatus::ENABLED.as_str()))
-                .expect("enabled status should parse"),
+            remote_bucket_versioning_from_status(Some(BucketVersioningStatus::ENABLED)).expect("enabled status should parse"),
             RemoteBucketVersioning::Enabled
         );
         assert_eq!(
-            remote_bucket_versioning_from_status(Some(BucketVersioningStatus::SUSPENDED.as_str()))
-                .expect("suspended status should parse"),
+            remote_bucket_versioning_from_status(Some(BucketVersioningStatus::SUSPENDED)).expect("suspended status should parse"),
             RemoteBucketVersioning::Suspended
         );
         let err = remote_bucket_versioning_from_status(Some("UnexpectedStatus"))

crates/ecstore/src/services/tier/warm_backend_wasabi.rs

@@ -18,6 +18,7 @@ use std::{
     sync::atomic::{AtomicBool, Ordering},
 };
 
+use s3s::header::{X_AMZ_DELETE_MARKER, X_AMZ_VERSION_ID};
 use uuid::Uuid;
 
 use crate::services::tier::{
@@ -127,7 +128,7 @@ impl WarmBackend for WarmBackendWasabi {
             self.ensure_unversioned().await?;
         }
         let (headers, reader) = self.s3.get_with_headers(object, rv, opts).await?;
-        if rv.is_empty() && (headers.contains_key("x-amz-version-id") || headers.contains_key("x-amz-delete-marker")) {
+        if rv.is_empty() && (headers.contains_key(X_AMZ_VERSION_ID) || headers.contains_key(X_AMZ_DELETE_MARKER)) {
             self.versioning_drifted.store(true, Ordering::Release);
             return Err(io::Error::other(
                 "Wasabi bucket versioning changed while reading an unversioned tier object",