fix: retain neutral audit header extraction after gateway sync

The shared connected preflight uses uuid.uuid7, which is unavailable in older runner Python environments. Select Python 3.14 with the existing pinned setup action in all eleven acceptance callers.

Co-authored-by: heihutu <heihutu@gmail.com>
Co-authored-by: zhi22915 <qiuzgang@gmail.com>

.agents/skills/security-advisory-lessons/references/advisory-patterns.md

@@ -45,11 +45,13 @@ Update this file only when an advisory adds or changes a reusable lesson, affect
 - `GHSA-6r96-hmgc-726c`: request headers collided with lowercase server-derived condition keys such as `userid`, `groups`, `versionid`, and JWT/LDAP claims. Lesson: never let caller-controlled headers append to or replace server-derived policy context; reserve trusted condition keys and keep intentional request-header keys separate.
 - `GHSA-v9cp-qfw9-9pfp`: quantified negated string conditions applied negation after aggregation, transposing `ForAllValues` and `ForAnyValue` semantics. Lesson: push negation into the per-value predicate for quantified operators and test partially overlapping multi-value sets.
 - `GHSA-5w8r-p896-6vq2`: OPA policy mode skipped `ExistingObjectTag/*` loading, so tagged objects looked untagged to external policies. Lesson: external authorization plugins need the same object-tag and request context as built-in policy evaluation before they decide.
+- `GHSA-9wgh-5r68-49p7`: substituted policy-variable values could recreate `${...}` and loop inside a single resolution pass, never reaching the outer iteration limit. Lesson: bound substitutions within each pass and test cycles formed across otherwise literal resolver values; confirm whether attacker-controlled claims can reach those values before assigning remote impact.
 
 ### S3 object actions, copy, multipart, and upload policy validation
 
 - `GHSA-g8w9-qw9q-fghr` and `GHSA-xm99-m3gq-83g8`: a valid presigned `PutObject` accepted extra `x-amz-*` headers omitted from `SignedHeaders`; `x-amz-copy-source` could turn that upload capability into a cross-bucket read performed as the signer. Lesson: a presigned URL is a bounded capability; inspect all received security-sensitive headers and reject unsigned ones before selecting the S3 operation or reaching storage.
 - `GHSA-3ppv-fx5m-m749`: explicit `versionId` reads and copy sources authorized `s3:GetObject` instead of `s3:GetObjectVersion`. Lesson: version-specific object access must select version-specific actions for direct reads, `CopyObject`, and `UploadPartCopy`, with tests proving the backend is not reached on denial.
+- `GHSA-cg33-pxfp-qcj3` and `GHSA-65wq-gjw8-xr9w`: `HeadObject` and object-tagging paths authorized current-object actions while operating on an explicit historical `versionId`. Lesson: select the corresponding version action for metadata and tag reads and writes too; audit every operation that accepts a version ID.
 - `GHSA-gjph-xcgw-4p7w`: `PutObject` and browser POST accepted a caller-supplied `versionId` and replaced an unlocked historical version with only `s3:PutObject`. Lesson: only replication-authorized writes may choose an existing version ID; ordinary uploads must not rewrite version history.
 - `GHSA-x298-9x87-fvjq`: anonymous `ListObjectVersions` fell back to `ListBucket` and returned before public-access-block gates. Lesson: compatibility fallbacks must converge on the same post-authorization checks as direct grants, especially `RestrictPublicBuckets` and anonymous data-plane denies.
 - `GHSA-mx42-j6wv-px98`: `UploadPartCopy` missed source authorization and allowed cross-bucket object exfiltration. Lesson: multipart copy must enforce the same source and destination contract as `CopyObject`.
@@ -110,6 +112,7 @@ Update this file only when an advisory adds or changes a reusable lesson, affect
 
 - `GHSA-j548-9grx-fh4f`: Object Lock enforcement treated unreadable, fabricated, or unparsable bucket metadata as absent configuration and allowed retained objects to be deleted or expired. Lesson: retention must fail closed unless Object Lock absence is authoritative, and every delete, lifecycle, scanner, force-delete, and default-retention path needs the same state distinction.
 - `GHSA-28w4-rp2p-636w` and `GHSA-c7hx-vr97-45rg`: an authorized `DeleteBucket` with `x-rustfs-force-delete: true` removed active COMPLIANCE-retained versions even though ordinary version deletion was denied. Lesson: bucket force-delete must enforce retention across every version, including against root/owner callers.
+- `GHSA-q23w-fp7p-472v`: `PutObjectRetention` policy conditions read `x-amz-object-lock-*` headers while the handler applied mode and date from the XML body. Lesson: authorize the parsed retention that will be stored, or reject mismatched header/body values; compare retention dates as instants so offsets cannot change the applied bound.
 
 ### Serde deserialization and input validation
 
@@ -143,13 +146,13 @@ rg -n "deny_unknown_fields|serde.default|as u32|as usize|as i32" rustfs crates
 - IAM fixes: include import/update/list service-account cases with attacker-controlled parent, claims, access key, secret key, and policy.
 - Copy/upload fixes: include cross-bucket, cross-user, source-denied, destination-denied, copy-source-condition, and multipart completion cases.
 - Presigned upload fixes: include a valid presign with extra unsigned tagging, redirect, storage-class, and cross-bucket copy-source headers; require rejection before source or destination storage access, and verify explicitly signed equivalents still work.
-- Version-action fixes: include historical UUID, explicit current version, `null`, range, partNumber, presigned, STS/session, service-account, anonymous bucket-policy, copy source, and multipart-copy source cases.
+- Version-action fixes: include historical UUID, explicit current version, `null`, range, partNumber, presigned, STS/session, service-account, anonymous bucket-policy, copy source, multipart-copy source, `HeadObject`, and tag read/write cases.
 - Version-write fixes: a `PutObject`-only principal must not replace an existing version through `PutObject` or browser POST `?versionId=`; verify replication-authorized writes and versioned, suspended, and unversioned buckets separately.
-- Policy-condition fixes: include reserved-key header collisions, missing keys, partially overlapping multi-value sets, plugin mode, and built-in policy mode.
+- Policy-condition fixes: include reserved-key header collisions, missing keys, partially overlapping multi-value sets, cyclic variable substitutions, plugin mode, and built-in policy mode.
 - Path fixes: include encoded traversal, absolute path, nested traversal, archive entries with `..`, valid object keys that resemble traversal text but should be rejected, and canonical bucket/prefix boundary checks; test per-key `DeleteObjects` errors and prefix-scoped denials on versioned buckets.
 - Logging fixes: assert redacted output for structs and response bodies that may contain credentials.
 - IAM export fixes: assert exported archives omit plaintext user and service-account secrets unless the format deliberately encrypts or seals them.
 - RPC auth fixes: include captured metadata replay across two concrete methods, stale timestamps, wrong path, wrong method surrogate, wrong secret, and valid same-method calls.
 - Browser/CORS fixes: assert no credentials on reflected/default origins, correct behavior for explicit allowlists, and no same-origin script execution for previewed object content.
 - SSE fixes: inspect stored bytes and verify API metadata, read-back behavior, and on-disk ciphertext together; for ETags, upload equal plaintext under different keys and assert list-only callers cannot derive or correlate the plaintext digest, including multipart objects.
-- Object Lock fixes: include unreadable metadata, fabricated metadata defaults, unparsable config, confirmed absent config, COMPLIANCE/GOVERNANCE retention, lifecycle expiry, scanner sweeps, and force-delete paths; an authorized force-delete must preserve active COMPLIANCE versions.
+- Object Lock fixes: include unreadable metadata, fabricated metadata defaults, unparsable config, confirmed absent config, COMPLIANCE/GOVERNANCE retention, lifecycle expiry, scanner sweeps, and force-delete paths; an authorized force-delete must preserve active COMPLIANCE versions. For `PutObjectRetention`, test conflicting header/body values and offset dates against policy bounds and read-back retention.

.config/e2e-full-selection.txt

@@ -1,2 +1,2 @@
-sha256-darwin=98dff0961c6eb21d6128bfd2d4a6100a624ddf1fb7db05dba360a5d3d0a310c2
-sha256-linux=890f5925b87e1bf3710c356253f3b2276d901e156df9ccc04ec9315374de6d22
+sha256-darwin=ff63177a6280d52ac29f4a14a0c1f7fdcf7cecd442c1f3d3bce0bbdffc0552a5
+sha256-linux=2cc8bb449f52bdf5d9e88f75dc5c3a94df121a55aa436d5ec95311d42ec5fa0b

.config/e2e-nightly-selection.txt

@@ -1,2 +1,2 @@
-sha256-darwin=efe68675f4676a0d2901242c5815b1f7af5cdec8231f7cccd408f6dacf5c75db
-sha256-linux=003ed6480935a0a95680cc061be143722fbd37bd477b8ff98607b1a064f3aef1
+sha256-darwin=8ef80b7353d1e0c4f863a2aca7e996d23858b8c32b9edf8fd272076fa63a83d8
+sha256-linux=99fe1cd128ade2df69b9e3ecf94b91021185a9769293420b1ba5d24760f2ea99

.config/e2e-smoke-selection.txt

@@ -1 +1 @@
-sha256=ba8f392b617cf9f2338142233a417e43927b28096635e628233ce8a60cf49d10
+sha256=bf77250d94604db7ba094c5d63bf8a284beb872983871bcfd0dc02e46ff53326

.config/make/lint-fmt.mak

@@ -70,6 +70,11 @@ s3s-footprint-check: ## Check the s3s dependency footprint ratchet stays frozen
 	@echo "📦 Checking s3s footprint ratchet..."
 	./scripts/check_s3s_footprint.sh
 
+.PHONY: test-count-ratchet-check
+test-count-ratchet-check: ## Check no workspace crate lost tests or gained #[ignore] against the committed baseline
+	@echo "🧪 Checking per-crate test-count ratchet..."
+	./scripts/check_test_count_ratchet.sh
+
 .PHONY: fips-wording-check
 fips-wording-check: ## Check docs and crates/kms do not over-claim crypto capabilities
 	@echo "📣 Checking cryptographic capability wording guard..."

.config/make/pre-commit.mak

@@ -20,11 +20,11 @@ planning-docs-check: ## Check that no planning-type documents are committed
 	./scripts/check_no_planning_docs.sh
 
 .PHONY: pre-commit
-pre-commit: fmt-check unsafe-code-check architecture-migration-check logging-guardrails-check error-other-ratchet-check tokio-io-uring-check extension-schema-check body-cache-whitelist-check s3s-footprint-check fips-wording-check embedded-secrets-check test-wiring-check doc-paths-check planning-docs-check quick-check ## Run fast pre-commit checks without clippy/full tests
+pre-commit: fmt-check unsafe-code-check architecture-migration-check logging-guardrails-check error-other-ratchet-check tokio-io-uring-check extension-schema-check body-cache-whitelist-check s3s-footprint-check test-count-ratchet-check fips-wording-check embedded-secrets-check test-wiring-check doc-paths-check planning-docs-check quick-check ## Run fast pre-commit checks without clippy/full tests
 	@echo "✅ All pre-commit checks passed!"
 
 .PHONY: pre-pr
-pre-pr: fmt-check unsafe-code-check architecture-migration-check logging-guardrails-check error-other-ratchet-check tokio-io-uring-check extension-schema-check body-cache-whitelist-check s3s-footprint-check fips-wording-check embedded-secrets-check test-wiring-check doc-paths-check planning-docs-check log-analyzer-rules-check offline-enrollment-e2e-check clippy-check test ## Run full pre-PR checks with clippy and tests
+pre-pr: fmt-check unsafe-code-check architecture-migration-check logging-guardrails-check error-other-ratchet-check tokio-io-uring-check extension-schema-check body-cache-whitelist-check s3s-footprint-check test-count-ratchet-check fips-wording-check embedded-secrets-check test-wiring-check doc-paths-check planning-docs-check log-analyzer-rules-check offline-enrollment-e2e-check clippy-check test ## Run full pre-PR checks with clippy and tests
 	@echo "✅ All pre-PR checks passed!"
 
 .PHONY: dev-check

.config/make/tests.mak

@@ -50,8 +50,13 @@ script-tests: ## Run shell script tests
 	./scripts/test_fuzz_runner.sh
 	./scripts/test_python_bin.sh
 	./scripts/check_embedded_secrets.sh --self-test
+	./scripts/check_s3s_footprint.sh --self-test
 	$(RUSTFS_PYTHON_BIN) ./scripts/check_test_wiring.py --self-test
+	$(RUSTFS_PYTHON_BIN) ./scripts/codemods/s3_error_to_s3_types.py --self-test
 	$(RUSTFS_PYTHON_BIN) ./scripts/test_e2e_binary.py
+	$(RUSTFS_PYTHON_BIN) ./scripts/test_release_build_workflow.py
+	$(RUSTFS_PYTHON_BIN) ./scripts/test_release_glibc.py
+	$(RUSTFS_PYTHON_BIN) ./scripts/test_upgrade_workflow.py
 	$(RUSTFS_PYTHON_BIN) ./scripts/test_migration_gate_evidence.py
 	$(RUSTFS_PYTHON_BIN) ./scripts/ci_gate.py --self-test
 	$(RUSTFS_PYTHON_BIN) ./scripts/check_security_coverage.py --self-test

.config/nextest.toml

@@ -88,6 +88,11 @@ threads-required = "num-test-threads"
 filter = 'package(rustfs-ecstore) & test(/^(bucket::lifecycle::bucket_lifecycle_ops::tests::manual_transition_worker_result_recovery_marks_unknown_for_corrupt_marker|services::rebalance::entry::tests::real_rebalance_run_fence_loss_blocks_multipart_publication|store::init::tests::(batch_transitioned_delete_uses_free_version_per_item|decommission_entry_(allows_free_version_consumed_before_source_lock|rejects_subquorum_free_version_conflict_and_retains_source|skips_cleanup_only_marker_when_free_version_is_present)|dispatched_tier_delete_recovery_(checks_later_pool_then_commits_after_source_removal|finds_directory_source_on_encoded_set|retains_journal_on_source_metadata_error)|force_tier_remove_blocks_on_physical_free_version_hidden_by_other_pool|legacy_unknown_transition_delete_falls_back_for_single_batch_and_blocks_prefix|multi_pool_(recursive_prefix_rejects_legacy_or_hidden_merge_loser_before_delete|same_remote_tuple_(batch|single)_delete_waits_for_all_sources|same_tuple_recursive_prefix_uses_one_journal_owner|transitioned_delete_persists_one_free_version_per_remote_tuple)|recursive_prefix_partial_(pool|set)_failure_keeps_prepared_cleanup_owners|restored_transitioned_delete_uses_free_version_as_cleanup_owner|stable_transitioned_recursive_prefix_delete_uses_journal_owners|suspended_null_transition_delete_uses_free_version_as_sole_owner|tier_mutation_peer_handler_applies_prepare_commit_and_abort_idempotently|transition_response_loss_persists_unknown_outcome_for_provider_recovery|transition_transaction_recovery_(drops_record_after_confirmed_local_commit|keeps_cleanup_pending_local_commit)|transitioned_delete_(free_version_replays_after_store_restart|local_quorum_failure_rolls_back_without_cleanup_owner|uses_free_version_as_cleanup_owner)|versioned_delete_marker_keeps_transitioned_source_and_remote_object|versioned_explicit_transition_delete_preserves_other_version_then_allows_bucket_delete))$/)'
 setup = 'ecstore-large-stack'
 
+# These cross-crate real-storage fixtures need the same bounded stack as ECStore tests.
+[[profile.default.scripts]]
+filter = '(package(rustfs) & test(/^(admin::handlers::site_replication::tests::durable_iam_state_apply_iam_item_rejects_a_stale_recreate_after_a_replicated_delete|admin::service::federated_identity::tests::(binding_keeps_explicit_iam_mapping_strict|markerless_legacy_oidc_session_crosses_auth_and_iam_deletion_boundaries)|storage::scanner_dirty_journal::tests::(clear_callback_keeps_unknown_coverage_until_all_pending_work_is_confirmed|owned_journals_keep_peer_records_after_interleaved_writes_and_clear))$/)) | (package(rustfs-heal) & ((binary(heal_b5_versioned_regression_test) & test(/^serial_tests::(test_heal_unversioned_bucket_e2e|test_historical_null_recursive_heal_restores_physical_versions|test_recursive_deep_heal_reports_stale_delete_marker_repair)$/)) | (binary(usage_observation_read_repair_test) & test(=usage_observation_read_repair_survives_snapshot_cleanup)))) | (package(rustfs-test-utils) & test(=data_usage_snapshot_tests::data_usage_snapshot_recovery_wires_primary_backup_and_negative_cache))'
+setup = 'ecstore-base-stack'
+
 [[profile.default.scripts]]
 filter = 'package(rustfs-ecstore) | package(rustfs-s3select-api) | package(rustfs-scanner) | (package(rustfs) & test(/^(app::multipart_usecase::tests::concurrent_completions_share_durable_bucket_quota_reservations|app::object::delete::tests::compressed_delete_requests_update_observed_usage_without_releasing_quota_floor|app::object::internal_put::tests::internal_multipart_roundtrip_completes_and_abort_leaves_nothing|app::object::restore::tests::execute_restore_object_maps_failures_to_typed_s3_errors|storage::access::tests::(delete_object_access_captures_authorized_bucket_incarnation|copy_operations_reject_recreated_source_bucket_after_authorization|request_slot_keeps_bucket_policy_bound_to_its_store))$/))'
 setup = 'ecstore-base-stack'
@@ -339,6 +344,11 @@ threads-required = "num-test-threads"
 filter = 'package(rustfs-ecstore) & test(/^(bucket::lifecycle::bucket_lifecycle_ops::tests::manual_transition_worker_result_recovery_marks_unknown_for_corrupt_marker|services::rebalance::entry::tests::real_rebalance_run_fence_loss_blocks_multipart_publication|store::init::tests::(batch_transitioned_delete_uses_free_version_per_item|decommission_entry_(allows_free_version_consumed_before_source_lock|rejects_subquorum_free_version_conflict_and_retains_source|skips_cleanup_only_marker_when_free_version_is_present)|dispatched_tier_delete_recovery_(checks_later_pool_then_commits_after_source_removal|finds_directory_source_on_encoded_set|retains_journal_on_source_metadata_error)|force_tier_remove_blocks_on_physical_free_version_hidden_by_other_pool|legacy_unknown_transition_delete_falls_back_for_single_batch_and_blocks_prefix|multi_pool_(recursive_prefix_rejects_legacy_or_hidden_merge_loser_before_delete|same_remote_tuple_(batch|single)_delete_waits_for_all_sources|same_tuple_recursive_prefix_uses_one_journal_owner|transitioned_delete_persists_one_free_version_per_remote_tuple)|recursive_prefix_partial_(pool|set)_failure_keeps_prepared_cleanup_owners|restored_transitioned_delete_uses_free_version_as_cleanup_owner|stable_transitioned_recursive_prefix_delete_uses_journal_owners|suspended_null_transition_delete_uses_free_version_as_sole_owner|tier_mutation_peer_handler_applies_prepare_commit_and_abort_idempotently|transition_response_loss_persists_unknown_outcome_for_provider_recovery|transition_transaction_recovery_(drops_record_after_confirmed_local_commit|keeps_cleanup_pending_local_commit)|transitioned_delete_(free_version_replays_after_store_restart|local_quorum_failure_rolls_back_without_cleanup_owner|uses_free_version_as_cleanup_owner)|versioned_delete_marker_keeps_transitioned_source_and_remote_object|versioned_explicit_transition_delete_preserves_other_version_then_allows_bucket_delete))$/)'
 setup = 'ecstore-large-stack'
 
+# These cross-crate real-storage fixtures need the same bounded stack as ECStore tests.
+[[profile.ci.scripts]]
+filter = '(package(rustfs) & test(/^(admin::handlers::site_replication::tests::durable_iam_state_apply_iam_item_rejects_a_stale_recreate_after_a_replicated_delete|admin::service::federated_identity::tests::(binding_keeps_explicit_iam_mapping_strict|markerless_legacy_oidc_session_crosses_auth_and_iam_deletion_boundaries)|storage::scanner_dirty_journal::tests::(clear_callback_keeps_unknown_coverage_until_all_pending_work_is_confirmed|owned_journals_keep_peer_records_after_interleaved_writes_and_clear))$/)) | (package(rustfs-heal) & ((binary(heal_b5_versioned_regression_test) & test(/^serial_tests::(test_heal_unversioned_bucket_e2e|test_historical_null_recursive_heal_restores_physical_versions|test_recursive_deep_heal_reports_stale_delete_marker_repair)$/)) | (binary(usage_observation_read_repair_test) & test(=usage_observation_read_repair_survives_snapshot_cleanup)))) | (package(rustfs-test-utils) & test(=data_usage_snapshot_tests::data_usage_snapshot_recovery_wires_primary_backup_and_negative_cache))'
+setup = 'ecstore-base-stack'
+
 [[profile.ci.scripts]]
 filter = 'package(rustfs-ecstore) | package(rustfs-s3select-api) | package(rustfs-scanner) | (package(rustfs) & test(/^(app::multipart_usecase::tests::concurrent_completions_share_durable_bucket_quota_reservations|app::object::delete::tests::compressed_delete_requests_update_observed_usage_without_releasing_quota_floor|app::object::internal_put::tests::internal_multipart_roundtrip_completes_and_abort_leaves_nothing|app::object::restore::tests::execute_restore_object_maps_failures_to_typed_s3_errors|storage::access::tests::(delete_object_access_captures_authorized_bucket_incarnation|copy_operations_reject_recreated_source_bucket_after_authorization|request_slot_keeps_bucket_policy_bound_to_its_store))$/))'
 setup = 'ecstore-base-stack'
@@ -589,6 +599,14 @@ test-group = 'ecstore-serial-flaky'
 # the PR-lane e2e coverage for scanner usage snapshots consumed by quota and
 # admin surfaces. It uses the same single-node, random-port, isolated-temp-dir
 # fixture as the existing smoke modules.
+#
+# Dual-stack differential (rustfs/backlog#2740): `dual_stack_differential::`
+# starts two single-node servers from the same binary, one per
+# RUSTFS_S3_STACK value, on random ports with isolated temp dirs, and runs one
+# SDK script covering every operation of `impl S3 for FS` against both. Its
+# comparator unit tests need no server. It meets the admission criteria
+# unchanged; the module clause admits a new differential test without another
+# filter edit.
 [profile.e2e-smoke]
 default-filter = """
     package(e2e_test) & (
@@ -596,6 +614,7 @@ default-filter = """
         | test(/^replication_extension_test::(test_replication_check_succeeds_with_remote_target|test_replication_check_rejects_target_without_object_lock|test_set_remote_target_rejects_unversioned_source_bucket|test_replication_check_rejects_unversioned_source_bucket|test_replication_check_rejects_missing_replication_config|test_replication_check_rejects_invalid_bucket|test_set_remote_target_rejects_same_bucket_on_same_deployment|test_set_remote_target_rejects_unversioned_target_bucket|test_set_remote_target_update_requires_arn|test_set_remote_target_update_rejects_missing_target|test_set_remote_target_rejects_invalid_target_url|test_set_remote_target_rejects_self_signed_https_target_without_skip_tls_verify|test_set_remote_target_rejects_private_ca_https_target_without_ca_cert_pem|test_list_remote_targets_rejects_empty_bucket|test_list_remote_targets_rejects_invalid_bucket|test_remove_remote_target_rejects_missing_target|test_remove_remote_target_rejects_missing_arn|test_remove_remote_target_rejects_invalid_bucket|test_remove_remote_target_rejects_target_used_by_replication|test_delete_bucket_replication_removes_remote_target)$/)
         | test(/^reliant::lifecycle::/)
         | test(/^reliant::tiering::/)
+        | test(/^dual_stack_differential::/)
         | test(/^upgrade_compatibility_test::upgrade_write_readiness_tests::/)
         | test(/^on_demand_migration::(get_basic_test::(get_miss_pulls_inline_and_serves_locally_afterwards|head_miss_answers_from_the_source_without_persisting)|interaction_test::test_odm_admin_config_is_redacted_and_status_counts_match_the_source)$/)
     )

.config/s3s-edge-allowlist.txt

@@ -0,0 +1,37 @@
+# Files that may still reference s3s (`s3s::` or `s3s_sigv4`) when Phase 1 of
+# the gateway migration ends (rustfs/backlog#2734 task T0.6, rustfs/backlog#2741).
+# Consumed only by `scripts/check_s3s_footprint.sh --mode allowlist`.
+#
+# One entry per line: a repository-relative file path or a git `:(glob)`
+# pathspec. `**` matches across directories, `*` stays within one path
+# component, and a directory entry matches everything under it. Blank lines
+# and lines starting with `#` are ignored; an entry may not contain whitespace,
+# so a note belongs on its own comment line.
+#
+# Every Rust file listed by `git grep -l '\bs3s\(::\|_sigv4\)' -- '*.rs'` that
+# no entry matches fails the allowlist mode. Task T1.9 switches CI to that
+# mode; until then `--dry-run` prints the files still outside this list, which
+# is the Phase 1 tracking list.
+#
+# Never add crates/** other than crates/e2e_test: the engine crates leave s3s
+# entirely, and the script rejects such an entry. Only the s3s-facing edge of
+# the rustfs binary and the e2e oracle keep it.
+
+# rustfs binary: the s3s-facing edge that survives Phase 1
+rustfs/src/storage/ecfs.rs
+rustfs/src/storage/s3_compat/**
+rustfs/src/storage/access.rs
+rustfs/src/auth.rs
+rustfs/src/error.rs
+rustfs/src/storage_api.rs
+rustfs/src/app/storage_api.rs
+rustfs/src/admin/router.rs
+rustfs/src/app/metadata_route.rs
+rustfs/src/server/http.rs
+rustfs/src/server/layer.rs
+rustfs/src/server/hybrid.rs
+rustfs/src/protocols/client.rs
+
+# e2e oracle: the test target that verifies S3 behaviour against s3s
+crates/e2e_test/src/fake_s3_target/**
+crates/e2e_test/src/**

.config/s3s-footprint-baseline.txt

@@ -0,0 +1,17 @@
+# Exact baseline for the direct s3s footprint, consumed only by
+# scripts/check_s3s_footprint.sh (rustfs/backlog#1677; rustfs/backlog#2734 T0.4).
+#
+# Each value is the exact measurement of the tree, not a ceiling. CI fails when a
+# measured counter differs from its line here in either direction:
+#   - a larger measurement means new code widened the s3s surface: route it
+#     through the gateway abstractions instead; never raise a value here;
+#   - a smaller measurement means the PR shrank the surface: lower the matching
+#     value to the measured number in the same PR.
+#
+# files           Rust files referencing s3s paths, repo-wide minus crates/e2e_test
+# s3_error_lines  lines holding s3_error!, same scope, minus those the script
+#                 header's rule clears (rustfs_s3_types's macro); fails closed
+# ecstore_files   Rust files referencing s3s paths under crates/ecstore/src
+files=194
+s3_error_lines=1565
+ecstore_files=36

.config/test-count-baseline.txt

@@ -0,0 +1,63 @@
+# Per-crate test-count baseline consumed by scripts/check_test_count_ratchet.sh
+# (rustfs/backlog#2734 section 1 rule 3; T0.3 in rustfs/backlog#2738).
+#
+# One row per workspace member: <package>, tests, ignored, tab-separated.
+# tests counts #[test], #[tokio::test], #[tokio::test(...)] and #[test_case(...)]
+# attribute lines under the package directory; ignored counts #[ignore] and
+# #[ignore = "..."]. Both columns are exact: CI fails when a crate's tests fall
+# below or rise above its row, or when its ignored count changes. Raise a row in
+# the PR that adds tests; lowering one is a reviewed statement that tests were
+# intentionally removed. Regenerate with:
+#   scripts/check_test_count_ratchet.sh --update
+e2e_test	902	18
+rustfs	5653	62
+rustfs-audit	70	0
+rustfs-checksums	26	0
+rustfs-common	25	0
+rustfs-concurrency	27	0
+rustfs-config	43	0
+rustfs-connect-symbol-catalog	6	0
+rustfs-credentials	24	0
+rustfs-crypto	67	0
+rustfs-data-usage	54	0
+rustfs-ecstore	6495	1
+rustfs-extension-schema	23	0
+rustfs-filemeta	306	0
+rustfs-heal	843	3
+rustfs-heal-contracts	6	0
+rustfs-iam	390	0
+rustfs-io-core	49	0
+rustfs-io-metrics	127	0
+rustfs-keystone	29	0
+rustfs-kms	715	25
+rustfs-license	2	0
+rustfs-lifecycle	152	0
+rustfs-lock	135	0
+rustfs-log-analyzer	89	0
+rustfs-madmin	209	0
+rustfs-notify	136	0
+rustfs-object-capacity	92	0
+rustfs-object-data-cache	140	0
+rustfs-obs	367	0
+rustfs-policy	681	0
+rustfs-protocols	736	1
+rustfs-protos	63	0
+rustfs-replication	200	0
+rustfs-rio	200	0
+rustfs-rio-v2	34	6
+rustfs-s3-client	65	0
+rustfs-s3-ops	8	0
+rustfs-s3-types	97	0
+rustfs-s3select-api	206	0
+rustfs-s3select-query	188	2
+rustfs-scanner	997	20
+rustfs-scanner-metrics	81	0
+rustfs-security-governance	20	0
+rustfs-signer	31	0
+rustfs-storage-api	31	0
+rustfs-targets	603	26
+rustfs-test-utils	1	0
+rustfs-tls-runtime	43	0
+rustfs-trusted-proxies	68	0
+rustfs-utils	217	0
+rustfs-zip	30	0

.docker/observability/grafana/dashboards/rustfs.json

@@ -1748,7 +1748,7 @@
             "uid": "${datasource}"
           },
           "editorMode": "code",
-          "expr": "max by (job, bucket) (rustfs_cluster_usage_buckets_objects_count{job=~\"$job\", bucket=~\"$bucket\"})",
+          "expr": "max by (rustfs_cluster_id, job, bucket) (rustfs_cluster_usage_buckets_objects_count{rustfs_cluster_id=\"$storage_cluster\", job=~\"$job\", bucket=~\"$bucket\"})",
           "legendFormat": "{{bucket}}",
           "range": true,
           "refId": "A"
@@ -1848,7 +1848,7 @@
             "uid": "${datasource}"
           },
           "editorMode": "code",
-          "expr": "max by (job, bucket) (rustfs_cluster_usage_buckets_total_bytes{job=~\"$job\", bucket=~\"$bucket\"})",
+          "expr": "max by (rustfs_cluster_id, job, bucket) (rustfs_cluster_usage_buckets_total_bytes{rustfs_cluster_id=\"$storage_cluster\", job=~\"$job\", bucket=~\"$bucket\"})",
           "legendFormat": "{{bucket}}",
           "range": true,
           "refId": "A"

.docker/observability/prometheus.yml

@@ -61,6 +61,7 @@ scrape_configs:
 otlp:
   promote_resource_attributes:
     - service.instance.id
+    - process.pid
     - service.name
     - service.namespace
     - cloud.availability_zone

.docker/observability/tests/bucket-usage.test.yml

@@ -0,0 +1,48 @@
+evaluation_interval: 1m
+tests:
+- name: bucket usage keeps clusters separate and deduplicates reporting nodes
+  interval: 1m
+  input_series:
+  - series: rustfs_cluster_usage_buckets_total_bytes{rustfs_cluster_id="a",job="rustfs",bucket="shared",service_instance_id="a1"}
+    values: '4096 1024'
+  - series: rustfs_cluster_usage_buckets_total_bytes{rustfs_cluster_id="a",job="rustfs",bucket="shared",service_instance_id="a2"}
+    values: '4096 1024'
+  - series: rustfs_cluster_usage_buckets_total_bytes{rustfs_cluster_id="b",job="rustfs",bucket="shared",service_instance_id="b1"}
+    values: '65536 65536'
+  - series: rustfs_cluster_usage_buckets_total_bytes{rustfs_cluster_id="a",job="rustfs",bucket="empty",service_instance_id="a1"}
+    values: '0 0'
+  - series: rustfs_cluster_usage_buckets_objects_count{rustfs_cluster_id="a",job="rustfs",bucket="shared",service_instance_id="a1"}
+    values: '3 1'
+  - series: rustfs_cluster_usage_buckets_objects_count{rustfs_cluster_id="a",job="rustfs",bucket="shared",service_instance_id="a2"}
+    values: '3 1'
+  - series: rustfs_cluster_usage_buckets_objects_count{rustfs_cluster_id="b",job="rustfs",bucket="shared",service_instance_id="b1"}
+    values: '70 70'
+  - series: rustfs_cluster_usage_buckets_objects_count{rustfs_cluster_id="a",job="rustfs",bucket="empty",service_instance_id="a1"}
+    values: '0 0'
+  promql_expr_test:
+  - expr: max by (rustfs_cluster_id, job, bucket) (rustfs_cluster_usage_buckets_total_bytes{rustfs_cluster_id="a",job=~"rustfs",bucket=~".*"})
+    eval_time: 1m
+    exp_samples:
+    - labels: '{rustfs_cluster_id="a",job="rustfs",bucket="shared"}'
+      value: 1024
+    - labels: '{rustfs_cluster_id="a",job="rustfs",bucket="empty"}'
+      value: 0
+  - expr: max by (rustfs_cluster_id, job, bucket) (rustfs_cluster_usage_buckets_objects_count{rustfs_cluster_id="a",job=~"rustfs",bucket=~".*"})
+    eval_time: 1m
+    exp_samples:
+    - labels: '{rustfs_cluster_id="a",job="rustfs",bucket="shared"}'
+      value: 1
+    - labels: '{rustfs_cluster_id="a",job="rustfs",bucket="empty"}'
+      value: 0
+  - expr: max by (rustfs_cluster_id, job, bucket) (rustfs_cluster_usage_buckets_total_bytes)
+    eval_time: 1m
+    exp_samples:
+    - labels: '{rustfs_cluster_id="a",job="rustfs",bucket="shared"}'
+      value: 1024
+    - labels: '{rustfs_cluster_id="a",job="rustfs",bucket="empty"}'
+      value: 0
+    - labels: '{rustfs_cluster_id="b",job="rustfs",bucket="shared"}'
+      value: 65536
+  - expr: rustfs_cluster_usage_buckets_total_bytes{rustfs_cluster_id="a",bucket="unknown"}
+    eval_time: 1m
+    exp_samples: []

.github/actions/quick-checks/action.yml

@@ -86,6 +86,10 @@ runs:
       shell: bash
       run: ./scripts/check_s3s_footprint.sh
 
+    - name: Check per-crate test-count ratchet
+      shell: bash
+      run: ./scripts/check_test_count_ratchet.sh
+
     - name: Check cryptographic capability wording
       shell: bash
       run: ./scripts/check_fips_wording.sh

.github/actions/setup/action.yml

@@ -36,6 +36,16 @@ inputs:
     description: "Install cross-compilation tools"
     required: false
     default: "false"
+  zig-version:
+    description: >-
+      Zig version for cargo-zigbuild builds. Pinned so the `<target>.<glibc>`
+      baseline semantics cannot drift with an untested zig release.
+    required: false
+    default: "0.17.0"
+  cargo-zigbuild-version:
+    description: "cargo-zigbuild version installed for cross builds."
+    required: false
+    default: "0.23.4"
   target:
     description: "Target architecture to add"
     required: false
@@ -99,10 +109,14 @@ runs:
     - name: Install Zig
       if: inputs.install-cross-tools == 'true'
       uses: mlugg/setup-zig@d1434d08867e3ee9daa34448df10607b98908d29 # v2
+      with:
+        version: ${{ inputs.zig-version }}
 
     - name: Install cargo-zigbuild
       if: inputs.install-cross-tools == 'true'
       uses: taiki-e/install-action@a21ae4029b089b9ddc45704028756f51ab8abe48 # cargo-zigbuild
+      with:
+        tool: cargo-zigbuild@${{ inputs.cargo-zigbuild-version }}
 
     - name: Install cargo-nextest
       if: inputs.install-test-tools == 'true'
@@ -126,5 +140,5 @@ runs:
         # normalises, so cargo fingerprints stay valid.
         cache-all-crates: false
         cache-on-failure: true
-        shared-key: ${{ inputs.cache-shared-key }}
+        shared-key: ${{ inputs.cache-shared-key }}${{ inputs.install-cross-tools == 'true' && format('-zig-{0}-zigbuild-{1}', inputs.zig-version, inputs.cargo-zigbuild-version) || '' }}
         save-if: ${{ inputs.cache-save-if }}

.github/workflows/audit.yml

@@ -20,6 +20,7 @@ on:
     paths:
       - '**/Cargo.toml'
       - '**/Cargo.lock'
+      - 'third-party/pyroscope/**'
       - 'deny.toml'
       - '.github/actions/**'
       - '.github/workflows/**'
@@ -39,6 +40,7 @@ on:
     paths:
       - '**/Cargo.toml'
       - '**/Cargo.lock'
+      - 'third-party/pyroscope/**'
       - 'deny.toml'
       - '.github/actions/**'
       - '.github/workflows/**'
@@ -129,6 +131,13 @@ jobs:
       - name: Run cargo-deny
         run: cargo deny check --hide-inclusion-graph advisories sources bans licenses
 
+      - name: Test patched profiling dependency
+        env:
+          CARGO_TARGET_DIR: ${{ github.workspace }}/target/pyroscope-dependency
+        run: >-
+          cargo test --manifest-path third-party/pyroscope/Cargo.toml
+          --features backend-pprof-rs --lib --test rustfs_zstd
+
   workflow-pin-report:
     name: Workflow Pin Report
     if: github.repository == 'rustfs/rustfs' && (github.event_name != 'pull_request' || github.event.action != 'closed')

.github/workflows/build.yml

@@ -213,12 +213,12 @@ jobs:
           fi
 
           all='{"include":[
-            {"target_id":"linux-x86_64-musl","os":"sm-standard-4","target":"x86_64-unknown-linux-musl","cross":false,"platform":"linux","rustflags":"","features":""},
-            {"target_id":"linux-aarch64-musl","os":"sm-standard-4-arm","target":"aarch64-unknown-linux-musl","cross":false,"platform":"linux","rustflags":"","features":""},
-            {"target_id":"linux-x86_64-gnu","os":"sm-standard-4","target":"x86_64-unknown-linux-gnu","cross":false,"platform":"linux","rustflags":"","features":"pyroscope"},
-            {"target_id":"linux-aarch64-gnu","os":"sm-standard-4-arm","target":"aarch64-unknown-linux-gnu","cross":false,"platform":"linux","rustflags":"","features":"pyroscope"},
-            {"target_id":"macos-aarch64","os":"macos-latest","target":"aarch64-apple-darwin","cross":false,"platform":"macos","rustflags":"","features":"pyroscope"},
-            {"target_id":"windows-x86_64","os":"windows-latest","target":"x86_64-pc-windows-msvc","cross":false,"platform":"windows","rustflags":"","features":""}
+            {"target_id":"linux-x86_64-musl","os":"sm-standard-4","target":"x86_64-unknown-linux-musl","cross":false,"platform":"linux","rustflags":"","features":"","glibc":""},
+            {"target_id":"linux-aarch64-musl","os":"sm-standard-4-arm","target":"aarch64-unknown-linux-musl","cross":false,"platform":"linux","rustflags":"","features":"","glibc":""},
+            {"target_id":"linux-x86_64-gnu","os":"sm-standard-4","target":"x86_64-unknown-linux-gnu","cross":true,"platform":"linux","rustflags":"","features":"pyroscope","glibc":"2.31"},
+            {"target_id":"linux-aarch64-gnu","os":"sm-standard-4-arm","target":"aarch64-unknown-linux-gnu","cross":true,"platform":"linux","rustflags":"","features":"pyroscope","glibc":"2.31"},
+            {"target_id":"macos-aarch64","os":"macos-latest","target":"aarch64-apple-darwin","cross":false,"platform":"macos","rustflags":"","features":"pyroscope","glibc":""},
+            {"target_id":"windows-x86_64","os":"windows-latest","target":"x86_64-pc-windows-msvc","cross":false,"platform":"windows","rustflags":"","features":"","glibc":""}
           ]}'
 
           if [[ "${selected}" == "all" ]]; then
@@ -246,7 +246,8 @@ jobs:
           echo "Selected platforms: ${selected}"
 
   build-rustfs:
-    name: Build RustFS
+    # Acceptance workflows select artifacts by this stable platform name.
+    name: Build RustFS (${{ matrix.target_id }})
     needs: [ build-check, prepare-platform-matrix ]
     if: needs.build-check.outputs.should_build == 'true' && needs.prepare-platform-matrix.result == 'success'
     runs-on: ${{ matrix.os }}
@@ -273,7 +274,7 @@ jobs:
         with:
           rust-version: stable
           target: ${{ matrix.target }}
-          cache-shared-key: build-${{ matrix.target }}
+          cache-shared-key: build-${{ matrix.target }}${{ matrix.glibc && format('-glibc-{0}', matrix.glibc) || '' }}
           # main only. A cache saved on refs/tags/X is scoped to that tag: no
           # other tag, no main run and no PR can restore it, so every release
           # cycle wrote up to 12 entries of 1-2GB (preview tag plus final tag,
@@ -410,35 +411,53 @@ jobs:
         shell: bash
         env:
           BUILD_TARGET: ${{ matrix.target }}
+          BUILD_OS: ${{ matrix.platform }}-${{ startsWith(matrix.target, 'aarch64-') && 'aarch64' || 'x86_64' }}
         run: |
           # Force rebuild by touching build.rs
           touch rustfs/build.rs
 
-          BINARY_NAMES=(rustfs)
-          if [[ "${{ matrix.platform }}" == "linux" ]]; then
-            BINARY_NAMES+=(rustfs-cli)
-          fi
           FEATURE_ARGS=()
           if [[ -n "${{ matrix.features }}" ]]; then
             FEATURE_ARGS=(--features "${{ matrix.features }}")
           fi
 
-          # Build one binary at a time so release LTO links cannot exhaust a
-          # self-hosted runner by running concurrently.
-          for binary in "${BINARY_NAMES[@]}"; do
-            PROFILE_ARGS=()
-            if [[ "$binary" == "rustfs" && "$BUILD_TARGET" == *-linux-gnu && "${{ matrix.features }}" == "pyroscope" ]]; then
-              # The in-process profiler resolves names from the final executable.
-              # Retain function symbols without enabling DWARF source information.
-              PROFILE_ARGS=(--config 'profile.release.strip="none"')
-            fi
-            if [[ "${{ matrix.cross }}" == "true" ]]; then
-              # All cross targets in the matrix are Linux; zigbuild handles them.
-              cargo zigbuild --release --target ${{ matrix.target }} "${FEATURE_ARGS[@]}" "${PROFILE_ARGS[@]}" -p rustfs --bin "$binary"
-            else
-              cargo build --release --target ${{ matrix.target }} "${FEATURE_ARGS[@]}" "${PROFILE_ARGS[@]}" -p rustfs --bin "$binary"
-            fi
-          done
+          # gnu targets pin an explicit glibc baseline (`<target>.<glibc>` for
+          # cargo-zigbuild) so release binaries link no symbols newer than it
+          # and run on older distros such as RHEL/Rocky 9 (glibc 2.34); #8062.
+          TARGET_ARG="$BUILD_TARGET"
+          if [[ -n "${{ matrix.glibc }}" ]]; then
+            TARGET_ARG="${TARGET_ARG}.${{ matrix.glibc }}"
+          fi
+
+          PROFILE_ARGS=()
+          if [[ "$BUILD_TARGET" == *-linux-gnu && "${{ matrix.features }}" == "pyroscope" ]]; then
+            # The in-process profiler resolves names from the final executable.
+            # Retain function symbols without enabling DWARF source information.
+            PROFILE_ARGS=(--config 'profile.release.strip="none"')
+          fi
+          if [[ "${{ matrix.cross }}" == "true" ]]; then
+            cargo zigbuild --release --locked --target "$TARGET_ARG" "${FEATURE_ARGS[@]}" "${PROFILE_ARGS[@]}" -p rustfs --bin rustfs
+          else
+            cargo build --release --locked --target "$TARGET_ARG" "${FEATURE_ARGS[@]}" "${PROFILE_ARGS[@]}" -p rustfs --bin rustfs
+          fi
+
+          # All release targets use same-architecture runners, including GNU
+          # builds that use Zig only to select the glibc ABI floor.
+          bin="target/${{ matrix.target }}/release/rustfs"
+          if [[ -f "${bin}.exe" ]]; then
+            bin="${bin}.exe"
+          fi
+          version_output=$("$bin" --version)
+          if ! grep -Fxq "build target : ${BUILD_TARGET}" <<< "$version_output" || \
+             ! grep -Fxq "build os     : ${BUILD_OS}" <<< "$version_output" || \
+             ! grep -Fxq "rust channel : stable-${BUILD_TARGET}" <<< "$version_output"; then
+            echo "release binary reports an unexpected compile target" >&2
+            exit 1
+          fi
+          if [[ -n "${{ matrix.glibc }}" ]]; then
+            python3 scripts/release/check_glibc.py "${{ matrix.glibc }}" \
+              "target/${{ matrix.target }}/release/rustfs"
+          fi
 
       - name: Verify macOS binary has no dynamic liblzma dependency
         if: matrix.platform == 'macos'
@@ -554,9 +573,6 @@ jobs:
           fi
 
           BINARY_FILES=("$BINARY_NAME")
-          if [[ "${{ matrix.platform }}" == "linux" ]]; then
-            BINARY_FILES+=("rustfs-cli")
-          fi
 
           if [[ "$TARGET" == *-linux-gnu && "${{ matrix.features }}" == "pyroscope" ]]; then
             BINARY_FILES+=(rustfs.cpu-symbol-catalog.json rustfs.cpu-symbol-catalog.sha256)
@@ -671,7 +687,7 @@ jobs:
           echo "🔧 Build type: ${BUILD_TYPE}"
           echo "📊 Version: ${VERSION}"
 
-      - name: Verify packaged Linux binaries
+      - name: Verify packaged Linux binary
         if: matrix.platform == 'linux'
         shell: bash
         run: |
@@ -679,7 +695,7 @@ jobs:
 
           package_file="${{ steps.package.outputs.package_file }}"
           archive_members="$(unzip -Z1 "$package_file" | sort)"
-          expected_members=$'rustfs\nrustfs-cli'
+          expected_members='rustfs'
           target="${{ matrix.target }}"
           features="${{ matrix.features }}"
           if [[ "$target" == *-linux-gnu && "$features" == "pyroscope" ]]; then
@@ -691,13 +707,14 @@ jobs:
             exit 1
           fi
 
-          # Every Linux target builds natively, so executing the binaries here
+          # musl targets build natively; the gnu legs are same-arch
+          # cargo-zigbuild builds pinned to the release glibc baseline. Either
+          # way the binaries run on their runner, so executing them here
           # proves each artifact matches its runner architecture.
-          cli_dir="$(mktemp -d)"
-          trap 'rm -rf "$cli_dir"' EXIT
-          unzip -q "$package_file" rustfs rustfs-cli -d "$cli_dir"
-          "$cli_dir/rustfs" --version
-          "$cli_dir/rustfs-cli" --help >/dev/null
+          binary_dir="$(mktemp -d)"
+          trap 'rm -rf "$binary_dir"' EXIT
+          unzip -q "$package_file" rustfs -d "$binary_dir"
+          "$binary_dir/rustfs" --version
 
       - name: Verify packaged console
         if: matrix.target == 'x86_64-unknown-linux-gnu' || matrix.target == 'aarch64-unknown-linux-gnu'

.github/workflows/cache-warm.yml

@@ -99,7 +99,7 @@ env:
   RUST_BACKTRACE: 1
 
 jobs:
-  # Readers: test-and-lint, test-ilm-integration-serial, build-rustfs-debug-binary,
+  # Readers: test-and-lint (workspace and ILM), build-rustfs-debug-binary,
   # e2e-tests, e2e-full.
   warm-ci-dev:
     name: Warm ci-dev

.github/workflows/ci.yml

@@ -16,7 +16,9 @@ name: Continuous Integration
 
 on:
   push:
-    branches: [ main ]
+    # gateway/integration receives the daily merge of main as a direct push
+    # (rustfs/backlog#2734), so its pushes must prove the result like main's.
+    branches: [ main, gateway/integration ]
     paths-ignore:
       - "**.md"
       - "docs/**"
@@ -168,13 +170,10 @@ jobs:
             echo "started_at=$(date --utc --iso-8601=seconds)"
           } > artifacts/test-and-lint/run-metadata.txt
 
-      # Clippy runs before the test pass: lint failures are the most common
-      # CI-only breakage and should surface in minutes, not after 20+ minutes
-      # of tests.
-      # Sampled too: clippy is the natural control arm for any CARGO_BUILD_JOBS
-      # experiment, since --all-targets is check-only for workspace members and
-      # never links the ~100 test binaries the limit exists to throttle.
+      # Keep full Clippy validation on main, merge groups, and scheduled/manual
+      # runs. PRs retain the complete default test suite and compile-free checks.
       - name: Run clippy lints
+        if: github.event_name != 'pull_request'
         run: |
           ./scripts/ci/resource_sampler.sh start clippy
           trap './scripts/ci/resource_sampler.sh stop' EXIT
@@ -283,6 +282,49 @@ jobs:
       - name: Verify rebalance/decommission migration proofs
         run: ./scripts/check_migration_gate_count.sh evidence artifacts/test-and-lint/core-test-listing.json target/nextest/ci/junit.xml
 
+      # The workspace already compiled these ignored tests. Keep the same Cargo
+      # package selection to reuse their binaries, then run them serially in
+      # separate processes because they share fixed ports and global state.
+      # test_transition_and_restore_flows was re-enabled by rustfs/backlog#1303:
+      # its "missing xl.meta on disk2" was a test-util bug (open_disk hardcoded
+      # disk_index 0), not an EC metadata-distribution issue.
+      # restore_object_usecase_reports_ongoing_conflict_and_completion was
+      # re-enabled by backlog#1304 (restore accepts serialize on a short CAS
+      # guard; the copy-back no longer holds the #4877 whole-copy-back lock,
+      # so the mid-restore ongoing read and fast 409 rejection it asserts are
+      # the implemented contract).
+      - name: Run ignored ILM integration tests serially
+        if: needs.classify-changes.outputs.mode == 'full'
+        env:
+          # Match the preceding workspace build and reuse its feature graph.
+          CARGO_BUILD_JOBS: ${{ (github.event_name == 'push' || github.event_name == 'workflow_dispatch') && '3' || '2' }}
+        run: |
+          mkdir -p artifacts/ilm-integration
+          set +e
+          NEXTEST_HIDE_PROGRESS_BAR=1 timeout --verbose --signal=TERM --kill-after=30s 80m \
+            cargo nextest run --profile default -j1 --run-ignored ignored-only \
+            --all --exclude e2e_test \
+            -E 'binary(lifecycle_integration_test) or (package(rustfs) and test(lifecycle_transition_api_test))' \
+            --status-level all --final-status-level all \
+            2>&1 | tee artifacts/ilm-integration/nextest.log
+          status=${PIPESTATUS[0]}
+          {
+            echo "exit_status=${status}"
+            echo "finished_at=$(date --utc --iso-8601=seconds)"
+            echo
+            echo "Remaining test-related processes:"
+            pgrep -af 'cargo|nextest|target/.*/deps/' || true
+          } > artifacts/ilm-integration/diagnostics.txt
+          exit "${status}"
+
+      - name: Upload ILM test diagnostics
+        if: always() && needs.classify-changes.outputs.mode == 'full'
+        uses: actions/upload-artifact@043fb46d1a93c77aae656e7c1c64a875d1fc6a0a # v7.0.1
+        with:
+          name: ilm-integration-${{ github.run_number }}-${{ github.run_attempt }}
+          path: artifacts/ilm-integration
+          retention-days: 3
+
   # This gate builds into a fresh target directory to isolate the E2E root.
   # Give its cold build a separate budget from workspace tests and migration proofs.
   offline-enrollment-e2e:
@@ -314,83 +356,11 @@ jobs:
       - name: Check offline enrollment E2E root boundary
         run: ./scripts/check_offline_enrollment_e2e.sh
 
-  # Dedicated serial lane for the ILM / lifecycle integration tests. These tests
-  # drive the object layer through process-global singletons (the GLOBAL_ENV
-  # ECStore, the global tier-config manager, background-expiry workers) and bind
-  # fixed ports (9002 for the scanner suite, 9003 for the app suite), so they are
-  # marked #[ignore] to keep them out of the default parallel `cargo nextest run
-  # --all` pass. Run them here explicitly with `--run-ignored ignored-only` and
-  # `-j1` so no two of them share global state or race for a port at once.
-  # serial_test's #[serial] does NOT serialize across nextest's process-per-test
-  # boundary (see .config/nextest.toml); `-j1` is what actually serializes them.
-  # See rustfs/backlog#1148 (ilm-1) and #1155.
-  test-ilm-integration-serial:
-    name: ILM Integration (serial)
-    if: needs.classify-changes.outputs.mode == 'full' && (github.event_name != 'pull_request' || github.event.action != 'closed')
-    needs: [ quick-checks, classify-changes ]
-    runs-on: sm-standard-4
-    timeout-minutes: 90
-    env:
-      FORCE_JAVASCRIPT_ACTIONS_TO_NODE24: "true"
-    steps:
-      - name: Checkout repository
-        uses: actions/checkout@f548e57e544e1ff5a4c46bf1e1b8685f8e4a348a # v7
-        with:
-          persist-credentials: false
-
-      - name: Setup Rust environment
-        uses: ./.github/actions/setup
-        with:
-          rust-version: stable
-          cache-shared-key: ci-dev
-          cache-save-if: 'false'
-          install-build-packaging-tools: 'false'
-
-      # test_transition_and_restore_flows was re-enabled by rustfs/backlog#1303:
-      # its "missing xl.meta on disk2" was a test-util bug (open_disk hardcoded
-      # disk_index 0), not an EC metadata-distribution issue.
-      # restore_object_usecase_reports_ongoing_conflict_and_completion was
-      # re-enabled by backlog#1304 (restore accepts serialize on a short CAS
-      # guard; the copy-back no longer holds the #4877 whole-copy-back lock,
-      # so the mid-restore ongoing read and fast 409 rejection it asserts are
-      # the implemented contract).
-      - name: Run ignored ILM integration tests serially
-        env:
-          # Match the measured Test and Lint link budget. The default exposed
-          # all 14 pod CPUs and a cold cache spent the full 80m compiling
-          # without starting one ILM test (main run 32982910990).
-          CARGO_BUILD_JOBS: ${{ (github.event_name == 'push' || github.event_name == 'workflow_dispatch') && '3' || '2' }}
-        run: |
-          mkdir -p artifacts/ilm-integration
-          set +e
-          NEXTEST_HIDE_PROGRESS_BAR=1 timeout --verbose --signal=TERM --kill-after=30s 80m \
-            cargo nextest run -j1 --run-ignored ignored-only \
-            -p rustfs-scanner -p rustfs \
-            -E 'binary(lifecycle_integration_test) or (package(rustfs) and test(lifecycle_transition_api_test))' \
-            --status-level all --final-status-level all \
-            2>&1 | tee artifacts/ilm-integration/nextest.log
-          status=${PIPESTATUS[0]}
-          {
-            echo "exit_status=${status}"
-            echo "finished_at=$(date --utc --iso-8601=seconds)"
-            echo
-            echo "Remaining test-related processes:"
-            pgrep -af 'cargo|nextest|target/.*/deps/' || true
-          } > artifacts/ilm-integration/diagnostics.txt
-          exit "${status}"
-
-      - name: Upload ILM test diagnostics
-        if: always()
-        uses: actions/upload-artifact@043fb46d1a93c77aae656e7c1c64a875d1fc6a0a # v7.0.1
-        with:
-          name: ilm-integration-${{ github.run_number }}-${{ github.run_attempt }}
-          path: |
-            artifacts/ilm-integration
-            target/nextest/ci/junit.xml
-
   test-and-lint-rio-v2:
     name: Test and Lint (rio-v2)
-    if: needs.classify-changes.outputs.mode == 'full' && (github.event_name != 'pull_request' || github.event.action != 'closed')
+    # Alternate feature configurations run on main, merge groups, and scheduled
+    # or manual validation; the default workspace remains the PR gate.
+    if: needs.classify-changes.outputs.mode == 'full' && github.event_name != 'pull_request'
     needs: [ quick-checks, classify-changes ]
     runs-on: sm-standard-4
     timeout-minutes: 90
@@ -506,7 +476,7 @@ jobs:
 
   test-and-lint-protocols:
     name: "Test and Lint (${{ matrix.features.name }})"
-    if: needs.classify-changes.outputs.mode == 'full' && (github.event_name != 'pull_request' || github.event.action != 'closed')
+    if: needs.classify-changes.outputs.mode == 'full' && github.event_name != 'pull_request'
     needs: [ quick-checks, classify-changes ]
     runs-on: sm-standard-4
     timeout-minutes: 90
@@ -780,6 +750,8 @@ jobs:
     needs: [ build-rustfs-debug-binary ]
     runs-on: sm-standard-2
     timeout-minutes: 30
+    env:
+      RUSTFS_S3_STACK: legacy
     steps:
       - name: Checkout repository
         uses: actions/checkout@f548e57e544e1ff5a4c46bf1e1b8685f8e4a348a # v7
@@ -898,18 +870,105 @@ jobs:
           path: ${{ runner.temp }}/rustfs-e2e-*/rustfs.log
           retention-days: 3
 
+  # The same e2e-smoke suite against the same debug binary with the gateway
+  # S3 stack selected (rustfs/backlog#2734). Report-only: it is outside the
+  # Test and Lint aggregate and becomes required at T2.10. The archive and run
+  # steps must stay identical to e2e-tests; scripts/ci_gate.py enforces that.
+  e2e-tests-gateway:
+    name: e2e-smoke (gateway, report-only until T2.10)
+    needs: [ build-rustfs-debug-binary ]
+    runs-on: sm-standard-2
+    timeout-minutes: 30
+    continue-on-error: true
+    env:
+      RUSTFS_S3_STACK: gateway
+    steps:
+      - name: Checkout repository
+        uses: actions/checkout@f548e57e544e1ff5a4c46bf1e1b8685f8e4a348a # v7
+        with:
+          persist-credentials: false
+
+      - name: Setup Rust environment
+        uses: ./.github/actions/setup
+        with:
+          rust-version: stable
+          cache-shared-key: ci-dev
+          cache-save-if: 'false'
+          install-build-packaging-tools: 'false'
+
+      - name: Set up Python
+        uses: actions/setup-python@5fda3b95a4ea91299a34e894583c3862153e4b97 # v7.0.0
+        with:
+          python-version: "3.12"
+
+      - name: Install awscurl
+        run: |
+          python3 -m pip install --user --upgrade pip "awscurl==0.44"
+          echo "AWSCURL_PATH=$HOME/.local/bin/awscurl" >> "$GITHUB_ENV"
+
+      - name: Verify awscurl
+        run: test -x "$AWSCURL_PATH"
+
+      - name: Download debug binary
+        uses: actions/download-artifact@3e5f45b2cfb9172054b4087a40e8e0b5a5461e7c # v8.0.1
+        with:
+          name: rustfs-debug-binary
+          path: target/debug
+
+      - name: Make binary executable
+        run: chmod +x ./target/debug/rustfs
+
+      - name: Archive e2e smoke test binaries
+        env:
+          NEXTEST_ARCHIVE: ${{ runner.temp }}/rustfs-e2e-smoke.tar.zst
+          NEXTEST_LISTING: ${{ runner.temp }}/rustfs-e2e-smoke-list.json
+        run: |
+          cargo nextest archive --profile e2e-smoke -p e2e_test --archive-file "${NEXTEST_ARCHIVE}"
+          cargo nextest list --profile e2e-smoke --archive-file "${NEXTEST_ARCHIVE}" --message-format json > "${NEXTEST_LISTING}"
+          python3 ./scripts/check_test_wiring.py --check-profile e2e-smoke "${NEXTEST_LISTING}"
+          ./scripts/check_security_smoke_count.sh check "${NEXTEST_LISTING}"
+
+      - name: Run e2e smoke suite
+        env:
+          NEXTEST_ARCHIVE: ${{ runner.temp }}/rustfs-e2e-smoke.tar.zst
+          RUSTFS_E2E_LOG_DIR: ${{ runner.temp }}/rustfs-e2e-smoke-logs
+          SAMPLER_INTERVAL_SECS: "15"
+        run: |
+          ./scripts/ci/resource_sampler.sh start e2e-smoke
+          trap './scripts/ci/resource_sampler.sh stop' EXIT
+          python3 scripts/e2e_binary.py run --features e2e-test-hooks -- cargo nextest run --profile e2e-smoke --archive-file "${NEXTEST_ARCHIVE}" \
+            --status-level all --final-status-level all --failure-output final
+
+      - name: Upload e2e smoke diagnostics
+        if: always()
+        uses: actions/upload-artifact@043fb46d1a93c77aae656e7c1c64a875d1fc6a0a # v7.0.1
+        with:
+          name: e2e-smoke-gateway-diagnostics-${{ github.run_number }}
+          path: |
+            ${{ runner.temp }}/rustfs-e2e-smoke-logs/
+            ${{ runner.temp }}/rustfs-e2e-smoke-list.json
+          if-no-files-found: warn
+
+      - name: Upload e2e smoke JUnit report
+        if: always()
+        uses: actions/upload-artifact@043fb46d1a93c77aae656e7c1c64a875d1fc6a0a # v7.0.1
+        with:
+          name: e2e-smoke-gateway-junit-${{ github.run_number }}
+          path: target/nextest/e2e-smoke/junit.xml
+          if-no-files-found: warn
+
   e2e-full:
     name: End-to-End Tests (full merge gate)
     # Merge gate only (backlog#1149 ci-5): the never-automated user-visible
     # suites — KMS, object_lock, multipart_auth, quota, checksum, encryption,
     # security-boundary, ... — via the e2e-full nextest profile. Too heavy for
-    # every PR, so it is gated to main pushes, the merge queue, and manual
-    # dispatch. protocols / the 7 cluster suites / replication / #[ignore] are
-    # owned by other lanes (see .config/nextest.toml profile.e2e-full).
+    # every PR, so it is gated to main and gateway/integration pushes, the merge
+    # queue, and manual dispatch. protocols / the 7 cluster suites / replication
+    # / #[ignore] are owned by other lanes (see .config/nextest.toml profile.e2e-full).
     if: >-
       github.event_name == 'workflow_dispatch' ||
       github.event_name == 'merge_group' ||
-      (github.event_name == 'push' && github.ref == 'refs/heads/main')
+      (github.event_name == 'push' && (github.ref == 'refs/heads/main' || github.ref == 'refs/heads/gateway/integration'))
     needs: [ build-rustfs-debug-binary ]
     runs-on: sm-standard-2
     timeout-minutes: 55
@@ -1264,7 +1323,6 @@ jobs:
       - quick-checks
       - test-and-lint
       - offline-enrollment-e2e
-      - test-ilm-integration-serial
       - test-and-lint-rio-v2
       - connect-short-credential-boundary
       - test-and-lint-protocols
@@ -1298,7 +1356,6 @@ jobs:
       - quick-checks
       - test-and-lint
       - offline-enrollment-e2e
-      - test-ilm-integration-serial
       - test-and-lint-rio-v2
       - test-and-lint-protocols
       - build-rustfs-debug-binary

.github/workflows/connect-diagnostic-schedule-acceptance.yml

@@ -57,6 +57,11 @@ jobs:
           ref: ${{ inputs.connect_sha }}
           token: ${{ secrets.PF_TESTING_GH_TOKEN }}
 
+      - name: Set up Python
+        uses: actions/setup-python@5fda3b95a4ea91299a34e894583c3862153e4b97 # v7.0.0
+        with:
+          python-version: "3.14"
+
       - name: Set up Node.js
         uses: actions/setup-node@820762786026740c76f36085b0efc47a31fe5020 # v7.0.0
         with:
@@ -94,7 +99,7 @@ jobs:
           ' <<<"$run" >/dev/null
           jobs=$(gh api --paginate --slurp "repos/${GITHUB_REPOSITORY}/actions/runs/${BUILD_RUN_ID}/jobs?per_page=100")
           jq -e '
-            [.[].jobs[] | select(.name | test("^Build RustFS \\(linux-x86_64-gnu, [a-z0-9-]+, x86_64-unknown-linux-gnu, false, linux, pyroscope\\)$"))] as $matches
+            [.[].jobs[] | select(.name | test("^Build RustFS \\(linux-x86_64-gnu\\)$"))] as $matches
             | ($matches | length) == 1 and $matches[0].conclusion == "success"
           ' <<<"$jobs" >/dev/null
           artifact=$(gh api "repos/${GITHUB_REPOSITORY}/actions/artifacts/${ARTIFACT_ID}")
@@ -137,7 +142,7 @@ jobs:
               assert outer.testzip() is None
               with zipfile.ZipFile(io.BytesIO(outer.read(package))) as inner:
                   entries = inner.infolist()
-                  assert {item.filename for item in entries} == {'rustfs', 'rustfs-cli'} and len(entries) == 2
+                  assert {item.filename for item in entries} == {'rustfs', 'rustfs.cpu-symbol-catalog.json', 'rustfs.cpu-symbol-catalog.sha256'} and len(entries) == 3
                   assert all(regular(item) and 0 < item.file_size <= 1073741824 for item in entries)
                   assert inner.testzip() is None
                   pathlib.Path('binary').mkdir()

.github/workflows/connect-health-service-acceptance.yml

@@ -57,6 +57,11 @@ jobs:
           ref: ${{ inputs.connect_sha }}
           token: ${{ secrets.PF_TESTING_GH_TOKEN }}
 
+      - name: Set up Python
+        uses: actions/setup-python@5fda3b95a4ea91299a34e894583c3862153e4b97 # v7.0.0
+        with:
+          python-version: "3.14"
+
       - name: Set up Node.js
         uses: actions/setup-node@820762786026740c76f36085b0efc47a31fe5020 # v7.0.0
         with:
@@ -127,7 +132,7 @@ jobs:
           ' <<<"$run" >/dev/null
           jobs=$(gh api --paginate --slurp "repos/rustfs/rustfs/actions/runs/${BUILD_RUN_ID}/jobs?per_page=100")
           jq -e '
-            [.[].jobs[] | select(.name | test("^Build RustFS \\(linux-x86_64-gnu, [a-z0-9-]+, x86_64-unknown-linux-gnu, false, linux, pyroscope\\)$"))] as $matches
+            [.[].jobs[] | select(.name | test("^Build RustFS \\(linux-x86_64-gnu\\)$"))] as $matches
             | ($matches | length) == 1 and $matches[0].conclusion == "success"
           ' <<<"$jobs" >/dev/null
           artifact=$(gh api "repos/rustfs/rustfs/actions/artifacts/${ARTIFACT_ID}")
@@ -172,7 +177,7 @@ jobs:
               assert outer.testzip() is None
               with zipfile.ZipFile(io.BytesIO(outer.read(package))) as inner:
                   entries = inner.infolist()
-                  assert {entry.filename for entry in entries} == {'rustfs', 'rustfs-cli', 'rustfs.cpu-symbol-catalog.json', 'rustfs.cpu-symbol-catalog.sha256'} and len(entries) == 4
+                  assert {entry.filename for entry in entries} == {'rustfs', 'rustfs.cpu-symbol-catalog.json', 'rustfs.cpu-symbol-catalog.sha256'} and len(entries) == 3
                   assert all(regular(entry) and 0 < entry.file_size <= 1073741824 for entry in entries)
                   assert inner.testzip() is None
                   pathlib.Path('binary').mkdir()

.github/workflows/connect-performance-drive-acceptance.yml

@@ -73,6 +73,11 @@ jobs:
           ref: ${{ inputs.connect_sha }}
           token: ${{ secrets.PF_TESTING_GH_TOKEN }}
 
+      - name: Set up Python
+        uses: actions/setup-python@5fda3b95a4ea91299a34e894583c3862153e4b97 # v7.0.0
+        with:
+          python-version: "3.14"
+
       - name: Set up Node.js
         uses: actions/setup-node@820762786026740c76f36085b0efc47a31fe5020 # v7.0.0
         with:
@@ -107,7 +112,7 @@ jobs:
           [[ $(jq -r '.head_repository.full_name' <<<"$run") == "$GITHUB_REPOSITORY" ]]
           [[ $(jq -r '.name' <<<"$run") == "Build and Release" ]]
 
-          expected_job='Build RustFS (linux-x86_64-gnu, sm-standard-4, x86_64-unknown-linux-gnu, false, linux, pyroscope)'
+          expected_job='Build RustFS (linux-x86_64-gnu)'
           jobs=$(gh api --paginate --slurp "repos/${GITHUB_REPOSITORY}/actions/runs/${BUILD_RUN_ID}/jobs?per_page=100")
           jq -e --arg name "$expected_job" '
             [.[].jobs[] | select(.name == $name)] as $matches

.github/workflows/connect-performance-network-acceptance.yml

@@ -73,6 +73,11 @@ jobs:
           ref: ${{ inputs.connect_sha }}
           token: ${{ secrets.PF_TESTING_GH_TOKEN }}
 
+      - name: Set up Python
+        uses: actions/setup-python@5fda3b95a4ea91299a34e894583c3862153e4b97 # v7.0.0
+        with:
+          python-version: "3.14"
+
       - name: Set up Node.js
         uses: actions/setup-node@820762786026740c76f36085b0efc47a31fe5020 # v7.0.0
         with:
@@ -107,7 +112,7 @@ jobs:
           [[ $(jq -r '.head_repository.full_name' <<<"$run") == "$GITHUB_REPOSITORY" ]]
           [[ $(jq -r '.name' <<<"$run") == "Build and Release" ]]
 
-          expected_job='Build RustFS (linux-x86_64-gnu, sm-standard-4, x86_64-unknown-linux-gnu, false, linux, pyroscope)'
+          expected_job='Build RustFS (linux-x86_64-gnu)'
           jobs=$(gh api --paginate --slurp "repos/${GITHUB_REPOSITORY}/actions/runs/${BUILD_RUN_ID}/jobs?per_page=100")
           jq -e --arg name "$expected_job" '
             [.[].jobs[] | select(.name == $name)] as $matches

.github/workflows/connect-profile-cpu-acceptance.yml

@@ -77,6 +77,11 @@ jobs:
           ref: ${{ inputs.connect_sha }}
           token: ${{ secrets.PF_TESTING_GH_TOKEN }}
 
+      - name: Set up Python
+        uses: actions/setup-python@5fda3b95a4ea91299a34e894583c3862153e4b97 # v7.0.0
+        with:
+          python-version: "3.14"
+
       - name: Set up Node.js
         uses: actions/setup-node@820762786026740c76f36085b0efc47a31fe5020 # v7.0.0
         with:
@@ -118,7 +123,7 @@ jobs:
           [[ $(jq -r '.head_repository.full_name' <<<"$run") == "$GITHUB_REPOSITORY" ]]
           [[ $(jq -r '.name' <<<"$run") == "Build and Release" ]]
 
-          expected_job='Build RustFS (linux-x86_64-gnu, sm-standard-4, x86_64-unknown-linux-gnu, false, linux, pyroscope)'
+          expected_job='Build RustFS (linux-x86_64-gnu)'
           jobs=$(github_api "repos/${GITHUB_REPOSITORY}/actions/runs/${BUILD_RUN_ID}/jobs?per_page=100")
           jq -e --arg name "$expected_job" '
             [.jobs[] | select(.name == $name)] as $matches

.github/workflows/connect-profile-memory-acceptance.yml

@@ -57,6 +57,11 @@ jobs:
           ref: ${{ inputs.connect_sha }}
           token: ${{ secrets.PF_TESTING_GH_TOKEN }}
 
+      - name: Set up Python
+        uses: actions/setup-python@5fda3b95a4ea91299a34e894583c3862153e4b97 # v7.0.0
+        with:
+          python-version: "3.14"
+
       - name: Set up Node.js
         uses: actions/setup-node@820762786026740c76f36085b0efc47a31fe5020 # v7.0.0
         with:
@@ -127,7 +132,7 @@ jobs:
           ' <<<"$run" >/dev/null
           jobs=$(gh api --paginate --slurp "repos/rustfs/rustfs/actions/runs/${BUILD_RUN_ID}/jobs?per_page=100")
           jq -e '
-            [.[].jobs[] | select(.name | test("^Build RustFS \\(linux-x86_64-gnu, [a-z0-9-]+, x86_64-unknown-linux-gnu, false, linux, pyroscope\\)$"))] as $matches
+            [.[].jobs[] | select(.name | test("^Build RustFS \\(linux-x86_64-gnu\\)$"))] as $matches
             | ($matches | length) == 1 and $matches[0].conclusion == "success"
           ' <<<"$jobs" >/dev/null
           artifact=$(gh api "repos/rustfs/rustfs/actions/artifacts/${ARTIFACT_ID}")
@@ -172,7 +177,7 @@ jobs:
               assert outer.testzip() is None
               with zipfile.ZipFile(io.BytesIO(outer.read(package))) as inner:
                   entries = inner.infolist()
-                  expected = {'rustfs', 'rustfs-cli', 'rustfs.cpu-symbol-catalog.json', 'rustfs.cpu-symbol-catalog.sha256'}
+                  expected = {'rustfs', 'rustfs.cpu-symbol-catalog.json', 'rustfs.cpu-symbol-catalog.sha256'}
                   assert {entry.filename for entry in entries} == expected and len(entries) == len(expected)
                   assert all(regular(entry) and 0 < entry.file_size <= 1073741824 for entry in entries)
                   assert inner.testzip() is None

.github/workflows/connect-profile-threads-acceptance.yml

@@ -81,6 +81,11 @@ jobs:
           ref: ${{ inputs.connect_sha }}
           token: ${{ secrets.PF_TESTING_GH_TOKEN }}
 
+      - name: Set up Python
+        uses: actions/setup-python@5fda3b95a4ea91299a34e894583c3862153e4b97 # v7.0.0
+        with:
+          python-version: "3.14"
+
       - name: Set up Node.js
         uses: actions/setup-node@820762786026740c76f36085b0efc47a31fe5020 # v7.0.0
         with:
@@ -148,7 +153,7 @@ jobs:
           [[ $(jq -r '.head_repository.full_name' <<<"$run") == "$GITHUB_REPOSITORY" ]]
           [[ $(jq -r '.name' <<<"$run") == "Build and Release" ]]
 
-          expected_job='Build RustFS (linux-x86_64-gnu, sm-standard-4, x86_64-unknown-linux-gnu, false, linux, pyroscope)'
+          expected_job='Build RustFS (linux-x86_64-gnu)'
           jobs=$(gh api --paginate --slurp "repos/${GITHUB_REPOSITORY}/actions/runs/${BUILD_RUN_ID}/jobs?per_page=100")
           jq -e --arg name "$expected_job" '
             [.[].jobs[] | select(.name == $name)] as $matches

.github/workflows/connect-top-api-acceptance.yml

@@ -73,6 +73,11 @@ jobs:
           ref: ${{ inputs.connect_sha }}
           token: ${{ secrets.PF_TESTING_GH_TOKEN }}
 
+      - name: Set up Python
+        uses: actions/setup-python@5fda3b95a4ea91299a34e894583c3862153e4b97 # v7.0.0
+        with:
+          python-version: "3.14"
+
       - name: Set up Node.js
         uses: actions/setup-node@820762786026740c76f36085b0efc47a31fe5020 # v7.0.0
         with:
@@ -107,7 +112,7 @@ jobs:
           [[ $(jq -r '.head_repository.full_name' <<<"$run") == "$GITHUB_REPOSITORY" ]]
           [[ $(jq -r '.name' <<<"$run") == "Build and Release" ]]
 
-          expected_job='Build RustFS (linux-x86_64-gnu, sm-standard-4, x86_64-unknown-linux-gnu, false, linux, pyroscope)'
+          expected_job='Build RustFS (linux-x86_64-gnu)'
           jobs=$(gh api --paginate --slurp "repos/${GITHUB_REPOSITORY}/actions/runs/${BUILD_RUN_ID}/jobs?per_page=100")
           jq -e --arg name "$expected_job" '
             [.[].jobs[] | select(.name == $name)] as $matches

.github/workflows/connect-top-disk-acceptance.yml

@@ -68,6 +68,11 @@ jobs:
           ref: ${{ inputs.connect_sha }}
           token: ${{ secrets.PF_TESTING_GH_TOKEN }}
 
+      - name: Set up Python
+        uses: actions/setup-python@5fda3b95a4ea91299a34e894583c3862153e4b97 # v7.0.0
+        with:
+          python-version: "3.14"
+
       - name: Set up Node.js
         uses: actions/setup-node@820762786026740c76f36085b0efc47a31fe5020 # v7.0.0
         with:
@@ -138,7 +143,7 @@ jobs:
           ' <<<"$run" >/dev/null
           jobs=$(gh api --paginate --slurp "repos/rustfs/rustfs/actions/runs/${BUILD_RUN_ID}/jobs?per_page=100")
           jq -e '
-            [.[].jobs[] | select(.name | test("^Build RustFS \\(linux-x86_64-gnu, [a-z0-9-]+, x86_64-unknown-linux-gnu, false, linux, pyroscope\\)$"))] as $matches
+            [.[].jobs[] | select(.name | test("^Build RustFS \\(linux-x86_64-gnu\\)$"))] as $matches
             | ($matches | length) == 1 and $matches[0].conclusion == "success"
           ' <<<"$jobs" >/dev/null
           artifact=$(gh api "repos/rustfs/rustfs/actions/artifacts/${ARTIFACT_ID}")
@@ -183,7 +188,7 @@ jobs:
               assert outer.testzip() is None
               with zipfile.ZipFile(io.BytesIO(outer.read(package))) as inner:
                   entries = inner.infolist()
-                  expected = {'rustfs', 'rustfs-cli', 'rustfs.cpu-symbol-catalog.json', 'rustfs.cpu-symbol-catalog.sha256'}
+                  expected = {'rustfs', 'rustfs.cpu-symbol-catalog.json', 'rustfs.cpu-symbol-catalog.sha256'}
                   assert {entry.filename for entry in entries} == expected and len(entries) == len(expected)
                   assert all(regular(entry) and 0 < entry.file_size <= 1073741824 for entry in entries)
                   assert inner.testzip() is None

.github/workflows/connect-top-locks-acceptance.yml

@@ -73,6 +73,11 @@ jobs:
           ref: ${{ inputs.connect_sha }}
           token: ${{ secrets.PF_TESTING_GH_TOKEN }}
 
+      - name: Set up Python
+        uses: actions/setup-python@5fda3b95a4ea91299a34e894583c3862153e4b97 # v7.0.0
+        with:
+          python-version: "3.14"
+
       - name: Set up Node.js
         uses: actions/setup-node@820762786026740c76f36085b0efc47a31fe5020 # v7.0.0
         with:
@@ -107,7 +112,7 @@ jobs:
           [[ $(jq -r '.head_repository.full_name' <<<"$run") == "$GITHUB_REPOSITORY" ]]
           [[ $(jq -r '.name' <<<"$run") == "Build and Release" ]]
 
-          expected_job='Build RustFS (linux-x86_64-gnu, sm-standard-4, x86_64-unknown-linux-gnu, false, linux, pyroscope)'
+          expected_job='Build RustFS (linux-x86_64-gnu)'
           jobs=$(gh api --paginate --slurp "repos/${GITHUB_REPOSITORY}/actions/runs/${BUILD_RUN_ID}/jobs?per_page=100")
           jq -e --arg name "$expected_job" '
             [.[].jobs[] | select(.name == $name)] as $matches

.github/workflows/connect-top-rpc-acceptance.yml

@@ -73,6 +73,11 @@ jobs:
           ref: ${{ inputs.connect_sha }}
           token: ${{ secrets.PF_TESTING_GH_TOKEN }}
 
+      - name: Set up Python
+        uses: actions/setup-python@5fda3b95a4ea91299a34e894583c3862153e4b97 # v7.0.0
+        with:
+          python-version: "3.14"
+
       - name: Set up Node.js
         uses: actions/setup-node@820762786026740c76f36085b0efc47a31fe5020 # v7.0.0
         with:
@@ -107,7 +112,7 @@ jobs:
           [[ $(jq -r '.head_repository.full_name' <<<"$run") == "$GITHUB_REPOSITORY" ]]
           [[ $(jq -r '.name' <<<"$run") == "Build and Release" ]]
 
-          expected_job='Build RustFS (linux-x86_64-gnu, sm-standard-4, x86_64-unknown-linux-gnu, false, linux, pyroscope)'
+          expected_job='Build RustFS (linux-x86_64-gnu)'
           jobs=$(gh api --paginate --slurp "repos/${GITHUB_REPOSITORY}/actions/runs/${BUILD_RUN_ID}/jobs?per_page=100")
           jq -e --arg name "$expected_job" '
             [.[].jobs[] | select(.name == $name)] as $matches

.github/workflows/e2e-upgrade.yml

@@ -96,38 +96,8 @@ jobs:
           overwrite: true
 
   upgrade:
-    name: ${{ matrix.name }}
+    name: Upgrade compatibility
     needs: build
-    strategy:
-      fail-fast: false
-      matrix:
-        include:
-          # The two `_from_rc2_` tests keep their names: they assert
-          # release-independent object contracts and pass unchanged against the
-          # newer pinned source, so renaming them would only churn history and
-          # the CI required-check names. UPGRADE_SOURCE_VERSION above is the
-          # single source of truth for which release they actually run against.
-          - name: Direct upgrade from the previous release
-            test: direct_upgrade_from_rc2_preserves_object_contracts
-            artifact: direct-upgrade
-          - name: Mixed-version rolling upgrade from the previous release
-            test: rolling_upgrade_from_rc2_preserves_mixed_version_contracts
-            artifact: mixed-version-upgrade
-          - name: Bucket configuration survives the upgrade
-            test: direct_upgrade_from_previous_release_preserves_bucket_configuration
-            artifact: bucket-config-upgrade
-          - name: Rollback reads current bucket metadata
-            test: rollback_to_previous_release_reads_current_bucket_metadata
-            artifact: bucket-config-rollback
-          - name: ODM configuration recovery after rc.5 rollback
-            test: rc5_rollback_requires_restoring_odm_configuration
-            artifact: odm-config-rollback
-          - name: Multipart layouts survive the rc.5 upgrade
-            test: direct_upgrade_from_rc5_preserves_multipart_layouts
-            artifact: multipart-layout-upgrade
-          - name: rc.5 multipart replication baseline
-            test: rc5_baseline_replicates_multipart_layouts
-            artifact: multipart-layout-baseline
     runs-on: sm-standard-2
     timeout-minutes: 60
     env:
@@ -142,8 +112,7 @@ jobs:
         uses: ./.github/actions/setup
         with:
           cache-shared-key: e2e-upgrade-tests
-          # One main-branch consumer warms the shared test-harness dependencies.
-          cache-save-if: ${{ github.ref == 'refs/heads/main' && matrix.artifact == 'direct-upgrade' }}
+          cache-save-if: ${{ github.ref == 'refs/heads/main' }}
           install-build-packaging-tools: "false"
           install-test-tools: "false"
 
@@ -170,22 +139,49 @@ jobs:
           echo "RUSTFS_UPGRADE_SOURCE_BINARY=$SOURCE_DIR/rustfs" >> "$GITHUB_ENV"
           echo "RUSTFS_E2E_LOG_DIR=$RUNNER_TEMP/rustfs-upgrade-logs" >> "$GITHUB_ENV"
 
-      - name: Run upgrade compatibility test
+      # Reuse one compiled harness, while keeping each case in its own process.
+      # Continue after a failure to retain the former matrix's complete verdict.
+      - name: Run upgrade compatibility tests
+        shell: bash
         env:
-          RUSTFS_SCANNER_HEAL_G09_EVIDENCE_DIR: ${{ runner.temp }}/rustfs-upgrade-g09-evidence/${{ matrix.artifact }}
           NO_PROXY: 127.0.0.1,localhost,::1
         run: |
           export no_proxy="$NO_PROXY"
-          python3 scripts/e2e_binary.py run -- cargo test --locked -p e2e_test \
-            "upgrade_compatibility_test::${{ matrix.test }}" \
-            -- --ignored --exact --nocapture
+          cases=(
+            direct-upgrade:direct_upgrade_from_rc2_preserves_object_contracts
+            mixed-version-upgrade:rolling_upgrade_from_rc2_preserves_mixed_version_contracts
+            bucket-config-upgrade:direct_upgrade_from_previous_release_preserves_bucket_configuration
+            bucket-config-rollback:rollback_to_previous_release_reads_current_bucket_metadata
+            odm-config-rollback:rc5_rollback_requires_restoring_odm_configuration
+            multipart-layout-upgrade:direct_upgrade_from_rc5_preserves_multipart_layouts
+            multipart-layout-baseline:rc5_baseline_replicates_multipart_layouts
+          )
+          status=0
+          for entry in "${cases[@]}"; do
+            artifact="${entry%%:*}"
+            test="${entry#*:}"
+            export RUSTFS_SCANNER_HEAL_G09_EVIDENCE_DIR="$RUNNER_TEMP/rustfs-upgrade-g09-evidence/$artifact"
+            export RUSTFS_E2E_LOG_DIR="$RUNNER_TEMP/rustfs-upgrade-logs/$artifact"
+            mkdir -p "$RUSTFS_E2E_LOG_DIR"
+            echo "::group::$test"
+            if python3 scripts/e2e_binary.py run -- cargo test --locked -p e2e_test \
+              "upgrade_compatibility_test::$test" -- --ignored --exact --nocapture \
+              2>&1 | tee "$RUSTFS_E2E_LOG_DIR/test.log"; then
+              echo "PASS: $test"
+            else
+              echo "::error::Upgrade compatibility failed: $test"
+              status=1
+            fi
+            echo "::endgroup::"
+          done
+          exit "$status"
 
       - name: Upload scanner/heal G09 evidence
         if: always()
         uses: actions/upload-artifact@043fb46d1a93c77aae656e7c1c64a875d1fc6a0a # v7.0.1
         with:
-          name: ${{ matrix.artifact }}-g09-evidence-${{ github.run_number }}
-          path: ${{ runner.temp }}/rustfs-upgrade-g09-evidence/${{ matrix.artifact }}
+          name: upgrade-g09-evidence-${{ github.run_number }}
+          path: ${{ runner.temp }}/rustfs-upgrade-g09-evidence
           if-no-files-found: ignore
           retention-days: 14
 
@@ -193,7 +189,7 @@ jobs:
         if: always()
         uses: actions/upload-artifact@043fb46d1a93c77aae656e7c1c64a875d1fc6a0a # v7.0.1
         with:
-          name: ${{ matrix.artifact }}-server-logs-${{ github.run_number }}
+          name: upgrade-server-logs-${{ github.run_number }}
           path: ${{ runner.temp }}/rustfs-upgrade-logs
           if-no-files-found: warn
           retention-days: 14

.github/workflows/nightly-gnu.yml

@@ -80,13 +80,24 @@ jobs:
       - name: Setup Rust environment
         uses: ./.github/actions/setup
         with:
-          cache-shared-key: build-x86_64-unknown-linux-gnu
+          cache-shared-key: build-x86_64-unknown-linux-gnu-glibc-2.31
           cache-save-if: 'false'
+          install-cross-tools: 'true'
+          target: x86_64-unknown-linux-gnu
           install-build-packaging-tools: 'false'
           install-test-tools: 'false'
 
       - name: Build RustFS
-        run: cargo build --release --locked --target x86_64-unknown-linux-gnu -p rustfs --bins
+        # Pin the glibc baseline (`<target>.<glibc>` for cargo-zigbuild) so
+        # nightlies run on older distros such as RHEL/Rocky 9 (glibc 2.34); #8062.
+        run: cargo zigbuild --release --locked --target x86_64-unknown-linux-gnu.2.31 -p rustfs --bins
+
+      - name: Verify glibc symbol baseline
+        run: |
+          python3 scripts/release/check_glibc.py 2.31 \
+            target/x86_64-unknown-linux-gnu/release/rustfs \
+            target/x86_64-unknown-linux-gnu/release/rustfs-cli
+          target/x86_64-unknown-linux-gnu/release/rustfs --version | grep -Fx 'build target : x86_64-unknown-linux-gnu'
 
       - name: Build DEB package
         id: deb

.github/workflows/oidc-keycloak.yml

@@ -20,8 +20,9 @@ on:
       - ".github/workflows/oidc-keycloak.yml"
       - "crates/config/src/constants/oidc.rs"
       - "crates/iam/src/federation/**"
-      - "crates/iam/src/oidc.rs"
+      - "crates/iam/src/oidc/**"
       - "rustfs/src/admin/handlers/oidc.rs"
+      - "rustfs/src/admin/service/oidc_config.rs"
       - "rustfs/src/admin/handlers/sts.rs"
       - "scripts/test/oidc_keycloak_live.sh"
       - "scripts/test/fixtures/keycloak-rustfs-ci-realm.json"
@@ -31,8 +32,9 @@ on:
       - ".github/workflows/oidc-keycloak.yml"
       - "crates/config/src/constants/oidc.rs"
       - "crates/iam/src/federation/**"
-      - "crates/iam/src/oidc.rs"
+      - "crates/iam/src/oidc/**"
       - "rustfs/src/admin/handlers/oidc.rs"
+      - "rustfs/src/admin/service/oidc_config.rs"
       - "rustfs/src/admin/handlers/sts.rs"
       - "scripts/test/oidc_keycloak_live.sh"
       - "scripts/test/fixtures/keycloak-rustfs-ci-realm.json"

.gitignore

@@ -90,3 +90,4 @@ worktrees/*
 # insta scratch files; the accepted .snap files ARE the assertions and are committed
 *.snap.new
 *.pending-snap
+.mimosa

.vscode/launch.json

@@ -7,7 +7,9 @@
             "name": "Debug(only) executable 'rustfs'",
             "env": {
               "RUST_LOG": "rustfs=info,ecstore=info,s3s=info,iam=info",
-              "RUSTFS_SKIP_BACKGROUND_TASK": "on"
+              "RUSTFS_SKIP_BACKGROUND_TASK": "on",
+              // Local test volumes share one physical disk.
+              "RUSTFS_UNSAFE_BYPASS_DISK_CHECK": "true"
               //"RUSTFS_OBS_LOG_DIRECTORY": "./deploy/logs",
               // "RUSTFS_POLICY_PLUGIN_URL":"http://localhost:8181/v1/data/rustfs/authz/allow",
               // "RUSTFS_POLICY_PLUGIN_AUTH_TOKEN":"your-opa-token"
@@ -44,6 +46,7 @@
             "env": {
                 "RUST_LOG": "rustfs=debug,ecstore=info,s3s=debug,iam=debug",
                 "RUSTFS_SKIP_BACKGROUND_TASK": "on",
+                "RUSTFS_UNSAFE_BYPASS_DISK_CHECK": "true",
                 //"RUSTFS_OBS_LOG_DIRECTORY": "./deploy/logs",
                 // "RUSTFS_POLICY_PLUGIN_URL":"http://localhost:8181/v1/data/rustfs/authz/allow",
                 // "RUSTFS_POLICY_PLUGIN_AUTH_TOKEN":"your-opa-token" 
@@ -117,7 +120,8 @@
             "env": {
                 "RUSTFS_ACCESS_KEY": "rustfsadmin",
                 "RUSTFS_SECRET_KEY": "rustfsadmin",
-                //"RUSTFS_VOLUMES": "./target/volume/test{1...4}",
+                "RUSTFS_VOLUMES": "./target/volume/test{1...4}",
+                "RUSTFS_UNSAFE_BYPASS_DISK_CHECK": "true",
                 "RUSTFS_ADDRESS": ":9000",
                 "RUSTFS_CONSOLE_ENABLE": "true",
                 // "RUSTFS_OBS_TRACE_ENDPOINT": "http://127.0.0.1:4318/v1/traces", // jeager otlp http endpoint
@@ -135,41 +139,30 @@
         {
             "type": "lldb",
             "request": "launch",
-            "name": "Debug test_lifecycle_transition_basic",
+            "name": "Debug test_transition_and_restore_flows",
             "cargo": {
                 "args": [
                     "test",
+                    "--no-run",
                     "-p",
                     "rustfs-scanner",
                     "--test",
                     "lifecycle_integration_test",
-                    "serial_tests::test_lifecycle_transition_basic",
                     "-j",
                     "1"
-                ]
+                ],
+                "filter": {
+                    "name": "lifecycle_integration_test",
+                    "kind": "test"
+                }
             },
-            "args": [],
-            "cwd": "${workspaceFolder}"
-        },
-        {
-            "name": "Debug executable target/debug/test",
-            "type":  "lldb",
-            "request": "launch",
-            "program": "${workspaceFolder}/target/debug/deps/lifecycle_integration_test-5915cbfcab491b3b",
             "args": [
-              "--skip",
-              "test_lifecycle_expiry_basic",
-              "--skip",
-              "test_lifecycle_expiry_deletemarker",
-              //"--skip",
-              //"test_lifecycle_transition_basic",
-            ],
-            "cwd": "${workspaceFolder}",
-            //"stopAtEntry": false,
-            //"preLaunchTask": "cargo build",
-            "sourceLanguages": [
-                "rust"
+                "serial_tests::test_transition_and_restore_flows",
+                "--exact",
+                "--ignored",
+                "--test-threads=1"
             ],
+            "cwd": "${workspaceFolder}"
         },
         {
             "name": "Debug executable target/debug/rustfs with sse kms",
@@ -285,7 +278,7 @@
                 "RUSTFS_COMPRESSION_ENABLED": "true",
                 "RUSTFS_KMS_ENABLE": "true",
                 "RUSTFS_KMS_BACKEND": "local",
-                "RUSTFS_KMS_KEY_DIR": "./target/rio-v2/kms-key-dir",
+                "RUSTFS_KMS_KEY_DIR": "${workspaceFolder}/target/rio-v2/kms-key-dir",
                 "RUSTFS_KMS_LOCAL_MASTER_KEY": "my-secret-key", // Some Password
                 "RUSTFS_KMS_DEFAULT_KEY_ID": "rustfs-master-key",
             },
@@ -307,6 +300,7 @@
                 "RUSTFS_ACCESS_KEY": "rustfsadmin",
                 "RUSTFS_SECRET_KEY": "rustfsadmin",
                 "RUSTFS_VOLUMES": "./target/e2e-test/test{1...4}",
+                "RUSTFS_UNSAFE_BYPASS_DISK_CHECK": "true",
                 "RUSTFS_REGION": "us-east-1",
                 "RUSTFS_ADDRESS": ":9000",
                 "RUSTFS_CONSOLE_ENABLE": "true",
@@ -315,7 +309,7 @@
 
                 "RUSTFS_KMS_ENABLE": "true",
                 "RUSTFS_KMS_BACKEND": "local",
-                "RUSTFS_KMS_KEY_DIR": "./target/e2e-key-dir",
+                "RUSTFS_KMS_KEY_DIR": "${workspaceFolder}/target/e2e-key-dir",
                 "RUSTFS_KMS_LOCAL_MASTER_KEY": "my-secret-key", // Some Password
                 "RUSTFS_KMS_DEFAULT_KEY_ID": "rustfs-master-key",
             },

ARCHITECTURE.md

@@ -155,7 +155,7 @@ default build (lifecycle:
    - ⚠️ PARTIALLY VIOLATED: serving-side `s3s` references remain in ecstore
      (bucket metadata/replication/lifecycle DTOs and error mapping). The
      count is ratcheted shrink-only by `scripts/check_s3s_footprint.sh`
-     (`S3S_ECSTORE_FILES_BASELINE`; the `object_lock` module was converted to
+     (the `ecstore_files` line of `.config/s3s-footprint-baseline.txt`; the `object_lock` module was converted to
      storage-level types as the first ratchet step). Target state: the
      baseline reaches zero and ecstore's `Cargo.toml` drops `s3s`.
 

Cargo.toml

@@ -48,7 +48,7 @@ members = [
     "crates/replication", # Replication contracts and wire formats
     "crates/concurrency", # Concurrency management for RustFS - timeout, locking, backpressure, and I/O scheduling
     "crates/s3-client", # S3 client for engine-side consumption of remote S3 endpoints (tiering, transition targets)
-    "crates/s3-types", # S3 event type definitions
+    "crates/s3-types", # S3 event and error type definitions
     "crates/s3-ops", # S3 operation definitions and mapping
     "crates/s3select-api", # S3 Select API interface
     "crates/s3select-query", # S3 Select query engine
@@ -67,6 +67,7 @@ members = [
     "crates/io-core", # Buffered Bytes, mmap-then-copy, and aligned pread I/O helpers
     "crates/zip", # ZIP file handling and compression
 ]
+exclude = ["third-party/pyroscope"]
 resolver = "3"
 
 [workspace.package]
@@ -156,8 +157,9 @@ futures-lite = "2.6.1"
 futures-util = "0.3.34"
 pollster = "1.0.1"
 pulsar = { default-features = false, version = "6.9.0" }
-lapin = { default-features = false, version = "4.12.0" }
+lapin = { default-features = false, version = "4.12.1" }
 hyper = { version = "1.12.0" }
+h2 = { version = "0.4.20" }
 hyper-rustls = { default-features = false, version = "0.27.10" }
 hyper-util = { version = "0.1.21" }
 http = "1.5.0"
@@ -172,7 +174,7 @@ tokio = { version = "1.53.2" }
 tokio-rustls = { default-features = false, version = "0.26.6" }
 tokio-stream = { version = "0.1.19" }
 tokio-test = "0.4.6"
-tokio-util = { version = "0.7.19" }
+tokio-util = { version = "0.7.20" }
 tonic = { version = "0.14.6" }
 tonic-prost = { version = "0.14.6" }
 tonic-prost-build = { version = "0.14.6" }
@@ -180,7 +182,7 @@ tower = { version = "0.5.3" }
 tower-http = { version = "0.7.1" }
 h3 = "0.0.8"
 h3-quinn = "0.0.10"
-quinn = { version = "0.11.11", default-features = false, features = [
+quinn = { version = "0.11.12", default-features = false, features = [
       "futures-io",
       "runtime-tokio",
       "rustls-aws-lc-rs",
@@ -200,7 +202,7 @@ rmp-serde = { version = "1.3.1" }
 serde = { version = "1.0.229" }
 serde_ignored = { version = "0.1" }
 serde_with = { version = "3", default-features = false, features = ["macros", "std"] }
-serde_json = { version = "1.0.151" }
+serde_json = { version = "1.0.154" }
 serde_urlencoded = "0.7.1"
 
 # Cryptography and Security
@@ -224,7 +226,7 @@ rustls-native-certs = "0.8"
 rustls-pki-types = "1.15.1"
 x509-parser = "0.18.1"
 sha1 = "0.11.0"
-sha2 = "0.11.0"
+sha2 = "0.11.1"
 subtle = "2.6"
 zeroize = { version = "1.9.1" }
 proptest = "1"
@@ -268,7 +270,7 @@ crossbeam-queue = "0.3.14"
 crossbeam-channel = "0.5.17"
 crossbeam-deque = "0.8.8"
 crossbeam-utils = "0.8.23"
-datafusion = { default-features = false, version = "55.1.0" }
+datafusion = { default-features = false, version = "55.2.0" }
 derive_builder = "0.20.2"
 ed25519-dalek = "3.0.0"
 enumset = "1.1.14"
@@ -319,18 +321,23 @@ rumqttc = { package = "rumqttc-next", version = "0.34.0" }
 redis = { version = "1.7.1" }
 rustify = { version = "0.7", default-features = false }
 rustix = { version = "1.1.5" }
-rust-embed = { version = "8.12.0" }
+rust-embed = { version = "8.13.0" }
 rustc-hash = { version = "2.1.3" }
 s3s = { git = "https://github.com/s3s-project/s3s", rev = "5761ddfe4c509fcac87e3b99fd5bd9e176b442c0", features = ["minio"] }
 s3s-sigv4 = { git = "https://github.com/s3s-project/s3s", rev = "5761ddfe4c509fcac87e3b99fd5bd9e176b442c0" }
+# The same s3s pin under the name the legacy edge of the gateway migration uses
+# (crates/s3-types, feature `compat-s3s`). Keep `rev` identical to `s3s` above.
+# DELETE BY T4.2 (rustfs/backlog#2784).
+legacy-s3s = { package = "s3s", git = "https://github.com/s3s-project/s3s", rev = "5761ddfe4c509fcac87e3b99fd5bd9e176b442c0", features = ["minio"] }
 serial_test = "4.0.1"
 shadow-rs = { default-features = false, version = "2.0.0" }
 siphasher = "1.0.4"
-smallvec = { version = "1.16.2" }
+smallvec = { version = "1.16.3" }
 compact_str = "0.10.0"
 snap = "1.1.2"
 starshard = { version = "2.4.0" }
 strum = { version = "0.28.0" }
+sync_wrapper = "1.0.2"
 sysinfo = "0.39.6"
 temp-env = "0.3.6"
 tempfile = "3.27.0"
@@ -340,12 +347,12 @@ tracing = { version = "0.1.44" }
 tracing-appender = "0.2.5"
 tracing-core = "0.1.36"
 tracing-error = "0.2.1"
-tracing-opentelemetry = { version = "0.34" }
+tracing-opentelemetry = { version = "0.34.0" }
 tracing-subscriber = { version = "0.3.23" }
 transform-stream = "0.3.1"
 url = "2.5.8"
 urlencoding = "2.1.3"
-uuid = { version = "1.27.0" }
+uuid = { version = "1.28.0" }
 vaultrs = { version = "0.8.0" }
 tar = "0.4.46"
 walkdir = "2.5.0"
@@ -353,26 +360,26 @@ winapi-util = "0.1.11"
 windows = { version = "0.62.2" }
 windows-sys = "0.61.2"
 xxhash-rust = { version = "0.8.19" }
-zip = "8.6.0"
-zstd = "0.14.0"
+zip = "9.0.2"
+zstd = "0.14.2"
 
 # Observability and Metrics
 metrics = "0.24.6"
 metrics-util = "0.20"
 dial9 = { version = "0.5.3", default-features = false }
-opentelemetry = { version = "0.33.0" }
-opentelemetry-appender-tracing = { version = "0.33.0" }
-opentelemetry-otlp = { version = "0.33.0" }
-opentelemetry-proto = { version = "0.33.0", default-features = false, features = ["metrics", "gen-tonic-messages"] }
-opentelemetry_sdk = { version = "0.33.0" }
-opentelemetry-semantic-conventions = { version = "0.33.0" }
-opentelemetry-stdout = { version = "0.33.0" }
+opentelemetry = { version = "0.33.1" }
+opentelemetry-appender-tracing = { version = "0.33.1" }
+opentelemetry-otlp = { version = "0.33.1" }
+opentelemetry-proto = { version = "0.33.1", default-features = false, features = ["metrics", "gen-tonic-messages"] }
+opentelemetry_sdk = { version = "0.33.1" }
+opentelemetry-semantic-conventions = { version = "0.33.1" }
+opentelemetry-stdout = { version = "0.33.1" }
 pyroscope = { version = "2.1.1" }
 
 # FTP and SFTP
 libunftp = { version = "0.23.1" }
 unftp-core = "0.1.0"
-suppaftp = { version = "12.1.1" }
+suppaftp = { version = "12.2.0" }
 rcgen = { version = "0.14.10", default-features = false, features = ["aws_lc_rs", "crypto", "pem"] }
 russh = { version = "0.64.1" }
 russh-sftp = "3.0.1"
@@ -383,7 +390,7 @@ dav-server = "0.11.0"
 # Performance Analysis and Memory Profiling
 rustfs-mimalloc = { version = "0.5.6" }
 # Preserve Unicode focus filters until rustfs/backlog#2302 is resolved.
-hotpath = { version = "0.28.5", default-features = false }
+hotpath = { version = "0.28.6", default-features = false }
 # Snapshot testing for output format regression detection
 insta = { version = "1.49" }
 
@@ -420,3 +427,8 @@ inherits = "release"
 inherits = "release"
 debug = true
 strip = "none"
+
+# Remove this patch when Pyroscope releases support for object 0.40 or later.
+# object 0.39 requires ruzstd 0.8, affected by RUSTSEC-2026-0344.
+[patch.crates-io]
+pyroscope = { path = "third-party/pyroscope" }

_typos.toml

@@ -51,6 +51,8 @@ unparseable = "unparseable"
 # Disaster-recovery objectives: recovery time and recovery point.
 RTO = "RTO"
 rto = "rto"
+# rustix OFlags variant (POSIX O_WRONLY equivalent).
+WRONLY = "WRONLY"
 
 [files]
 extend-exclude = []

crates/common/Cargo.toml

@@ -34,6 +34,7 @@ hotpath-alloc = ["hotpath", "hotpath/hotpath-alloc"]
 hotpath-cpu = ["hotpath", "hotpath/hotpath-cpu"]
 
 [dependencies]
+arc-swap.workspace = true
 hotpath.workspace = true
 tokio = { workspace = true, features = ["fs", "rt-multi-thread"] }
 tonic = { workspace = true, features = ["gzip", "deflate"] }

crates/common/src/globals.rs

@@ -12,9 +12,10 @@
 // See the License for the specific language governing permissions and
 // limitations under the License.
 
+use arc_swap::ArcSwap;
 use std::collections::HashMap;
-use std::sync::LazyLock;
 use std::sync::atomic::{AtomicU64, Ordering};
+use std::sync::{Arc, LazyLock};
 use tokio::sync::RwLock;
 use tonic::transport::Channel;
 
@@ -27,6 +28,42 @@ pub static GLOBAL_ROOT_CERT: LazyLock<RwLock<Option<Vec<u8>>>> = LazyLock::new(|
 pub static GLOBAL_MTLS_IDENTITY: LazyLock<RwLock<Option<MtlsIdentityPem>>> = LazyLock::new(|| RwLock::new(None));
 pub static GLOBAL_OUTBOUND_TLS_GENERATION: LazyLock<AtomicU64> = LazyLock::new(|| AtomicU64::new(0));
 
+#[derive(Clone, Default)]
+pub struct OutboundTlsSnapshot {
+    pub generation: u64,
+    pub root_ca_pem: Option<Vec<u8>>,
+    pub mtls_identity: Option<MtlsIdentityPem>,
+}
+
+static GLOBAL_OUTBOUND_TLS_STATE: LazyLock<ArcSwap<OutboundTlsSnapshot>> =
+    LazyLock::new(|| ArcSwap::from_pointee(OutboundTlsSnapshot::default()));
+
+pub fn get_global_outbound_tls_snapshot() -> Arc<OutboundTlsSnapshot> {
+    GLOBAL_OUTBOUND_TLS_STATE.load_full()
+}
+
+/// Publish identity, trust and generation together; the individual globals are legacy views.
+pub async fn publish_global_outbound_tls_snapshot(snapshot: OutboundTlsSnapshot) -> u64 {
+    GLOBAL_OUTBOUND_TLS_STATE.rcu(|current| {
+        if current.generation != 0
+            && current.root_ca_pem == snapshot.root_ca_pem
+            && current.mtls_identity == snapshot.mtls_identity
+        {
+            return Arc::clone(current);
+        }
+        Arc::new(OutboundTlsSnapshot {
+            generation: snapshot.generation.max(current.generation.saturating_add(1)),
+            root_ca_pem: snapshot.root_ca_pem.clone(),
+            mtls_identity: snapshot.mtls_identity.clone(),
+        })
+    });
+    let accepted = get_global_outbound_tls_snapshot();
+    *GLOBAL_ROOT_CERT.write().await = accepted.root_ca_pem.clone();
+    *GLOBAL_MTLS_IDENTITY.write().await = accepted.mtls_identity.clone();
+    GLOBAL_OUTBOUND_TLS_GENERATION.store(accepted.generation, Ordering::Relaxed);
+    accepted.generation
+}
+
 /// Log level to use when reporting cached gRPC connection eviction.
 #[derive(Clone, Copy, Debug, Eq, PartialEq)]
 pub enum ConnectionEvictionLogLevel {
@@ -85,17 +122,27 @@ pub async fn get_global_addr() -> String {
 /// # Arguments
 /// * `cert` - A vector of bytes representing the PEM-encoded root CA certificate.
 pub async fn set_global_root_cert(cert: Vec<u8>) {
+    GLOBAL_OUTBOUND_TLS_STATE.rcu(|current| {
+        let mut next = (**current).clone();
+        next.root_ca_pem = Some(cert.clone());
+        Arc::new(next)
+    });
     *GLOBAL_ROOT_CERT.write().await = Some(cert);
 }
 
 /// Clear the global root CA certificate for outbound gRPC clients.
 pub async fn clear_global_root_cert() {
+    GLOBAL_OUTBOUND_TLS_STATE.rcu(|current| {
+        let mut next = (**current).clone();
+        next.root_ca_pem = None;
+        Arc::new(next)
+    });
     *GLOBAL_ROOT_CERT.write().await = None;
 }
 
 /// Get the global root CA certificate for outbound gRPC clients.
 pub async fn get_global_root_cert() -> Option<Vec<u8>> {
-    GLOBAL_ROOT_CERT.read().await.clone()
+    get_global_outbound_tls_snapshot().root_ca_pem.clone()
 }
 
 /// Set the global mTLS identity (cert+key PEM) for outbound gRPC clients.
@@ -105,22 +152,32 @@ pub async fn get_global_root_cert() -> Option<Vec<u8>> {
 /// # Arguments
 /// * `identity` - An optional MtlsIdentityPem struct containing the cert and key PEM.
 pub async fn set_global_mtls_identity(identity: Option<MtlsIdentityPem>) {
+    GLOBAL_OUTBOUND_TLS_STATE.rcu(|current| {
+        let mut next = (**current).clone();
+        next.mtls_identity = identity.clone();
+        Arc::new(next)
+    });
     *GLOBAL_MTLS_IDENTITY.write().await = identity;
 }
 
 /// Get the global mTLS identity for outbound gRPC clients.
 pub async fn get_global_mtls_identity() -> Option<MtlsIdentityPem> {
-    GLOBAL_MTLS_IDENTITY.read().await.clone()
+    get_global_outbound_tls_snapshot().mtls_identity.clone()
 }
 
 /// Set the global outbound TLS generation.
 pub fn set_global_outbound_tls_generation(generation: u64) {
+    GLOBAL_OUTBOUND_TLS_STATE.rcu(|current| {
+        let mut next = (**current).clone();
+        next.generation = generation;
+        Arc::new(next)
+    });
     GLOBAL_OUTBOUND_TLS_GENERATION.store(generation, Ordering::Relaxed);
 }
 
 /// Get the global outbound TLS generation.
 pub fn get_global_outbound_tls_generation() -> u64 {
-    GLOBAL_OUTBOUND_TLS_GENERATION.load(Ordering::Relaxed)
+    get_global_outbound_tls_snapshot().generation
 }
 
 /// Evict a stale/dead connection from the global connection cache.
@@ -194,8 +251,77 @@ pub async fn clear_all_connections() {
 ///
 /// When present, gRPC clients will present this identity to servers requesting/requiring mTLS.
 /// When absent, clients proceed with standard server-authenticated TLS.
-#[derive(Clone, Debug)]
+#[derive(Clone, PartialEq, Eq)]
 pub struct MtlsIdentityPem {
     pub cert_pem: Vec<u8>,
     pub key_pem: Vec<u8>,
 }
+
+impl std::fmt::Debug for MtlsIdentityPem {
+    fn fmt(&self, f: &mut std::fmt::Formatter<'_>) -> std::fmt::Result {
+        f.debug_struct("MtlsIdentityPem")
+            .field("cert_bytes", &self.cert_pem.len())
+            .field("key_pem", &"[REDACTED]")
+            .finish()
+    }
+}
+
+#[cfg(test)]
+mod tls_snapshot_tests {
+    use super::*;
+
+    #[tokio::test]
+    async fn outbound_snapshot_publication_is_coherent_and_skips_unchanged_material() {
+        let start = get_global_outbound_tls_generation().saturating_add(1);
+        let snapshot = OutboundTlsSnapshot {
+            generation: start,
+            root_ca_pem: Some(vec![1]),
+            mtls_identity: Some(MtlsIdentityPem {
+                cert_pem: vec![1],
+                key_pem: vec![1],
+            }),
+        };
+        let first = publish_global_outbound_tls_snapshot(snapshot.clone()).await;
+        assert_eq!(
+            publish_global_outbound_tls_snapshot(OutboundTlsSnapshot {
+                generation: first.saturating_add(1),
+                ..snapshot
+            })
+            .await,
+            first
+        );
+        let writer = tokio::spawn(async move {
+            for tag in 2_u8..=255 {
+                publish_global_outbound_tls_snapshot(OutboundTlsSnapshot {
+                    generation: start.saturating_add(u64::from(tag)),
+                    root_ca_pem: Some(vec![tag]),
+                    mtls_identity: Some(MtlsIdentityPem {
+                        cert_pem: vec![tag],
+                        key_pem: vec![tag],
+                    }),
+                })
+                .await;
+                tokio::task::yield_now().await;
+            }
+        });
+        while !writer.is_finished() {
+            let state = get_global_outbound_tls_snapshot();
+            let identity = state.mtls_identity.as_ref().expect("identity in complete snapshot");
+            assert_eq!(state.root_ca_pem.as_ref().expect("roots in complete snapshot"), &identity.cert_pem);
+            assert_eq!(identity.cert_pem, identity.key_pem);
+            tokio::task::yield_now().await;
+        }
+        writer.await.expect("writer completes");
+    }
+
+    #[test]
+    fn mtls_identity_debug_redacts_private_key() {
+        let identity = MtlsIdentityPem {
+            cert_pem: b"public certificate".to_vec(),
+            key_pem: b"PRIVATE-KEY-MARKER".to_vec(),
+        };
+        let debug = format!("{identity:?}");
+        assert!(!debug.contains("PRIVATE-KEY-MARKER"));
+        assert!(debug.contains("[REDACTED]"));
+    }
+}

crates/config/src/observability/mod.rs

@@ -51,6 +51,11 @@ pub const ENV_OBS_PROFILING_EXPORT_ENABLED: &str = "RUSTFS_OBS_PROFILING_EXPORT_
 /// PUT records multiple timers and histograms when attribution is active.
 pub const ENV_OBS_PUT_STAGE_METRICS_ENABLED: &str = "RUSTFS_OBS_PUT_STAGE_METRICS_ENABLED";
 
+/// Enables sampled in-process GetObject stage summaries without OTLP histograms.
+pub const ENV_OBS_GET_STAGE_LOCAL_SUMMARY_ENABLED: &str = "RUSTFS_OBS_GET_STAGE_LOCAL_SUMMARY_ENABLED";
+pub const ENV_OBS_GET_STAGE_LOCAL_SUMMARY_SAMPLE_RATE: &str = "RUSTFS_OBS_GET_STAGE_LOCAL_SUMMARY_SAMPLE_RATE";
+pub const ENV_OBS_GET_STAGE_LOCAL_SUMMARY_INTERVAL_SECS: &str = "RUSTFS_OBS_GET_STAGE_LOCAL_SUMMARY_INTERVAL_SECS";
+
 pub const ENV_OBS_LOGGER_LEVEL: &str = "RUSTFS_OBS_LOGGER_LEVEL";
 pub const ENV_OBS_LOG_STDOUT_ENABLED: &str = "RUSTFS_OBS_LOG_STDOUT_ENABLED";
 pub const ENV_OBS_LOG_DIRECTORY: &str = "RUSTFS_OBS_LOG_DIRECTORY";
@@ -159,6 +164,15 @@ mod tests {
         assert_eq!(ENV_OBS_LOGS_EXPORT_ENABLED, "RUSTFS_OBS_LOGS_EXPORT_ENABLED");
         assert_eq!(ENV_OBS_PROFILING_EXPORT_ENABLED, "RUSTFS_OBS_PROFILING_EXPORT_ENABLED");
         assert_eq!(ENV_OBS_PUT_STAGE_METRICS_ENABLED, "RUSTFS_OBS_PUT_STAGE_METRICS_ENABLED");
+        assert_eq!(ENV_OBS_GET_STAGE_LOCAL_SUMMARY_ENABLED, "RUSTFS_OBS_GET_STAGE_LOCAL_SUMMARY_ENABLED");
+        assert_eq!(
+            ENV_OBS_GET_STAGE_LOCAL_SUMMARY_SAMPLE_RATE,
+            "RUSTFS_OBS_GET_STAGE_LOCAL_SUMMARY_SAMPLE_RATE"
+        );
+        assert_eq!(
+            ENV_OBS_GET_STAGE_LOCAL_SUMMARY_INTERVAL_SECS,
+            "RUSTFS_OBS_GET_STAGE_LOCAL_SUMMARY_INTERVAL_SECS"
+        );
         // Test log cleanup related env keys
         assert_eq!(ENV_OBS_LOG_MAX_TOTAL_SIZE_BYTES, "RUSTFS_OBS_LOG_MAX_TOTAL_SIZE_BYTES");
         assert_eq!(ENV_OBS_LOG_MAX_SINGLE_FILE_SIZE_BYTES, "RUSTFS_OBS_LOG_MAX_SINGLE_FILE_SIZE_BYTES");

crates/e2e_test/src/common.rs

@@ -1542,6 +1542,15 @@ impl RustFSTestClusterEnvironment {
         self.start_node_from_binary(node_idx, &binary_path).await
     }
 
+    /// Start a process and wait only for TCP, allowing tests to bring up a
+    /// quorum one node at a time before the S3 service can become ready.
+    pub async fn start_node_process(&mut self, node_idx: usize) -> Result<(), Box<dyn std::error::Error + Send + Sync>> {
+        let binary_path = rustfs_binary_path();
+        let volumes_arg = self.build_volumes_arg();
+        self.spawn_node(node_idx, &binary_path, &volumes_arg)?;
+        self.wait_for_node_ready(&self.nodes[node_idx].address, node_idx).await
+    }
+
     /// Start one stopped cluster node with a specific RustFS binary while
     /// preserving the cluster's volume layout and that node's data directory.
     pub async fn start_node_from_binary(

crates/e2e_test/src/dual_stack_differential/compare.rs

@@ -0,0 +1,512 @@
+// Copyright 2024 RustFS Team
+//
+// Licensed under the Apache License, Version 2.0 (the "License");
+// you may not use this file except in compliance with the License.
+// You may obtain a copy of the License at
+//
+//     http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing, software
+// distributed under the License is distributed on an "AS IS" BASIS,
+// WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+// See the License for the specific language governing permissions and
+// limitations under the License.
+
+//! What one stack answered, and how two answers are compared.
+//!
+//! An [`Observation`] keeps only what a client acts on: the HTTP status, the S3 error code, the
+//! allowlisted response headers and a few typed fields decoded from the response. The error
+//! message is kept for diagnostics and never compared: message wording is judged by the wire
+//! corpus, not here. The comparator is pure so it can be tested without a server.
+
+use serde::Serialize;
+use std::collections::{BTreeMap, BTreeSet};
+use std::path::{Path, PathBuf};
+
+/// Response headers compared exactly (after [`normalized_header_value`]).
+const HEADER_ALLOWLIST: [&str; 7] = [
+    "content-type",
+    "x-amz-version-id",
+    "etag",
+    "x-amz-delete-marker",
+    "content-range",
+    "accept-ranges",
+    "x-amz-request-charged",
+];
+
+/// Every response header with this prefix is compared as well.
+const HEADER_ALLOWLIST_PREFIX: &str = "x-amz-server-side-encryption";
+
+/// Placeholder for a field one side did not send.
+const ABSENT: &str = "<absent>";
+
+/// The lowercase header name if the comparator looks at this header.
+pub(crate) fn compared_header(name: &str) -> Option<String> {
+    let name = name.to_ascii_lowercase();
+    (HEADER_ALLOWLIST.contains(&name.as_str()) || name.starts_with(HEADER_ALLOWLIST_PREFIX)).then_some(name)
+}
+
+/// A version ID is minted per process, so its value never matches across two servers; only
+/// whether it is the literal `null` (an unversioned write) carries meaning.
+fn normalized_header_value(name: &str, value: &str) -> String {
+    if name == "x-amz-version-id" && value != "null" {
+        "<version-id>".to_owned()
+    } else {
+        value.to_owned()
+    }
+}
+
+/// One request's outcome as a client sees it.
+#[derive(Clone, Debug, Serialize)]
+pub(crate) struct Observation {
+    /// The S3 operation, named as in the `s3s::S3` trait (`GetObject`).
+    pub op: &'static str,
+    /// Which request of the script this was (`normal`, `negative`, or a setup label).
+    pub case: &'static str,
+    /// HTTP status; 0 when no response arrived.
+    pub status: u16,
+    /// The S3 error code, or the SDK's error kind when there was no service error.
+    pub error_code: Option<String>,
+    /// Allowlisted headers only, lowercase names, normalized values.
+    pub headers: BTreeMap<String, String>,
+    /// Typed values decoded from the response body, never free text.
+    pub fields: BTreeMap<String, String>,
+    /// Diagnostic only; never compared.
+    #[serde(skip)]
+    pub message: Option<String>,
+}
+
+impl Observation {
+    /// Builds an observation, keeping only the allowlisted headers.
+    pub(crate) fn new<'h>(
+        op: &'static str,
+        case: &'static str,
+        status: u16,
+        error_code: Option<String>,
+        headers: impl IntoIterator<Item = (&'h str, &'h str)>,
+    ) -> Self {
+        let headers = headers
+            .into_iter()
+            .filter_map(|(name, value)| compared_header(name).map(|name| (name, value)))
+            .map(|(name, value)| {
+                let value = normalized_header_value(&name, value);
+                (name, value)
+            })
+            .collect();
+        Self {
+            op,
+            case,
+            status,
+            error_code,
+            headers,
+            fields: BTreeMap::new(),
+            message: None,
+        }
+    }
+
+    pub(crate) fn with_field(mut self, name: &str, value: impl Into<String>) -> Self {
+        self.fields.insert(name.to_owned(), value.into());
+        self
+    }
+
+    pub(crate) fn is_success(&self) -> bool {
+        (200..300).contains(&self.status)
+    }
+}
+
+/// One field on which the two stacks answered differently.
+#[derive(Clone, Debug, PartialEq, Eq, Serialize)]
+pub(crate) struct Difference {
+    pub op: String,
+    pub case: String,
+    pub field: String,
+    pub a: String,
+    pub b: String,
+}
+
+/// Every difference between two transcripts of the same script, matched by `(op, case)`.
+pub(crate) fn compare(a: &[Observation], b: &[Observation]) -> Vec<Difference> {
+    let mut differences = Vec::new();
+    let (index_a, duplicates_a) = index(a);
+    let (index_b, duplicates_b) = index(b);
+    for (op, case) in duplicates_a.union(&duplicates_b) {
+        let count = |side: &[Observation]| {
+            side.iter()
+                .filter(|observation| (observation.op, observation.case) == (*op, *case))
+                .count()
+                .to_string()
+        };
+        differences.push(difference(op, case, "duplicate_case", count(a), count(b)));
+    }
+    let keys: BTreeSet<_> = index_a.keys().chain(index_b.keys()).copied().collect();
+    for key @ (op, case) in keys {
+        match (index_a.get(&key), index_b.get(&key)) {
+            (Some(a), Some(b)) => compare_one(a, b, &mut differences),
+            (a, b) => {
+                let presence = |side: Option<&&Observation>| if side.is_some() { "present" } else { ABSENT }.to_owned();
+                differences.push(difference(op, case, "observation", presence(a), presence(b)));
+            }
+        }
+    }
+    differences
+}
+
+type Key = (&'static str, &'static str);
+
+fn index(side: &[Observation]) -> (BTreeMap<Key, &Observation>, BTreeSet<Key>) {
+    let mut index = BTreeMap::new();
+    let mut duplicates = BTreeSet::new();
+    for observation in side {
+        let key = (observation.op, observation.case);
+        if index.insert(key, observation).is_some() {
+            duplicates.insert(key);
+        }
+    }
+    (index, duplicates)
+}
+
+fn difference(op: &str, case: &str, field: &str, a: String, b: String) -> Difference {
+    Difference {
+        op: op.to_owned(),
+        case: case.to_owned(),
+        field: field.to_owned(),
+        a,
+        b,
+    }
+}
+
+fn compare_one(a: &Observation, b: &Observation, differences: &mut Vec<Difference>) {
+    let mut field = |name: String, value_a: Option<&str>, value_b: Option<&str>| {
+        if value_a != value_b {
+            let show = |value: Option<&str>| value.unwrap_or(ABSENT).to_owned();
+            differences.push(difference(a.op, a.case, &name, show(value_a), show(value_b)));
+        }
+    };
+    field(
+        "status".to_owned(),
+        Some(a.status.to_string().as_str()),
+        Some(b.status.to_string().as_str()),
+    );
+    field("error_code".to_owned(), a.error_code.as_deref(), b.error_code.as_deref());
+    let headers: BTreeSet<_> = a.headers.keys().chain(b.headers.keys()).collect();
+    for name in headers {
+        field(
+            format!("header.{name}"),
+            a.headers.get(name).map(String::as_str),
+            b.headers.get(name).map(String::as_str),
+        );
+    }
+    let fields: BTreeSet<_> = a.fields.keys().chain(b.fields.keys()).collect();
+    for name in fields {
+        field(
+            format!("body.{name}"),
+            a.fields.get(name).map(String::as_str),
+            b.fields.get(name).map(String::as_str),
+        );
+    }
+}
+
+/// Operations observed on both sides.
+pub(crate) fn compared_ops(a: &[Observation], b: &[Observation]) -> BTreeSet<&'static str> {
+    let ops_b: BTreeSet<_> = b.iter().map(|observation| observation.op).collect();
+    a.iter()
+        .map(|observation| observation.op)
+        .filter(|op| ops_b.contains(op))
+        .collect()
+}
+
+/// `(op, case)` pairs observed on both sides.
+pub(crate) fn compared_cases(a: &[Observation], b: &[Observation]) -> usize {
+    let cases_b: BTreeSet<_> = b.iter().map(|observation| (observation.op, observation.case)).collect();
+    a.iter()
+        .map(|observation| (observation.op, observation.case))
+        .collect::<BTreeSet<_>>()
+        .intersection(&cases_b)
+        .count()
+}
+
+/// The JSON report written for every run.
+#[derive(Debug, Serialize)]
+pub(crate) struct Report {
+    pub schema: u32,
+    pub run: String,
+    pub stack_a: String,
+    pub stack_b: String,
+    pub region_a: Option<String>,
+    pub region_b: Option<String>,
+    pub compared_ops: usize,
+    pub compared_cases: usize,
+    pub differences: Vec<Difference>,
+}
+
+impl Report {
+    /// Writes the report as `<directory>/<run>.json` and returns its path.
+    pub(crate) fn write_to(&self, directory: &Path) -> std::io::Result<PathBuf> {
+        std::fs::create_dir_all(directory)?;
+        let path = directory.join(format!("{}.json", self.run));
+        let json = serde_json::to_vec_pretty(self).map_err(std::io::Error::other)?;
+        std::fs::write(&path, json)?;
+        Ok(path)
+    }
+}
+
+/// One line per difference, for a failing assertion.
+pub(crate) fn render(differences: &[Difference]) -> String {
+    differences
+        .iter()
+        .map(|d| format!("({}, {}, {}, a={:?}, b={:?})", d.op, d.case, d.field, d.a, d.b))
+        .collect::<Vec<_>>()
+        .join("\n")
+}
+
+#[cfg(test)]
+mod tests {
+    use super::*;
+
+    fn ok(op: &'static str) -> Observation {
+        Observation::new(op, "normal", 200, None, [("content-type", "application/xml")])
+    }
+
+    fn fields(differences: &[Difference]) -> Vec<&str> {
+        differences.iter().map(|d| d.field.as_str()).collect()
+    }
+
+    #[test]
+    fn identical_transcripts_have_no_difference() {
+        let a = vec![
+            ok("HeadBucket"),
+            ok("GetBucketLocation").with_field("location_constraint", "us-east-1"),
+        ];
+        let b = a.clone();
+        assert_eq!(compare(&a, &b), Vec::new());
+        assert_eq!(compared_ops(&a, &b).len(), 2);
+    }
+
+    #[test]
+    fn a_status_difference_is_reported() {
+        let a = vec![ok("PutBucketAcl")];
+        let b = vec![Observation::new(
+            "PutBucketAcl",
+            "normal",
+            403,
+            None,
+            [("content-type", "application/xml")],
+        )];
+        assert_eq!(
+            compare(&a, &b),
+            vec![Difference {
+                op: "PutBucketAcl".into(),
+                case: "normal".into(),
+                field: "status".into(),
+                a: "200".into(),
+                b: "403".into(),
+            }]
+        );
+    }
+
+    #[test]
+    fn an_error_code_difference_is_reported() {
+        let a = vec![Observation::new("GetObject", "negative", 404, Some("NoSuchKey".into()), [])];
+        let b = vec![Observation::new(
+            "GetObject",
+            "negative",
+            404,
+            Some("NoSuchBucket".into()),
+            [],
+        )];
+        let differences = compare(&a, &b);
+        assert_eq!(fields(&differences), ["error_code"]);
+        assert_eq!((differences[0].a.as_str(), differences[0].b.as_str()), ("NoSuchKey", "NoSuchBucket"));
+    }
+
+    #[test]
+    fn an_error_code_on_one_side_only_is_reported() {
+        let a = vec![Observation::new("HeadBucket", "negative", 404, Some("NotFound".into()), [])];
+        let b = vec![Observation::new("HeadBucket", "negative", 404, None, [])];
+        let differences = compare(&a, &b);
+        assert_eq!(fields(&differences), ["error_code"]);
+        assert_eq!(differences[0].b, ABSENT);
+    }
+
+    #[test]
+    fn an_allowlisted_header_difference_is_reported() {
+        let a = vec![Observation::new("HeadObject", "normal", 200, None, [("ETag", "\"aa\"")])];
+        let b = vec![Observation::new("HeadObject", "normal", 200, None, [("etag", "\"bb\"")])];
+        assert_eq!(fields(&compare(&a, &b)), ["header.etag"]);
+    }
+
+    #[test]
+    fn a_header_sent_by_one_side_only_is_reported() {
+        let a = vec![Observation::new(
+            "DeleteObject",
+            "normal",
+            204,
+            None,
+            [("x-amz-delete-marker", "true")],
+        )];
+        let b = vec![Observation::new("DeleteObject", "normal", 204, None, [])];
+        let differences = compare(&a, &b);
+        assert_eq!(fields(&differences), ["header.x-amz-delete-marker"]);
+        assert_eq!((differences[0].a.as_str(), differences[0].b.as_str()), ("true", ABSENT));
+    }
+
+    #[test]
+    fn every_server_side_encryption_header_is_compared() {
+        let header = "x-amz-server-side-encryption-customer-key-md5";
+        let a = vec![Observation::new("PutObject", "sse-c", 200, None, [(header, "one")])];
+        let b = vec![Observation::new("PutObject", "sse-c", 200, None, [(header, "two")])];
+        assert_eq!(fields(&compare(&a, &b)), [format!("header.{header}")]);
+    }
+
+    #[test]
+    fn headers_outside_the_allowlist_are_not_compared() {
+        let a = vec![Observation::new(
+            "ListBuckets",
+            "normal",
+            200,
+            None,
+            [
+                ("date", "Mon"),
+                ("x-amz-request-id", "1"),
+                ("server", "a"),
+                ("content-length", "10"),
+            ],
+        )];
+        let b = vec![Observation::new(
+            "ListBuckets",
+            "normal",
+            200,
+            None,
+            [
+                ("date", "Tue"),
+                ("x-amz-request-id", "2"),
+                ("server", "b"),
+                ("content-length", "11"),
+            ],
+        )];
+        assert_eq!(compare(&a, &b), Vec::new());
+        assert!(a[0].headers.is_empty(), "non-allowlisted headers were kept: {:?}", a[0].headers);
+    }
+
+    #[test]
+    fn distinct_version_ids_are_not_a_difference() {
+        let a = vec![Observation::new(
+            "PutObject",
+            "normal",
+            200,
+            None,
+            [("x-amz-version-id", "0b1c")],
+        )];
+        let b = vec![Observation::new(
+            "PutObject",
+            "normal",
+            200,
+            None,
+            [("x-amz-version-id", "9f8e")],
+        )];
+        assert_eq!(compare(&a, &b), Vec::new());
+    }
+
+    #[test]
+    fn a_null_version_id_against_a_real_one_is_reported() {
+        let a = vec![Observation::new(
+            "PutObject",
+            "normal",
+            200,
+            None,
+            [("x-amz-version-id", "null")],
+        )];
+        let b = vec![Observation::new(
+            "PutObject",
+            "normal",
+            200,
+            None,
+            [("x-amz-version-id", "9f8e")],
+        )];
+        assert_eq!(fields(&compare(&a, &b)), ["header.x-amz-version-id"]);
+    }
+
+    #[test]
+    fn a_body_field_difference_is_reported() {
+        let a = vec![ok("GetBucketLocation").with_field("location_constraint", "us-east-1")];
+        let b = vec![ok("GetBucketLocation").with_field("location_constraint", "eu-west-1")];
+        let differences = compare(&a, &b);
+        assert_eq!(fields(&differences), ["body.location_constraint"]);
+        assert_eq!((differences[0].a.as_str(), differences[0].b.as_str()), ("us-east-1", "eu-west-1"));
+    }
+
+    #[test]
+    fn a_body_field_on_one_side_only_is_reported() {
+        let a = vec![ok("ListObjectsV2").with_field("keys", "[]")];
+        let b = vec![ok("ListObjectsV2")];
+        assert_eq!(fields(&compare(&a, &b)), ["body.keys"]);
+    }
+
+    #[test]
+    fn error_messages_are_never_compared() {
+        let mut a = Observation::new("GetObject", "negative", 404, Some("NoSuchKey".into()), []);
+        let mut b = a.clone();
+        a.message = Some("The specified key does not exist.".into());
+        b.message = Some("Object not found".into());
+        assert_eq!(compare(&[a], &[b]), Vec::new());
+    }
+
+    #[test]
+    fn a_case_observed_on_one_side_only_is_reported() {
+        let a = vec![ok("HeadBucket"), ok("ListBuckets")];
+        let b = vec![ok("HeadBucket")];
+        let differences = compare(&a, &b);
+        assert_eq!(fields(&differences), ["observation"]);
+        assert_eq!(differences[0].op, "ListBuckets");
+        assert_eq!(compared_ops(&a, &b).into_iter().collect::<Vec<_>>(), ["HeadBucket"]);
+    }
+
+    #[test]
+    fn a_case_recorded_twice_is_reported() {
+        let a = vec![ok("HeadBucket"), ok("HeadBucket")];
+        let b = vec![ok("HeadBucket")];
+        assert_eq!(fields(&compare(&a, &b)), ["duplicate_case"]);
+    }
+
+    #[test]
+    fn compared_ops_counts_operations_not_cases() {
+        let negative = Observation::new("HeadBucket", "negative", 404, Some("NotFound".into()), []);
+        let a = vec![ok("HeadBucket"), negative, ok("ListBuckets")];
+        let b = a.clone();
+        assert_eq!(compared_ops(&a, &b).len(), 2);
+    }
+
+    #[test]
+    fn the_report_lists_each_difference_as_op_field_a_b() {
+        let directory = std::env::temp_dir().join(format!("dual-stack-report-{}", uuid::Uuid::new_v4()));
+        let report = Report {
+            schema: 1,
+            run: "unit".into(),
+            stack_a: "legacy".into(),
+            stack_b: "legacy".into(),
+            region_a: None,
+            region_b: Some("eu-west-1".into()),
+            compared_ops: 1,
+            compared_cases: 1,
+            differences: compare(
+                &[ok("GetBucketLocation").with_field("location_constraint", "us-east-1")],
+                &[ok("GetBucketLocation").with_field("location_constraint", "eu-west-1")],
+            ),
+        };
+        let path = report.write_to(&directory).expect("write the report");
+        let json: serde_json::Value = serde_json::from_slice(&std::fs::read(&path).expect("read the report")).expect("parse");
+        let _ = std::fs::remove_dir_all(&directory);
+        assert_eq!(path.file_name().and_then(|name| name.to_str()), Some("unit.json"));
+        assert_eq!(json["compared_ops"], 1);
+        assert_eq!(
+            json["differences"],
+            serde_json::json!([{
+                "op": "GetBucketLocation",
+                "case": "normal",
+                "field": "body.location_constraint",
+                "a": "us-east-1",
+                "b": "eu-west-1",
+            }])
+        );
+    }
+}

crates/e2e_test/src/dual_stack_differential/mod.rs

@@ -0,0 +1,197 @@
+// Copyright 2024 RustFS Team
+//
+// Licensed under the Apache License, Version 2.0 (the "License");
+// you may not use this file except in compliance with the License.
+// You may obtain a copy of the License at
+//
+//     http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing, software
+// distributed under the License is distributed on an "AS IS" BASIS,
+// WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+// See the License for the specific language governing permissions and
+// limitations under the License.
+
+//! Dual-stack differential: one `rustfs` binary, two processes, one SDK script (rustfs/backlog#2740).
+//!
+//! Stack A always runs `RUSTFS_S3_STACK=legacy`. Stack B runs the value of `RUSTFS_E2E_STACK_B`
+//! (default `legacy` until the gateway stack is assembled), and `RUSTFS_E2E_STACK_B_REGION`, when
+//! set, starts B in another region so that a difference can be injected by hand. Both servers
+//! answer the script in `scenarios`, which covers every operation of `impl S3 for FS`; `compare`
+//! reports each `(op, field, a, b)` on which the answers differ, and every run writes its report
+//! to `<target>/dual-stack/<run>.json`. Two processes, not two embedded servers: bucket metadata
+//! is process-global, so two stacks in one process would read each other's buckets.
+
+mod compare;
+mod scenarios;
+
+use crate::common::{RustFSTestEnvironment, init_logging, workspace_root};
+use compare::{Difference, Observation, Report, compare, compared_cases, compared_ops, render};
+use std::path::PathBuf;
+use std::time::{SystemTime, UNIX_EPOCH};
+
+type BoxError = Box<dyn std::error::Error + Send + Sync>;
+
+const STACK_A: &str = "legacy";
+const ENV_STACK_B: &str = "RUSTFS_E2E_STACK_B";
+const ENV_STACK_B_REGION: &str = "RUSTFS_E2E_STACK_B_REGION";
+/// Becomes `gateway` once rustfs/backlog#2734 T2.5 assembles the gateway stack.
+const DEFAULT_STACK_B: &str = "legacy";
+/// The region the negative control gives stack A explicitly, so an inherited `RUSTFS_REGION`
+/// cannot erase the injected difference.
+const CONTROL_REGION_A: &str = "us-east-1";
+const CONTROL_REGION_B: &str = "eu-west-1";
+/// `normal` cases the legacy stack must answer with a 2xx. The comparison alone cannot tell a
+/// script that exercises real behavior from one in which both stacks fail the same way. Five of
+/// the 73 fail on legacy by design of this script: PutBucketReplication (no registered remote
+/// target), GetBucketReplication (so no configuration), PutBucketNotificationConfiguration (no
+/// notification target configured), RestoreObject (the object is not archived) and
+/// GetObjectTorrent (legacy answers `NoSuchKey`).
+const MIN_NORMAL_SUCCESSES: usize = 68;
+
+struct PairConfig {
+    stack_b: String,
+    region_a: Option<String>,
+    region_b: Option<String>,
+}
+
+impl PairConfig {
+    fn from_env() -> Self {
+        Self {
+            stack_b: std::env::var(ENV_STACK_B).unwrap_or_else(|_| DEFAULT_STACK_B.to_owned()),
+            region_a: None,
+            region_b: std::env::var(ENV_STACK_B_REGION).ok(),
+        }
+    }
+}
+
+/// Starts one server on `stack`, overriding its region only when asked.
+async fn start_stack(stack: &str, region: Option<&str>) -> Result<RustFSTestEnvironment, BoxError> {
+    let mut env = RustFSTestEnvironment::new().await?;
+    let mut vars = vec![("RUSTFS_S3_STACK", stack)];
+    if let Some(region) = region {
+        vars.push(("RUSTFS_REGION", region));
+    }
+    env.start_rustfs_server_without_cleanup_with_env(&vars).await?;
+    Ok(env)
+}
+
+/// Starts stack A and stack B side by side, each with its own port and data directory.
+async fn spawn_pair(config: &PairConfig) -> Result<(RustFSTestEnvironment, RustFSTestEnvironment), BoxError> {
+    tokio::try_join!(
+        start_stack(STACK_A, config.region_a.as_deref()),
+        start_stack(&config.stack_b, config.region_b.as_deref())
+    )
+}
+
+fn report_directory() -> PathBuf {
+    let target = std::env::var_os("CARGO_TARGET_DIR").map_or_else(|| PathBuf::from("target"), PathBuf::from);
+    let target = if target.is_absolute() {
+        target
+    } else {
+        workspace_root().join(target)
+    };
+    target.join("dual-stack")
+}
+
+struct Run {
+    report: Report,
+    path: PathBuf,
+    a: Vec<Observation>,
+}
+
+/// Runs the script against both stacks concurrently and writes the report.
+async fn run_differential(label: &str, config: &PairConfig) -> Run {
+    let (server_a, server_b) = spawn_pair(config).await.expect("start stack A and stack B");
+    let (a, b) = tokio::join!(Box::pin(scenarios::run(&server_a)), Box::pin(scenarios::run(&server_b)));
+    let millis = SystemTime::now()
+        .duration_since(UNIX_EPOCH)
+        .map_or(0, |elapsed| elapsed.as_millis());
+    let report = Report {
+        schema: 1,
+        run: format!("{label}-{millis}-{}", std::process::id()),
+        stack_a: STACK_A.to_owned(),
+        stack_b: config.stack_b.clone(),
+        region_a: config.region_a.clone(),
+        region_b: config.region_b.clone(),
+        compared_ops: compared_ops(&a, &b).len(),
+        compared_cases: compared_cases(&a, &b),
+        differences: compare(&a, &b),
+    };
+    let path = report.write_to(&report_directory()).expect("write the dual-stack report");
+    Run { report, path, a }
+}
+
+#[cfg(test)]
+mod tests {
+    use super::*;
+
+    #[tokio::test]
+    async fn both_stacks_answer_every_operation_identically() {
+        init_logging();
+        let config = PairConfig::from_env();
+        let run = run_differential("self-comparison", &config).await;
+        let report = &run.report;
+
+        let mut observed: Vec<_> = run.a.iter().map(|observation| observation.op).collect();
+        observed.sort_unstable();
+        observed.dedup();
+        assert_eq!(
+            observed,
+            scenarios::OPERATIONS,
+            "the script must drive exactly the operations of impl S3 for FS"
+        );
+        assert_eq!(report.compared_ops, scenarios::OPERATIONS.len(), "report: {}", run.path.display());
+        assert!(
+            report.differences.is_empty(),
+            "stack {} and stack {} differ on {} of {} cases (report: {}):\n{}",
+            report.stack_a,
+            report.stack_b,
+            report.differences.len(),
+            report.compared_cases,
+            run.path.display(),
+            render(&report.differences)
+        );
+
+        let not_4xx: Vec<_> = run
+            .a
+            .iter()
+            .filter(|observation| observation.case == "negative" && !(400..500).contains(&observation.status))
+            .collect();
+        assert!(not_4xx.is_empty(), "negative cases must fail with a 4xx on {STACK_A}: {not_4xx:#?}");
+        let successes = run
+            .a
+            .iter()
+            .filter(|observation| observation.case == "normal" && observation.is_success())
+            .count();
+        assert!(
+            successes >= MIN_NORMAL_SUCCESSES,
+            "only {successes} normal cases succeeded on {STACK_A}: {:#?}",
+            run.a
+        );
+    }
+
+    #[tokio::test]
+    async fn differential_detects_an_injected_difference() {
+        init_logging();
+        let config = PairConfig {
+            stack_b: STACK_A.to_owned(),
+            region_a: Some(CONTROL_REGION_A.to_owned()),
+            region_b: Some(CONTROL_REGION_B.to_owned()),
+        };
+        let run = run_differential("injected-region", &config).await;
+        let expected = Difference {
+            op: "GetBucketLocation".to_owned(),
+            case: "normal".to_owned(),
+            field: "body.location_constraint".to_owned(),
+            a: CONTROL_REGION_A.to_owned(),
+            b: CONTROL_REGION_B.to_owned(),
+        };
+        assert!(
+            run.report.differences.contains(&expected),
+            "B started in {CONTROL_REGION_B} but GetBucketLocation was not reported (report: {}):\n{}",
+            run.path.display(),
+            render(&run.report.differences)
+        );
+    }
+}

crates/e2e_test/src/heal_erasure_disk_rebuild_test.rs

@@ -1191,11 +1191,59 @@ mod tests {
         .into())
     }
 
+    fn scanner_cycle_completion_budget(
+        cycle_end: u64,
+        interval: Option<u64>,
+        now: u64,
+        active: bool,
+    ) -> Result<Duration, &'static str> {
+        if cycle_end > now {
+            return Err("scanner cycle timestamp is in the future");
+        }
+        if active {
+            return Ok(Duration::from_secs(60));
+        }
+        let interval = interval
+            .filter(|interval| *interval != 0)
+            .ok_or("scanner schedule has no positive interval")?;
+        let next_cycle = cycle_end
+            .checked_add(interval)
+            .ok_or("scanner schedule timestamp overflows")?;
+        let budget = next_cycle
+            .saturating_sub(now)
+            .checked_add(60)
+            .ok_or("scanner completion budget overflows")?;
+        Ok(Duration::from_secs(budget))
+    }
+
+    #[test]
+    fn scanner_cycle_completion_budget_accounts_for_captured_retry_schedule() {
+        assert_eq!(
+            scanner_cycle_completion_budget(1_791_815_390, Some(256), 1_791_815_447, false),
+            Ok(Duration::from_secs(259)),
+        );
+        assert_eq!(scanner_cycle_completion_budget(100, Some(1), 110, false), Ok(Duration::from_secs(60)));
+    }
+
+    #[test]
+    fn scanner_cycle_completion_budget_does_not_grant_planned_sleep_to_active_cycle() {
+        assert_eq!(scanner_cycle_completion_budget(100, Some(256), 110, true), Ok(Duration::from_secs(60)));
+        assert_eq!(scanner_cycle_completion_budget(0, None, 110, true), Ok(Duration::from_secs(60)));
+    }
+
+    #[test]
+    fn scanner_cycle_completion_budget_rejects_invalid_schedule() {
+        for (end, interval, now) in [(101, 1, 100), (100, 0, 100), (u64::MAX, 1, u64::MAX), (1, u64::MAX - 1, 1)] {
+            assert!(scanner_cycle_completion_budget(end, Some(interval), now, false).is_err());
+        }
+    }
+
     async fn wait_for_scanner_cycle_after(
         cluster: &RustFSTestClusterEnvironment,
         previous_cycle_end: u64,
     ) -> Result<u64, Box<dyn Error + Send + Sync>> {
-        let deadline = Instant::now() + Duration::from_secs(60);
+        let mut deadline = Instant::now() + Duration::from_secs(60);
+        let mut schedule_accounted = false;
         loop {
             let mut latest_cycle_end = 0;
             let mut versions_observed = false;
@@ -1237,14 +1285,36 @@ mod tests {
                 let versions_scanned = metrics["versions_scanned"]
                     .as_u64()
                     .ok_or("scanner status is missing its version-coverage counter")?;
-                latest_cycle_end = latest_cycle_end.max(cycle_end);
+                let now = std::time::SystemTime::now().duration_since(std::time::UNIX_EPOCH)?.as_secs();
+                if cycle_end > now {
+                    return Err("scanner cycle timestamp is in the future".into());
+                }
+                let schedule = &status["cycle_schedule"];
+                if !schedule_accounted && schedule["execution_role"].as_str() == Some("leader") {
+                    let active = metrics["current_cycle_active"]
+                        .as_bool()
+                        .ok_or("scanner status is missing its active-cycle flag")?;
+                    if active || (cycle_end != 0 && schedule["effective_interval_available"].as_bool() == Some(true)) {
+                        let interval = schedule["effective_interval_seconds"].as_u64();
+                        let budget = scanner_cycle_completion_budget(cycle_end, interval, now, active)?;
+                        deadline = Instant::now().checked_add(budget).ok_or("scanner deadline overflows")?;
+                        schedule_accounted = true;
+                    }
+                }
+                // Superseded walks complete all sets while withholding quota authority.
+                // Deferred and partial attempts do not certify completed scanning.
+                if matches!(metrics["last_cycle_result"].as_str(), Some("success" | "superseded")) {
+                    latest_cycle_end = latest_cycle_end.max(cycle_end);
+                }
                 versions_observed |= versions_scanned > 0;
                 observations.push(format!(
-                    "node{node_index}: end={cycle_end}, versions={versions_scanned}, cycle={}, active={}, leader={}, result={}",
+                    "node{node_index}: end={cycle_end}, versions={versions_scanned}, cycle={}, active={}, leader={}, result={}, partial_reason={}, schedule={}",
                     metrics["current_cycle"],
                     metrics["current_cycle_active"],
                     metrics["leader_lock_state"],
                     metrics["last_cycle_result"],
+                    metrics["last_cycle_partial_reason"],
+                    schedule,
                 ));
             }
             // The coordinator records cycle completion, but remote workers
@@ -1555,7 +1625,8 @@ mod tests {
         }
 
         if background_enabled {
-            wait_for_scanner_cycle_after(&cluster, 0).await?;
+            let baseline_cycle_floor = std::time::SystemTime::now().duration_since(std::time::UNIX_EPOCH)?.as_secs();
+            wait_for_scanner_cycle_after(&cluster, baseline_cycle_floor).await?;
         }
 
         cluster.stop_node(1)?;
@@ -2562,4 +2633,156 @@ mod tests {
         };
         assert!(!outage_candidate_targets_replacement(&insufficient_peer_shards, 12, replacement_set_slot));
     }
+    async fn two_node_heal_status(
+        cluster: &RustFSTestClusterEnvironment,
+        bucket: Option<&str>,
+        recreate: bool,
+        scan_mode: u8,
+    ) -> Result<serde_json::Value, Box<dyn Error + Send + Sync>> {
+        let url = format!("{}/rustfs/admin/v3/heal/{}", cluster.nodes[0].url, bucket.unwrap_or_default());
+        let options = serde_json::json!({
+            "recursive": true, "dryRun": false, "remove": false,
+            "recreate": recreate, "scanMode": scan_mode, "updateParity": false, "nolock": false,
+        });
+        let started =
+            start_root_heal_when_control_ready(&url, &options.to_string(), &cluster.access_key, &cluster.secret_key).await?;
+        let started: serde_json::Value = serde_json::from_str(&started)?;
+        let token = started["clientToken"]
+            .as_str()
+            .ok_or_else(|| format!("missing heal token: {started}"))?;
+        let status_url = format!("{url}?clientToken={token}");
+        let deadline = Instant::now() + Duration::from_secs(240);
+        loop {
+            let body = timeout(
+                Duration::from_secs(20),
+                signed_admin_post(&status_url, None, &cluster.access_key, &cluster.secret_key),
+            )
+            .await??;
+            let status: serde_json::Value = serde_json::from_str(&body)?;
+            match status["summary"].as_str() {
+                Some("finished" | "stopped") => return Ok(status),
+                Some("running") if Instant::now() < deadline => sleep(Duration::from_millis(250)).await,
+                _ => return Err(format!("heal did not reach a terminal state: {status}").into()),
+            }
+        }
+    }
+
+    async fn verify_two_node_directory_recovery(remove_bucket: bool, scan_mode: u8) -> Result<(), Box<dyn Error + Send + Sync>> {
+        init_logging();
+        let mut cluster = RustFSTestClusterEnvironment::new(2).await?;
+        // Isolate explicit administrator recovery from scanner, MRF, and disk healing.
+        for key in [
+            "RUSTFS_HEAL_AUTO_HEAL_ENABLE",
+            "RUSTFS_HEAL_MRF_ENABLE",
+            "RUSTFS_SCANNER_ENABLED",
+            "RUSTFS_PUT_RENAME_EARLY_ACK_ENABLE",
+        ] {
+            cluster.set_env(key, "false");
+        }
+        cluster.start().await?;
+        let source = cluster.create_s3_client(0)?;
+        let target = cluster.create_s3_client(1)?;
+        let bucket = "two-node-directory-recovery";
+        source.create_bucket().bucket(bucket).send().await?;
+        let objects = [
+            ("empty", Vec::new()),
+            ("nested/inline", vec![0x3c; 1024]),
+            ("nested/external", (0_u8..=255).cycle().take(1024 * 1024).collect::<Vec<_>>()),
+        ];
+        for (key, bytes) in &objects {
+            source
+                .put_object()
+                .bucket(bucket)
+                .key(*key)
+                .body(ByteStream::from(bytes.clone()))
+                .send()
+                .await?;
+        }
+        let target_bucket = Path::new(&cluster.nodes[1].data_dir).join(bucket);
+        let original = objects
+            .iter()
+            .map(|(key, _)| census_object_version_on_disk(Path::new(&cluster.nodes[1].data_dir), bucket, key, None))
+            .collect::<Result<Vec<_>, _>>()?;
+        for census in &original {
+            assert!(census.has_xl_meta, "fixture must have a target metadata copy: {census:?}");
+        }
+        assert!(
+            !original[2].present_part_fingerprints.is_empty(),
+            "large fixture must exercise external parts"
+        );
+        let removed = Path::new(&cluster.temp_dir).join("removed");
+        tokio::fs::create_dir(&removed).await?;
+        if remove_bucket {
+            tokio::fs::rename(&target_bucket, removed.join(bucket)).await?;
+            let status = two_node_heal_status(&cluster, None, false, scan_mode).await?;
+            assert_eq!(
+                status["summary"], "stopped",
+                "missing bucket must not look like an empty successful scan: {status}"
+            );
+            assert_eq!(status["outcome"]["execution"]["state"], "aborted");
+            assert_eq!(status["outcome"]["coverage"], "partial");
+            assert_eq!(status["outcome"]["counters"]["processed"], 0);
+            assert!(!target_bucket.exists(), "default discovery must not silently recreate a bucket");
+            let scoped_default = two_node_heal_status(&cluster, Some(bucket), false, scan_mode).await?;
+            assert_eq!(
+                scoped_default["summary"], "stopped",
+                "default namespace quorum protection must remain: {scoped_default}"
+            );
+        } else {
+            for (index, (key, _)) in objects.iter().enumerate() {
+                tokio::fs::rename(target_bucket.join(key), removed.join(index.to_string())).await?;
+            }
+        }
+        let status = two_node_heal_status(&cluster, remove_bucket.then_some(bucket), remove_bucket, scan_mode).await?;
+        assert_eq!(status["summary"], "finished", "explicit recovery should complete: {status}");
+        assert_eq!(status["outcome"]["execution"]["state"], "completed");
+        assert_eq!(status["outcome"]["coverage"], "complete");
+        for ((key, _), before) in objects.iter().zip(&original) {
+            let after = census_object_version_on_disk(Path::new(&cluster.nodes[1].data_dir), bucket, key, None)?;
+            assert!(after.has_xl_meta, "metadata must be restored before GET: {key}");
+            assert_eq!(
+                after.present_part_fingerprints, before.present_part_fingerprints,
+                "external shard bytes must match: {key}"
+            );
+            assert_eq!(
+                after.inline_data_fingerprint, before.inline_data_fingerprint,
+                "inline shard bytes must match: {key}"
+            );
+        }
+        cluster.stop_node(0)?;
+        for (key, expected) in &objects {
+            let actual = timeout(Duration::from_secs(30), async {
+                target
+                    .get_object()
+                    .bucket(bucket)
+                    .key(*key)
+                    .send()
+                    .await?
+                    .body
+                    .collect()
+                    .await
+                    .map(|body| body.into_bytes())
+                    .map_err(Box::<dyn Error + Send + Sync>::from)
+            })
+            .await??;
+            assert_eq!(actual.len(), expected.len(), "target-only read length: {key}");
+            assert_eq!(sha256_hex(&actual), sha256_hex(expected), "target-only content: {key}");
+        }
+        Ok(())
+    }
+
+    #[tokio::test]
+    async fn test_two_node_bucket_discovery_and_explicit_recreate() -> Result<(), Box<dyn Error + Send + Sync>> {
+        verify_two_node_directory_recovery(true, 2).await
+    }
+
+    #[tokio::test]
+    async fn test_two_node_object_directory_recovery_normal() -> Result<(), Box<dyn Error + Send + Sync>> {
+        verify_two_node_directory_recovery(false, 1).await
+    }
+
+    #[tokio::test]
+    async fn test_two_node_object_directory_recovery_deep() -> Result<(), Box<dyn Error + Send + Sync>> {
+        verify_two_node_directory_recovery(false, 2).await
+    }
 }

crates/e2e_test/src/internode_rpc_signature_e2e_test.rs

@@ -264,7 +264,7 @@ async fn call_make_volume_response(
     request: MakeVolumeRequest,
     headers: HeaderMap,
 ) -> Result<Response<MakeVolumeResponse>, Status> {
-    let mut client = node_service_time_out_client_no_auth(&url.to_string())
+    let mut client = node_service_time_out_client_no_auth(url)
         .await
         .map_err(|err| Status::unavailable(format!("cannot reach the node service: {err}")))?;
     let mut rpc_request = Request::new(request);
@@ -273,7 +273,7 @@ async fn call_make_volume_response(
 }
 
 async fn call_ping_response(url: &str, headers: HeaderMap) -> Result<Response<PingResponse>, Status> {
-    let mut client = node_service_time_out_client_no_auth(&url.to_string())
+    let mut client = node_service_time_out_client_no_auth(url)
         .await
         .map_err(|err| Status::unavailable(format!("cannot reach the node service: {err}")))?;
     let mut rpc_request = Request::new(PingRequest {

crates/e2e_test/src/lib.rs

@@ -31,6 +31,12 @@ pub mod chaos;
 #[cfg(test)]
 pub mod fake_s3_target;
 
+// Dual-stack differential (rustfs/backlog#2740): one binary, two processes on
+// different RUSTFS_S3_STACK values, the same SDK script for every S3 operation,
+// compared on status, error code, allowlisted headers and typed body fields.
+#[cfg(test)]
+mod dual_stack_differential;
+
 // On-demand migration (backlog#2147): shared two-server environment, admin
 // wrappers, and the harness self-test (backlog#2151). Behavior scenarios are
 // added by later ODM tasks.

crates/e2e_test/src/namespace_lock_quorum_test.rs

@@ -12,7 +12,7 @@
 // See the License for the specific language governing permissions and
 // limitations under the License.
 
-use crate::common::RustFSTestClusterEnvironment;
+use crate::common::{ClusterTopology, RustFSTestClusterEnvironment};
 use aws_sdk_s3::Client;
 use aws_sdk_s3::error::SdkError;
 use bytes::Bytes;
@@ -185,21 +185,113 @@ async fn wait_for_quorum_read_admission(clients: &[Client], bucket: &str) -> Tes
             return Ok(());
         }
         if tokio::time::Instant::now() >= deadline {
-            return Err(format!("read quorum did not become available after lease convergence for {bucket}").into());
+            let diagnostics = clients[0].get_object().bucket(bucket).key("warm-small").send().await;
+            return Err(format!(
+                "read quorum did not become available after lease convergence for {bucket}: GET={diagnostics:?}"
+            )
+            .into());
         }
         tokio::time::sleep(std::time::Duration::from_millis(100)).await;
     }
 }
 
+#[tokio::test]
+async fn test_cold_degraded_object_mutations_require_only_storage_quorum() -> TestResult {
+    crate::common::init_logging();
+    let mut cluster = RustFSTestClusterEnvironment::new(4).await?;
+    cluster.set_env("RUSTFS_STORAGE_CLASS_STANDARD", "EC:2");
+    cluster.set_env("RUSTFS_PUT_RENAME_EARLY_ACK_ENABLE", "false");
+    cluster.set_env("RUSTFS_OBS_METRICS_EXPORT_ENABLED", "false");
+    cluster.set_env("RUSTFS_GET_OBJECT_METADATA_CACHE_DISTRIBUTED_ENABLE", "false");
+    cluster.start().await?;
+    let clients = cluster
+        .create_all_clients()?
+        .into_iter()
+        .map(|client| {
+            Client::from_conf(
+                client
+                    .config()
+                    .to_builder()
+                    .retry_config(aws_sdk_s3::config::retry::RetryConfig::standard().with_max_attempts(1))
+                    .build(),
+            )
+        })
+        .collect::<Vec<_>>();
+    let bucket = "cold-degraded-mutations";
+    clients[0].create_bucket().bucket(bucket).send().await?;
+    // No object mutation may warm a process-wide configuration probe before
+    // the missing peer is introduced. Three shards still satisfy EC:2 writes.
+    cluster.stop_node(3)?;
+    let admission_deadline = Instant::now() + Duration::from_secs(30);
+    for client in &clients[..3] {
+        loop {
+            match client
+                .get_object()
+                .bucket(bucket)
+                .key("never-created-readiness-probe")
+                .send()
+                .await
+            {
+                Err(error) if error.as_service_error().and_then(|error| error.meta().code()) == Some("NoSuchKey") => break,
+                Err(error)
+                    if error.raw_response().is_some_and(|response| response.status().as_u16() == 503)
+                        && Instant::now() < admission_deadline =>
+                {
+                    tokio::time::sleep(Duration::from_millis(100)).await;
+                }
+                Err(error) => return Err(error.into()),
+                Ok(_) => panic!("a cold readiness probe must not create an object"),
+            }
+        }
+    }
+    let body = b"first write while one peer is offline";
+    clients[0]
+        .put_object()
+        .bucket(bucket)
+        .key("object")
+        .body(Bytes::from_static(body).into())
+        .send()
+        .await?;
+    for client in &clients[..3] {
+        assert_quorum_object_body(client, bucket, "object", body).await?;
+    }
+    // This coordinator has not previously committed an object mutation either.
+    clients[1].delete_object().bucket(bucket).key("object").send().await?;
+    let error = clients[2]
+        .get_object()
+        .bucket(bucket)
+        .key("object")
+        .send()
+        .await
+        .expect_err("acknowledged deletion must be visible at another entrypoint");
+    assert_eq!(error.as_service_error().and_then(|error| error.meta().code()), Some("NoSuchKey"));
+
+    cluster.stop_node(2)?;
+    let error = clients[0]
+        .put_object()
+        .bucket(bucket)
+        .key("below-quorum")
+        .body(Bytes::from_static(body).into())
+        .send()
+        .await
+        .expect_err("removing the cache gate must not weaken storage write quorum");
+    assert_eq!(error.as_service_error().and_then(|error| error.meta().code()), Some("ServiceUnavailable"));
+    cluster.stop();
+    Ok(())
+}
+
 #[tokio::test]
 async fn test_degraded_cluster_read_quorum_follows_erasure_layout() -> TestResult {
     crate::common::init_logging();
 
-    for (node_count, parity) in [(4, 2), (6, 3), (6, 2)] {
+    for (node_count, parity) in [(3, 1), (4, 2), (6, 3), (6, 2), (4, 0)] {
         let read_quorum = node_count - parity;
         let write_quorum = read_quorum + usize::from(read_quorum == parity);
         let mut cluster = RustFSTestClusterEnvironment::new(node_count).await?;
         cluster.set_env("RUSTFS_STORAGE_CLASS_STANDARD", format!("EC:{parity}"));
+        if parity == 0 {
+            cluster.set_env("RUSTFS_STORAGE_CLASS_RRS", "EC:0");
+        }
         // Wait for every seed fanout before removing any physical shard.
         cluster.set_env("RUSTFS_PUT_RENAME_EARLY_ACK_ENABLE", "false");
         cluster.set_env("RUSTFS_OBS_METRICS_EXPORT_ENABLED", "false");
@@ -338,6 +430,277 @@ async fn test_degraded_cluster_read_quorum_follows_erasure_layout() -> TestResul
     Ok(())
 }
 
+#[tokio::test]
+async fn test_cold_start_read_quorum_follows_erasure_layout() -> TestResult {
+    crate::common::init_logging();
+    for (node_count, parity) in [(3, 1), (4, 2), (6, 3), (6, 2), (4, 0)] {
+        let read_quorum = node_count - parity;
+        let write_quorum = read_quorum + usize::from(read_quorum == parity);
+        let mut cluster = RustFSTestClusterEnvironment::new(node_count).await?;
+        cluster.set_env("RUSTFS_STORAGE_CLASS_STANDARD", format!("EC:{parity}"));
+        if parity == 0 {
+            cluster.set_env("RUSTFS_STORAGE_CLASS_RRS", "EC:0");
+        }
+        cluster.set_env("RUSTFS_PUT_RENAME_EARLY_ACK_ENABLE", "false");
+        cluster.set_env("RUSTFS_OBS_METRICS_EXPORT_ENABLED", "false");
+        cluster.set_env("RUST_LOG", "warn");
+        cluster.start().await?;
+        let clients = cluster
+            .create_all_clients()?
+            .into_iter()
+            .map(|client| {
+                Client::from_conf(
+                    client
+                        .config()
+                        .to_builder()
+                        .retry_config(aws_sdk_s3::config::retry::RetryConfig::standard().with_max_attempts(1))
+                        .build(),
+                )
+            })
+            .collect::<Vec<_>>();
+        let bucket = format!("cold-read-quorum-{node_count}-{parity}");
+        clients[0].create_bucket().bucket(&bucket).send().await?;
+        let small = b"cold startup follows the stored EC layout".to_vec();
+        let large = vec![73; 256 * 1024];
+        for (key, body) in [
+            ("warm-small", &small),
+            ("warm-large", &large),
+            ("cold-large", &large),
+            ("below-quorum", &large),
+        ] {
+            clients[0]
+                .put_object()
+                .bucket(&bucket)
+                .key(key)
+                .body(Bytes::copy_from_slice(body).into())
+                .send()
+                .await?;
+            for node in &cluster.nodes {
+                let census =
+                    crate::chaos::census_object_version_on_disk(std::path::Path::new(&node.data_dir), &bucket, key, None)?;
+                assert!(census.is_complete(), "seed shard must exist before cold startup: {census:?}");
+                assert_eq!((census.data_blocks, census.parity_blocks), (Some(read_quorum), Some(parity)));
+            }
+        }
+        for node in 0..node_count {
+            cluster.stop_node(node)?;
+        }
+        // Rebuild every process's in-memory state below read quorum, then
+        // cross the EC boundary. A format bootstrap must not admit S3 reads
+        // merely because half the peers are online (e.g. six disks, EC:2).
+        for node in 0..read_quorum - 1 {
+            cluster.start_node_process(node).await?;
+        }
+        let error = clients[0]
+            .get_object()
+            .bucket(&bucket)
+            .key("below-quorum")
+            .send()
+            .await
+            .expect_err("cold startup below the configured EC read quorum must stay unavailable");
+        assert_eq!(error.raw_response().map(|response| response.status().as_u16()), Some(503));
+        cluster.start_node_process(read_quorum - 1).await?;
+        wait_for_quorum_read_admission(&clients[..read_quorum], &bucket).await?;
+        for client in clients.iter().take(read_quorum) {
+            assert_quorum_object_body(client, &bucket, "warm-small", &small).await?;
+            assert_quorum_object_body(client, &bucket, "cold-large", &large).await?;
+            let write = client
+                .put_object()
+                .bucket(&bucket)
+                .key("cold-quorum-write")
+                .body(Bytes::copy_from_slice(&small).into())
+                .send()
+                .await;
+            if read_quorum >= write_quorum {
+                write?;
+                assert_quorum_object_body(client, &bucket, "cold-quorum-write", &small).await?;
+            } else {
+                let error = write.expect_err("cold startup at read quorum must preserve the EC write quorum");
+                assert_eq!(error.raw_response().map(|response| response.status().as_u16()), Some(503));
+            }
+        }
+        for node in read_quorum..node_count {
+            cluster.start_node(node).await?;
+        }
+        for client in &clients {
+            assert_quorum_object_body(client, &bucket, "cold-large", &large).await?;
+        }
+    }
+    Ok(())
+}
+
+#[tokio::test]
+async fn test_four_by_six_cold_start_at_read_quorum() -> TestResult {
+    crate::common::init_logging();
+    let mut cluster = RustFSTestClusterEnvironment::with_topology(ClusterTopology::single_pool_multidrive(4, 6)).await?;
+    // Explicit URLs are consumed in order. Place one drive from each node in
+    // every set, matching a four-host, six-drive-per-host deployment.
+    let volumes = (0..6)
+        .flat_map(|drive| {
+            cluster
+                .nodes
+                .iter()
+                .map(move |node| format!("http://{}{}", node.address, node.data_dirs[drive]))
+        })
+        .collect::<Vec<_>>()
+        .join(" ");
+    cluster.set_env("RUSTFS_VOLUMES", volumes);
+    cluster.set_env("RUSTFS_ERASURE_SET_DRIVE_COUNT", "4");
+    cluster.set_env("RUSTFS_STORAGE_CLASS_STANDARD", "EC:2");
+    cluster.set_env("RUSTFS_PUT_RENAME_EARLY_ACK_ENABLE", "false");
+    cluster.set_env("RUSTFS_OBS_METRICS_EXPORT_ENABLED", "false");
+    cluster.set_env("RUST_LOG", "error");
+    cluster.start().await?;
+    let clients = cluster
+        .create_all_clients()?
+        .into_iter()
+        .map(|client| {
+            Client::from_conf(
+                client
+                    .config()
+                    .to_builder()
+                    .retry_config(aws_sdk_s3::config::retry::RetryConfig::standard().with_max_attempts(1))
+                    .build(),
+            )
+        })
+        .collect::<Vec<_>>();
+    let bucket = "cold-start-read-quorum";
+    clients[0].create_bucket().bucket(bucket).send().await?;
+    let mut fixtures = Vec::new();
+    let mut covered = [[false; 2]; 6];
+    for index in 0..128 {
+        let key = match index {
+            0 => "warm-small".to_string(),
+            1 => "warm-large".to_string(),
+            _ => format!("cold-{index}"),
+        };
+        let size_class = index % 2;
+        let size = if size_class == 0 { 71 } else { 256 * 1024 };
+        let body = (0..size)
+            .map(|offset| u8::try_from(offset % 251).expect("bounded fixture byte"))
+            .collect::<Vec<_>>();
+        clients[0]
+            .put_object()
+            .bucket(bucket)
+            .key(&key)
+            .body(Bytes::copy_from_slice(&body).into())
+            .send()
+            .await?;
+        for node in &cluster.nodes {
+            let mut complete_shards = 0;
+            for (drive, path) in node.data_dirs.iter().enumerate() {
+                let census = crate::chaos::census_object_version_on_disk(std::path::Path::new(path), bucket, &key, None)?;
+                if census.has_xl_meta {
+                    assert!(census.is_complete(), "seed shard must be complete before shutdown: {census:?}");
+                    assert_eq!((census.data_blocks, census.parity_blocks), (Some(2), Some(2)));
+                    covered[drive][size_class] = true;
+                    complete_shards += 1;
+                }
+            }
+            assert_eq!(complete_shards, 1, "each node must own one shard of every seed object");
+        }
+        fixtures.push((key, body));
+        if covered.iter().flatten().all(|covered| *covered) {
+            break;
+        }
+    }
+    assert!(
+        covered.iter().flatten().all(|covered| *covered),
+        "fixtures must cover all six sets with inline and large objects"
+    );
+
+    for first in 0..4 {
+        for second in first + 1..4 {
+            for node in 0..4 {
+                cluster.stop_node(node)?;
+            }
+            cluster.start_node_process(first).await?;
+            let error = clients[first]
+                .get_object()
+                .bucket(bucket)
+                .key("warm-large")
+                .send()
+                .await
+                .expect_err("one node cannot satisfy read quorum after a cold start");
+            assert_eq!(error.raw_response().map(|response| response.status().as_u16()), Some(503));
+            cluster.start_node_process(second).await?;
+            wait_for_quorum_read_admission(&[clients[first].clone(), clients[second].clone()], bucket).await?;
+            for node in [first, second] {
+                for (key, body) in &fixtures {
+                    assert_quorum_object_body(&clients[node], bucket, key, body).await?;
+                }
+                let range = clients[node]
+                    .get_object()
+                    .bucket(bucket)
+                    .key("warm-large")
+                    .range("bytes=17-1023")
+                    .send()
+                    .await?;
+                assert_eq!(range.body.collect().await?.into_bytes().as_ref(), &fixtures[1].1[17..1024]);
+                let listing = clients[node].list_objects_v2().bucket(bucket).send().await?;
+                for (key, _) in &fixtures {
+                    assert!(
+                        listing.contents().iter().any(|object| object.key() == Some(key.as_str())),
+                        "listing omitted {key}"
+                    );
+                }
+                let error = clients[node]
+                    .put_object()
+                    .bucket(bucket)
+                    .key("recovery-write")
+                    .body(Bytes::from_static(b"after recovery").into())
+                    .send()
+                    .await
+                    .expect_err("two nodes must not authorize a write");
+                assert_eq!(error.raw_response().map(|response| response.status().as_u16()), Some(503));
+            }
+            let error = clients[first]
+                .delete_object()
+                .bucket(bucket)
+                .key("warm-large")
+                .send()
+                .await
+                .expect_err("two nodes must not authorize a delete");
+            assert_eq!(error.raw_response().map(|response| response.status().as_u16()), Some(503));
+            assert_quorum_object_body(&clients[first], bucket, "warm-large", &fixtures[1].1).await?;
+
+            let remaining = (0..4).filter(|node| *node != first && *node != second).collect::<Vec<_>>();
+            cluster.start_node(remaining[0]).await?;
+            let deadline = tokio::time::Instant::now() + Duration::from_secs(90);
+            loop {
+                match clients[first]
+                    .put_object()
+                    .bucket(bucket)
+                    .key("recovery-write")
+                    .body(Bytes::from_static(b"after recovery").into())
+                    .send()
+                    .await
+                {
+                    Ok(_) => break,
+                    Err(error)
+                        if error.raw_response().is_some_and(|response| response.status().as_u16() == 503)
+                            && tokio::time::Instant::now() < deadline =>
+                    {
+                        tokio::time::sleep(Duration::from_millis(200)).await
+                    }
+                    Err(error) => return Err(error.into()),
+                }
+            }
+            for node in [first, second, remaining[0]] {
+                assert_quorum_object_body(&clients[node], bucket, "recovery-write", b"after recovery").await?;
+            }
+            cluster.start_node(remaining[1]).await?;
+            for client in &clients {
+                for (key, body) in &fixtures {
+                    assert_quorum_object_body(client, bucket, key, body).await?;
+                }
+                assert_quorum_object_body(client, bucket, "recovery-write", b"after recovery").await?;
+            }
+        }
+    }
+    Ok(())
+}
+
 async fn put_object(client: Client, payload: Vec<u8>, writer_id: usize) -> Result<(), String> {
     client
         .put_object()

crates/e2e_test/src/reliant/grpc_lock_server.rs

@@ -465,6 +465,13 @@ impl NodeService for MinimalLockNodeService {
         Err(Status::unimplemented("lock-only test server"))
     }
 
+    async fn recover_delete_transaction(
+        &self,
+        _request: Request<rustfs_protos::proto_gen::node_service::RecoverDeleteTransactionRequest>,
+    ) -> Result<Response<rustfs_protos::proto_gen::node_service::RecoverDeleteTransactionResponse>, Status> {
+        Err(Status::unimplemented("delete recovery is not supported by the lock fixture"))
+    }
+
     async fn settle_part_transaction(
         &self,
         _request: Request<rustfs_protos::proto_gen::node_service::SettlePartTransactionRequest>,

crates/e2e_test/src/sts_query_compat_test.rs

@@ -740,7 +740,7 @@ async fn test_bucket_tags_opa_contract() -> TestResult {
         .version_id(version)
         .send()
         .await?;
-    opa.expect_bucket_tags("s3:GetObject", source, Some("finance")).await?;
+    opa.expect_bucket_tags("s3:GetObjectVersion", source, Some("finance")).await?;
     client.list_objects_v2().bucket(source).send().await?;
     opa.expect_bucket_tags("s3:ListBucket", source, Some("finance")).await?;
     client.list_object_versions().bucket(source).send().await?;

crates/e2e_test/src/upgrade_compatibility_test.rs

@@ -2345,7 +2345,7 @@ fn layout_reported_parts(attributes: &aws_sdk_s3::operation::get_object_attribut
     attributes.object_parts().map(|parts| parts.parts().len())
 }
 
-async fn assert_layout_readable(client: &Client, case: &LayoutCase, context: &str) -> TestResult {
+async fn assert_layout_full_body(client: &Client, case: &LayoutCase, context: &str) -> TestResult {
     let label = case.label();
     let head = layout_head(client, case).await?;
     assert_eq!(
@@ -2362,7 +2362,12 @@ async fn assert_layout_readable(client: &Client, case: &LayoutCase, context: &st
     let full = layout_get(client, case, None, None).await?.body.collect().await?.into_bytes();
     assert_eq!(full.len(), case.body.len(), "{context}: {label}: full GET length");
     assert!(full == case.body, "{context}: {label}: full GET body must equal the rc.5 upload");
+    Ok(())
+}
 
+async fn assert_layout_readable(client: &Client, case: &LayoutCase, context: &str) -> TestResult {
+    assert_layout_full_body(client, case, context).await?;
+    let label = case.label();
     if case.is_multipart_layout() {
         let first = case.part_sizes[0];
         let range = format!("bytes={}-{}", first - 32, first + 31);
@@ -2641,10 +2646,11 @@ async fn direct_upgrade_from_rc5_preserves_multipart_layouts() -> TestResult {
             "rc.5 wrote a legacy layout"
         );
     }
-    // The rc.5 writer must itself still read what it wrote, so a later
-    // failure is attributable to the upgrade rather than to the fixture.
+    // Prove the rc.5 writer persisted every original byte before upgrading.
+    // Its compressed cross-part range reader has a legacy limitation; the
+    // upgraded reader must still satisfy every range and partNumber assertion.
     for case in &cases {
-        assert_layout_readable(&old_client, case, "rc.5").await?;
+        assert_layout_full_body(&old_client, case, "rc.5").await?;
     }
 
     // Upgrade in place.

crates/ecstore/Cargo.toml

@@ -183,6 +183,7 @@ sha2 = { workspace = true }
 hex-simd = { workspace = true }
 tempfile.workspace = true
 hyper = { workspace = true, features = ["http2", "http1", "server"] }
+h2 = { workspace = true }
 hyper-util = { workspace = true, features = ["tokio", "server-auto", "server-graceful", "tracing"] }
 hyper-rustls = { workspace = true, default-features = false, features = ["native-tokio", "http1", "tls12", "logging", "http2", "aws-lc-rs"] }
 hostname.workspace = true
@@ -200,8 +201,8 @@ md-5.workspace = true
 memmap2 = { workspace = true }
 libc.workspace = true
 # "process" adds getrlimit for the io_uring fd-cache RLIMIT_NOFILE gate
-# (backlog#1178); "fs" comes from the workspace default.
-rustix = { workspace = true, features = ["process", "fs"] }
+# (backlog#1178); "thread" provides cleanup affinity; "fs" is a workspace default.
+rustix = { workspace = true, features = ["process", "fs", "thread"] }
 rustfs-madmin.workspace = true
 reqwest = { workspace = true }
 aws-sdk-s3 = { workspace = true, default-features = false, features = ["sigv4a", "default-https-client", "rt-tokio"] }

crates/ecstore/src/api/mod.rs

@@ -199,10 +199,10 @@ pub mod bucket {
             get_object_lock_config, get_object_lock_config_state, get_on_demand_migration_config,
             get_on_demand_migration_config_in, get_public_access_block_config, get_quota_config, get_replication_config,
             get_request_payment_config, get_sse_config, get_tagging_config, get_versioning_config, get_website_config,
-            init_bucket_metadata_sys, list_bucket_targets, reload_bucket_metadata, remove_bucket_metadata, set_bucket_metadata,
-            update, update_bucket_targets_under_transaction_lock, update_config_with, update_if_incarnation,
-            update_if_incarnation_at, update_quota_if_incarnation, update_quota_if_incarnation_at, update_under_transaction_lock,
-            update_under_transaction_lock_at,
+            init_bucket_metadata_sys, list_bucket_targets, reload_bucket_metadata, remove_bucket_metadata,
+            remove_bucket_metadata_if_absent, set_bucket_metadata, update, update_bucket_targets_under_transaction_lock,
+            update_config_with, update_if_incarnation, update_if_incarnation_at, update_quota_if_incarnation,
+            update_quota_if_incarnation_at, update_under_transaction_lock, update_under_transaction_lock_at,
         };
         #[cfg(feature = "test-util")]
         pub use crate::bucket::metadata_sys::{ConfigWriteLockProbe, test_support};
@@ -210,7 +210,8 @@ pub mod bucket {
 
     pub mod migration {
         pub use crate::bucket::migration::{
-            LegacyBlobDecryptFn, migration_startup_error, try_migrate_bucket_metadata, try_migrate_iam_config,
+            LegacyBlobDecryptFn, migration_startup_error, try_migrate_bucket_metadata, try_migrate_bucket_metadata_for_buckets,
+            try_migrate_iam_config,
         };
     }
 
@@ -422,11 +423,12 @@ pub mod disk {
     pub use crate::disk::os::{LocalPublicationPause, LocalPublicationStage};
     pub use crate::disk::{
         BATCH_READ_VERSION_MAX_ITEMS, BUCKET_META_PREFIX, BatchReadVersionItem, BatchReadVersionReq, BatchReadVersionResp,
-        CheckPartsResp, ConditionalFileUpdate, DeleteOptions, Disk, DiskAPI, DiskInfo, DiskInfoOptions, DiskLocation, DiskOption,
-        DiskStore, FileInfoVersions, FileReader, FileWriter, HEALING_MARKER_PATH, NsScannerOpenRequest, OldCurrentSize,
-        PartTransactionAction, RUSTFS_META_BUCKET, ReadMultipleReq, ReadMultipleResp, ReadOptions, RenameDataResp,
-        STORAGE_FORMAT_FILE, SnapshotLeaseToken, UpdateMetadataOpts, VolumeInfo, WalkDirOptions, new_disk,
-        validate_batch_read_version_item_count,
+        CheckPartsResp, ConditionalFileUpdate, DELETE_RECOVERY_PROTOCOL_VERSION, DeleteOptions, DeleteRecoveryAction,
+        DeleteRecoveryCursor, DeleteRecoveryDigest, DeleteRecoveryInspection, DeleteRecoveryPage, DeleteRecoveryRpcOperation,
+        DeleteRecoveryRpcReply, Disk, DiskAPI, DiskInfo, DiskInfoOptions, DiskLocation, DiskOption, DiskStore, FileInfoVersions,
+        FileReader, FileWriter, HEALING_MARKER_PATH, NsScannerOpenRequest, OldCurrentSize, PartTransactionAction,
+        RUSTFS_META_BUCKET, ReadMultipleReq, ReadMultipleResp, ReadOptions, RenameDataResp, STORAGE_FORMAT_FILE,
+        SnapshotLeaseToken, UpdateMetadataOpts, VolumeInfo, WalkDirOptions, new_disk, validate_batch_read_version_item_count,
     };
     pub use bytes::Bytes;
     pub use endpoint::Endpoint;
@@ -519,13 +521,14 @@ pub mod integrity {
 pub mod object {
     pub use crate::object_api::{
         BLOCK_SIZE_V2, ERASURE_ALGORITHM, EncryptionResolutionError, EncryptionResolutionErrorKind, GetObjectBodyCacheHook,
-        GetObjectBodyCacheHookLookup, GetObjectBodySource, GetObjectReader, NamespaceLockFence, ObjectEncryptionResolver,
-        ObjectInfo, ObjectLockConfigSnapshot, ObjectMutationHook, ObjectOptions, PutObjReader, QuotaAdmission,
-        RangedDecompressReader, ReadEncryptionMaterial, ReadEncryptionMode, ReadEncryptionRequest,
+        GetObjectBodyCacheHookLookup, GetObjectBodySource, GetObjectReadCondition, GetObjectReadMetadata, GetObjectReader,
+        NamespaceLockFence, ObjectEncryptionResolver, ObjectInfo, ObjectLockConfigSnapshot, ObjectMutationHook, ObjectOptions,
+        PutObjReader, QuotaAdmission, RangedDecompressReader, ReadEncryptionMaterial, ReadEncryptionMode, ReadEncryptionRequest,
         SCANNER_PUBLICATION_LEASE_FENCE_METADATA_KEY, ScannerPublicationCommitScope, ScannerPublicationCommitStartError,
         ScannerPublicationCommitState, ShardIntegrityWriteMode, StreamConsumer, WriteCommitGuard, WriteCompletion,
         get_object_body_cache_plaintext_len, lookup_get_object_body_cache_hook, register_get_object_body_cache_hook,
         register_object_mutation_hook, unregister_get_object_body_cache_hook, unregister_object_mutation_hook,
+        with_get_object_read_condition,
     };
     pub use crate::store::{
         PrepareSelectObjectSnapshotError, PreparedGetObjectReader, SelectObjectSnapshot, SelectObjectSnapshotReadError,
@@ -590,6 +593,7 @@ pub mod set_disk {
     #[cfg(feature = "test-util")]
     pub mod test_util {
         pub use crate::bucket::quota::reservation::fail_next_quota_ledger_save_for_test;
+        pub use crate::set_disk::ConditionalReadBenchmarkMetadataGuard;
         pub use crate::set_disk::{MultipartCommitBarrier, MultipartCommitPause, PutObjectCommitBarrier, PutObjectCommitPause};
 
         /// Keep a namespace commit pending until the returned owner is dropped.

crates/ecstore/src/bucket/lifecycle/bucket_lifecycle_ops.rs

@@ -2417,6 +2417,16 @@ fn is_slow_down(err: &Error) -> bool {
     matches!(err, Error::SlowDown)
 }
 
+/// Lifecycle expiry is idempotent when the exact object or version has already
+/// disappeared between the scanner listing and the delete admission.
+///
+/// Keep this predicate narrower than bucket/volume "not found" helpers: an
+/// unavailable bucket or disk is an operational failure and must not be
+/// reported as a completed expiry.
+fn is_lifecycle_expiry_already_gone(err: &Error) -> bool {
+    is_err_object_not_found(err) || is_err_version_not_found(err)
+}
+
 /// Resolves the expiry worker count from the single documented knob,
 /// `RUSTFS_MAX_EXPIRY_WORKERS`: a set, parsable, non-zero value wins;
 /// anything else falls back to `min(cpus, 16)`. The historical
@@ -4920,6 +4930,20 @@ async fn expire_transitioned_object_with_lock_lost_signal(
                 //audit_log_lifecycle(*oi, ILMExpiry, tags, traceFn);
                 Ok(dobj)
             }
+            Err(err) if is_lifecycle_expiry_already_gone(&err) => {
+                debug!(
+                    event = EVENT_LIFECYCLE_DELETE_COMPLETED,
+                    component = LOG_COMPONENT_ECSTORE,
+                    subsystem = LOG_SUBSYSTEM_LIFECYCLE,
+                    bucket = %oi.bucket,
+                    object = %oi.name,
+                    operation = "delete_restored_object",
+                    reason = "already_gone",
+                    "Lifecycle delete target was already absent"
+                );
+                crate::object_api::notify_object_mutation(&oi.bucket, &oi.name).await;
+                Ok(oi.clone())
+            }
             Err(err) => Err(std::io::Error::other(err)),
         };
     }
@@ -4930,6 +4954,8 @@ async fn expire_transitioned_object_with_lock_lost_signal(
     // pointing to a remote tier version that has already been removed. If this
     // only creates a delete marker, remote cleanup must be driven by persisted
     // free-version recovery rather than the visible delete result.
+    // Keep remote transitioned not-found errors visible: local metadata
+    // absence alone does not prove that the tier object was cleaned up.
     let dobj = match api.delete_object(&oi.bucket, &oi.name, opts).await {
         Ok(obj) => obj,
         Err(e) => {
@@ -5802,6 +5828,20 @@ async fn apply_expiry_on_non_transitioned_objects_with_lock_lost_signal(
         .await
     {
         Ok(dobj) => dobj,
+        Err(err) if is_lifecycle_expiry_already_gone(&err) => {
+            debug!(
+                event = EVENT_LIFECYCLE_DELETE_COMPLETED,
+                component = LOG_COMPONENT_ECSTORE,
+                subsystem = LOG_SUBSYSTEM_LIFECYCLE,
+                bucket = %oi.bucket,
+                object = %oi.name,
+                operation = "delete_object",
+                reason = "already_gone",
+                "Lifecycle delete target was already absent"
+            );
+            crate::object_api::notify_object_mutation(&oi.bucket, &oi.name).await;
+            return true;
+        }
         Err(e) => {
             error!(
                 event = EVENT_LIFECYCLE_DELETE_FAILED,
@@ -6041,18 +6081,19 @@ mod tests {
         cleanup_empty_multipart_sha_dirs_on_local_disks, cleanup_stale_multipart_uploads_once_at,
         enqueue_recovered_free_version_with_state, enqueue_transition_for_existing_objects_scoped,
         enqueue_transition_with_lifecycle, enqueue_transition_with_lifecycle_report, eval_action_from_lifecycle,
-        get_lock_acquire_timeout, jitter_tier_free_version_recovery_delay, lifecycle_action_blocked_by_replication,
-        lifecycle_delete_all_versions_replication_scan, lifecycle_deleted_object, lifecycle_replication_blocks_action,
-        lifecycle_rule_has_date_expiration, lifecycle_version_delete_target, manual_transition_duration_elapsed,
-        manual_transition_has_more_after_limit, manual_transition_recovery_progress_sink, manual_transition_version_marker,
-        manual_transition_worker_failure_reason, mark_delete_opts_skip_decommissioned_on_remote_success,
-        merge_stale_multipart_candidate, persist_manual_transition_job_progress_if_owned,
-        persist_manual_transition_page_checkpoint, recover_manual_transition_job, recover_manual_transition_jobs,
-        resolve_tier_free_version_recovery_enabled, resolve_transition_queue_capacity, resolve_transition_queue_send_timeout,
-        resolve_transition_worker_count, resolve_transition_workers_absolute_max, run_tier_free_version_recovery_coordinator,
-        run_tier_free_version_recovery_loop, select_restore_s3_location, set_lifecycle_observability_observer,
-        set_recovered_free_version_enqueue_observer, should_defer_date_expiry_for_recent_config_update,
-        transitioned_cleanup_tuple, transitioned_object_delete_opts, wait_for_tier_free_version_recovery,
+        get_lock_acquire_timeout, is_lifecycle_expiry_already_gone, jitter_tier_free_version_recovery_delay,
+        lifecycle_action_blocked_by_replication, lifecycle_delete_all_versions_replication_scan, lifecycle_deleted_object,
+        lifecycle_replication_blocks_action, lifecycle_rule_has_date_expiration, lifecycle_version_delete_target,
+        manual_transition_duration_elapsed, manual_transition_has_more_after_limit, manual_transition_recovery_progress_sink,
+        manual_transition_version_marker, manual_transition_worker_failure_reason,
+        mark_delete_opts_skip_decommissioned_on_remote_success, merge_stale_multipart_candidate,
+        persist_manual_transition_job_progress_if_owned, persist_manual_transition_page_checkpoint,
+        recover_manual_transition_job, recover_manual_transition_jobs, resolve_tier_free_version_recovery_enabled,
+        resolve_transition_queue_capacity, resolve_transition_queue_send_timeout, resolve_transition_worker_count,
+        resolve_transition_workers_absolute_max, run_tier_free_version_recovery_coordinator, run_tier_free_version_recovery_loop,
+        select_restore_s3_location, set_lifecycle_observability_observer, set_recovered_free_version_enqueue_observer,
+        should_defer_date_expiry_for_recent_config_update, transitioned_cleanup_tuple, transitioned_object_delete_opts,
+        wait_for_tier_free_version_recovery,
     };
     #[cfg(feature = "test-util")]
     use super::{delete_free_version_remote_object_then, encode_dir_object, get_transitioned_object_reader_with_tier_manager};
@@ -10511,6 +10552,26 @@ mod tests {
         );
     }
 
+    #[test]
+    fn lifecycle_expiry_already_gone_only_accepts_object_or_version_absence() {
+        assert!(is_lifecycle_expiry_already_gone(&Error::FileNotFound));
+        assert!(is_lifecycle_expiry_already_gone(&Error::ObjectNotFound(
+            "bucket".to_string(),
+            "object".to_string(),
+        )));
+        assert!(is_lifecycle_expiry_already_gone(&Error::FileVersionNotFound));
+        assert!(is_lifecycle_expiry_already_gone(&Error::VersionNotFound(
+            "bucket".to_string(),
+            "object".to_string(),
+            "version".to_string(),
+        )));
+
+        assert!(!is_lifecycle_expiry_already_gone(&Error::VolumeNotFound));
+        assert!(!is_lifecycle_expiry_already_gone(&Error::DiskNotFound));
+        assert!(!is_lifecycle_expiry_already_gone(&Error::ErasureReadQuorum));
+        assert!(!is_lifecycle_expiry_already_gone(&Error::MethodNotAllowed));
+    }
+
     #[tokio::test]
     async fn manual_transition_counts_already_transitioned_object() {
         let lc = latest_transition_lifecycle();
@@ -13385,7 +13446,9 @@ mod tests {
                 .bucket_incarnation_id_from_disk(&bucket)
                 .await
                 .expect("bucket incarnation");
-            for (attempt_incarnation, expected) in [(Uuid::new_v4(), false), (incarnation, true), (incarnation, false)] {
+            // Replaying the same scanner snapshot after the delete is an
+            // already-gone success and must not be reported as a failure.
+            for (attempt_incarnation, expected) in [(Uuid::new_v4(), false), (incarnation, true), (incarnation, true)] {
                 assert_eq!(
                     super::apply_expiry_on_non_transitioned_objects(
                         ecstore.clone(),
@@ -14214,7 +14277,11 @@ mod tests {
     #[tokio::test]
     #[serial]
     async fn tier_free_version_recovery_production_entrypoint_enqueues_seeded_item() {
-        let (disk_paths, ecstore) = setup_test_env().await;
+        let (disk_paths, ecstore) =
+            temp_env::async_with_vars([(super::ENV_TIER_FREE_VERSION_RECOVERY_ENABLED, Some("false"))], async {
+                setup_test_env().await
+            })
+            .await;
         let runtime_state = install_unconsumed_runtime_expiry_worker(&ecstore, 1).await;
         let recovery_rx = {
             let state = runtime_state.read().await;

crates/ecstore/src/bucket/metadata.rs

@@ -439,6 +439,8 @@ pub struct BucketMetadata {
     pub bucket_incarnation_id: Uuid,
     pub(crate) bucket_incarnation_sidecar: bool,
     pub(crate) bucket_creation_committed: bool,
+    // A legacy default is not proof of an interrupted creation.
+    pub(crate) bucket_creation_commit_record_present: bool,
     pub policy_config_json: Vec<u8>,
     pub notification_config_xml: Vec<u8>,
     pub lifecycle_config_xml: Vec<u8>,
@@ -514,6 +516,7 @@ impl Default for BucketMetadata {
             bucket_incarnation_id: Uuid::nil(),
             bucket_incarnation_sidecar: false,
             bucket_creation_committed: false,
+            bucket_creation_commit_record_present: false,
             policy_config_json: Default::default(),
             notification_config_xml: Default::default(),
             lifecycle_config_xml: Default::default(),
@@ -752,7 +755,10 @@ impl BucketMetadata {
                 "TableBucketConfigUpdatedAt" => self.table_bucket_config_updated_at = read_msgp_time_value(rd)?,
                 "DurabilityConfigUpdatedAt" => self.durability_config_updated_at = read_msgp_time_value(rd)?,
                 "OnDemandMigrationConfigUpdatedAt" => self.on_demand_migration_config_updated_at = read_msgp_time_value(rd)?,
-                "BucketCreationCommitted" => self.bucket_creation_committed = read_msgp_bool(rd)?,
+                "BucketCreationCommitted" => {
+                    self.bucket_creation_committed = read_msgp_bool(rd)?;
+                    self.bucket_creation_commit_record_present = true;
+                }
                 other => {
                     tracing::debug!(field = %other, "BucketMetadata decode_from: skipping unknown field");
                     skip_msgp_value(rd)?;
@@ -765,8 +771,10 @@ impl BucketMetadata {
 
     /// Encode to msgp bytes. Field order follows MinIO BucketMetadata for compatibility.
     pub fn encode_to<W: Write>(&self, wr: &mut W) -> Result<()> {
-        // Map size: MinIO fields (25) + RustFS extensions (22)
-        let map_len: u32 = 47;
+        // Preserve the absence of a legacy commit record across config rewrites.
+        let commit_record_present = self.bucket_creation_commit_record_present || self.bucket_creation_committed;
+        // Map size: MinIO fields (25) + RustFS extensions (21) + optional commit record.
+        let map_len: u32 = 46 + u32::from(commit_record_present);
         rmp::encode::write_map_len(wr, map_len)?;
 
         // MinIO field order (same as Go struct)
@@ -847,8 +855,10 @@ impl BucketMetadata {
         write_msgp_time(wr, self.durability_config_updated_at)?;
         rmp::encode::write_str(wr, "OnDemandMigrationConfigUpdatedAt")?;
         write_msgp_time(wr, self.on_demand_migration_config_updated_at)?;
-        rmp::encode::write_str(wr, "BucketCreationCommitted")?;
-        rmp::encode::write_bool(wr, self.bucket_creation_committed)?;
+        if commit_record_present {
+            rmp::encode::write_str(wr, "BucketCreationCommitted")?;
+            rmp::encode::write_bool(wr, self.bucket_creation_committed)?;
+        }
 
         Ok(())
     }
@@ -1691,6 +1701,13 @@ mod test {
         let mut bm = BucketMetadata::unmarshal(&body[4..]).expect("unmarshal recovered blob");
         assert_eq!(bm.name, "interop");
         assert!(!bm.bucket_creation_committed, "legacy metadata must default to uncommitted");
+        assert!(!bm.bucket_creation_commit_record_present, "legacy metadata has no creation-commit record");
+        let rewritten = BucketMetadata::unmarshal(&bm.marshal_msg().expect("rewrite legacy metadata"))
+            .expect("read rewritten legacy metadata");
+        assert!(
+            !rewritten.bucket_creation_commit_record_present,
+            "rewriting legacy metadata must not fabricate a creation intent"
+        );
         bm.parse_all_configs().expect("parse recovered configs");
         assert!(bm.lifecycle_config.is_some());
     }
@@ -1708,6 +1725,7 @@ mod test {
 
         assert_eq!(bm.name, new.name);
         assert!(new.bucket_creation_committed);
+        assert!(new.bucket_creation_commit_record_present);
         assert!(!bm.bucket_incarnation_id.is_nil());
         assert_eq!(bm.bucket_incarnation_id, new.bucket_incarnation_id);
     }

crates/ecstore/src/bucket/metadata_sys.rs

@@ -178,6 +178,13 @@ pub(crate) fn object_lock_config_state_from_authoritative_metadata(bm: &BucketMe
     Ok(ObjectLockConfigState::ConfirmedAbsent)
 }
 
+#[derive(Debug, Clone, PartialEq)]
+pub(crate) struct CachedQuotaMetadataSnapshot {
+    pub(crate) quota: Option<BucketQuota>,
+    pub(crate) bucket_incarnation: Uuid,
+    pub(crate) quota_revision: OffsetDateTime,
+}
+
 /// Convert the persisted serving-layer configuration into the storage-level
 /// [`DefaultRetention`](crate::bucket::object_lock::types::DefaultRetention)
 /// the WORM evaluation code consumes (rustfs/backlog#1842). A rule without a
@@ -314,6 +321,33 @@ pub async fn remove_bucket_metadata(bucket: &str) -> Result<bool> {
     Ok(lock.remove(bucket).await)
 }
 
+/// Remove a peer's cached metadata only when the bucket is absent under the
+/// namespace read fence. A delayed delete notification must not withdraw the
+/// metadata of a same-name bucket that has already been recreated.
+pub async fn remove_bucket_metadata_if_absent(api: Arc<ECStore>, bucket: &str) -> Result<bool> {
+    if is_meta_bucketname(bucket) {
+        return Ok(false);
+    }
+    // Lock order: bucket namespace read -> metadata system read (released)
+    // -> metadata publish, matching peer reloads.
+    let namespace_lock = api.new_ns_lock(bucket, bucket).await?;
+    let namespace_guard = namespace_lock
+        .get_read_lock(crate::set_disk::get_lock_acquire_timeout())
+        .await?;
+    let sys = bucket_metadata_sys_of(&api.ctx)?;
+    let sys = sys.read().await.clone();
+    if sys
+        .bucket_exists(bucket, &namespace_guard, "peer bucket metadata removal existence check")
+        .await?
+    {
+        return Ok(false);
+    }
+    await_bucket_namespace_operation(Some(&namespace_guard), bucket, "peer bucket metadata removal", async {
+        Ok(sys.remove(bucket).await)
+    })
+    .await
+}
+
 fn start_refresh_buckets_metadata_loop(sys: Arc<RwLock<BucketMetadataSys>>) {
     let Some(cancel_token) = runtime_sources::background_services_cancel_token() else {
         warn!("bucket metadata refresh loop skipped because background cancellation token is not initialized");
@@ -1423,6 +1457,9 @@ fn quota_config_and_incarnation_from_authority(
         BucketMetadataAuthority::Authoritative(metadata)
             if metadata.bucket_incarnation_sidecar && !metadata.bucket_incarnation_id.is_nil() =>
         {
+            if !metadata.quota_config_json.is_empty() && metadata.quota_config.is_none() {
+                return Err(Error::other("persisted bucket quota configuration is invalid"));
+            }
             Ok((
                 metadata.quota_config.clone(),
                 metadata.bucket_incarnation_id,
@@ -1437,6 +1474,15 @@ fn quota_config_and_incarnation_from_authority(
     }
 }
 
+pub(crate) async fn get_cached_quota_config_and_incarnation_in(
+    ctx: &crate::runtime::instance::InstanceContext,
+    bucket: &str,
+) -> Result<Option<CachedQuotaMetadataSnapshot>> {
+    let bucket_meta_sys_lock = bucket_metadata_sys_of(ctx)?;
+    let bucket_meta_sys = bucket_meta_sys_lock.read().await.clone();
+    bucket_meta_sys.cached_quota_config_and_incarnation(bucket).await
+}
+
 pub async fn get_replication_config(bucket: &str) -> Result<(ReplicationConfiguration, OffsetDateTime)> {
     let bucket_meta_sys_lock = get_bucket_metadata_sys()?;
     let bucket_meta_sys = bucket_meta_sys_lock.read().await;
@@ -1573,9 +1619,27 @@ struct MetadataPublishGuard {
     _guard: tokio::sync::OwnedMutexGuard<MetadataPublishLockState>,
 }
 
+#[derive(Debug, Clone)]
+pub(crate) struct PendingBucketCreationProof {
+    pub(crate) generation: Uuid,
+    pub(crate) intent: Vec<u8>,
+    pub(crate) versioning_requested: bool,
+    pub(crate) witnesses: Vec<crate::disk::os::NativeBucketCreationWitness>,
+}
+
+impl PendingBucketCreationProof {
+    pub(crate) fn matches_metadata(&self, metadata: &BucketMetadata) -> Result<bool> {
+        Ok(!metadata.bucket_creation_committed
+            && metadata.bucket_incarnation_sidecar
+            && metadata.bucket_incarnation_id == self.generation
+            && metadata.marshal_msg()? == self.intent)
+    }
+}
+
 #[derive(Debug, Clone)]
 pub struct BucketMetadataSys {
     metadata_map: Arc<RwLock<HashMap<String, Arc<BucketMetadata>>>>,
+    pending_bucket_creations: Arc<RwLock<HashMap<String, PendingBucketCreationProof>>>,
     /// Serializes metadata-map commits and their derived cache updates for one
     /// bucket. Namespace locks, when present, are acquired before this lock.
     metadata_publish_locks: Arc<MetadataPublishLockRegistry>,
@@ -1619,6 +1683,7 @@ impl BucketMetadataSys {
     pub fn new(api: Arc<ECStore>) -> Self {
         Self {
             metadata_map: Arc::new(RwLock::new(HashMap::new())),
+            pending_bucket_creations: Arc::new(RwLock::new(HashMap::new())),
             metadata_publish_locks: Arc::new(MetadataPublishLockRegistry {
                 locks: StdMutex::new(HashMap::new()),
             }),
@@ -1649,6 +1714,81 @@ impl BucketMetadataSys {
             .expect("bucket metadata object store should still be live")
     }
 
+    pub(crate) async fn pending_bucket_creation(&self, bucket: &str) -> Option<PendingBucketCreationProof> {
+        self.pending_bucket_creations.read().await.get(bucket).cloned()
+    }
+
+    pub(crate) async fn grant_pending_bucket_creation(&self, bucket: &str, proof: PendingBucketCreationProof) {
+        self.pending_bucket_creations.write().await.insert(bucket.to_owned(), proof);
+    }
+
+    pub(crate) async fn revoke_pending_bucket_creation(&self, bucket: &str) {
+        self.pending_bucket_creations.write().await.remove(bucket);
+    }
+
+    pub(crate) async fn consume_pending_bucket_creation(
+        &self,
+        bucket: &str,
+        metadata: &BucketMetadata,
+        witnesses: &[crate::disk::os::NativeBucketCreationWitness],
+    ) -> Result<bool> {
+        let mut pending = self.pending_bucket_creations.write().await;
+        let Some(proof) = pending.get(bucket) else {
+            return Ok(false);
+        };
+        if !proof.matches_metadata(metadata)?
+            || proof.witnesses.len() != witnesses.len()
+            || !proof
+                .witnesses
+                .iter()
+                .zip(witnesses)
+                .all(|(old, current)| old.matches(current))
+            || !crate::disk::os::consume_native_creation_witnesses(witnesses)
+        {
+            return Ok(false);
+        }
+        pending.remove(bucket);
+        Ok(true)
+    }
+
+    #[cfg(test)]
+    pub(crate) async fn refresh_bucket_for_test(&self, bucket: &str) -> Result<()> {
+        let expected = self.metadata_map.read().await.get(bucket).cloned();
+        let lock = self.object_store().new_ns_lock(bucket, bucket).await?;
+        let guard = lock.get_read_lock(crate::set_disk::get_lock_acquire_timeout()).await?;
+        self.load_bucket_under_namespace(bucket, MetadataLoadMode::Refresh, expected.as_ref(), &guard)
+            .await
+    }
+
+    #[cfg(test)]
+    pub(crate) async fn publish_persisted_metadata_for_test(&self, bucket: &str) -> Result<()> {
+        let expected = self.metadata_map.read().await.get(bucket).cloned();
+        let lock = self.object_store().new_ns_lock(bucket, bucket).await?;
+        let guard = lock.get_read_lock(crate::set_disk::get_lock_acquire_timeout()).await?;
+        let (metadata, persisted) = await_bucket_namespace_operation(
+            Some(&guard),
+            bucket,
+            "test cached metadata load",
+            load_bucket_metadata_parse_with_presence(self.object_store(), bucket, true),
+        )
+        .await?;
+        self.publish_if_unchanged(bucket, expected.as_ref(), metadata, persisted, &guard)
+            .await
+    }
+
+    #[cfg(test)]
+    pub(crate) async fn migrate_bucket_creation_commit_for_test(&self, bucket: &str) -> Result<()> {
+        self.migrate_bucket_creation_commit(bucket).await.map(|_| ())
+    }
+
+    #[cfg(test)]
+    pub(crate) async fn legacy_migration_reports_missing_bucket_for_test(&self, bucket: &str) -> Result<bool> {
+        Ok(matches!(
+            self.migrate_legacy_metadata(bucket).await?,
+            BucketMetadataAuthority::MissingBucket
+        ))
+    }
+
     fn object_store_if_live(&self) -> Option<Arc<ECStore>> {
         self.api.upgrade()
     }
@@ -1839,6 +1979,7 @@ impl BucketMetadataSys {
         expected: Option<&Arc<BucketMetadata>>,
         namespace_guard: &rustfs_lock::NamespaceLockGuard,
     ) -> Result<()> {
+        self.revoke_pending_bucket_creation(bucket).await;
         if !await_bucket_namespace_operation(
             Some(namespace_guard),
             bucket,
@@ -1922,6 +2063,7 @@ impl BucketMetadataSys {
         persisted: bool,
         namespace_guard: &rustfs_lock::NamespaceLockGuard,
     ) -> Result<()> {
+        self.revoke_pending_bucket_creation(bucket).await;
         if !persisted {
             let _publish_guard = self
                 .lock_metadata_publish(bucket, namespace_guard, "refreshed bucket metadata absence publish")
@@ -1983,6 +2125,7 @@ impl BucketMetadataSys {
     }
 
     pub async fn set(&self, bucket: String, bm: Arc<BucketMetadata>) {
+        self.revoke_pending_bucket_creation(&bucket).await;
         if !is_meta_bucketname(&bucket) {
             let publish_lock = self.metadata_publish_lock(&bucket);
             let _publish_guard = publish_lock.lock().await;
@@ -2004,6 +2147,7 @@ impl BucketMetadataSys {
         if is_meta_bucketname(bucket) {
             return false;
         }
+        self.revoke_pending_bucket_creation(bucket).await;
         let publish_lock = self.metadata_publish_lock(bucket);
         let _publish_guard = publish_lock.lock().await;
         let mut map = self.metadata_map.write().await;
@@ -2021,6 +2165,7 @@ impl BucketMetadataSys {
     }
 
     async fn _reset(&mut self) {
+        self.pending_bucket_creations.write().await.clear();
         let mut map = self.metadata_map.write().await;
         map.clear();
         drop(map);
@@ -2159,6 +2304,7 @@ impl BucketMetadataSys {
     /// server's metadata never leaks into the ambient (first) instance.
     pub(crate) async fn persist_and_set(&self, bm: BucketMetadata) -> Result<()> {
         let mut bm = bm;
+        self.revoke_pending_bucket_creation(&bm.name).await;
         bm.save_with_store(self.object_store()).await?;
 
         self.set(bm.name.clone(), Arc::new(bm)).await;
@@ -2167,6 +2313,7 @@ impl BucketMetadataSys {
     }
 
     async fn persist_new_and_set(&self, mut bm: BucketMetadata) -> Result<()> {
+        self.revoke_pending_bucket_creation(&bm.name).await;
         bm.bucket_creation_committed = true;
         bm.save_with_store_committed(self.object_store()).await?;
         save_bucket_incarnation(self.object_store(), &bm.name, bm.bucket_incarnation_id).await?;
@@ -2176,7 +2323,9 @@ impl BucketMetadataSys {
     }
 
     async fn persist_new_bucket_metadata_intent(&self, mut bm: BucketMetadata) -> Result<()> {
+        self.revoke_pending_bucket_creation(&bm.name).await;
         bm.bucket_creation_committed = false;
+        bm.bucket_creation_commit_record_present = true;
         bm.save_with_store(self.object_store()).await?;
         save_bucket_incarnation(self.object_store(), &bm.name, bm.bucket_incarnation_id).await?;
         bm.bucket_incarnation_sidecar = true;
@@ -2186,6 +2335,7 @@ impl BucketMetadataSys {
     }
 
     async fn commit_bucket_metadata(&self, mut bm: BucketMetadata) -> Result<()> {
+        self.revoke_pending_bucket_creation(&bm.name).await;
         bm.parse_all_configs()?;
         bm.bucket_incarnation_sidecar = true;
         // The caller holds the full bucket creation fence (lifecycle, metadata
@@ -2234,6 +2384,7 @@ impl BucketMetadataSys {
         bucket: &str,
         namespace_guard: &rustfs_lock::NamespaceLockGuard,
     ) -> Result<()> {
+        self.revoke_pending_bucket_creation(bucket).await;
         let expected = self.metadata_map.read().await.get(bucket).cloned();
         if !self
             .bucket_exists(bucket, namespace_guard, "peer bucket metadata existence check")
@@ -2290,6 +2441,7 @@ impl BucketMetadataSys {
             #[cfg(test)]
             self.lazy_disk_loads.fetch_add(1, std::sync::atomic::Ordering::Relaxed);
 
+            self.revoke_pending_bucket_creation(bucket).await;
             let lock = self.object_store().new_ns_lock(bucket, bucket).await?;
             let guard = lock.get_read_lock(crate::set_disk::get_lock_acquire_timeout()).await?;
             #[cfg(test)]
@@ -2598,6 +2750,7 @@ impl BucketMetadataSys {
     /// revalidated at write quorum, so a stale snapshot can neither revert an
     /// acknowledged configuration update nor outlive a delete/recreate.
     async fn migrate_bucket_creation_commit(&self, bucket: &str) -> Result<BucketMetadataAuthority> {
+        self.revoke_pending_bucket_creation(bucket).await;
         let transaction_lock = self
             .object_store()
             .new_ns_lock(RUSTFS_META_BUCKET, &bucket_metadata_transaction_lock_key(bucket))
@@ -2675,6 +2828,7 @@ impl BucketMetadataSys {
         &self,
         bucket: &str,
     ) -> Result<BucketMetadataAuthority> {
+        self.revoke_pending_bucket_creation(bucket).await;
         #[cfg(test)]
         if self.object_lock_disk_read_errors.write().await.remove(bucket) {
             return Err(Error::other(format!("injected Object Lock metadata disk read failure: {bucket}")));
@@ -2952,6 +3106,27 @@ impl BucketMetadataSys {
         }
     }
 
+    async fn cached_quota_config_and_incarnation(&self, bucket: &str) -> Result<Option<CachedQuotaMetadataSnapshot>> {
+        let metadata = {
+            let metadata_map = self.metadata_map.read().await;
+            metadata_map.get(bucket).cloned()
+        };
+        let Some(metadata) = metadata else {
+            return Ok(None);
+        };
+        if !metadata.bucket_incarnation_sidecar || metadata.bucket_incarnation_id.is_nil() {
+            return Ok(None);
+        }
+        if !metadata.quota_config_json.is_empty() && metadata.quota_config.is_none() {
+            return Err(Error::other("persisted bucket quota configuration is invalid"));
+        }
+        Ok(Some(CachedQuotaMetadataSnapshot {
+            quota: metadata.quota_config.clone(),
+            bucket_incarnation: metadata.bucket_incarnation_id,
+            quota_revision: metadata.quota_config_updated_at,
+        }))
+    }
+
     pub async fn get_replication_config(&self, bucket: &str) -> Result<(ReplicationConfiguration, OffsetDateTime)> {
         let (bm, _) = self.get_config(bucket).await?;
 
@@ -3046,6 +3221,7 @@ mod tests {
     };
     use crate::bucket::target::{BucketTarget, BucketTargetType, Credentials};
     use crate::config::com::read_config;
+    use crate::disk::{DiskAPI as _, RUSTFS_META_TMP_BUCKET};
     use crate::storage_api_contracts::bucket::{BucketOperations as _, DeleteBucketOptions, MakeBucketOptions};
     use byteorder::{ByteOrder as _, LittleEndian};
     use serial_test::serial;
@@ -3100,6 +3276,180 @@ mod tests {
         ),
     ];
 
+    #[tokio::test]
+    #[serial(storage_class_env)]
+    async fn persistence_entry_failures_revoke_pending_creation_before_cache_publish() {
+        let (dirs, store) = isolated_store_over_temp_disks().await;
+        init_bucket_metadata_sys(store.clone(), Vec::new()).await;
+        let sys = bucket_metadata_sys_of(&store.ctx).expect("instance metadata system");
+        let sys = sys.read().await.clone();
+        let options = MakeBucketOptions {
+            lock_enabled: true,
+            ..Default::default()
+        };
+        for entry in ["save", "persist", "persist-new", "commit"] {
+            let bucket = format!("revoke-{entry}-{}", Uuid::new_v4());
+            store.fail_next_bucket_creation_after_intent_for_test(&bucket);
+            store
+                .make_bucket(&bucket, &options)
+                .await
+                .expect_err("stop the real producer before physical publication");
+            sys.pending_bucket_creation(&bucket)
+                .await
+                .expect("the producer must leave a real live creation permission");
+            let (intent, persisted) = get_config_from_disk_with_presence_in(&store.ctx, &bucket)
+                .await
+                .expect("read the original producer intent");
+            assert!(persisted, "{entry}: the complete original intent must be persisted");
+            let original_intent = intent.marshal_msg().expect("encode the original intent");
+
+            // Keep the creation fence order used by the real producer.
+            let lifecycle_guard = store
+                .acquire_bucket_lifecycle_write_lock(&bucket)
+                .await
+                .expect("hold the bucket lifecycle fence");
+            let transaction_guard = acquire_bucket_metadata_transaction_lock_in(&store.ctx, &bucket)
+                .await
+                .expect("hold the metadata transaction fence");
+            let namespace_lock = store
+                .new_ns_lock(&bucket, &bucket)
+                .await
+                .expect("create the exact bucket namespace lock");
+            let namespace_guard = namespace_lock
+                .get_write_lock(crate::set_disk::get_lock_acquire_timeout())
+                .await
+                .expect("hold the exact bucket namespace fence");
+
+            // A file at each native staging root causes actual IO to fail before
+            // metadata can reach the cache-publishing set call. Preserve the
+            // original staging directories so no fixture data is discarded.
+            for dir in &dirs {
+                let staging = dir.path().join(RUSTFS_META_TMP_BUCKET);
+                let saved = dir.path().join(".rustfs.sys/revocation-saved-tmp");
+                std::fs::rename(&staging, &saved).expect("preserve the native staging directory");
+                std::fs::write(&staging, b"block metadata staging with a non-directory")
+                    .expect("install an actual non-directory IO obstruction");
+            }
+            let result = match entry {
+                "save" => sys.save(intent).await,
+                "persist" => sys.persist_and_set(intent).await,
+                "persist-new" => sys.persist_new_and_set(intent).await,
+                "commit" => sys.commit_bucket_metadata(intent).await,
+                _ => unreachable!(),
+            };
+            for dir in &dirs {
+                let staging = dir.path().join(RUSTFS_META_TMP_BUCKET);
+                let saved = dir.path().join(".rustfs.sys/revocation-saved-tmp");
+                std::fs::remove_file(&staging).expect("remove only the fixture's staging obstruction");
+                std::fs::rename(&saved, &staging).expect("restore the complete original staging directory");
+            }
+            drop((namespace_guard, transaction_guard, lifecycle_guard));
+            result.expect_err("native persistence must fail before its downstream cache set");
+            assert!(
+                sys.pending_bucket_creation(&bucket).await.is_none(),
+                "{entry}: the entry revoker must retire live permission even when persistence fails"
+            );
+            let (after_failure, present) = get_config_from_disk_with_presence_in(&store.ctx, &bucket)
+                .await
+                .expect("read the intent after the failed persistence entry");
+            assert!(present, "{entry}: a failed persistence entry must retain the original metadata");
+            assert_eq!(
+                after_failure
+                    .marshal_msg()
+                    .expect("encode the original intent after persistence failure"),
+                original_intent,
+                "{entry}: native IO failure must not change the original generation or complete intent"
+            );
+            let error = store
+                .make_bucket(&bucket, &options)
+                .await
+                .expect_err("a failed ordinary persistence entry cannot preserve creation permission");
+            assert!(error.to_string().contains("administrator recovery"), "{entry}: {error}");
+            let (after_retry, _) = get_config_from_disk_with_presence_in(&store.ctx, &bucket)
+                .await
+                .expect("read the original intent after the rejected retry");
+            assert_eq!(
+                after_retry.marshal_msg().expect("encode the intent after the rejected retry"),
+                original_intent,
+                "{entry}: the rejected retry must preserve the complete original intent"
+            );
+            for disk in store.pools[0].disk_set[0].disks.read().await.iter() {
+                assert!(
+                    matches!(
+                        disk.as_ref().expect("native disk").stat_volume(&bucket).await,
+                        Err(crate::disk::error::DiskError::VolumeNotFound)
+                    ),
+                    "{entry}: failed persistence and the rejected retry must not publish a physical volume"
+                );
+            }
+        }
+    }
+
+    #[tokio::test]
+    #[serial(storage_class_env)]
+    async fn metadata_reset_revokes_real_pending_creation_without_changing_persisted_intent() {
+        let (_dirs, store) = isolated_store_over_temp_disks().await;
+        init_bucket_metadata_sys(store.clone(), Vec::new()).await;
+        let bucket = format!("revoke-reset-{}", Uuid::new_v4());
+        let options = MakeBucketOptions {
+            lock_enabled: true,
+            ..Default::default()
+        };
+        store.fail_next_bucket_creation_after_intent_for_test(&bucket);
+        store
+            .make_bucket(&bucket, &options)
+            .await
+            .expect_err("stop the real producer before physical publication");
+        let sys = bucket_metadata_sys_of(&store.ctx).expect("instance metadata system");
+        let mut sys = sys.read().await.clone();
+        sys.pending_bucket_creation(&bucket)
+            .await
+            .expect("the real producer must leave live creation permission before reset");
+        let (intent, persisted) = get_config_from_disk_with_presence_in(&store.ctx, &bucket)
+            .await
+            .expect("read the original intent before cache reset");
+        assert!(persisted, "cache reset starts from a complete real persisted intent");
+        let original_intent = intent.marshal_msg().expect("encode the original reset intent");
+        sys._reset().await;
+        assert!(
+            sys.pending_bucket_creation(&bucket).await.is_none(),
+            "reset must revoke the producer-issued permission shared with the instance metadata system"
+        );
+        let (after_reset, present) = get_config_from_disk_with_presence_in(&store.ctx, &bucket)
+            .await
+            .expect("read the original intent after cache reset");
+        assert!(present, "cache reset must retain the persisted intent");
+        assert_eq!(
+            after_reset.marshal_msg().expect("encode the intent after cache reset"),
+            original_intent,
+            "cache reset must leave the original generation and complete persisted intent unchanged"
+        );
+        let error = store
+            .make_bucket(&bucket, &options)
+            .await
+            .expect_err("reset cannot regrant the producer's old creation permission");
+        assert!(error.to_string().contains("administrator recovery"), "{error}");
+        let (after_retry, _) = get_config_from_disk_with_presence_in(&store.ctx, &bucket)
+            .await
+            .expect("read the intent after the reset-induced retry rejection");
+        assert_eq!(
+            after_retry
+                .marshal_msg()
+                .expect("encode the intent after the rejected reset retry"),
+            original_intent,
+            "the reset-induced rejection must preserve the original complete intent"
+        );
+        for disk in store.pools[0].disk_set[0].disks.read().await.iter() {
+            assert!(
+                matches!(
+                    disk.as_ref().expect("native disk").stat_volume(&bucket).await,
+                    Err(crate::disk::error::DiskError::VolumeNotFound)
+                ),
+                "cache reset and the rejected retry must not publish a physical volume"
+            );
+        }
+    }
+
     #[tokio::test]
     async fn g_d3_003_new_writer_replication_loads_without_fail_closed_state() {
         let (dirs, store) = isolated_store_over_temp_disks().await;
@@ -4856,6 +5206,154 @@ mod tests {
         );
     }
 
+    #[tokio::test]
+    #[serial]
+    async fn quota_admission_rejects_stale_cached_no_quota() {
+        let (_dirs, store) = isolated_store_over_temp_disks().await;
+        init_bucket_metadata_sys(store.clone(), Vec::new()).await;
+        let sys = bucket_metadata_sys_of(&store.ctx).expect("scoped metadata system");
+        let bucket = "stale-no-quota-admission";
+        store
+            .make_bucket(bucket, &MakeBucketOptions::default())
+            .await
+            .expect("create bucket");
+        let (stale, _) = sys.read().await.get_config(bucket).await.expect("cache initial metadata");
+        store
+            .update_bucket_metadata_config(bucket, "quota.json", br#"{"quota":1}"#.to_vec())
+            .await
+            .expect("publish quota to authoritative storage");
+        // Model a peer that has not received the newer configuration snapshot.
+        sys.read().await.set(bucket.to_owned(), stale).await;
+        assert!(
+            get_cached_quota_config_and_incarnation_in(&store.ctx, bucket)
+                .await
+                .expect("read stale cache")
+                .expect("cached bucket identity")
+                .quota
+                .is_none()
+        );
+
+        temp_env::async_with_vars([("RUSTFS_QUOTA_BEGIN_SAFE_NO_QUOTA_CACHE_FAST_PATH", Some("true"))], async {
+            let error = crate::bucket::quota::reservation::begin(&store.ctx, bucket, "object", &ObjectOptions::default(), 0, 0)
+                .await
+                .err()
+                .expect("persisted quota must reject admission without its snapshot");
+            assert!(matches!(error, Error::PartMissingOrCorrupt), "{error:?}");
+        })
+        .await;
+    }
+
+    #[tokio::test]
+    #[serial]
+    async fn peer_reload_clears_negative_cache_after_remote_bucket_creation() {
+        let (dirs, node_a, node_b) = two_nodes_over_one_store().await;
+        let bucket = "cross-node-negative-cache-refresh";
+        let node_b_sys = node_b.read().await.clone();
+
+        assert_eq!(
+            node_b_sys.get_bucket_incarnation_id(bucket).await.unwrap_err(),
+            Error::BucketNotFound(bucket.to_string()),
+            "the probe must populate node B's missing-bucket cache"
+        );
+
+        for dir in &dirs {
+            std::fs::create_dir_all(dir.path().join(bucket)).expect("physical bucket should exist before creation metadata");
+        }
+        let created = BucketMetadata::new(bucket);
+        let created_incarnation = created.bucket_incarnation_id;
+        node_a
+            .read()
+            .await
+            .persist_new_and_set(created)
+            .await
+            .expect("node A should publish the created bucket metadata");
+
+        node_b_sys
+            .reload_from_store(bucket)
+            .await
+            .expect("peer reload should clear the stale negative cache");
+        assert_eq!(
+            node_b_sys.get_bucket_incarnation_id(bucket).await.unwrap(),
+            created_incarnation,
+            "node B should resolve the newly created bucket immediately"
+        );
+    }
+
+    #[tokio::test(flavor = "multi_thread", worker_threads = 4)]
+    #[serial]
+    async fn peer_delete_clears_metadata_for_absent_bucket() {
+        let (_dirs, store) = isolated_store_over_temp_disks().await;
+        init_bucket_metadata_sys(store.clone(), Vec::new()).await;
+        let sys = bucket_metadata_sys_of(&store.ctx).expect("metadata owner");
+        let bucket = "peer-delete-absent-bucket";
+
+        store
+            .make_bucket(bucket, &MakeBucketOptions::default())
+            .await
+            .expect("initial bucket creation");
+        let cached = sys.read().await.get(bucket).await.expect("initial cached metadata");
+        store
+            .delete_bucket(bucket, &DeleteBucketOptions::default())
+            .await
+            .expect("bucket deletion");
+
+        // Model a peer that still caches the metadata after the physical delete.
+        sys.read().await.set(bucket.to_string(), cached).await;
+        assert!(
+            remove_bucket_metadata_if_absent(store.clone(), bucket)
+                .await
+                .expect("conditional peer delete"),
+            "the peer must evict the deleted bucket's cached metadata"
+        );
+        let sys = sys.read().await;
+        assert!(matches!(sys.get(bucket).await, Err(Error::ConfigNotFound)));
+        assert_eq!(
+            sys.get_bucket_incarnation_id(bucket)
+                .await
+                .expect_err("deleted bucket lookup"),
+            Error::BucketNotFound(bucket.to_string()),
+        );
+    }
+
+    #[tokio::test(flavor = "multi_thread", worker_threads = 4)]
+    #[serial]
+    async fn delayed_peer_delete_preserves_recreated_bucket_metadata() {
+        let (_dirs, store) = isolated_store_over_temp_disks().await;
+        init_bucket_metadata_sys(store.clone(), Vec::new()).await;
+        let sys = bucket_metadata_sys_of(&store.ctx).expect("metadata owner");
+        let bucket = "delayed-peer-delete-recreated-bucket";
+
+        store
+            .make_bucket(bucket, &MakeBucketOptions::default())
+            .await
+            .expect("initial bucket creation");
+        let old_incarnation = store.bucket_incarnation_id_from_disk(bucket).await.expect("old incarnation");
+        store
+            .delete_bucket(bucket, &DeleteBucketOptions::default())
+            .await
+            .expect("bucket deletion");
+        store
+            .make_bucket(bucket, &MakeBucketOptions::default())
+            .await
+            .expect("bucket recreation");
+        let new_incarnation = store.bucket_incarnation_id_from_disk(bucket).await.expect("new incarnation");
+        assert_ne!(old_incarnation, new_incarnation);
+
+        let removed = remove_bucket_metadata_if_absent(store.clone(), bucket)
+            .await
+            .expect("conditional peer delete");
+        assert!(!removed, "a delayed delete must not remove metadata for a live recreated bucket");
+        assert_eq!(
+            sys.read()
+                .await
+                .get(bucket)
+                .await
+                .expect("recreated metadata should remain cached")
+                .bucket_incarnation_id,
+            new_incarnation,
+        );
+    }
+
     #[tokio::test(flavor = "multi_thread", worker_threads = 4)]
     #[serial]
     async fn stale_config_request_cannot_mutate_a_recreated_bucket() {

crates/ecstore/src/bucket/migration.rs

@@ -273,6 +273,30 @@ where
 
     let buckets: Vec<String> = buckets_list.into_iter().map(|b| b.name).collect();
 
+    try_migrate_bucket_metadata_for_buckets(store, &buckets).await
+}
+
+/// Migrate the already discovered startup candidates without repeating a
+/// majority-only namespace listing. Every migration write retains its quorum.
+pub async fn try_migrate_bucket_metadata_for_buckets<S>(store: Arc<S>, buckets: &[String]) -> Result<()>
+where
+    S: ObjectIO<
+            Error = Error,
+            RangeSpec = HTTPRangeSpec,
+            HeaderMap = HeaderMap,
+            ObjectOptions = ObjectOptions,
+            ObjectInfo = ObjectInfo,
+            GetObjectReader = GetObjectReader,
+            PutObjectReader = PutObjReader,
+        > + ObjectOperations<
+            Error = Error,
+            ObjectInfo = ObjectInfo,
+            ObjectOptions = ObjectOptions,
+            FileInfo = FileInfo,
+            ObjectToDelete = ObjectToDelete,
+            DeletedObject = DeletedObject,
+        >,
+{
     if buckets.is_empty() {
         debug!("No migrating bucket metadata found");
         return Ok(());

crates/ecstore/src/bucket/quota/reservation.rs

@@ -850,6 +850,11 @@ pub(crate) async fn begin(
     let snapshot_admission = opts.quota_admission;
     let data_movement = opts.data_movement;
     if crate::bucket::utils::is_meta_bucketname(bucket) {
+        let meta_bucket_fast_path_started = rustfs_io_metrics::put_stage_timer();
+        rustfs_io_metrics::record_put_object_stage_duration_from(
+            rustfs_io_metrics::PUT_STAGE_PUT_OBJECT_QUOTA_BEGIN_META_BUCKET_FAST_PATH,
+            meta_bucket_fast_path_started,
+        );
         return Ok(QuotaContext {
             store: None,
             bucket: bucket.to_string(),
@@ -906,9 +911,63 @@ pub(crate) async fn begin(
         });
     }
 
+    let metadata_lock_started = rustfs_io_metrics::put_stage_timer();
     let metadata_guard = metadata_sys::acquire_bucket_metadata_transaction_read_lock_for_options_in(ctx, bucket, opts).await?;
+    rustfs_io_metrics::record_put_object_stage_duration_from(
+        rustfs_io_metrics::PUT_STAGE_PUT_OBJECT_QUOTA_BEGIN_METADATA_LOCK,
+        metadata_lock_started,
+    );
+    let cache_lookup_started = rustfs_io_metrics::put_stage_timer();
+    let cached_quota_snapshot = if rustfs_io_metrics::put_stage_metrics_enabled() {
+        match metadata_sys::get_cached_quota_config_and_incarnation_in(ctx, bucket).await {
+            Ok(Some(snapshot)) => {
+                rustfs_io_metrics::record_put_object_quota_cache_decision(
+                    "candidate",
+                    quota_cache_reason(snapshot.quota.as_ref()),
+                );
+                Some(snapshot)
+            }
+            Ok(None) => {
+                rustfs_io_metrics::record_put_object_quota_cache_decision("miss", "no_authoritative_cache");
+                None
+            }
+            Err(_) => {
+                rustfs_io_metrics::record_put_object_quota_cache_decision("miss", "invalid_cached_quota");
+                None
+            }
+        }
+    } else {
+        None
+    };
+    rustfs_io_metrics::record_put_object_stage_duration_from(
+        rustfs_io_metrics::PUT_STAGE_PUT_OBJECT_QUOTA_BEGIN_CACHE_LOOKUP,
+        cache_lookup_started,
+    );
+    let config_read_started = rustfs_io_metrics::put_stage_timer();
     let (quota, bucket_incarnation, quota_revision) =
         metadata_sys::get_quota_config_and_incarnation_from_disk_for_options_in(ctx, bucket, opts, &metadata_guard).await?;
+    if let Some(cached) = cached_quota_snapshot.as_ref() {
+        let reason = quota_cache_reason(quota.as_ref());
+        if cached.quota == quota && cached.bucket_incarnation == bucket_incarnation && cached.quota_revision == quota_revision {
+            rustfs_io_metrics::record_put_object_quota_cache_decision("authoritative_match", reason);
+        } else {
+            rustfs_io_metrics::record_put_object_quota_cache_decision("authoritative_mismatch", reason);
+            if cached.quota != quota {
+                rustfs_io_metrics::record_put_object_quota_cache_decision("authoritative_mismatch_field", "quota");
+            }
+            if cached.bucket_incarnation != bucket_incarnation {
+                rustfs_io_metrics::record_put_object_quota_cache_decision("authoritative_mismatch_field", "bucket_incarnation");
+            }
+            if cached.quota_revision != quota_revision {
+                rustfs_io_metrics::record_put_object_quota_cache_decision("authoritative_mismatch_field", "quota_revision");
+            }
+        }
+    }
+    rustfs_io_metrics::record_put_object_stage_duration_from(
+        rustfs_io_metrics::PUT_STAGE_PUT_OBJECT_QUOTA_BEGIN_CONFIG_READ,
+        config_read_started,
+    );
+    let lock_lost_check_started = rustfs_io_metrics::put_stage_timer();
     if metadata_guard.is_lock_lost() {
         return Err(StorageError::NamespaceLockQuorumUnavailable {
             mode: "quota_config",
@@ -918,6 +977,11 @@ pub(crate) async fn begin(
             achieved: 0,
         });
     }
+    rustfs_io_metrics::record_put_object_stage_duration_from(
+        rustfs_io_metrics::PUT_STAGE_PUT_OBJECT_QUOTA_BEGIN_LOCK_LOST_CHECK,
+        lock_lost_check_started,
+    );
+    let policy_evaluate_started = rustfs_io_metrics::put_stage_timer();
     if quota
         .as_ref()
         .is_some_and(|quota| quota.has_unsupported_reservation_protocol())
@@ -926,6 +990,11 @@ pub(crate) async fn begin(
         return Err(StorageError::PartMissingOrCorrupt);
     }
     let durable_quota = quota.as_ref().filter(|quota| quota.uses_durable_reservations());
+    rustfs_io_metrics::record_put_object_stage_duration_from(
+        rustfs_io_metrics::PUT_STAGE_PUT_OBJECT_QUOTA_BEGIN_POLICY_EVALUATE,
+        policy_evaluate_started,
+    );
+    let capability_proof_started = rustfs_io_metrics::put_stage_timer();
     let capability_proof = if durable_quota.is_some() {
         Some(
             crate::services::notification_sys::acquire_cross_pool_fence_fleet_proof()
@@ -934,7 +1003,12 @@ pub(crate) async fn begin(
     } else {
         None
     };
+    rustfs_io_metrics::record_put_object_stage_duration_from(
+        rustfs_io_metrics::PUT_STAGE_PUT_OBJECT_QUOTA_BEGIN_CAPABILITY_PROOF,
+        capability_proof_started,
+    );
     let durable_quota_limit = durable_quota.and_then(|quota| quota.quota);
+    let snapshot_admission_started = rustfs_io_metrics::put_stage_timer();
     let reservation_protocol = durable_quota.and_then(|quota| quota.reservation_protocol);
     let snapshot_admission = match quota.as_ref().filter(|quota| !quota.uses_durable_reservations()) {
         Some(quota) => match (quota.quota, snapshot_admission) {
@@ -954,6 +1028,10 @@ pub(crate) async fn begin(
         },
         None => None,
     };
+    rustfs_io_metrics::record_put_object_stage_duration_from(
+        rustfs_io_metrics::PUT_STAGE_PUT_OBJECT_QUOTA_BEGIN_SNAPSHOT_ADMISSION,
+        snapshot_admission_started,
+    );
     let legacy_data_movement = durable_quota_limit.is_none()
         && quota.as_ref().and_then(|quota| quota.quota).is_some()
         && snapshot_admission.is_none()
@@ -965,11 +1043,21 @@ pub(crate) async fn begin(
                 .then(|| quota.as_ref().and_then(|quota| quota.quota))
                 .flatten()
         });
+    let object_store_lookup_started = rustfs_io_metrics::put_stage_timer();
     let store = if durable_quota_limit.is_some() {
         Some(metadata_sys::object_store_in(ctx).await?)
     } else {
         None
     };
+    rustfs_io_metrics::record_put_object_stage_duration_from(
+        rustfs_io_metrics::PUT_STAGE_PUT_OBJECT_QUOTA_BEGIN_OBJECT_STORE_LOOKUP,
+        object_store_lookup_started,
+    );
+    let context_build_started = rustfs_io_metrics::put_stage_timer();
+    rustfs_io_metrics::record_put_object_stage_duration_from(
+        rustfs_io_metrics::PUT_STAGE_PUT_OBJECT_QUOTA_BEGIN_CONTEXT_BUILD,
+        context_build_started,
+    );
     Ok(QuotaContext {
         store,
         bucket: bucket.to_string(),
@@ -988,6 +1076,15 @@ pub(crate) async fn begin(
     })
 }
 
+fn quota_cache_reason(quota: Option<&crate::bucket::quota::BucketQuota>) -> &'static str {
+    match quota {
+        None => "no_quota",
+        Some(quota) if quota.uses_durable_reservations() => "durable_quota",
+        Some(quota) if quota.quota.is_some() => "snapshot_quota",
+        Some(_) => "quota_without_limit",
+    }
+}
+
 fn quota_capability_error(bucket: &str, object: &str) -> StorageError {
     StorageError::NamespaceLockQuorumUnavailable {
         mode: "quota_capability",

crates/ecstore/src/bucket/replication/replication_resyncer.rs

@@ -448,6 +448,26 @@ fn is_version_id_format_mismatch(err: &SdkError<HeadObjectError>) -> bool {
     is_version_id_mismatch(code, raw_status)
 }
 
+/// A version-addressed HEAD rejected because the target cannot address the
+/// source version id (AWS S3: bodiless 400). Narrower than
+/// [`is_version_id_format_mismatch`]: a bodiless 403 may be a genuine
+/// `s3:GetObjectVersion` denial, so it is not accepted where a mutation or a
+/// completion verdict follows instead of another non-destructive HEAD.
+fn rejects_source_version_id(err: &SdkError<HeadObjectError>) -> bool {
+    is_version_id_format_mismatch(err) && !has_raw_status(err, 403)
+}
+
+/// Whether the target's latest version of `object` is a delete marker. For
+/// targets that mint their own version ids, which cannot be asked about the
+/// source marker id: S3 answers a versionless HEAD on such a key with 404
+/// (405 on some targets) plus `x-amz-delete-marker: true`.
+async fn latest_target_version_is_delete_marker(tgt_client: &TargetClient, object: &str) -> bool {
+    match head_object_for_worker(tgt_client, &tgt_client.bucket, object, None).await {
+        Ok(head) => head.delete_marker == Some(true),
+        Err(e) => raw_header_value(e.as_ref(), "x-amz-delete-marker").as_deref() == Some("true"),
+    }
+}
+
 async fn mark_replication_target_offline_if_needed(target_client: &Arc<TargetClient>, err: &(impl Display + ?Sized)) {
     if is_replication_target_offline_error(err) {
         ReplicationTargetStore::mark_target_offline(target_client).await;
@@ -1846,7 +1866,17 @@ async fn verify_resync_head_result(
             // only absence proves completion. All other outcomes remain
             // failed (backlog#862 / #799 B13).
             let retryable = if roi.version_purge_status.is_empty() {
-                matches!(classify_delete_marker_head_error(&err), DeleteMarkerHeadOutcome::Ambiguous)
+                match classify_delete_marker_head_error(&err) {
+                    // A target that mints its own version ids never holds the
+                    // source marker id; verify the marker on the latest version.
+                    DeleteMarkerHeadOutcome::Ambiguous => {
+                        let mints_own = rejects_source_version_id(&err)
+                            || ReplicationTargetStore::version_identity_capability(&target_client.arn)
+                                .version_addressing_unreliable();
+                        !(mints_own && latest_target_version_is_delete_marker(target_client, &roi.name).await)
+                    }
+                    DeleteMarkerHeadOutcome::Exists | DeleteMarkerHeadOutcome::Missing => false,
+                }
             } else {
                 !matches!(classify_delete_marker_head_error(&err), DeleteMarkerHeadOutcome::Missing)
             };
@@ -3567,7 +3597,16 @@ async fn replicate_delete_to_target(
         }
     }
 
-    if !is_version_purge && dobj.delete_object.delete_marker && dobj.delete_object.delete_marker_version_id.is_some() {
+    // A target that mints its own version ids never holds the source marker
+    // id, so a version-addressed pre-check cannot succeed there; the marker
+    // is created by the versionless DELETE below.
+    let version_addressable =
+        !ReplicationTargetStore::version_identity_capability(&tgt_client.arn).version_addressing_unreliable();
+    if version_addressable
+        && !is_version_purge
+        && dobj.delete_object.delete_marker
+        && dobj.delete_object.delete_marker_version_id.is_some()
+    {
         match head_object_for_worker(
             tgt_client.as_ref(),
             &tgt_client.bucket,
@@ -3576,6 +3615,11 @@ async fn replicate_delete_to_target(
         )
         .await
         {
+            // The verdict is not known yet (e.g. no PUT since restart) and the
+            // target rejected the source id: learn it as the PUT path does.
+            Err(e) if rejects_source_version_id(e.as_ref()) => {
+                ReplicationTargetStore::record_version_identity_capability(&tgt_client.arn, VersionIdentityCapability::MintsOwn);
+            }
             Ok(head) if head.delete_marker == Some(true) => {
                 // Some targets expose an existing marker as a successful HEAD
                 // with the marker bit instead of the usual 405. Treat that as
@@ -6675,6 +6719,93 @@ mod tests {
         assert!(seen[0].starts_with("HEAD "), "{seen:?}");
     }
 
+    /// AWS S3 answers a HEAD for a version id it did not mint with a bodiless
+    /// 400. On a target with no verdict yet, that teaches `MintsOwn` and the
+    /// marker is still created with one versionless DELETE.
+    #[tokio::test]
+    async fn live_delete_marker_worker_learns_mints_own_from_bodiless_400() {
+        let (endpoint, server) = spawn_scripted_target_server(2, |line| {
+            if line.starts_with("HEAD ") {
+                empty_response("400 Bad Request")
+            } else if line.starts_with("DELETE ") {
+                empty_response("204 No Content")
+            } else {
+                empty_response("500 Unexpected")
+            }
+        });
+        let target = test_target_client(endpoint);
+        register_test_target(&target).await;
+        let dobj = marker_creation_dobj(&target.arn);
+
+        let rinfo = replicate_delete_to_target(&dobj, target.clone(), None).await;
+
+        assert_eq!(rinfo.replication_status, ReplicationStatusType::Completed, "{rinfo:?}");
+        assert_eq!(
+            ReplicationTargetStore::version_identity_capability(&target.arn),
+            VersionIdentityCapability::MintsOwn
+        );
+        let seen = server.join().expect("test HTTP server should finish");
+        assert_eq!(seen.len(), 2, "{seen:?}");
+        assert!(seen[0].starts_with("HEAD "), "{seen:?}");
+        assert!(seen[1].starts_with("DELETE "), "{seen:?}");
+        assert!(!seen[1].contains("versionId="), "marker creation must remain versionless: {seen:?}");
+    }
+
+    /// A bodiless 403 on the version-addressed HEAD may be a genuine
+    /// GetObjectVersion denial on an id-adopting target. It must fail closed:
+    /// no DELETE, and no `MintsOwn` verdict cached for the ARN.
+    #[tokio::test]
+    async fn live_delete_marker_worker_keeps_bodiless_403_failure_closed() {
+        let (endpoint, server) = spawn_scripted_target_server(1, |line| {
+            if line.starts_with("HEAD ") {
+                empty_response("403 Forbidden")
+            } else if line.starts_with("DELETE ") {
+                empty_response("204 No Content")
+            } else {
+                empty_response("500 Unexpected")
+            }
+        });
+        let target = test_target_client(endpoint);
+        register_test_target(&target).await;
+        let dobj = marker_creation_dobj(&target.arn);
+
+        let rinfo = replicate_delete_to_target(&dobj, target.clone(), None).await;
+
+        assert_eq!(rinfo.replication_status, ReplicationStatusType::Failed, "{rinfo:?}");
+        assert_ne!(
+            ReplicationTargetStore::version_identity_capability(&target.arn),
+            VersionIdentityCapability::MintsOwn
+        );
+        let seen = server.join().expect("test HTTP server should finish");
+        assert_eq!(seen.len(), 1, "a denied HEAD must not trigger DELETE: {seen:?}");
+        assert!(seen[0].starts_with("HEAD "), "{seen:?}");
+    }
+
+    /// A target already known to mint its own version ids is never asked for
+    /// the source marker id: the marker is created by the versionless DELETE.
+    #[tokio::test]
+    async fn live_delete_marker_worker_skips_version_head_on_mints_own_target() {
+        let (endpoint, server) = spawn_scripted_target_server(1, |line| {
+            if line.starts_with("DELETE ") {
+                empty_response("204 No Content")
+            } else {
+                empty_response("500 Unexpected")
+            }
+        });
+        let target = test_target_client(endpoint);
+        register_test_target(&target).await;
+        ReplicationTargetStore::record_version_identity_capability(&target.arn, VersionIdentityCapability::MintsOwn);
+        let dobj = marker_creation_dobj(&target.arn);
+
+        let rinfo = replicate_delete_to_target(&dobj, target.clone(), None).await;
+
+        assert_eq!(rinfo.replication_status, ReplicationStatusType::Completed, "{rinfo:?}");
+        let seen = server.join().expect("test HTTP server should finish");
+        assert_eq!(seen.len(), 1, "no version-addressed HEAD on a MintsOwn target: {seen:?}");
+        assert!(seen[0].starts_with("DELETE "), "{seen:?}");
+        assert!(!seen[0].contains("versionId="), "marker creation must remain versionless: {seen:?}");
+    }
+
     /// A target answers `HEAD ?versionId=<delete marker>` with a bodiless
     /// 405, so the SDK reports no error code. That is the marker having
     /// propagated, not a failure (rustfs/backlog#2479).
@@ -6765,6 +6896,76 @@ mod tests {
         server.join().expect("test HTTP server should finish");
     }
 
+    /// AWS answers `HEAD ?versionId=<source marker id>` with a bodiless 400.
+    /// The marker is verified through the latest version instead: a
+    /// versionless 404 with `x-amz-delete-marker: true` proves it propagated.
+    #[tokio::test]
+    async fn resync_verification_accepts_latest_delete_marker_when_target_rejects_source_id() {
+        let (endpoint, server) = spawn_scripted_target_server(2, |line| {
+            if line.contains("versionId=") {
+                empty_response("400 Bad Request")
+            } else {
+                "HTTP/1.1 404 Not Found\r\nX-Amz-Delete-Marker: true\r\nX-Amz-Version-Id: target-marker\r\nContent-Length: 0\r\nConnection: close\r\n\r\n"
+                    .to_string()
+            }
+        });
+        let target = test_target_client(endpoint);
+        let roi = delete_marker_roi();
+        let mut st = TargetReplicationResyncStatus::default();
+
+        let head_result =
+            head_object_for_worker(target.as_ref(), &target.bucket, &roi.name, roi.version_id.map(|v| v.to_string())).await;
+        let (size, err) = verify_resync_head_result(head_result, &roi, &mut st, &target).await;
+
+        assert!(err.is_none(), "{err:?}");
+        assert_eq!((size, st.replicated_count, st.failed_count), (0, 1, 0));
+        let seen = server.join().expect("test HTTP server should finish");
+        assert!(!seen[1].contains("versionId="), "the re-check must address the latest version: {seen:?}");
+    }
+
+    /// A live latest version on a target that rejects the source id means the
+    /// marker has not propagated: the verification stays failed.
+    #[tokio::test]
+    async fn resync_verification_fails_delete_marker_when_latest_target_version_is_live() {
+        let (endpoint, server) = spawn_scripted_target_server(2, |line| {
+            if line.contains("versionId=") {
+                empty_response("400 Bad Request")
+            } else {
+                empty_response("200 OK")
+            }
+        });
+        let target = test_target_client(endpoint);
+        let roi = delete_marker_roi();
+        let mut st = TargetReplicationResyncStatus::default();
+
+        let head_result =
+            head_object_for_worker(target.as_ref(), &target.bucket, &roi.name, roi.version_id.map(|v| v.to_string())).await;
+        let (size, err) = verify_resync_head_result(head_result, &roi, &mut st, &target).await;
+
+        assert!(err.is_some(), "a live latest version must not verify the marker");
+        assert_eq!((size, st.replicated_count, st.failed_count), (0, 0, 1));
+        server.join().expect("test HTTP server should finish");
+    }
+
+    /// A bodiless 403 may be a genuine permission denial: it is not evidence
+    /// that the target mints its own ids, so no latest-version re-check runs.
+    #[tokio::test]
+    async fn resync_verification_keeps_delete_marker_failed_on_bodiless_403() {
+        let (endpoint, server) = spawn_scripted_target_server(1, |_| empty_response("403 Forbidden"));
+        let target = test_target_client(endpoint);
+        let roi = delete_marker_roi();
+        let mut st = TargetReplicationResyncStatus::default();
+
+        let head_result =
+            head_object_for_worker(target.as_ref(), &target.bucket, &roi.name, roi.version_id.map(|v| v.to_string())).await;
+        let (size, err) = verify_resync_head_result(head_result, &roi, &mut st, &target).await;
+
+        assert!(err.is_some(), "a 403 must not count as a propagated delete marker");
+        assert_eq!((size, st.replicated_count, st.failed_count), (0, 0, 1));
+        let seen = server.join().expect("test HTTP server should finish");
+        assert_eq!(seen.len(), 1, "{seen:?}");
+    }
+
     /// Serves exactly `requests` connections, answering each from
     /// `respond(request_line)`, and returns the request lines it saw. Reads
     /// the whole request (headers plus `Content-Length` body, honoring

crates/ecstore/src/cluster/rpc/client.rs

@@ -24,10 +24,11 @@ use crate::cluster::rpc::{gen_tonic_replay_scope_headers, gen_tonic_signature_he
 #[cfg(test)]
 use crate::cluster::rpc::{tonic_boot_epoch_challenge, tonic_boot_epoch_response_headers};
 use crate::disk::error::{DiskError, Error as DiskErrorType, RpcStatusError};
+#[cfg(test)]
 use crate::runtime::sources as runtime_sources;
 use http::{Request as HttpRequest, Response as HttpResponse, Uri};
 use rustfs_protos::{
-    ChannelClass, create_new_channel, get_channel_for_class,
+    ChannelClass, get_channel_for_class,
     proto_gen::node_service::{
         heal_control_service_client::HealControlServiceClient, node_service_client::NodeServiceClient,
         scanner_control_service_client::ScannerControlServiceClient,
@@ -53,7 +54,7 @@ use super::context_propagation::{inject_request_id_into_metadata, inject_trace_c
 /// 3. Subsequent calls will attempt fresh connections
 /// 4. If node is still down, connection will fail fast (3s timeout)
 pub async fn node_service_time_out_client(
-    addr: &String,
+    addr: &str,
     interceptor: TonicInterceptor,
 ) -> Result<NodeServiceClient<InterceptedService<AuthenticatedChannel, TonicInterceptor>>, Box<dyn Error>> {
     // Default to the latency-sensitive control channel; bulk `bytes` RPCs opt in via the
@@ -66,12 +67,9 @@ pub(crate) async fn scanner_control_time_out_client(
     interceptor: TonicInterceptor,
 ) -> crate::error::Result<ScannerControlServiceClient<InterceptedService<AuthenticatedChannel, TonicInterceptor>>> {
     let interceptor = interceptor.with_rpc_audience(addr)?;
-    let channel = match runtime_sources::cached_node_channel(addr).await {
-        Some(channel) => channel,
-        None => create_new_channel(addr)
-            .await
-            .map_err(|err| crate::error::Error::other(err.to_string()))?,
-    };
+    let channel = get_channel_for_class(addr, ChannelClass::Control)
+        .await
+        .map_err(|err| crate::error::Error::other(err.to_string()))?;
     let channel = ReplayScopeChannel::new(channel, interceptor.replay_scope_audience());
     let limit = rustfs_protos::scoped_dirty_usage::SCOPED_DIRTY_USAGE_MAX_REQUEST_BYTES as usize;
     Ok(ScannerControlServiceClient::with_interceptor(channel, interceptor)
@@ -84,10 +82,7 @@ pub async fn heal_control_time_out_client(
     interceptor: TonicInterceptor,
 ) -> Result<HealControlServiceClient<InterceptedService<AuthenticatedChannel, TonicInterceptor>>, Box<dyn Error>> {
     let interceptor = interceptor.with_rpc_audience(addr)?;
-    let channel = match runtime_sources::cached_node_channel(addr).await {
-        Some(channel) => channel,
-        None => create_new_channel(addr).await?,
-    };
+    let channel = get_channel_for_class(addr, ChannelClass::Control).await?;
     let max_message_size = rustfs_protos::HEAL_CONTROL_RPC_MAX_MESSAGE_SIZE;
     let channel = ReplayScopeChannel::new(channel, interceptor.replay_scope_audience());
     Ok(HealControlServiceClient::with_interceptor(channel, interceptor)
@@ -100,10 +95,7 @@ pub async fn tier_mutation_control_time_out_client(
     interceptor: TonicInterceptor,
 ) -> Result<TierMutationControlServiceClient<InterceptedService<AuthenticatedChannel, TonicInterceptor>>, Box<dyn Error>> {
     let interceptor = interceptor.with_rpc_audience(addr)?;
-    let channel = match runtime_sources::cached_node_channel(addr).await {
-        Some(channel) => channel,
-        None => create_new_channel(addr).await?,
-    };
+    let channel = get_channel_for_class(addr, ChannelClass::Control).await?;
     let max_message_size = rustfs_protos::TIER_MUTATION_RPC_MAX_MESSAGE_SIZE;
     let channel = ReplayScopeChannel::new(channel, interceptor.replay_scope_audience());
     Ok(TierMutationControlServiceClient::with_interceptor(channel, interceptor)
@@ -118,22 +110,12 @@ pub async fn tier_mutation_control_time_out_client(
 /// from lock/health RPCs; everything else uses [`ChannelClass::Control`]. When isolation is
 /// disabled the two classes resolve to the same cached channel, i.e. legacy behavior.
 pub async fn node_service_time_out_client_for_class(
-    addr: &String,
+    addr: &str,
     interceptor: TonicInterceptor,
     class: ChannelClass,
 ) -> Result<NodeServiceClient<InterceptedService<AuthenticatedChannel, TonicInterceptor>>, Box<dyn Error>> {
     let interceptor = interceptor.with_rpc_audience(addr)?;
-    let channel = match class {
-        ChannelClass::Control => match runtime_sources::cached_node_channel(addr).await {
-            Some(channel) => {
-                debug!("Using cached gRPC channel for: {}", addr);
-                channel
-            }
-            // No cached connection, create new one.
-            None => create_new_channel(addr).await?,
-        },
-        ChannelClass::Bulk => get_channel_for_class(addr, ChannelClass::Bulk).await?,
-    };
+    let channel = get_channel_for_class(addr, class).await?;
 
     let max_message_size = rustfs_protos::internode_rpc_max_message_size();
     let channel = ReplayScopeChannel::new(channel, interceptor.replay_scope_audience());
@@ -142,8 +124,24 @@ pub async fn node_service_time_out_client_for_class(
         .max_encoding_message_size(max_message_size))
 }
 
+pub(crate) async fn node_service_delete_recovery_client(
+    addr: &str,
+    interceptor: TonicInterceptor,
+    timeout: std::time::Duration,
+) -> Result<NodeServiceClient<InterceptedService<AuthenticatedChannel, TonicInterceptor>>, DiskError> {
+    let interceptor = interceptor.with_rpc_audience(addr).map_err(DiskError::from)?;
+    let channel = rustfs_protos::create_channel_with_timeout(addr, timeout)
+        .await
+        .map_err(|error| DiskError::RemoteClientUnavailable(error.to_string()))?;
+    let channel = ReplayScopeChannel::new(channel, interceptor.replay_scope_audience());
+    let max_message_size = rustfs_protos::internode_rpc_max_message_size();
+    Ok(NodeServiceClient::with_interceptor(channel, interceptor)
+        .max_decoding_message_size(max_message_size)
+        .max_encoding_message_size(max_message_size))
+}
+
 pub async fn node_service_time_out_client_no_auth(
-    addr: &String,
+    addr: &str,
 ) -> Result<NodeServiceClient<InterceptedService<AuthenticatedChannel, TonicInterceptor>>, Box<dyn Error>> {
     node_service_time_out_client(addr, TonicInterceptor::NoOp(NoOpInterceptor)).await
 }
@@ -355,6 +353,13 @@ fn apply_peer_replay_response(
     }
 }
 
+fn rename_data_grpc_stage(path: &str) -> bool {
+    matches!(
+        path,
+        "/node_service.NodeService/RenameData" | "/node_service.NodeService/RenameDataAtIncarnation"
+    )
+}
+
 impl<S, ReqBody, ResBody> Service<HttpRequest<ReqBody>> for ReplayScopeChannel<S>
 where
     S: Service<HttpRequest<ReqBody>, Response = HttpResponse<ResBody>>,
@@ -372,6 +377,12 @@ where
     }
 
     fn call(&mut self, mut request: HttpRequest<ReqBody>) -> Self::Future {
+        let observe_rename_data = rustfs_io_metrics::put_stage_metrics_enabled() && rename_data_grpc_stage(request.uri().path());
+        let request_scope_started = if observe_rename_data {
+            rustfs_io_metrics::put_stage_timer()
+        } else {
+            None
+        };
         let authenticated = self.audience.as_ref().is_some_and(|_| {
             request
                 .headers()
@@ -406,11 +417,30 @@ where
                 }
             }
         }
+        rustfs_io_metrics::record_put_object_stage_duration_from(
+            rustfs_io_metrics::PUT_STAGE_SET_DISK_RENAME_REMOTE_CLIENT_REQUEST_SCOPE,
+            request_scope_started,
+        );
 
         let audience = self.audience.clone();
         let future = self.inner.call(request);
         Box::pin(async move {
-            let response = future.await?;
+            let response = if observe_rename_data {
+                rustfs_io_metrics::observe_put_stage_future(
+                    future,
+                    rustfs_io_metrics::PUT_STAGE_SET_DISK_RENAME_REMOTE_CLIENT_TRANSPORT_CALL,
+                    rustfs_io_metrics::PUT_STAGE_SET_DISK_RENAME_REMOTE_CLIENT_TRANSPORT_CALL_POLL_PENDING_COUNT,
+                    rustfs_io_metrics::PUT_STAGE_SET_DISK_RENAME_REMOTE_CLIENT_TRANSPORT_CALL_FIRST_PENDING_TO_READY,
+                )
+                .await?
+            } else {
+                future.await?
+            };
+            let replay_response_started = if observe_rename_data {
+                rustfs_io_metrics::put_stage_timer()
+            } else {
+                None
+            };
             if let (Some(audience), Some(challenge)) = (audience, challenge) {
                 let response_state = verify_tonic_peer_replay_capabilities_response(&audience, challenge, response.headers());
                 if let Err(error) = &response_state
@@ -428,6 +458,10 @@ where
                 }
                 apply_peer_replay_response(audience, sent_state, response_state);
             }
+            rustfs_io_metrics::record_put_object_stage_duration_from(
+                rustfs_io_metrics::PUT_STAGE_SET_DISK_RENAME_REMOTE_CLIENT_REPLAY_RESPONSE,
+                replay_response_started,
+            );
             Ok(response)
         })
     }
@@ -546,6 +580,126 @@ mod tests {
     use tracing_opentelemetry::OpenTelemetrySpanExt;
     use tracing_subscriber::{Registry, layer::SubscriberExt};
 
+    #[tokio::test]
+    async fn delete_recovery_channel_uses_its_scoped_deadline() {
+        use bytes::Bytes;
+        use http_body_util::StreamBody;
+        use hyper::{body::Frame, service::service_fn};
+        use hyper_util::rt::{TokioExecutor, TokioIo};
+        use rustfs_protos::proto_gen::node_service::{RecoverDeleteTransactionRequest, RecoverDeleteTransactionResponse};
+        use tonic_prost::prost::Message;
+        let listener = tokio::net::TcpListener::bind("127.0.0.1:0")
+            .await
+            .expect("bind recovery peer");
+        let addr = format!("http://{}", listener.local_addr().expect("peer address"));
+        let server = tokio::spawn(async move {
+            loop {
+                let (stream, _) = listener.accept().await.expect("accept recovery connection");
+                tokio::spawn(async move {
+                    let service = service_fn(|_request: HttpRequest<hyper::body::Incoming>| async {
+                        tokio::time::sleep(std::time::Duration::from_millis(500)).await;
+                        let payload = RecoverDeleteTransactionResponse {
+                            success: true,
+                            protocol_version: 1,
+                            result: Bytes::new(),
+                            error: None,
+                        }
+                        .encode_to_vec();
+                        let mut data = vec![0];
+                        data.extend_from_slice(&u32::try_from(payload.len()).expect("small response").to_be_bytes());
+                        data.extend_from_slice(&payload);
+                        let mut trailers = http::HeaderMap::new();
+                        trailers.insert("grpc-status", "0".parse().expect("status"));
+                        let frames: Vec<Result<Frame<Bytes>, std::convert::Infallible>> =
+                            vec![Ok(Frame::data(data.into())), Ok(Frame::trailers(trailers))];
+                        let mut response = HttpResponse::new(StreamBody::new(futures::stream::iter(frames)));
+                        response
+                            .headers_mut()
+                            .insert("content-type", "application/grpc".parse().expect("content type"));
+                        Ok::<_, std::convert::Infallible>(response)
+                    });
+                    let _ = hyper::server::conn::http2::Builder::new(TokioExecutor::new())
+                        .serve_connection(TokioIo::new(stream), service)
+                        .await;
+                });
+            }
+        });
+        let request = || RecoverDeleteTransactionRequest {
+            disk: "disk".into(),
+            volume: "bucket".into(),
+            protocol_version: 1,
+            operation: Bytes::new(),
+        };
+        // Seed an existing short-lived control channel to reproduce the inherited deadline.
+        let control = tonic::transport::Endpoint::from_shared(addr.clone())
+            .expect("test endpoint")
+            .timeout(std::time::Duration::from_millis(50))
+            .connect()
+            .await
+            .expect("control connect");
+        rustfs_common::cache_connection(addr.clone(), control).await;
+        let mut foreground = node_service_time_out_client(&addr, TonicInterceptor::NoOp(NoOpInterceptor))
+            .await
+            .expect("control client");
+        let error = foreground
+            .recover_delete_transaction(request())
+            .await
+            .expect_err("short control deadline");
+        assert!(matches!(error.code(), tonic::Code::Cancelled | tonic::Code::DeadlineExceeded));
+        // The 2-second test override bypasses the production range parser, which rejects 2.
+        let mut recovery = node_service_delete_recovery_client(
+            &addr,
+            TonicInterceptor::NoOp(NoOpInterceptor),
+            std::time::Duration::from_secs(2),
+        )
+        .await
+        .expect("scoped recovery client");
+        assert!(
+            recovery
+                .recover_delete_transaction(request())
+                .await
+                .expect("scoped deadline must survive control deadline")
+                .into_inner()
+                .success
+        );
+        server.abort();
+        rustfs_protos::evict_failed_connection(&addr).await;
+    }
+
+    #[tokio::test]
+    async fn rpc_client_factories_do_not_reuse_https_channels_without_tls_generation() {
+        let listener = std::net::TcpListener::bind("127.0.0.1:0").expect("reserve local address");
+        let addr = format!("https://{}", listener.local_addr().expect("local address"));
+        drop(listener);
+        for factory in 0..4 {
+            let placeholder = tonic::transport::Endpoint::from_static("http://127.0.0.1:1").connect_lazy();
+            runtime_sources::cache_test_node_channel(addr.clone(), placeholder).await;
+            let interceptor = TonicInterceptor::NoOp(NoOpInterceptor);
+            let _result = match factory {
+                0 => scanner_control_time_out_client(&addr, interceptor)
+                    .await
+                    .map(|_| ())
+                    .map_err(|e| e.to_string()),
+                1 => heal_control_time_out_client(&addr, interceptor)
+                    .await
+                    .map(|_| ())
+                    .map_err(|e| e.to_string()),
+                2 => tier_mutation_control_time_out_client(&addr, interceptor)
+                    .await
+                    .map(|_| ())
+                    .map_err(|e| e.to_string()),
+                _ => node_service_time_out_client(&addr, interceptor)
+                    .await
+                    .map(|_| ())
+                    .map_err(|e| e.to_string()),
+            };
+            assert!(
+                !runtime_sources::test_node_channel_is_cached(&addr).await,
+                "factory {factory} must evict an unbound HTTPS cache entry before attempting a fresh dial"
+            );
+        }
+    }
+
     #[derive(Clone)]
     struct EpochProofService {
         audience: String,

crates/ecstore/src/cluster/rpc/network_probe.rs

@@ -289,13 +289,9 @@ async fn client(
     >,
     NetworkPeerProbeError,
 > {
-    node_service_time_out_client_for_class(
-        &address.to_owned(),
-        TonicInterceptor::Signature(gen_tonic_signature_interceptor()),
-        class,
-    )
-    .await
-    .map_err(|_| NetworkPeerProbeError::Unreachable)
+    node_service_time_out_client_for_class(address, TonicInterceptor::Signature(gen_tonic_signature_interceptor()), class)
+        .await
+        .map_err(|_| NetworkPeerProbeError::Unreachable)
 }
 
 fn ping_request(payload: &[u8]) -> PingRequest {
@@ -329,6 +325,24 @@ fn map_status(status: tonic::Status) -> NetworkPeerProbeError {
     match status.code() {
         Code::Unavailable | Code::Unknown => NetworkPeerProbeError::Unreachable,
         Code::DeadlineExceeded => NetworkPeerProbeError::TimedOut,
+        Code::Cancelled | Code::Internal => {
+            let mut source = std::error::Error::source(&status);
+            while let Some(error) = source {
+                if error
+                    .downcast_ref::<h2::Error>()
+                    .is_some_and(|error| matches!(error.reason(), Some(h2::Reason::CANCEL | h2::Reason::NO_ERROR)))
+                {
+                    return NetworkPeerProbeError::Unreachable;
+                }
+                source = if let Some(error) = error.downcast_ref::<std::io::Error>() {
+                    // io::Error::source skips its immediate custom error.
+                    error.get_ref().map(|inner| inner as &(dyn std::error::Error + 'static))
+                } else {
+                    error.source()
+                };
+            }
+            NetworkPeerProbeError::ProtocolFailure
+        }
         _ => NetworkPeerProbeError::ProtocolFailure,
     }
 }
@@ -339,6 +353,35 @@ mod tests {
     use crate::layout::endpoint::Endpoint;
     use crate::layout::endpoints::{Endpoints, PoolEndpoints};
 
+    #[test]
+    fn peer_transport_resets_are_unreachable_without_reclassifying_application_errors() {
+        for reason in [h2::Reason::CANCEL, h2::Reason::NO_ERROR] {
+            let status = tonic::Status::from_error(Box::new(h2::Error::from(reason)));
+            assert!(matches!(status.code(), Code::Cancelled | Code::Internal));
+            let code = status.code();
+            assert_eq!(map_status(status), NetworkPeerProbeError::Unreachable);
+            let mut wrapped = tonic::Status::new(code, "wrapped transport reset");
+            wrapped.set_source(Arc::new(std::io::Error::other(h2::Error::from(reason))));
+            assert_eq!(map_status(wrapped), NetworkPeerProbeError::Unreachable);
+        }
+        for reason in [h2::Reason::PROTOCOL_ERROR, h2::Reason::COMPRESSION_ERROR] {
+            let status = tonic::Status::from_error(Box::new(h2::Error::from(reason)));
+            assert_eq!(map_status(status), NetworkPeerProbeError::ProtocolFailure);
+        }
+        for status in [
+            tonic::Status::cancelled("peer cancelled the application request"),
+            tonic::Status::internal("peer rejected the probe"),
+            tonic::Status::unauthenticated("authentication rejected"),
+            tonic::Status::permission_denied("authorization rejected"),
+        ] {
+            assert_eq!(map_status(status), NetworkPeerProbeError::ProtocolFailure);
+        }
+        assert_eq!(
+            map_status(tonic::Status::deadline_exceeded("probe deadline elapsed")),
+            NetworkPeerProbeError::TimedOut
+        );
+    }
+
     fn pacing(duration: Duration) -> Arc<DiagnosticPacing> {
         NetworkPeerProbeClient::from_endpoint_pools(&topology())
             .with_diagnostic_pacing(Instant::now(), duration)

crates/ecstore/src/cluster/rpc/remote_disk.rs

@@ -14,7 +14,8 @@
 
 use crate::cluster::rpc::client::{
     AuthenticatedChannel, TonicInterceptor, gen_tonic_signature_interceptor, is_network_like_disk_error,
-    node_service_time_out_client, node_service_time_out_client_for_class, node_service_time_out_client_no_auth,
+    node_service_delete_recovery_client, node_service_time_out_client, node_service_time_out_client_for_class,
+    node_service_time_out_client_no_auth,
 };
 use crate::cluster::rpc::internode_data_transport::{
     InternodeDataTransport, NsScannerCapabilityRequest, NsScannerStreamRequest, ReadStreamRequest, WalkDirStreamRequest,
@@ -22,10 +23,11 @@ use crate::cluster::rpc::internode_data_transport::{
 };
 use crate::disk::error::{Error, Result};
 use crate::disk::{
-    BatchReadVersionReq, BatchReadVersionResp, CheckPartsResp, ConditionalFileUpdate, DeleteOptions, DiskAPI, DiskInfo,
-    DiskInfoOptions, DiskLocation, DiskOption, FileInfoVersions, FileReader, FileWriter, PartTransactionAction, ReadMultipleReq,
-    ReadMultipleResp, ReadOptions, RenameDataResp, SnapshotLeaseToken, UpdateMetadataOpts, VolumeInfo, WalkDirOptions,
-    batch_read_version_one_by_one,
+    BatchReadVersionReq, BatchReadVersionResp, CheckPartsResp, ConditionalFileUpdate, DELETE_RECOVERY_PROTOCOL_VERSION,
+    DeleteOptions, DeleteRecoveryAction, DeleteRecoveryCursor, DeleteRecoveryDigest, DeleteRecoveryInspection,
+    DeleteRecoveryPage, DeleteRecoveryRpcOperation, DeleteRecoveryRpcReply, DiskAPI, DiskInfo, DiskInfoOptions, DiskLocation,
+    DiskOption, FileInfoVersions, FileReader, FileWriter, PartTransactionAction, ReadMultipleReq, ReadMultipleResp, ReadOptions,
+    RenameDataResp, SnapshotLeaseToken, UpdateMetadataOpts, VolumeInfo, WalkDirOptions, batch_read_version_one_by_one,
     disk_store::{
         DEFAULT_RUSTFS_DRIVE_ACTIVE_MONITORING, ENV_RUSTFS_DRIVE_ACTIVE_MONITORING, SKIP_IF_SUCCESS_BEFORE,
         get_drive_active_check_interval, get_drive_active_check_timeout, get_drive_disk_info_timeout, get_drive_list_dir_timeout,
@@ -59,10 +61,10 @@ use rustfs_protos::proto_gen::node_service::{
     CompareAndUpdateFileRequest, DeletePathsRequest, DeleteRequest, DeleteVersionRequest, DeleteVersionsRequest,
     DeleteVersionsResponse, DeleteVolumeRequest, DiskInfoRequest, ListDirRequest, ListVolumesRequest, MakeVolumeRequest,
     MakeVolumesRequest, PreparePartTransactionRequest, ReadAllRequest, ReadMetadataRequest, ReadMultipleRequest,
-    ReadMultipleResponse, ReadPartsRequest, ReadVersionRequest, ReadXlRequest, RenameDataRequest, RenameFileRequest,
-    SettlePartTransactionRequest, SnapshotLeaseReleaseRequest, SnapshotLeaseRenewRequest, SnapshotLeaseRequest,
-    SnapshotLeaseResponse, StatVolumeRequest, UpdateMetadataRequest, VerifyFileRequest, WriteAllRequest, WriteMetadataRequest,
-    node_service_client::NodeServiceClient,
+    ReadMultipleResponse, ReadPartsRequest, ReadVersionRequest, ReadXlRequest, RecoverDeleteTransactionRequest,
+    RenameDataRequest, RenameFileRequest, SettlePartTransactionRequest, SnapshotLeaseReleaseRequest, SnapshotLeaseRenewRequest,
+    SnapshotLeaseRequest, SnapshotLeaseResponse, StatVolumeRequest, UpdateMetadataRequest, VerifyFileRequest, WriteAllRequest,
+    WriteMetadataRequest, node_service_client::NodeServiceClient,
 };
 use serde::{Serialize, de::DeserializeOwned};
 use std::{
@@ -142,6 +144,45 @@ fn decode_delete_versions_errors(response: DeleteVersionsResponse, expected_len:
         .collect()
 }
 
+fn decode_delete_recovery_batch_errors(codes: Vec<Option<u32>>, expected_len: usize) -> Result<Vec<Option<DiskError>>> {
+    if codes.len() != expected_len {
+        return Err(Error::other("malformed recoverable batch delete response length"));
+    }
+    codes
+        .into_iter()
+        .map(|code| match code {
+            None => Ok(None),
+            Some(code) => DiskError::from_u32(code)
+                .map(Some)
+                .ok_or_else(|| Error::other("invalid recoverable batch delete error code")),
+        })
+        .collect()
+}
+
+fn validate_delete_recovery_inspection(
+    volume: &str,
+    object: &str,
+    transaction: Uuid,
+    inspection: &DeleteRecoveryInspection,
+) -> Result<()> {
+    if inspection.protocol_version != DELETE_RECOVERY_PROTOCOL_VERSION
+        || inspection.id.object != object
+        || inspection.id.transaction != transaction
+        || transaction.is_nil()
+    {
+        return Err(Error::other("delete recovery inspection does not match request"));
+    }
+    if let Some(journal) = &inspection.journal
+        && (journal.protocol_version != DELETE_RECOVERY_PROTOCOL_VERSION
+            || journal.volume != volume
+            || journal.object != object
+            || journal.transaction != transaction)
+    {
+        return Err(Error::other("delete recovery journal does not match request"));
+    }
+    Ok(())
+}
+
 fn snapshot_lease_token_from_response(response: SnapshotLeaseResponse) -> Result<SnapshotLeaseToken> {
     if !response.success {
         return Err(response.error.unwrap_or_default().into());
@@ -858,6 +899,75 @@ fn spawn_control_channel_prewarm(addr: String) {
 }
 
 impl RemoteDisk {
+    async fn recover_delete_transaction(
+        &self,
+        volume: &str,
+        operation: DeleteRecoveryRpcOperation,
+    ) -> Result<DeleteRecoveryRpcReply> {
+        let payload = rmp_serde::to_vec_named(&operation)?;
+        if payload.len() > operation.max_wire_payload_bytes() {
+            return Err(Error::other("delete recovery operation exceeds payload limit"));
+        }
+        let settle_timeout = if matches!(&operation, DeleteRecoveryRpcOperation::Settle { .. }) {
+            Some(crate::disk::get_delete_recovery_timeout()?)
+        } else {
+            None
+        };
+        self.execute_with_timeout(
+            || async {
+                let mut client = if let Some(timeout) = settle_timeout {
+                    if let Some(error) = self.offline_bypass_error() {
+                        return Err(error);
+                    }
+                    node_service_delete_recovery_client(
+                        &self.addr,
+                        TonicInterceptor::Signature(gen_tonic_signature_interceptor()),
+                        timeout,
+                    )
+                    .await?
+                } else {
+                    self.get_client().await?
+                };
+                let mut request = Request::new(RecoverDeleteTransactionRequest {
+                    disk: self.endpoint.to_string(),
+                    volume: volume.to_owned(),
+                    protocol_version: DELETE_RECOVERY_PROTOCOL_VERSION,
+                    operation: payload.into(),
+                });
+                let canonical_body = rustfs_protos::canonical_recover_delete_transaction_request_body(request.get_ref())
+                    .map_err(|_| Error::other("recover_delete_transaction request length cannot be represented"))?;
+                crate::cluster::rpc::set_tonic_canonical_body_digest(&mut request, &canonical_body).map_err(Error::other)?;
+                // UNIMPLEMENTED is deliberately propagated: older peers cannot settle transactions.
+                let response = client
+                    .recover_delete_transaction(request)
+                    .await
+                    .map_err(|status| {
+                        if status.code() == Code::Unimplemented {
+                            DiskError::UnsupportedDisk
+                        } else {
+                            status.into()
+                        }
+                    })?
+                    .into_inner();
+                if response.protocol_version != DELETE_RECOVERY_PROTOCOL_VERSION {
+                    return Err(DiskError::UnsupportedDisk);
+                }
+                if !response.success {
+                    return Err(response
+                        .error
+                        .ok_or_else(|| Error::other("missing delete recovery error"))?
+                        .into());
+                }
+                if response.error.is_some() {
+                    return Err(Error::other("malformed delete recovery success"));
+                }
+                rustfs_protos::decode_delete_recovery_payload(&response.result).map_err(Into::into)
+            },
+            settle_timeout.unwrap_or_else(get_max_timeout_duration),
+        )
+        .await
+    }
+
     async fn rename_file_with_durability(
         &self,
         src_volume: &str,
@@ -1920,6 +2030,12 @@ fn encode_msgpack<T: Serialize>(value: &T) -> Result<Vec<u8>> {
     encode_msgpack_with_capacity(value, MSGPACK_ENCODE_CAPACITY_HINT)
 }
 
+// Optional fields are omitted by serde; named maps keep rollback flags and
+// transaction UUIDs independent of field order across peer versions.
+fn encode_delete_options_msgpack(value: &DeleteOptions) -> Result<Vec<u8>> {
+    Ok(rmp_serde::to_vec_named(value)?)
+}
+
 fn encode_file_info_msgpack(value: &FileInfo) -> Result<Vec<u8>> {
     encode_msgpack_with_capacity(value, FILE_INFO_MSGPACK_ENCODE_CAPACITY_HINT)
 }
@@ -2163,6 +2279,7 @@ impl RemoteDisk {
         self.execute_with_timeout_for_op(
             "rename_data",
             || async {
+                let prepare_started = rustfs_io_metrics::put_stage_timer();
                 let file_info = compat_json(fi)?;
                 let file_info_bin = encode_file_info_msgpack(fi)?;
                 let mut client = self.get_client().await?;
@@ -2190,24 +2307,67 @@ impl RemoteDisk {
                 } else {
                     attach_mutation_body_digest(&mut request, canonical_body, "rename_data")?;
                 }
+                rustfs_io_metrics::record_put_object_stage_duration_from(
+                    rustfs_io_metrics::PUT_STAGE_SET_DISK_RENAME_REMOTE_CLIENT_PREPARE,
+                    prepare_started,
+                );
 
-                let response = if incarnation_bound {
-                    // Older peers return Unimplemented before mutation; never downgrade.
-                    client.rename_data_at_incarnation(request).await?
-                } else {
-                    client.rename_data(request).await?
-                }
-                .into_inner();
+                let rpc_started = rustfs_io_metrics::put_stage_timer();
+                let response = rustfs_io_metrics::observe_put_stage_future(
+                    async {
+                        if incarnation_bound {
+                            // Older peers return Unimplemented before mutation; never downgrade.
+                            client.rename_data_at_incarnation(request).await
+                        } else {
+                            client.rename_data(request).await
+                        }
+                    },
+                    rustfs_io_metrics::PUT_STAGE_SET_DISK_RENAME_REMOTE_CLIENT_RPC_AWAIT,
+                    rustfs_io_metrics::PUT_STAGE_SET_DISK_RENAME_REMOTE_CLIENT_RPC_AWAIT_POLL_PENDING_COUNT,
+                    rustfs_io_metrics::PUT_STAGE_SET_DISK_RENAME_REMOTE_CLIENT_RPC_AWAIT_FIRST_PENDING_TO_READY,
+                )
+                .await?;
+                let response_process_started = rustfs_io_metrics::put_stage_timer();
+                let response_headers_started = rustfs_io_metrics::put_stage_timer();
+                let _ = response.metadata().len();
+                rustfs_io_metrics::record_put_object_stage_duration_from(
+                    rustfs_io_metrics::PUT_STAGE_SET_DISK_RENAME_REMOTE_CLIENT_RESPONSE_HEADERS,
+                    response_headers_started,
+                );
+                let into_inner_started = rustfs_io_metrics::put_stage_timer();
+                let response = response.into_inner();
+                rustfs_io_metrics::record_put_object_stage_duration_from(
+                    rustfs_io_metrics::PUT_STAGE_SET_DISK_RENAME_REMOTE_CLIENT_INTO_INNER,
+                    into_inner_started,
+                );
+                rustfs_io_metrics::record_put_object_stage_duration_from(
+                    rustfs_io_metrics::PUT_STAGE_SET_DISK_RENAME_REMOTE_CLIENT_RPC,
+                    rpc_started,
+                );
 
+                let success_check_started = rustfs_io_metrics::put_stage_timer();
                 if !response.success {
                     return Err(response.error.unwrap_or_default().into());
                 }
+                rustfs_io_metrics::record_put_object_stage_duration_from(
+                    rustfs_io_metrics::PUT_STAGE_SET_DISK_RENAME_REMOTE_CLIENT_SUCCESS_CHECK,
+                    success_check_started,
+                );
 
+                let decode_started = rustfs_io_metrics::put_stage_timer();
                 let rename_data_resp = decode_msgpack_or_json::<RenameDataResp>(
                     &response.rename_data_resp_bin,
                     &response.rename_data_resp,
                     "RenameDataResp",
                 )?;
+                rustfs_io_metrics::record_put_object_stage_duration_from(
+                    rustfs_io_metrics::PUT_STAGE_SET_DISK_RENAME_REMOTE_CLIENT_DECODE,
+                    decode_started,
+                );
+                rustfs_io_metrics::record_put_object_stage_duration_from(
+                    rustfs_io_metrics::PUT_STAGE_SET_DISK_RENAME_REMOTE_CLIENT_RESPONSE_PROCESS,
+                    response_process_started,
+                );
 
                 Ok(rename_data_resp)
             },
@@ -2566,7 +2726,7 @@ impl DiskAPI for RemoteDisk {
                 // `_bin` support for DeleteVersion is new (grpc-optimization P2); always dual-write
                 // JSON + msgpack until its fallback counter has read zero across a release window.
                 let file_info_bin = encode_file_info_msgpack(&fi)?;
-                let opts_bin = encode_msgpack(&opts)?;
+                let opts_bin = encode_delete_options_msgpack(&opts)?;
                 let conditional_marker = opts.expected_delete_marker.is_some();
                 let file_info = serde_json::to_string(&fi)?;
                 let opts = serde_json::to_string(&opts)?;
@@ -2638,7 +2798,7 @@ impl DiskAPI for RemoteDisk {
 
         // `_bin` support for DeleteVersions is new (grpc-optimization P2); always dual-write JSON +
         // msgpack until its fallback counter has read zero across a release window.
-        let opts_bin = match encode_msgpack(&opts) {
+        let opts_bin = match encode_delete_options_msgpack(&opts) {
             Ok(opts_bin) => opts_bin,
             Err(err) => {
                 let mut errors = Vec::with_capacity(versions.len());
@@ -3617,6 +3777,138 @@ impl DiskAPI for RemoteDisk {
     }
 
     #[tracing::instrument(level = "trace", skip_all)]
+    async fn delete_version_recoverable(
+        &self,
+        volume: &str,
+        object: &str,
+        file_info: FileInfo,
+        force_del_marker: bool,
+        opts: DeleteOptions,
+    ) -> Result<()> {
+        match self
+            .recover_delete_transaction(
+                volume,
+                DeleteRecoveryRpcOperation::ApplyVersion {
+                    object: object.to_owned(),
+                    file_info: Box::new(file_info),
+                    force_del_marker,
+                    opts,
+                },
+            )
+            .await?
+        {
+            DeleteRecoveryRpcReply::AppliedVersion => Ok(()),
+            _ => Err(Error::other("unexpected recoverable delete reply")),
+        }
+    }
+
+    async fn delete_versions_recoverable(
+        &self,
+        volume: &str,
+        versions: Vec<FileInfoVersions>,
+        opts: DeleteOptions,
+    ) -> Vec<Option<DiskError>> {
+        let expected_len = versions.len();
+        let result = match self
+            .recover_delete_transaction(volume, DeleteRecoveryRpcOperation::ApplyVersions { versions, opts })
+            .await
+        {
+            Ok(DeleteRecoveryRpcReply::AppliedVersions(codes)) => decode_delete_recovery_batch_errors(codes, expected_len),
+            Ok(_) => Err(Error::other("unexpected recoverable batch delete reply")),
+            Err(error) => Err(error),
+        };
+        match result {
+            Ok(errors) => errors,
+            Err(error) => vec![Some(error); expected_len],
+        }
+    }
+
+    async fn discover_delete_transactions(
+        &self,
+        volume: &str,
+        object: Option<&str>,
+        cursor: Option<DeleteRecoveryCursor>,
+        limit: u32,
+    ) -> Result<DeleteRecoveryPage> {
+        match self
+            .recover_delete_transaction(
+                volume,
+                DeleteRecoveryRpcOperation::Discover {
+                    object: object.map(str::to_owned),
+                    cursor,
+                    limit,
+                },
+            )
+            .await?
+        {
+            DeleteRecoveryRpcReply::Discovered(page) => {
+                if object.is_some_and(|object| page.transactions.iter().any(|id| id.object != object)) {
+                    return Err(Error::other("delete recovery discovery escaped requested object"));
+                }
+                if page.transactions.len() > usize::try_from(limit).map_err(|_| Error::other("invalid discovery limit"))?
+                    || page
+                        .transactions
+                        .iter()
+                        .any(|id| id.transaction.is_nil() || id.object.is_empty())
+                {
+                    return Err(Error::other("malformed delete discovery page"));
+                }
+                Ok(page)
+            }
+            _ => Err(Error::other("unexpected delete discovery reply")),
+        }
+    }
+    async fn inspect_delete_transaction(
+        &self,
+        volume: &str,
+        object: &str,
+        transaction: Uuid,
+    ) -> Result<DeleteRecoveryInspection> {
+        match self
+            .recover_delete_transaction(
+                volume,
+                DeleteRecoveryRpcOperation::Inspect {
+                    object: object.to_owned(),
+                    transaction,
+                },
+            )
+            .await?
+        {
+            DeleteRecoveryRpcReply::Inspected(inspection) => {
+                validate_delete_recovery_inspection(volume, object, transaction, &inspection)?;
+                Ok(inspection)
+            }
+            _ => Err(Error::other("unexpected delete inspection reply")),
+        }
+    }
+    async fn settle_delete_transaction(
+        &self,
+        volume: &str,
+        object: &str,
+        transaction: Uuid,
+        expected_fingerprint: DeleteRecoveryDigest,
+        action: DeleteRecoveryAction,
+    ) -> Result<DeleteRecoveryInspection> {
+        match self
+            .recover_delete_transaction(
+                volume,
+                DeleteRecoveryRpcOperation::Settle {
+                    object: object.to_owned(),
+                    transaction,
+                    expected_fingerprint,
+                    action,
+                },
+            )
+            .await?
+        {
+            DeleteRecoveryRpcReply::Settled(inspection) => {
+                validate_delete_recovery_inspection(volume, object, transaction, &inspection)?;
+                Ok(inspection)
+            }
+            _ => Err(Error::other("unexpected delete settlement reply")),
+        }
+    }
+
     async fn settle_part_transaction(&self, volume: &str, path: &str, action: PartTransactionAction) -> Result<()> {
         self.execute_with_timeout(
             || async {
@@ -4030,6 +4322,45 @@ mod tests {
 
     static INIT: Once = Once::new();
 
+    #[test]
+    fn recoverable_batch_delete_rejects_invalid_error_codes_and_length() {
+        let errors = decode_delete_recovery_batch_errors(vec![None, Some(DiskError::FileNotFound.to_u32())], 2)
+            .expect("valid per-item errors");
+        assert!(errors[0].is_none());
+        assert_eq!(errors[1], Some(DiskError::FileNotFound));
+        assert!(decode_delete_recovery_batch_errors(vec![None], 2).is_err());
+        assert!(decode_delete_recovery_batch_errors(vec![Some(0)], 1).is_err());
+        assert!(decode_delete_recovery_batch_errors(vec![Some(u32::MAX)], 1).is_err());
+    }
+
+    #[test]
+    fn delete_recovery_inspection_rejects_wrong_transaction_object_and_protocol() {
+        let transaction = Uuid::new_v4();
+        let inspection = DeleteRecoveryInspection {
+            protocol_version: DELETE_RECOVERY_PROTOCOL_VERSION,
+            id: crate::disk::DeleteTransactionId {
+                object: "object".into(),
+                transaction,
+            },
+            state: crate::disk::DeleteRecoveryState::Absent,
+            journal: None,
+            current_digest: None,
+            current_versions: Vec::new(),
+            backup_digest: None,
+            rollback_data_complete: false,
+            commit_data_safe: false,
+            explicit_delete_safe: false,
+            fingerprint: [0; 32],
+        };
+        validate_delete_recovery_inspection("bucket", "object", transaction, &inspection).expect("matching inspection");
+        assert!(validate_delete_recovery_inspection("bucket", "other", transaction, &inspection).is_err());
+        assert!(validate_delete_recovery_inspection("bucket", "object", Uuid::new_v4(), &inspection).is_err());
+        assert!(validate_delete_recovery_inspection("bucket", "object", Uuid::nil(), &inspection).is_err());
+        let mut incompatible = inspection;
+        incompatible.protocol_version += 1;
+        assert!(validate_delete_recovery_inspection("bucket", "object", transaction, &incompatible).is_err());
+    }
+
     #[test]
     fn compare_and_update_wire_outcome_fails_closed() {
         assert_eq!(
@@ -4063,32 +4394,19 @@ mod tests {
     }
 
     #[test]
-    fn retired_marker_options_preserve_legacy_positional_wire_shape() {
+    fn delete_options_named_wire_preserves_rollback_and_legacy_marker_fields() {
         #[derive(Debug, serde::Serialize, serde::Deserialize, PartialEq)]
         struct LegacyDeleteOptions {
             recursive: bool,
             immediate: bool,
             undo_write: bool,
+            #[serde(default)]
             undo_delete: bool,
             old_data_dir: Option<Uuid>,
+            #[serde(default, skip_serializing_if = "Option::is_none")]
+            expected_delete_marker: Option<rustfs_filemeta::MetaDeleteMarker>,
         }
-        let legacy = LegacyDeleteOptions {
-            recursive: false,
-            immediate: false,
-            undo_write: false,
-            undo_delete: false,
-            old_data_dir: None,
-        };
-        let original = encode_msgpack(&legacy).unwrap();
-        let current = encode_msgpack(&DeleteOptions::default()).unwrap();
-        assert_eq!(current, original, "ordinary deletes must retain the older peer's positional payload");
-        assert_eq!(rmp_serde::from_slice::<LegacyDeleteOptions>(&current).unwrap(), legacy);
-        assert!(
-            rmp_serde::from_slice::<DeleteOptions>(&original)
-                .unwrap()
-                .expected_delete_marker
-                .is_none()
-        );
+        let transaction = Uuid::new_v4();
         let mut marker = FileInfo {
             deleted: true,
             version_id: Some(Uuid::new_v4()),
@@ -4097,12 +4415,69 @@ mod tests {
         };
         marker.set_delete_marker_incarnation(Uuid::new_v4());
         let marker = rustfs_filemeta::MetaDeleteMarker::from(marker);
+        for expected_delete_marker in [None, Some(marker)] {
+            for (undo_write, undo_delete) in [(true, false), (false, true)] {
+                let legacy = LegacyDeleteOptions {
+                    recursive: false,
+                    immediate: false,
+                    undo_write,
+                    undo_delete,
+                    old_data_dir: Some(transaction),
+                    expected_delete_marker: expected_delete_marker.clone(),
+                };
+                // Older senders use positional arrays. Their unchanged field prefix and
+                // missing optional fields must also decode on the named-map sender's peer.
+                for (encoding, payload) in [
+                    ("positional", rmp_serde::to_vec(&legacy).expect("encode legacy positional options")),
+                    ("named", rmp_serde::to_vec_named(&legacy).expect("encode legacy named options")),
+                ] {
+                    let decoded: DeleteOptions = rmp_serde::from_slice(&payload).expect("decode legacy options on current peer");
+                    assert_eq!(decoded.undo_write, undo_write, "{encoding} undo-write flag");
+                    assert_eq!(decoded.undo_delete, undo_delete, "{encoding} undo-delete flag");
+                    assert_eq!(decoded.old_data_dir, Some(transaction), "{encoding} transaction UUID");
+                    assert_eq!(decoded.expected_delete_marker, expected_delete_marker, "{encoding} marker precondition");
+                    assert!(decoded.delete_recovery_bucket_incarnation.is_none(), "{encoding} omitted incarnation");
+                    assert!(!decoded.recoverable_delete, "{encoding} omitted recovery flag");
+                }
+                for incarnation in [None, Some(Uuid::new_v4())] {
+                    let current = DeleteOptions {
+                        undo_write,
+                        undo_delete,
+                        old_data_dir: Some(transaction),
+                        expected_delete_marker: expected_delete_marker.clone(),
+                        delete_recovery_bucket_incarnation: incarnation,
+                        recoverable_delete: true,
+                        ..Default::default()
+                    };
+                    let payload = encode_delete_options_msgpack(&current).expect("encode current named options");
+                    let decoded: DeleteOptions = rmp_serde::from_slice(&payload).expect("decode current options on current peer");
+                    assert_eq!(decoded.undo_write, undo_write);
+                    assert_eq!(decoded.undo_delete, undo_delete);
+                    assert_eq!(decoded.old_data_dir, Some(transaction));
+                    assert_eq!(decoded.expected_delete_marker, expected_delete_marker);
+                    assert_eq!(decoded.delete_recovery_bucket_incarnation, incarnation);
+                    assert!(decoded.recoverable_delete);
+                    let decoded: LegacyDeleteOptions =
+                        rmp_serde::from_slice(&payload).expect("decode current options on legacy peer");
+                    assert_eq!(decoded, legacy, "unknown recovery fields cannot shift rollback or marker values");
+                }
+            }
+        }
+    }
+
+    #[test]
+    fn malformed_nonempty_delete_options_do_not_fall_back_to_json() {
         let options = DeleteOptions {
-            expected_delete_marker: Some(marker.clone()),
+            undo_write: true,
+            old_data_dir: Some(Uuid::new_v4()),
             ..Default::default()
         };
-        let decoded: DeleteOptions = rmp_serde::from_slice(&encode_msgpack(&options).unwrap()).unwrap();
-        assert_eq!(decoded.expected_delete_marker, Some(marker));
+        let json = serde_json::to_string(&options).expect("valid compatibility JSON");
+        assert!(decode_msgpack_or_json::<DeleteOptions>(&[0xc1], &json, "DeleteOptions").is_err());
+        let decoded: DeleteOptions =
+            decode_msgpack_or_json(&[], &json, "DeleteOptions").expect("absent MessagePack permits JSON compatibility");
+        assert!(decoded.undo_write);
+        assert_eq!(decoded.old_data_dir, options.old_data_dir);
     }
 
     #[test]
@@ -4785,6 +5160,8 @@ mod tests {
             cleanup_data_dir: Some(Uuid::new_v4()),
             sign: Some(vec![0x14, 0x35]),
             old_current_size: Some(crate::disk::OldCurrentSize::Present(64 * 1024)),
+            old_current_source_checked: false,
+            old_current_source: None,
         };
         let json = serde_json::to_string(&response).expect("legacy rename_data JSON response should encode");
         let decode_before = rustfs_io_metrics::internode_metrics::global_internode_metrics().msgpack_json_decode_total_for_test();
@@ -5187,6 +5564,8 @@ mod tests {
             cleanup_data_dir: Some(Uuid::new_v4()),
             sign: Some(vec![1_u8; 32]),
             old_current_size: Some(crate::disk::OldCurrentSize::Present(4096)),
+            old_current_source_checked: false,
+            old_current_source: None,
         };
         let json = serde_json::to_vec(&response).expect("rename data response json should encode");
         let named_msgpack = rmp_serde::encode::to_vec_named(&response).expect("rename data response named msgpack should encode");

crates/ecstore/src/core/sets.rs

@@ -72,6 +72,38 @@ type WalkOptions = StorageWalkOptions<fn(&FileInfo) -> bool>;
 
 const LIST_MULTIPART_SETS_CONCURRENCY: usize = 4;
 
+#[cfg(test)]
+type TestSetLockers = HashMap<usize, Vec<Arc<dyn rustfs_lock::LockClient>>>;
+
+#[cfg(test)]
+static TEST_SET_LOCKERS: std::sync::LazyLock<std::sync::Mutex<TestSetLockers>> =
+    std::sync::LazyLock::new(|| std::sync::Mutex::new(HashMap::new()));
+
+/// Keep a fixture's real locker domain scoped to its owning instance.
+#[cfg(test)]
+pub(crate) struct ScopedSetLockers(Arc<InstanceContext>);
+
+#[cfg(test)]
+impl ScopedSetLockers {
+    pub(crate) fn install(ctx: Arc<InstanceContext>, lockers: Vec<Arc<dyn rustfs_lock::LockClient>>) -> Self {
+        let key = Arc::as_ptr(&ctx) as usize;
+        let mut domains = TEST_SET_LOCKERS.lock().expect("test set lockers mutex");
+        assert!(!domains.contains_key(&key), "the instance already owns a test locker domain");
+        domains.insert(key, lockers);
+        Self(ctx)
+    }
+}
+
+#[cfg(test)]
+impl Drop for ScopedSetLockers {
+    fn drop(&mut self) {
+        TEST_SET_LOCKERS
+            .lock()
+            .expect("test set lockers mutex")
+            .remove(&(Arc::as_ptr(&self.0) as usize));
+    }
+}
+
 fn is_idempotent_delete_prefix_error(err: &Error) -> bool {
     is_err_object_not_found(err) || is_err_strict_volume_not_found(err)
 }
@@ -273,6 +305,14 @@ impl Sets {
                 }
             }
 
+            #[cfg(test)]
+            let pool_lockers = TEST_SET_LOCKERS
+                .lock()
+                .expect("test set lockers mutex")
+                .get(&(Arc::as_ptr(&instance_ctx) as usize))
+                .cloned()
+                .unwrap_or_else(|| pool_lockers.clone());
+
             let set_disks = SetDisks::new_with_instance_ctx(
                 runtime_sources::local_node_name().await,
                 Arc::new(RwLock::new(set_drive)),

crates/ecstore/src/diagnostics/admin_server_info.rs

@@ -15,7 +15,9 @@
 use crate::cluster::rpc::{
     ScannerBucketListing, TonicInterceptor, gen_tonic_signature_interceptor, node_service_time_out_client,
 };
-use crate::data_usage::{DATA_USAGE_CACHE_NAME, DATA_USAGE_ROOT, load_data_usage_from_backend_cached};
+use crate::data_usage::{
+    DATA_USAGE_CACHE_NAME, DATA_USAGE_ROOT, load_admin_data_usage_from_backend_cached as load_server_info_usage_snapshot,
+};
 use crate::error::{Error, Result};
 use crate::{
     disk::endpoint::{Endpoint, EndpointType},
@@ -42,7 +44,11 @@ use std::{
 use time::OffsetDateTime;
 use tokio::time::timeout;
 use tonic::Request;
-use tracing::warn;
+use tracing::{error, trace, warn};
+
+const EVENT_SERVER_PING: &str = "server_ping";
+const LOG_COMPONENT_ECSTORE: &str = "ecstore";
+const LOG_SUBSYSTEM_DIAGNOSTICS: &str = "diagnostics";
 
 use shadow_rs::shadow;
 
@@ -157,11 +163,22 @@ async fn is_server_resolvable(endpoint: &Endpoint) -> Result<()> {
 
         let response: PingResponse = client.ping(request).await?.into_inner();
 
-        let ping_response_body = flatbuffers::root::<PingBody>(&response.body);
-        if let Err(e) = ping_response_body {
-            eprintln!("{e}");
+        if flatbuffers::root::<PingBody>(&response.body).is_err() {
+            error!(
+                event = EVENT_SERVER_PING,
+                component = LOG_COMPONENT_ECSTORE,
+                subsystem = LOG_SUBSYSTEM_DIAGNOSTICS,
+                result = "invalid_response",
+                "Server ping response decoding failed"
+            );
         } else {
-            println!("ping_resp:body(flatbuffer): {ping_response_body:?}");
+            trace!(
+                event = EVENT_SERVER_PING,
+                component = LOG_COMPONENT_ECSTORE,
+                subsystem = LOG_SUBSYSTEM_DIAGNOSTICS,
+                result = "success",
+                "Server ping completed"
+            );
         }
 
         Ok(())
@@ -294,7 +311,7 @@ pub async fn get_server_info(get_pools: bool) -> InfoMessage {
     if let Some(store) = runtime_sources::object_store_handle() {
         mode = ITEM_ONLINE;
         apply_data_usage_result(
-            load_data_usage_from_backend_cached(store.clone()).await,
+            load_server_info_usage_snapshot(store.clone()).await,
             &mut buckets,
             &mut objects,
             &mut versions,
@@ -967,6 +984,93 @@ mod tests {
         assert_eq!(usage.size, 6);
     }
 
+    #[tokio::test]
+    #[serial]
+    async fn server_info_reads_complete_observation_without_promoting_authority() {
+        use crate::bucket::metadata_sys::test_support::isolated_store_over_temp_disks;
+        use crate::config::com::{read_config, save_config};
+        use crate::data_usage::{
+            DATA_USAGE_OBJ_NAME_PATH, DATA_USAGE_OBSERVED_OBJ_NAME_PATH, invalidate_data_usage_snapshot_cache,
+            load_data_usage_from_backend_cached,
+        };
+        use rustfs_data_usage::{BucketUsageInfo, DataUsageInfo};
+        use std::time::{Duration, SystemTime};
+
+        let (_dirs, store) = isolated_store_over_temp_disks().await;
+        let primary = DataUsageInfo {
+            last_update: Some(SystemTime::UNIX_EPOCH + Duration::from_secs(1)),
+            scanner_epoch: Some(1),
+            usage_snapshot_bootstrap_pending: true,
+            usage_snapshot_converged: Some(false),
+            ..Default::default()
+        };
+        let observed = DataUsageInfo {
+            last_update: Some(SystemTime::UNIX_EPOCH + Duration::from_secs(2)),
+            scanner_epoch: Some(1),
+            scanner_cycle: Some(1),
+            buckets_count: 1,
+            objects_total_count: 3,
+            versions_total_count: 4,
+            delete_markers_total_count: 1,
+            objects_total_size: 6,
+            buckets_usage: [(
+                "bucket".to_string(),
+                BucketUsageInfo {
+                    objects_count: 3,
+                    versions_count: 4,
+                    delete_markers_count: 1,
+                    size: 6,
+                    ..Default::default()
+                },
+            )]
+            .into_iter()
+            .collect(),
+            usage_snapshot_complete: true,
+            usage_snapshot_converged: Some(false),
+            usage_snapshot_authoritative_baseline: Some(primary.snapshot_identity()),
+            ..Default::default()
+        };
+        let primary_bytes = serde_json::to_vec(&primary).expect("bootstrap should encode");
+        save_config(store.clone(), DATA_USAGE_OBJ_NAME_PATH.as_str(), primary_bytes.clone())
+            .await
+            .expect("bootstrap should persist");
+        save_config(
+            store.clone(),
+            DATA_USAGE_OBSERVED_OBJ_NAME_PATH.as_str(),
+            serde_json::to_vec(&observed).expect("observation should encode"),
+        )
+        .await
+        .expect("observation should persist");
+        invalidate_data_usage_snapshot_cache().await;
+
+        let displayed = super::load_server_info_usage_snapshot(store.clone())
+            .await
+            .expect("server info should load the complete observation");
+        assert_eq!(displayed.usage_snapshot_converged, Some(false));
+        let mut buckets = rustfs_madmin::Buckets::default();
+        let mut objects = rustfs_madmin::Objects::default();
+        let mut versions = rustfs_madmin::Versions::default();
+        let mut delete_markers = rustfs_madmin::DeleteMarkers::default();
+        let mut usage = rustfs_madmin::Usage::default();
+        apply_data_usage_result(Ok(displayed), &mut buckets, &mut objects, &mut versions, &mut delete_markers, &mut usage);
+        assert!(objects.error.is_none());
+        assert_eq!(
+            (buckets.count, objects.count, versions.count, delete_markers.count, usage.size),
+            (1, 3, 4, 1, 6)
+        );
+        let quota = load_data_usage_from_backend_cached(store.clone())
+            .await
+            .expect("quota should retain its bootstrap state");
+        assert!(!quota.is_complete_bucket_usage_snapshot());
+        assert_eq!(
+            read_config(store, DATA_USAGE_OBJ_NAME_PATH.as_str())
+                .await
+                .expect("primary should remain readable"),
+            primary_bytes
+        );
+        invalidate_data_usage_snapshot_cache().await;
+    }
+
     #[test]
     fn incomplete_data_usage_is_unavailable_in_server_info() {
         let mut buckets = rustfs_madmin::Buckets::default();

crates/ecstore/src/diagnostics/get.rs

@@ -182,6 +182,7 @@ pub(crate) const GET_METADATA_CACHE_REASON_DELETE_MARKER: &str = "delete_marker"
 pub(crate) const GET_METADATA_CACHE_REASON_DIST_ERASURE: &str = "dist_erasure";
 pub(crate) const GET_METADATA_CACHE_REASON_INCL_FREE_VERSIONS: &str = "incl_free_versions";
 pub(crate) const GET_METADATA_CACHE_REASON_INSUFFICIENT_CACHED_QUORUM: &str = "insufficient_cached_quorum";
+pub(crate) const GET_METADATA_CACHE_REASON_BODY_LAYOUT_REQUIRED: &str = "body_layout_required";
 pub(crate) const GET_METADATA_CACHE_REASON_META_BUCKET: &str = "meta_bucket";
 pub(crate) const GET_METADATA_CACHE_REASON_NO_LOCK: &str = "no_lock";
 pub(crate) const GET_METADATA_CACHE_REASON_NOT_FOUND_OR_EXPIRED: &str = "not_found_or_expired";

crates/ecstore/src/disk/disk_store.rs

@@ -12,6 +12,7 @@
 // See the License for the specific language governing permissions and
 // limitations under the License.
 
+use crate::disk::cleanup_runtime;
 use crate::disk::io_admission::{
     AdmissionReader, BoxedChunkReader, DiskIoAdmission, DiskIoClass, DiskIoContext, current_disk_io_context,
     with_disk_io_context, with_disk_io_permit,
@@ -466,12 +467,11 @@ impl LocalDiskWrapper {
         self.track_disk_health_mutation(
             "delete_version",
             DiskMetricMutation::Delete,
-            || async {
+            || {
                 Box::pin(
                     self.disk
                         .delete_version_with_namespace_owner(volume, path, fi, force_del_marker, opts, namespace_owner),
                 )
-                .await
             },
             get_max_timeout_duration(),
         )
@@ -485,10 +485,10 @@ impl LocalDiskWrapper {
         opts: DeleteOptions,
         namespace_owner: Option<Arc<dyn Send + Sync>>,
     ) -> Result<()> {
-        self.track_disk_health_mutation(
+        self.track_cleanup_mutation(
             "delete",
             DiskMetricMutation::Delete,
-            || async { Box::pin(self.disk.delete_with_namespace_owner(volume, path, opts, namespace_owner)).await },
+            || async { Box::pin(self.cleanup_delete(volume, path, opts, namespace_owner)).await },
             get_max_timeout_duration(),
         )
         .await
@@ -1476,6 +1476,28 @@ impl LocalDiskWrapper {
         )
     }
 
+    async fn cleanup_delete(
+        &self,
+        volume: &str,
+        path: &str,
+        options: DeleteOptions,
+        owner: Option<Arc<dyn Send + Sync>>,
+    ) -> Result<()> {
+        if !cleanup_runtime::enabled()? {
+            return self.disk.delete_with_namespace_owner(volume, path, options, owner).await;
+        }
+        let disk = self.disk.clone();
+        let volume = volume.to_owned();
+        let path = path.to_owned();
+        let guarded = owner.is_some();
+        let operation = async move { disk.delete_with_namespace_owner(&volume, &path, options, owner).await };
+        if guarded {
+            cleanup_runtime::guarded_disk_operation(operation).await
+        } else {
+            cleanup_runtime::disk_operation(operation).await
+        }
+    }
+
     /// Run a delete under an owned coordinator task when a publication guard
     /// is present. This keeps the guard alive if the RPC waiter is cancelled
     /// while the local namespace mutation is still in progress.
@@ -1494,12 +1516,13 @@ impl LocalDiskWrapper {
         } else {
             get_max_timeout_duration()
         };
+        let cleanup_owner = external_guard.clone();
         run_owned_mutation(external_guard, move || async move {
             operation
-                .track_disk_health_mutation(
+                .track_cleanup_mutation(
                     "delete",
                     DiskMetricMutation::Delete,
-                    || async { operation.disk.delete(&volume, &path, options).await },
+                    || async { operation.cleanup_delete(&volume, &path, options, cleanup_owner).await },
                     timeout_duration,
                 )
                 .await
@@ -1816,6 +1839,8 @@ impl LocalDiskWrapper {
                     undo_delete: false,
                     old_data_dir: None,
                     expected_delete_marker: None,
+                    recoverable_delete: false,
+                    delete_recovery_bucket_incarnation: None,
                 },
             )
             .await?;
@@ -2001,10 +2026,86 @@ impl LocalDiskWrapper {
     }
 
     /// Get the current disk ID
+    pub(crate) async fn delete_versions_with_namespace_owner(
+        &self,
+        volume: &str,
+        versions: Vec<FileInfoVersions>,
+        opts: DeleteOptions,
+        namespace_owner: Option<Arc<dyn Send + Sync>>,
+    ) -> Vec<Option<Error>> {
+        let version_count = versions.len();
+        self.metrics.record_operation_call("delete_versions");
+        // Check if disk is faulty before proceeding
+        if self.health.is_faulty() {
+            self.metrics.record_availability_error();
+            return vec![Some(DiskError::FaultyDisk); versions.len()];
+        }
+
+        // Check if disk is stale
+        if let Err(e) = self.check_disk_stale().await {
+            self.metrics.record_availability_error();
+            return vec![Some(e); versions.len()];
+        }
+
+        let _io_permit = match self.acquire_disk_io_permit().await {
+            Ok(permit) => permit,
+            Err(error) => return vec![Some(error); version_count],
+        };
+
+        // Record operation start
+        self.health.last_started.store(current_unix_nanos(), Ordering::Relaxed);
+        let waiting_guard = self.health.waiting_guard();
+        let metric_waiting_guard = self.metrics.waiting_guard();
+        let started = Instant::now();
+
+        // Execute the operation
+        let result = self
+            .disk
+            .delete_versions_with_namespace_owner(volume, versions, opts, namespace_owner)
+            .await;
+        self.metrics.record_operation_latency("delete_versions", started.elapsed());
+        self.record_batch_delete_error_metrics(&result);
+
+        drop(waiting_guard);
+        drop(metric_waiting_guard);
+        let has_err = result.iter().any(|e| e.is_some());
+        if !has_err {
+            // Log success and decrement waiting counter
+            self.health.record_operation_success(&self.endpoint(), "operation_success");
+            self.metrics.record_mutation_success(DiskMetricMutation::Delete);
+        }
+
+        result
+    }
+
     pub async fn get_current_disk_id(&self) -> Option<Uuid> {
         *self.disk_id.read().await
     }
 
+    pub(crate) async fn settle_delete_transaction_with_namespace_owner(
+        &self,
+        volume: &str,
+        object: &str,
+        transaction: uuid::Uuid,
+        expected_fingerprint: crate::disk::DeleteRecoveryDigest,
+        action: crate::disk::DeleteRecoveryAction,
+        namespace_owner: Option<Arc<dyn Send + Sync>>,
+    ) -> Result<crate::disk::DeleteRecoveryInspection> {
+        self.track_disk_health_mutation(
+            "settle_delete_transaction",
+            DiskMetricMutation::Delete,
+            || {
+                Box::pin(async {
+                    self.disk
+                        .delete_recovery_settle(volume, object, transaction, expected_fingerprint, action, namespace_owner)
+                        .await
+                })
+            },
+            crate::disk::get_delete_recovery_timeout()?,
+        )
+        .await
+    }
+
     /// Track disk health for an operation.
     /// This method should wrap disk operations to ensure health checking.
     pub async fn track_disk_health<T, F, Fut>(&self, operation: F, timeout_duration: Duration) -> Result<T>
@@ -2015,6 +2116,31 @@ impl LocalDiskWrapper {
         self.track_disk_health_with_op("unknown", operation, timeout_duration).await
     }
 
+    /// Cleanup latency includes waiting behind a node-wide execution budget,
+    /// so a waiter deadline alone is not evidence that this disk is faulty.
+    /// The active disk health probe retains its original timeout policy.
+    async fn track_cleanup_mutation<T, F, Fut>(
+        &self,
+        op: &'static str,
+        mutation: DiskMetricMutation,
+        operation: F,
+        timeout_duration: Duration,
+    ) -> Result<T>
+    where
+        F: FnOnce() -> Fut,
+        Fut: std::future::Future<Output = Result<T>>,
+    {
+        // Runtime initialization must not prevent control-plane release of
+        // reader tokens. Disk I/O entries still propagate configuration errors.
+        let action = if cleanup_runtime::enabled().unwrap_or(false) {
+            TimeoutHealthAction::IgnoreFailure
+        } else {
+            TimeoutHealthAction::MarkFailure
+        };
+        self.track_disk_health_with_op_timeout_action_and_mutation(op, operation, timeout_duration, action, mutation)
+            .await
+    }
+
     async fn track_disk_health_mutation<T, F, Fut>(
         &self,
         op: &'static str,
@@ -2195,6 +2321,87 @@ impl LocalDiskWrapper {
 
 #[async_trait::async_trait]
 impl DiskAPI for LocalDiskWrapper {
+    async fn delete_version_recoverable(
+        &self,
+        volume: &str,
+        object: &str,
+        file_info: FileInfo,
+        force_del_marker: bool,
+        mut opts: DeleteOptions,
+    ) -> Result<()> {
+        crate::disk::validate_delete_recovery_options(&opts)?;
+        if object.starts_with('/') {
+            return Err(DiskError::InvalidPath);
+        }
+        opts.recoverable_delete = true;
+        self.delete_version(volume, object, file_info, force_del_marker, opts).await
+    }
+    async fn delete_versions_recoverable(
+        &self,
+        volume: &str,
+        versions: Vec<FileInfoVersions>,
+        mut opts: DeleteOptions,
+    ) -> Vec<Option<DiskError>> {
+        if let Err(err) = crate::disk::validate_delete_recovery_options(&opts) {
+            return vec![Some(err); versions.len()];
+        }
+        if versions.iter().any(|version| version.name.starts_with('/')) {
+            return vec![Some(DiskError::InvalidPath); versions.len()];
+        }
+        opts.recoverable_delete = true;
+        self.delete_versions(volume, versions, opts).await
+    }
+    async fn discover_delete_transactions(
+        &self,
+        volume: &str,
+        object: Option<&str>,
+        cursor: Option<crate::disk::DeleteRecoveryCursor>,
+        limit: u32,
+    ) -> Result<crate::disk::DeleteRecoveryPage> {
+        self.track_disk_health_with_op(
+            "discover_delete_transactions",
+            || async {
+                self.disk
+                    .discover_delete_transactions(volume, object, cursor.clone(), limit)
+                    .await
+            },
+            get_max_timeout_duration(),
+        )
+        .await
+    }
+    async fn inspect_delete_transaction(
+        &self,
+        volume: &str,
+        object: &str,
+        transaction: uuid::Uuid,
+    ) -> Result<crate::disk::DeleteRecoveryInspection> {
+        self.track_disk_health_with_op(
+            "inspect_delete_transaction",
+            || async { self.disk.inspect_delete_transaction(volume, object, transaction).await },
+            get_max_timeout_duration(),
+        )
+        .await
+    }
+    async fn settle_delete_transaction(
+        &self,
+        volume: &str,
+        object: &str,
+        transaction: uuid::Uuid,
+        expected_fingerprint: crate::disk::DeleteRecoveryDigest,
+        action: crate::disk::DeleteRecoveryAction,
+    ) -> Result<crate::disk::DeleteRecoveryInspection> {
+        self.track_disk_health_mutation(
+            "settle_delete_transaction",
+            DiskMetricMutation::Delete,
+            || async {
+                self.disk
+                    .settle_delete_transaction(volume, object, transaction, expected_fingerprint, action)
+                    .await
+            },
+            crate::disk::get_delete_recovery_timeout()?,
+        )
+        .await
+    }
     fn has_replacement_mount_lease(&self) -> bool {
         self.disk.has_replacement_mount_lease()
     }
@@ -2320,6 +2527,15 @@ impl DiskAPI for LocalDiskWrapper {
         })
     }
 
+    async fn bucket_creation_witness(&self, volume: &str) -> Result<Option<super::os::NativeBucketCreationWitness>> {
+        self.track_disk_health_with_op(
+            "bucket_creation_witness",
+            || async { self.disk.bucket_creation_witness(volume).await },
+            Duration::ZERO,
+        )
+        .await
+    }
+
     async fn make_volume(&self, volume: &str) -> Result<()> {
         // Scoped heal must drain directory creation before releasing its lifecycle owner.
         let timeout = if crate::store::bucket_heal_scope(volume).is_some() {
@@ -2408,46 +2624,7 @@ impl DiskAPI for LocalDiskWrapper {
     }
 
     async fn delete_versions(&self, volume: &str, versions: Vec<FileInfoVersions>, opts: DeleteOptions) -> Vec<Option<Error>> {
-        let version_count = versions.len();
-        self.metrics.record_operation_call("delete_versions");
-        // Check if disk is faulty before proceeding
-        if self.health.is_faulty() {
-            self.metrics.record_availability_error();
-            return vec![Some(DiskError::FaultyDisk); versions.len()];
-        }
-
-        // Check if disk is stale
-        if let Err(e) = self.check_disk_stale().await {
-            self.metrics.record_availability_error();
-            return vec![Some(e); versions.len()];
-        }
-
-        let _io_permit = match self.acquire_disk_io_permit().await {
-            Ok(permit) => permit,
-            Err(error) => return vec![Some(error); version_count],
-        };
-
-        // Record operation start
-        self.health.last_started.store(current_unix_nanos(), Ordering::Relaxed);
-        let waiting_guard = self.health.waiting_guard();
-        let metric_waiting_guard = self.metrics.waiting_guard();
-        let started = Instant::now();
-
-        // Execute the operation
-        let result = self.disk.delete_versions(volume, versions, opts).await;
-        self.metrics.record_operation_latency("delete_versions", started.elapsed());
-        self.record_batch_delete_error_metrics(&result);
-
-        drop(waiting_guard);
-        drop(metric_waiting_guard);
-        let has_err = result.iter().any(|e| e.is_some());
-        if !has_err {
-            // Log success and decrement waiting counter
-            self.health.record_operation_success(&self.endpoint(), "operation_success");
-            self.metrics.record_mutation_success(DiskMetricMutation::Delete);
-        }
-
-        result
+        self.delete_versions_with_namespace_owner(volume, versions, opts, None).await
     }
 
     async fn delete_paths(&self, volume: &str, paths: &[String]) -> Result<()> {
@@ -2470,9 +2647,37 @@ impl DiskAPI for LocalDiskWrapper {
     }
 
     async fn release_snapshot_lease(&self, volume: &str, path: &str, token: SnapshotLeaseToken) -> Result<()> {
-        self.track_disk_health_with_op(
+        // Quota fence release is control-plane work and may wait for in-flight
+        // mutations. Keep its original waiter lifecycle instead of creating
+        // detached cleanup tasks for repeated fence-release RPCs.
+        if volume == super::RUSTFS_META_BUCKET && super::is_quota_mutation_fence_path(path) {
+            return self
+                .track_disk_health_with_op(
+                    "release_snapshot_lease",
+                    || async { self.disk.release_snapshot_lease(volume, path, token).await },
+                    get_max_timeout_duration(),
+                )
+                .await;
+        }
+        self.track_cleanup_mutation(
             "release_snapshot_lease",
-            || async { self.disk.release_snapshot_lease(volume, path, token).await },
+            DiskMetricMutation::None,
+            || async {
+                if self.disk.release_snapshot_lease_without_cleanup(volume, path, token).await {
+                    return Ok(());
+                }
+                if !cleanup_runtime::enabled().unwrap_or(false) {
+                    // The raw release removes the token before trying the
+                    // reclaim budget. Initialization failure therefore keeps
+                    // the cleanup intent and data, without leaking a reader
+                    // or falling back to unbudgeted filesystem deletion.
+                    return self.disk.release_snapshot_lease(volume, path, token).await;
+                }
+                let disk = self.disk.clone();
+                let volume = volume.to_owned();
+                let path = path.to_owned();
+                cleanup_runtime::release_lease(async move { disk.release_snapshot_lease(&volume, &path, token).await }).await
+            },
             get_max_timeout_duration(),
         )
         .await
@@ -2497,10 +2702,22 @@ impl DiskAPI for LocalDiskWrapper {
         } else {
             get_max_timeout_duration()
         };
-        self.track_disk_health_mutation(
+        self.track_cleanup_mutation(
             "delete_data_dir",
             DiskMetricMutation::Delete,
-            || async { self.disk.delete_data_dir(volume, path, opts).await },
+            || async {
+                if !cleanup_runtime::enabled()? {
+                    return self.disk.delete_data_dir(volume, path, opts).await;
+                }
+                let disk = self.disk.clone();
+                let volume = volume.to_owned();
+                let path = path.to_owned();
+                cleanup_runtime::disk_operation(async move {
+                    let _scope = scope;
+                    disk.delete_data_dir(&volume, &path, opts).await
+                })
+                .await
+            },
             timeout,
         )
         .await
@@ -2747,10 +2964,10 @@ impl DiskAPI for LocalDiskWrapper {
     }
 
     async fn delete(&self, volume: &str, path: &str, opt: DeleteOptions) -> Result<()> {
-        self.track_disk_health_mutation(
+        self.track_cleanup_mutation(
             "delete",
             DiskMetricMutation::Delete,
-            || async { self.disk.delete(volume, path, opt).await },
+            || async { self.cleanup_delete(volume, path, opt, None).await },
             get_max_timeout_duration(),
         )
         .await
@@ -2783,11 +3000,25 @@ impl DiskAPI for LocalDiskWrapper {
     }
 
     async fn write_all(&self, volume: &str, path: &str, data: Bytes) -> Result<()> {
-        self.track_disk_health_mutation(
+        let cleanup = path.rsplit('/').next() == Some(cleanup_runtime::OLD_DATA_CLEANUP_RECEIPT_FILE);
+        self.track_disk_health_with_op_timeout_action_and_mutation(
             "write_all",
-            DiskMetricMutation::Write,
-            || async { self.disk.write_all(volume, path, data).await },
+            || async {
+                if !cleanup || !cleanup_runtime::enabled()? {
+                    return self.disk.write_all(volume, path, data).await;
+                }
+                let disk = self.disk.clone();
+                let volume = volume.to_owned();
+                let path = path.to_owned();
+                cleanup_runtime::disk_operation(async move { disk.write_all(&volume, &path, data).await }).await
+            },
             get_max_timeout_duration(),
+            if cleanup && cleanup_runtime::enabled()? {
+                TimeoutHealthAction::IgnoreFailure
+            } else {
+                TimeoutHealthAction::MarkFailure
+            },
+            DiskMetricMutation::Write,
         )
         .await
     }
@@ -3180,6 +3411,256 @@ mod tests {
         let _ = walk.await;
     }
 
+    #[tokio::test]
+    async fn invalid_cleanup_config_does_not_leak_snapshot_readers() {
+        const CHILD: &str = "RUSTFS_CLEANUP_INVALID_LEASE_TEST_CHILD";
+        if std::env::var_os(CHILD).is_none() {
+            let output = tokio::task::spawn_blocking(|| {
+                std::process::Command::new(std::env::current_exe().expect("test executable"))
+                    .args([
+                        "--exact",
+                        "disk::disk_store::tests::invalid_cleanup_config_does_not_leak_snapshot_readers",
+                        "--nocapture",
+                    ])
+                    .env(CHILD, "1")
+                    .env("RUSTFS_CLEANUP_ISOLATE_ENABLE", "true")
+                    .env("RUSTFS_CLEANUP_ASYNC_THREADS", "0")
+                    .output()
+                    .expect("run with invalid cleanup configuration")
+            })
+            .await
+            .expect("child process task");
+            assert!(
+                output.status.success(),
+                "{}\n{}",
+                String::from_utf8_lossy(&output.stdout),
+                String::from_utf8_lossy(&output.stderr)
+            );
+            assert!(String::from_utf8_lossy(&output.stdout).contains("1 passed"));
+            return;
+        }
+        let dir = tempfile::tempdir().expect("temporary disk");
+        let endpoint = Endpoint::try_from(dir.path().to_str().expect("UTF-8 path")).expect("endpoint");
+        let disk = Arc::new(LocalDisk::new(&endpoint, false).await.expect("local disk"));
+        let wrapper = LocalDiskWrapper::new(disk.clone(), false);
+        let volume = "invalid-cleanup-config";
+        let path = "object/data";
+        wrapper.make_volume(volume).await.expect("bucket");
+        wrapper
+            .write_all(volume, "object/data/part.1", Bytes::from_static(b"live shard"))
+            .await
+            .expect("write shard");
+        let token = wrapper.acquire_snapshot_lease(volume, path).await.expect("GET lease");
+        wrapper
+            .release_snapshot_lease(volume, path, token)
+            .await
+            .expect("control-only release must not depend on cleanup initialization");
+        assert!(matches!(
+            wrapper.renew_snapshot_lease(volume, path, token).await,
+            Err(DiskError::FileNotFound)
+        ));
+
+        let token = wrapper
+            .acquire_snapshot_lease(volume, path)
+            .await
+            .expect("reader of old data");
+        // Register reclaim intent directly; this only updates the lease
+        // registry while a reader is present and performs no filesystem I/O.
+        assert_eq!(
+            disk.delete_data_dir(
+                volume,
+                path,
+                DeleteOptions {
+                    recursive: true,
+                    ..Default::default()
+                }
+            )
+            .await
+            .expect("defer reclaim"),
+            DataDirDeleteStatus::Deferred
+        );
+        assert!(
+            wrapper.release_snapshot_lease(volume, path, token).await.is_err(),
+            "invalid configuration must still reject reclamation"
+        );
+        assert!(
+            matches!(wrapper.renew_snapshot_lease(volume, path, token).await, Err(DiskError::FileNotFound)),
+            "failed reclamation must not strand a reader token"
+        );
+        assert_eq!(
+            wrapper.read_all(volume, "object/data/part.1").await.expect("retained data"),
+            Bytes::from_static(b"live shard"),
+            "configuration failure must not fall back to unbudgeted deletion"
+        );
+    }
+
+    #[tokio::test]
+    async fn rpc_cleanup_entries_share_disk_budget() {
+        const CHILD: &str = "RUSTFS_CLEANUP_RPC_TEST_CHILD";
+        if std::env::var_os(CHILD).is_none() {
+            let output = tokio::task::spawn_blocking(|| {
+                std::process::Command::new(std::env::current_exe().unwrap())
+                    .args([
+                        "--exact",
+                        "disk::disk_store::tests::rpc_cleanup_entries_share_disk_budget",
+                        "--nocapture",
+                    ])
+                    .env(CHILD, "1")
+                    .env("RUSTFS_CLEANUP_ISOLATE_ENABLE", "true")
+                    .env("RUSTFS_CLEANUP_ASYNC_THREADS", "1")
+                    .env("RUSTFS_CLEANUP_DISK_MAX_PENDING", "1")
+                    .env("RUSTFS_CLEANUP_DISK_WORKERS", "1")
+                    .env("RUSTFS_CLEANUP_GC_WORKERS", "1")
+                    .env("RUSTFS_CLEANUP_CPUS", "none")
+                    .env_remove("RUSTFS_PUT_RENAME_TAIL_CLEANUP_COUNTERFACTUAL_SKIP")
+                    .env_remove("RUSTFS_PUT_RENAME_TAIL_CLEANUP_DEFER_HOLD_WORKER")
+                    .env_remove("RUSTFS_PUT_RENAME_TAIL_CLEANUP_ZERO_TARGET_TMP_DELETE_SKIP")
+                    .output()
+                    .unwrap()
+            })
+            .await
+            .unwrap();
+            assert!(
+                output.status.success(),
+                "{}\n{}",
+                String::from_utf8_lossy(&output.stdout),
+                String::from_utf8_lossy(&output.stderr)
+            );
+            assert!(String::from_utf8_lossy(&output.stdout).contains("1 passed"));
+            return;
+        }
+        let dir = tempfile::tempdir().unwrap();
+        let endpoint = Endpoint::try_from(dir.path().to_str().unwrap()).unwrap();
+        let disk = Arc::new(LocalDisk::new(&endpoint, false).await.unwrap());
+        let wrapper = LocalDiskWrapper::new(disk, false);
+        let volume = "cleanup-receiver";
+        let path = "object/old-data";
+        let body_path = format!("{path}/part.1");
+        let receipt_path = format!("{path}/{}", cleanup_runtime::OLD_DATA_CLEANUP_RECEIPT_FILE);
+        wrapper.make_volume(volume).await.unwrap();
+        wrapper
+            .write_all(volume, &body_path, Bytes::from_static(b"old body"))
+            .await
+            .unwrap();
+        let read_lease = wrapper
+            .acquire_snapshot_lease(volume, path)
+            .await
+            .expect("ordinary GET lease");
+        let (entered, entry) = tokio::sync::oneshot::channel();
+        let (release, released) = std::sync::mpsc::channel();
+        let occupied = tokio::spawn(cleanup_runtime::disk_operation(async move {
+            entered.send(()).unwrap();
+            // Deliberately stall the only cleanup scheduler to prove that
+            // ordinary GET completion does not dispatch to it.
+            released
+                .recv_timeout(Duration::from_secs(30))
+                .expect("release stalled scheduler");
+            Ok(())
+        }));
+        tokio::time::timeout(Duration::from_secs(20), entry).await.unwrap().unwrap();
+        tokio::time::timeout(Duration::from_secs(5), wrapper.release_snapshot_lease(volume, path, read_lease))
+            .await
+            .expect("ordinary GET release must progress while the cleanup scheduler is stalled")
+            .expect("release without reclamation");
+        let fence_path = super::super::quota_mutation_fence_path(volume, "object");
+        let fence = wrapper
+            .acquire_snapshot_lease(super::super::RUSTFS_META_BUCKET, &fence_path)
+            .await
+            .unwrap();
+        wrapper
+            .release_snapshot_lease(super::super::RUSTFS_META_BUCKET, &fence_path, fence)
+            .await
+            .unwrap();
+        let timeout = wrapper
+            .track_cleanup_mutation(
+                "delete",
+                DiskMetricMutation::Delete,
+                std::future::pending::<Result<()>>,
+                Duration::from_millis(10),
+            )
+            .await;
+        assert!(matches!(timeout, Err(DiskError::Timeout)));
+        assert!(!wrapper.health.is_faulty(), "cleanup queue delay must not mark a healthy disk faulty");
+        let options = DeleteOptions {
+            recursive: true,
+            ..Default::default()
+        };
+        let drops = Arc::new(std::sync::atomic::AtomicUsize::new(0));
+        // This is the same entry point used after RPC authentication and
+        // scanner publication-lease validation by handle_delete.
+        let result = wrapper
+            .delete_with_publication_guard(volume, path, options.clone(), Some(Arc::new(DropProbe(drops.clone()))))
+            .await;
+        assert!(matches!(result, Err(DiskError::Io(err)) if err.kind() == io::ErrorKind::WouldBlock));
+        assert_eq!(drops.load(Ordering::SeqCst), 1);
+        assert!(matches!(wrapper.delete_data_dir(volume, path, options.clone()).await,
+            Err(DiskError::Io(err)) if err.kind() == io::ErrorKind::WouldBlock));
+        assert!(matches!(wrapper.write_all(volume, &receipt_path, Bytes::from_static(b"receipt")).await,
+            Err(DiskError::Io(err)) if err.kind() == io::ErrorKind::WouldBlock));
+        assert_eq!(wrapper.read_all(volume, &body_path).await.unwrap(), Bytes::from_static(b"old body"));
+        release.send(()).unwrap();
+        occupied.await.unwrap().unwrap();
+        wrapper
+            .write_all(volume, &receipt_path, Bytes::from_static(b"receipt"))
+            .await
+            .unwrap();
+        wrapper
+            .delete_with_publication_guard(volume, path, options, None)
+            .await
+            .unwrap();
+        assert!(matches!(wrapper.read_all(volume, &body_path).await, Err(DiskError::FileNotFound)));
+        // Snapshot readers defer old-data cleanup. Saturation must release
+        // their token while retaining reclaim intent, rather than leaking a
+        // live token or performing an unbudgeted delete on the RPC runtime.
+        wrapper
+            .write_all(volume, &body_path, Bytes::from_static(b"leased body"))
+            .await
+            .unwrap();
+        let lease = wrapper.acquire_snapshot_lease(volume, path).await.unwrap();
+        assert_eq!(
+            wrapper
+                .delete_data_dir(
+                    volume,
+                    path,
+                    DeleteOptions {
+                        recursive: true,
+                        ..Default::default()
+                    }
+                )
+                .await
+                .unwrap(),
+            DataDirDeleteStatus::Deferred
+        );
+        let (entered, entry) = tokio::sync::oneshot::channel();
+        let (release, released) = tokio::sync::oneshot::channel();
+        let occupied = tokio::spawn(cleanup_runtime::disk_operation(async move {
+            entered.send(()).unwrap();
+            released.await.unwrap();
+            Ok(())
+        }));
+        tokio::time::timeout(Duration::from_secs(20), entry).await.unwrap().unwrap();
+        assert!(matches!(wrapper.release_snapshot_lease(volume, path, lease).await,
+            Err(DiskError::Io(err)) if err.kind() == io::ErrorKind::WouldBlock));
+        assert_eq!(wrapper.read_all(volume, &body_path).await.unwrap(), Bytes::from_static(b"leased body"));
+        release.send(()).unwrap();
+        occupied.await.unwrap().unwrap();
+        assert_eq!(
+            wrapper
+                .delete_data_dir(
+                    volume,
+                    path,
+                    DeleteOptions {
+                        recursive: true,
+                        ..Default::default()
+                    }
+                )
+                .await
+                .unwrap(),
+            DataDirDeleteStatus::Deleted
+        );
+        assert!(matches!(wrapper.read_all(volume, &body_path).await, Err(DiskError::FileNotFound)));
+    }
+
     #[tokio::test]
     async fn owned_mutation_keeps_publication_guard_after_waiter_cancellation() {
         let drops = Arc::new(std::sync::atomic::AtomicUsize::new(0));

crates/ecstore/src/disk/error_reduce.rs

@@ -131,6 +131,15 @@ pub fn build_write_quorum_failure_summary(
 
 fn dominant_error_label(errors: &[Option<Error>], ignored_errs: &[Error], dominant_error: Option<&Error>) -> &'static str {
     let Some(dominant_error) = dominant_error else {
+        // The reducer also returns None when every failure is ignored for
+        // quorum voting. That case contains no successful nil entry.
+        if !errors.is_empty() && errors.iter().all(Option::is_some) {
+            return if errors.iter().all(|error| error.as_ref() == Some(&Error::DiskNotFound)) {
+                "disk_not_found"
+            } else {
+                "all_ignored"
+            };
+        }
         return "nil_dominated";
     };
 
@@ -387,9 +396,9 @@ mod tests {
     }
 
     #[test]
-    fn test_build_write_quorum_failure_summary_all_ignored_is_nil_dominated() {
-        // All disks offline (all DiskNotFound). The dominant error must be None and
-        // the label must be "nil_dominated" rather than falling back to "other_error".
+    fn test_build_write_quorum_failure_summary_all_missing_is_disk_not_found() {
+        // Ignoring unavailable disks for quorum reduction must not imply a
+        // successful nil entry in the diagnostic label.
         let errors = vec![
             Some(Error::DiskNotFound),
             Some(Error::DiskNotFound),
@@ -397,7 +406,21 @@ mod tests {
         ];
         let summary = build_write_quorum_failure_summary(&errors, OBJECT_OP_IGNORED_ERRS, 2);
         assert_eq!(summary.dominant_error, None);
-        assert_eq!(summary.dominant_error_label, "nil_dominated");
+        assert_eq!(summary.achieved, 0);
+        assert_eq!(summary.dominant_error_label, "disk_not_found");
+    }
+
+    #[test]
+    fn test_build_write_quorum_failure_summary_all_ignored_has_no_success() {
+        let errors = [Some(Error::FaultyDisk), Some(Error::DiskNotFound)];
+        let summary = build_write_quorum_failure_summary(&errors, OBJECT_OP_IGNORED_ERRS, 2);
+        assert_eq!(summary.dominant_error, None);
+        assert_eq!(summary.achieved, 0);
+        assert_eq!(summary.dominant_error_label, "all_ignored");
+        assert_eq!(
+            reduce_write_quorum_errs(&errors, OBJECT_OP_IGNORED_ERRS, 2),
+            Some(Error::ErasureWriteQuorum)
+        );
     }
 
     #[test]

crates/ecstore/src/disk/fs.rs

@@ -117,6 +117,9 @@ pub const O_APPEND: FileMode = 0x00400;
 //     create_new: bool,
 
 pub async fn open_file(path: impl AsRef<Path>, mode: FileMode) -> io::Result<File> {
+    if mode & O_CREATE != 0 {
+        super::os::record_native_publication(path.as_ref());
+    }
     let base_opts = match mode & (O_RDONLY | O_WRONLY | O_RDWR) {
         O_RDONLY => get_readonly_options(),
         O_WRONLY => get_writeonly_options(),
@@ -160,6 +163,7 @@ pub fn lstat_std(path: impl AsRef<Path>) -> io::Result<Metadata> {
 }
 
 pub async fn make_dir_all(path: impl AsRef<Path>) -> io::Result<()> {
+    super::os::record_native_publication(path.as_ref());
     fs::create_dir_all(path.as_ref()).await
 }
 
@@ -211,24 +215,21 @@ pub fn remove_all_std(path: impl AsRef<Path>) -> io::Result<()> {
 }
 
 pub async fn mkdir(path: impl AsRef<Path>) -> io::Result<()> {
+    super::os::record_native_publication(path.as_ref());
     fs::create_dir(path.as_ref()).await
 }
 
 pub async fn rename(from: impl AsRef<Path>, to: impl AsRef<Path>) -> io::Result<()> {
+    super::os::record_native_publication(to.as_ref());
     fs::rename(from, to).await
 }
 
 #[cfg(any(not(windows), test))]
 pub fn rename_std(from: impl AsRef<Path>, to: impl AsRef<Path>) -> io::Result<()> {
+    super::os::record_native_publication(to.as_ref());
     std::fs::rename(from, to)
 }
 
-#[cfg(any(not(windows), test))]
-#[tracing::instrument(level = "debug", skip_all)]
-pub async fn read_file(path: impl AsRef<Path>) -> io::Result<Vec<u8>> {
-    fs::read(path.as_ref()).await
-}
-
 // Bucket existence cache - reduces statx syscalls for repeated bucket checks
 
 /// Cache for bucket directory existence checks.
@@ -614,28 +615,6 @@ mod tests {
         assert_eq!(content, "test content");
     }
 
-    #[tokio::test]
-    async fn test_read_file() {
-        let temp_dir = TempDir::new().unwrap();
-        let file_path = temp_dir.path().join("test_read.txt");
-
-        let test_content = b"This is test content for reading";
-        tokio::fs::write(&file_path, test_content).await.unwrap();
-
-        // Read file
-        let read_content = read_file(&file_path).await.unwrap();
-        assert_eq!(read_content, test_content);
-    }
-
-    #[tokio::test]
-    async fn test_read_file_nonexistent() {
-        let temp_dir = TempDir::new().unwrap();
-        let file_path = temp_dir.path().join("nonexistent.txt");
-
-        // Should fail for non-existent file
-        assert!(read_file(&file_path).await.is_err());
-    }
-
     #[tokio::test]
     async fn test_same_file() {
         let temp_dir = TempDir::new().unwrap();

crates/ecstore/src/disk/local/commit.rs

@@ -47,6 +47,33 @@ use tokio::fs;
 use tracing::{info, warn};
 use uuid::Uuid;
 
+fn transitioned_old_current_source(
+    dst_meta_existed: bool,
+    dst_meta_unparsable: bool,
+    xlmeta: &FileMeta,
+    volume: &str,
+    path: &str,
+    version_id: Uuid,
+) -> (bool, Option<FileInfo>) {
+    if dst_meta_unparsable {
+        return (false, None);
+    }
+    if !dst_meta_existed {
+        return (true, None);
+    }
+    let Ok(old_current) = xlmeta.into_fileinfo(volume, path, "", false, false, true) else {
+        return (false, None);
+    };
+    if old_current.version_id.unwrap_or_default() != version_id {
+        return (true, None);
+    }
+    if old_current.transition_status == rustfs_filemeta::TRANSITION_COMPLETE {
+        (true, Some(old_current))
+    } else {
+        (true, None)
+    }
+}
+
 /// Hold later repair publications after admitting one baseline object. The
 /// fixture arms this on one replacement disk before rejoining the cluster.
 #[cfg(feature = "e2e-test-hooks")]
@@ -214,7 +241,7 @@ pub(super) async fn lock_rename_commit_directories(
     publication_root: &os::PublicationRoot,
     mutation_lease: Arc<os::NamespaceMutationLease>,
 ) -> Result<os::RenameCommitGuard> {
-    #[cfg(windows)]
+    #[cfg(any(windows, target_os = "linux"))]
     let result = {
         let source_parent = source_parent.to_path_buf();
         let destination_parent = destination_parent.to_path_buf();
@@ -222,7 +249,7 @@ pub(super) async fn lock_rename_commit_directories(
         let publication_root = publication_root.clone();
         os::run_blocking_namespace_operation(mutation_lease, move || {
             let result = os::prepare_rename_commit_guard(&source_parent, &destination_parent, &base_dir, &publication_root);
-            #[cfg(test)]
+            #[cfg(all(test, windows))]
             if result.is_ok() {
                 run_destination_commit_directory_preparation(&destination_parent);
             }
@@ -230,7 +257,7 @@ pub(super) async fn lock_rename_commit_directories(
         })
         .await
     };
-    #[cfg(not(windows))]
+    #[cfg(not(any(windows, target_os = "linux")))]
     let result = {
         let _ = mutation_lease;
         os::prepare_rename_commit_guard(source_parent, destination_parent, base_dir, publication_root)
@@ -244,12 +271,12 @@ pub(super) async fn lock_rename_commit_directories(
     result.map_err(to_file_error).map_err(DiskError::from)
 }
 
-async fn read_rename_destination_metadata(
+async fn prepare_rename_destination_metadata(
     file_path: &Path,
     rename_commit_guard: &os::RenameCommitGuard,
     mutation_lease: Arc<os::NamespaceMutationLease>,
 ) -> Result<Option<Bytes>> {
-    #[cfg(windows)]
+    #[cfg(any(windows, target_os = "linux"))]
     let result = {
         let file_path = file_path.to_path_buf();
         let rename_commit_guard = rename_commit_guard.clone();
@@ -258,10 +285,10 @@ async fn read_rename_destination_metadata(
         })
         .await
     };
-    #[cfg(not(windows))]
+    #[cfg(not(any(windows, target_os = "linux")))]
     let _ = (rename_commit_guard, mutation_lease);
-    #[cfg(not(windows))]
-    let result = match super::super::fs::read_file(file_path).await {
+    #[cfg(not(any(windows, target_os = "linux")))]
+    let result = match fs::read(file_path).await {
         Ok(data) => Ok(Some(data)),
         Err(err) if err.kind() == ErrorKind::NotFound => Ok(None),
         Err(err) => Err(err),
@@ -398,7 +425,7 @@ impl LocalDisk {
 
         // xl.meta path
         let src_file_path = self.io_get_object_path(src_volume, format!("{}/{}", src_path, STORAGE_FORMAT_FILE).as_str())?;
-        let dst_file_path = self.io_get_object_path(dst_volume, format!("{}/{}", dst_path, STORAGE_FORMAT_FILE).as_str())?;
+        let dst_file_path = self.io_get_object_open_path(dst_volume, format!("{}/{}", dst_path, STORAGE_FORMAT_FILE).as_str())?;
 
         // data_dir path
         let has_data_dir_path = {
@@ -453,8 +480,7 @@ impl LocalDisk {
             fs::create_dir_all(src_file_parent).await.map_err(to_file_error)?;
         }
         // Acquire the common trees before reading destination metadata. On
-        // Windows this pins the object directory identity across metadata
-        // preparation, data publication, rollback backup, and final commit.
+        // Windows this pins the object directory identity across publication.
         let rename_commit_guard = lock_rename_commit_directories(
             src_file_parent,
             dst_file_parent,
@@ -472,9 +498,12 @@ impl LocalDisk {
             source_volume_dir: &src_volume_dir,
             data_paths: has_data_dir_path.as_ref(),
         };
-        let has_dst_buf =
-            read_rename_destination_metadata(commit.destination_metadata, &commit.directory_guard, commit.mutation_lease.clone())
-                .await?;
+        let has_dst_buf = prepare_rename_destination_metadata(
+            commit.destination_metadata,
+            &commit.directory_guard,
+            commit.mutation_lease.clone(),
+        )
+        .await?;
 
         if no_inline {
             // Non-inline: read xl.meta, parse, write, rename data dir, rename xl.meta
@@ -508,6 +537,14 @@ impl LocalDisk {
             }
 
             let version_id = fi.version_id.unwrap_or_default();
+            let (old_current_source_checked, old_current_source) = transitioned_old_current_source(
+                has_dst_buf.is_some(),
+                dst_meta_unparsable,
+                &xlmeta,
+                dst_volume,
+                dst_path,
+                version_id,
+            );
             let has_old_data_dir = xlmeta.find_unshared_data_dir_for_version(Some(version_id));
             let old_version_exists = xlmeta.find_version(Some(version_id)).is_ok();
             let rollback_data_dir = has_old_data_dir.or_else(|| {
@@ -798,7 +835,9 @@ impl LocalDisk {
                 && let Some(parent) = dst_file_path.parent()
             {
                 let fsync_started = rustfs_io_metrics::put_stage_timer();
-                if let Err(err) = os::fsync_dst_dir_group_commit(parent, Some(commit.mutation_lease.clone())).await {
+                if let Err(err) =
+                    os::fsync_commit_directory(parent, &commit.directory_guard, commit.mutation_lease.clone(), None).await
+                {
                     rustfs_io_metrics::record_put_object_stage_duration_from(
                         rustfs_io_metrics::PUT_STAGE_SET_DISK_RENAME_DST_DIR_FSYNC,
                         fsync_started,
@@ -897,6 +936,8 @@ impl LocalDisk {
                 cleanup_data_dir: has_old_data_dir,
                 sign: version_signature,
                 old_current_size,
+                old_current_source_checked,
+                old_current_source,
             })
         } else {
             // Inline metadata preparation is blocking. The transaction lease is
@@ -910,6 +951,8 @@ impl LocalDisk {
                 None
             };
             let dst_path_for_failpoint = dst_path.to_string();
+            let dst_volume_for_source = dst_volume.to_string();
+            let dst_path_for_source = dst_path.to_string();
             #[cfg(windows)]
             let source_parent = src_file_parent.to_path_buf();
             let rename_commit_guard_for_preparation = commit.directory_guard.clone();
@@ -954,6 +997,14 @@ impl LocalDisk {
                 };
 
                 let version_id = fi.version_id.unwrap_or_default();
+                let (old_current_source_checked, old_current_source) = transitioned_old_current_source(
+                    has_dst_buf.is_some(),
+                    dst_meta_unparsable,
+                    &xlmeta,
+                    &dst_volume_for_source,
+                    &dst_path_for_source,
+                    version_id,
+                );
                 let old_data_dir = xlmeta.find_unshared_data_dir_for_version(Some(version_id));
                 let old_version_exists = xlmeta.find_version(Some(version_id)).is_ok();
                 let rollback_data_dir = old_data_dir.or_else(|| {
@@ -1003,6 +1054,8 @@ impl LocalDisk {
                     old_data_dir,
                     version_signature,
                     old_current_size,
+                    old_current_source_checked,
+                    old_current_source,
                     staged_rollback_path,
                     has_dst_buf.is_none(),
                     prepared_metadata_source,
@@ -1022,6 +1075,8 @@ impl LocalDisk {
                 cleanup_data_dir,
                 version_signature,
                 old_current_size,
+                old_current_source_checked,
+                old_current_source,
                 mut local_rollback_path,
                 destination_was_absent,
                 prepared_metadata_source,
@@ -1147,10 +1202,11 @@ impl LocalDisk {
                     && let Some(dst_parent) = dst_file_path.parent()
                 {
                     let fsync_started = rustfs_io_metrics::put_stage_timer();
-                    if let Err(err) = os::fsync_dst_dir_group_commit_or_namespace_file_sync_limit(
+                    if let Err(err) = os::fsync_commit_directory(
                         dst_parent,
+                        &commit.directory_guard,
                         commit.mutation_lease.clone(),
-                        admission,
+                        Some(admission),
                     )
                     .await
                     {
@@ -1262,6 +1318,8 @@ impl LocalDisk {
                 cleanup_data_dir,
                 sign: version_signature,
                 old_current_size,
+                old_current_source_checked,
+                old_current_source,
             })
         }
     }

crates/ecstore/src/disk/mod.rs

@@ -14,6 +14,7 @@
 
 // #730: disk abstractions still carry staged health and direct-I/O migration paths.
 
+pub(crate) mod cleanup_runtime;
 pub mod disk_store;
 pub mod endpoint;
 pub mod error;
@@ -230,6 +231,105 @@ pub enum Disk {
 }
 
 impl Disk {
+    pub(crate) async fn delete_versions_with_namespace_owner(
+        &self,
+        volume: &str,
+        versions: Vec<FileInfoVersions>,
+        opts: DeleteOptions,
+        namespace_owner: Option<Arc<dyn Send + Sync>>,
+    ) -> Vec<Option<Error>> {
+        match self {
+            Self::Local(disk) => {
+                disk.delete_versions_with_namespace_owner(volume, versions, opts, namespace_owner)
+                    .await
+            }
+            Self::Remote(disk) => {
+                let _owner = namespace_owner;
+                disk.delete_versions(volume, versions, opts).await
+            }
+        }
+    }
+
+    pub(crate) async fn delete_versions_recoverable_with_namespace_owner(
+        &self,
+        volume: &str,
+        versions: Vec<FileInfoVersions>,
+        mut opts: DeleteOptions,
+        namespace_owner: Option<Arc<dyn Send + Sync>>,
+    ) -> Vec<Option<Error>> {
+        if let Err(error) = validate_delete_recovery_options(&opts) {
+            return vec![Some(error); versions.len()];
+        }
+        if versions.iter().any(|version| version.name.starts_with('/')) {
+            return vec![Some(DiskError::InvalidPath); versions.len()];
+        }
+        opts.recoverable_delete = true;
+        match self {
+            Self::Local(disk) => {
+                disk.delete_versions_with_namespace_owner(volume, versions, opts, namespace_owner)
+                    .await
+            }
+            Self::Remote(disk) => {
+                let _owner = namespace_owner;
+                disk.delete_versions_recoverable(volume, versions, opts).await
+            }
+        }
+    }
+
+    pub(crate) async fn settle_delete_transaction_with_namespace_owner(
+        &self,
+        volume: &str,
+        object: &str,
+        transaction: Uuid,
+        expected_fingerprint: DeleteRecoveryDigest,
+        action: DeleteRecoveryAction,
+        namespace_owner: Option<Arc<dyn Send + Sync>>,
+    ) -> Result<DeleteRecoveryInspection> {
+        match self {
+            Self::Local(disk) => {
+                disk.settle_delete_transaction_with_namespace_owner(
+                    volume,
+                    object,
+                    transaction,
+                    expected_fingerprint,
+                    action,
+                    namespace_owner,
+                )
+                .await
+            }
+            Self::Remote(disk) => {
+                let _owner = namespace_owner;
+                disk.settle_delete_transaction(volume, object, transaction, expected_fingerprint, action)
+                    .await
+            }
+        }
+    }
+
+    pub(crate) async fn delete_version_recoverable_with_namespace_owner(
+        &self,
+        volume: &str,
+        path: &str,
+        fi: FileInfo,
+        force_del_marker: bool,
+        mut opts: DeleteOptions,
+        namespace_owner: Option<Arc<dyn Send + Sync>>,
+    ) -> Result<()> {
+        validate_delete_recovery_options(&opts)?;
+        if path.starts_with('/') {
+            return Err(DiskError::InvalidPath);
+        }
+        opts.recoverable_delete = true;
+        match self {
+            Self::Local(disk) => {
+                disk.delete_version_with_namespace_owner(volume, path, fi, force_del_marker, opts, namespace_owner)
+                    .await
+            }
+            Self::Remote(disk) => {
+                disk.delete_version_recoverable(volume, path, fi, force_del_marker, opts)
+                    .await
+            }
+        }
+    }
     pub(crate) async fn set_disk_id_state(&self, id: Option<Uuid>) -> Result<()> {
         match self {
             Disk::Local(local_disk) => {
@@ -257,6 +357,70 @@ impl Disk {
 
 #[async_trait::async_trait]
 impl DiskAPI for Disk {
+    async fn delete_version_recoverable(
+        &self,
+        volume: &str,
+        object: &str,
+        file_info: FileInfo,
+        force_del_marker: bool,
+        opts: DeleteOptions,
+    ) -> Result<()> {
+        self.delete_version_recoverable_with_namespace_owner(volume, object, file_info, force_del_marker, opts, None)
+            .await
+    }
+    async fn delete_versions_recoverable(
+        &self,
+        volume: &str,
+        versions: Vec<FileInfoVersions>,
+        opts: DeleteOptions,
+    ) -> Vec<Option<DiskError>> {
+        match self {
+            Self::Local(disk) => disk.delete_versions_recoverable(volume, versions, opts).await,
+            Self::Remote(disk) => disk.delete_versions_recoverable(volume, versions, opts).await,
+        }
+    }
+    async fn discover_delete_transactions(
+        &self,
+        volume: &str,
+        object: Option<&str>,
+        cursor: Option<DeleteRecoveryCursor>,
+        limit: u32,
+    ) -> Result<DeleteRecoveryPage> {
+        match self {
+            Self::Local(disk) => disk.discover_delete_transactions(volume, object, cursor, limit).await,
+            Self::Remote(disk) => disk.discover_delete_transactions(volume, object, cursor, limit).await,
+        }
+    }
+    async fn inspect_delete_transaction(
+        &self,
+        volume: &str,
+        object: &str,
+        transaction: Uuid,
+    ) -> Result<DeleteRecoveryInspection> {
+        match self {
+            Self::Local(disk) => disk.inspect_delete_transaction(volume, object, transaction).await,
+            Self::Remote(disk) => disk.inspect_delete_transaction(volume, object, transaction).await,
+        }
+    }
+    async fn settle_delete_transaction(
+        &self,
+        volume: &str,
+        object: &str,
+        transaction: Uuid,
+        expected_fingerprint: DeleteRecoveryDigest,
+        action: DeleteRecoveryAction,
+    ) -> Result<DeleteRecoveryInspection> {
+        match self {
+            Self::Local(disk) => {
+                disk.settle_delete_transaction(volume, object, transaction, expected_fingerprint, action)
+                    .await
+            }
+            Self::Remote(disk) => {
+                disk.settle_delete_transaction(volume, object, transaction, expected_fingerprint, action)
+                    .await
+            }
+        }
+    }
     fn to_string(&self) -> String {
         match self {
             Disk::Local(local_disk) => local_disk.to_string(),
@@ -327,6 +491,13 @@ impl DiskAPI for Disk {
         }
     }
 
+    async fn bucket_creation_witness(&self, volume: &str) -> Result<Option<os::NativeBucketCreationWitness>> {
+        match self {
+            Disk::Local(local_disk) => local_disk.bucket_creation_witness(volume).await,
+            Disk::Remote(_) => Ok(None),
+        }
+    }
+
     #[tracing::instrument(level = "trace", skip_all)]
     async fn make_volume(&self, volume: &str) -> Result<()> {
         match self {
@@ -1121,6 +1292,11 @@ pub trait DiskAPI: Debug + Send + Sync + 'static {
     // NSScanner
 
     // Volume operations.
+    /// Unknown or remote backends cannot attest process-local publication history.
+    async fn bucket_creation_witness(&self, _volume: &str) -> Result<Option<os::NativeBucketCreationWitness>> {
+        Ok(None)
+    }
+
     async fn make_volume(&self, volume: &str) -> Result<()>;
     async fn make_volumes(&self, volume: Vec<&str>) -> Result<()>;
     async fn list_volumes(&self) -> Result<Vec<VolumeInfo>>;
@@ -1246,6 +1422,51 @@ pub trait DiskAPI: Debug + Send + Sync + 'static {
         Err(DiskError::MethodNotAllowed)
     }
     async fn delete(&self, volume: &str, path: &str, opt: DeleteOptions) -> Result<()>;
+    async fn discover_delete_transactions(
+        &self,
+        _volume: &str,
+        _object: Option<&str>,
+        _cursor: Option<DeleteRecoveryCursor>,
+        _limit: u32,
+    ) -> Result<DeleteRecoveryPage> {
+        Err(DiskError::MethodNotAllowed)
+    }
+    async fn delete_version_recoverable(
+        &self,
+        _volume: &str,
+        _object: &str,
+        _file_info: FileInfo,
+        _force_del_marker: bool,
+        _opts: DeleteOptions,
+    ) -> Result<()> {
+        Err(DiskError::MethodNotAllowed)
+    }
+    async fn delete_versions_recoverable(
+        &self,
+        _volume: &str,
+        versions: Vec<FileInfoVersions>,
+        _opts: DeleteOptions,
+    ) -> Vec<Option<DiskError>> {
+        vec![Some(DiskError::MethodNotAllowed); versions.len()]
+    }
+    async fn inspect_delete_transaction(
+        &self,
+        _volume: &str,
+        _object: &str,
+        _transaction: Uuid,
+    ) -> Result<DeleteRecoveryInspection> {
+        Err(DiskError::MethodNotAllowed)
+    }
+    async fn settle_delete_transaction(
+        &self,
+        _volume: &str,
+        _object: &str,
+        _transaction: Uuid,
+        _expected_fingerprint: DeleteRecoveryDigest,
+        _action: DeleteRecoveryAction,
+    ) -> Result<DeleteRecoveryInspection> {
+        Err(DiskError::MethodNotAllowed)
+    }
     // VerifyFile
     async fn verify_file(&self, volume: &str, path: &str, fi: &FileInfo) -> Result<CheckPartsResp>;
     // CheckParts
@@ -1568,6 +1789,14 @@ pub struct RenameDataResp {
     /// "cannot vote", never as `Absent`.
     #[serde(default)]
     pub old_current_size: Option<OldCurrentSize>,
+    /// Whether this disk decoded enough of the overwritten current version to
+    /// decide if a transitioned source has to be returned.
+    #[serde(default)]
+    pub old_current_source_checked: bool,
+    /// Transitioned old-current source used by PUT free-version cleanup receipt.
+    /// Empty when the old current is not transitioned or no source was known.
+    #[serde(default)]
+    pub old_current_source: Option<FileInfo>,
 }
 
 #[derive(Debug, Clone, Default, Serialize, Deserialize)]
@@ -1582,6 +1811,269 @@ pub struct DeleteOptions {
     /// Remote calls carrying it must use DeleteRetiredMarker, never DeleteVersion.
     #[serde(default, skip_serializing_if = "Option::is_none")]
     pub expected_delete_marker: Option<rustfs_filemeta::MetaDeleteMarker>,
+    #[serde(default, skip_serializing_if = "Option::is_none")]
+    pub delete_recovery_bucket_incarnation: Option<Uuid>,
+    #[serde(default)]
+    pub recoverable_delete: bool,
+}
+
+pub const DELETE_RECOVERY_PROTOCOL_VERSION: u32 = 1;
+pub(crate) fn get_delete_recovery_timeout() -> Result<std::time::Duration> {
+    let value = std::env::var("RUSTFS_DELETE_RECOVERY_TIMEOUT_SECS");
+    match value {
+        Ok(value) => parse_delete_recovery_timeout(Some(&value)),
+        Err(std::env::VarError::NotPresent) => parse_delete_recovery_timeout(None),
+        Err(std::env::VarError::NotUnicode(_)) => Err(DiskError::other("invalid delete recovery timeout configuration")),
+    }
+}
+
+fn parse_delete_recovery_timeout(value: Option<&str>) -> Result<std::time::Duration> {
+    let seconds = match value {
+        None => 900,
+        Some(value) => value
+            .parse::<u64>()
+            .ok()
+            .filter(|seconds| (30..=86400).contains(seconds))
+            .ok_or_else(|| DiskError::other("delete recovery timeout must be an integer between 30 and 86400 seconds"))?,
+    };
+    Ok(std::time::Duration::from_secs(seconds))
+}
+
+pub(crate) fn validate_delete_recovery_options(opts: &DeleteOptions) -> Result<()> {
+    if opts.old_data_dir.is_none_or(|id| id.is_nil())
+        || opts.undo_write
+        || opts.undo_delete
+        || opts.recursive
+        || opts.immediate
+        || opts.expected_delete_marker.is_some()
+        || opts.delete_recovery_bucket_incarnation.is_some_and(|id| id.is_nil())
+    {
+        return Err(DiskError::FileCorrupt);
+    }
+    Ok(())
+}
+pub type DeleteRecoveryDigest = [u8; 32];
+
+#[derive(Debug, Clone, PartialEq, Eq, Serialize, Deserialize)]
+#[serde(deny_unknown_fields)]
+pub struct DeleteTransactionId {
+    pub object: String,
+    pub transaction: Uuid,
+}
+
+#[derive(Debug, Clone, PartialEq, Eq, Serialize, Deserialize)]
+#[serde(deny_unknown_fields)]
+pub struct DeleteRecoveryCursor {
+    pub token: Uuid,
+}
+
+#[derive(Debug, Clone, PartialEq, Eq, Serialize, Deserialize)]
+#[serde(deny_unknown_fields)]
+pub struct DeleteRecoveryPage {
+    pub transactions: Vec<DeleteTransactionId>,
+    pub next: Option<DeleteRecoveryCursor>,
+}
+
+#[derive(Debug, Clone, PartialEq, Serialize, Deserialize)]
+#[serde(deny_unknown_fields)]
+pub struct DeleteRecoveryJournal {
+    pub protocol_version: u32,
+    pub volume: String,
+    pub object: String,
+    pub transaction: Uuid,
+    pub bucket_incarnation: Option<Uuid>,
+    pub original_digest: Option<DeleteRecoveryDigest>,
+    pub target_digest: Option<DeleteRecoveryDigest>,
+    pub data_dirs: Vec<Uuid>,
+    pub deleted_versions: Vec<Uuid>,
+    pub requests: Vec<FileInfo>,
+    pub original_versions: Vec<DeleteRecoveryVersionIdentity>,
+    #[serde(default)]
+    pub predecessors: Vec<Uuid>,
+}
+
+impl DeleteRecoveryJournal {
+    /// Checks source generations only; the coordinator must supply a certified
+    /// decision and the receiver must retain its namespace/incarnation fences.
+    pub(crate) fn matches_adoption_source(&self, current: &[DeleteRecoveryVersionIdentity]) -> bool {
+        if self.requests.is_empty() {
+            return false;
+        }
+        let mut originals = std::collections::HashMap::with_capacity(self.original_versions.len());
+        for version in &self.original_versions {
+            originals.entry(version.version_id.unwrap_or_default()).or_insert(version);
+        }
+        let mut locals = std::collections::HashMap::with_capacity(current.len());
+        for version in current {
+            locals.entry(version.version_id.unwrap_or_default()).or_insert(version);
+        }
+        self.requests.iter().all(|request| {
+            if request.deleted && request.mark_deleted {
+                return current.first() == self.original_versions.first();
+            }
+            let id = request.version_id.unwrap_or_default();
+            let Some(original) = originals.get(&id) else {
+                return false;
+            };
+            match locals.get(&id) {
+                Some(local) => local == original,
+                None => !id.is_nil(),
+            }
+        })
+    }
+
+    /// Matches immutable local preparation intent against an independently
+    /// inspected same-transaction commit receipt. This match is not authority
+    /// by itself; the coordinator must supply the certified receipt.
+    pub(crate) fn matches_prepared_authority(&self, authority: &Self) -> bool {
+        if self.protocol_version != DELETE_RECOVERY_PROTOCOL_VERSION
+            || authority.protocol_version != DELETE_RECOVERY_PROTOCOL_VERSION
+            || self.transaction.is_nil()
+            || self.transaction != authority.transaction
+            || self.volume != authority.volume
+            || self.object != authority.object
+            || self.bucket_incarnation != authority.bucket_incarnation
+            || self.bucket_incarnation.is_some_and(|id| id.is_nil())
+            || self.requests.is_empty()
+            || self.requests != authority.requests
+            || self.deleted_versions.iter().collect::<std::collections::HashSet<_>>()
+                != authority.deleted_versions.iter().collect::<std::collections::HashSet<_>>()
+        {
+            return false;
+        }
+        let sources: std::collections::HashMap<_, _> = self
+            .original_versions
+            .iter()
+            .map(|version| (version.version_id.unwrap_or_default(), version))
+            .collect();
+        let certified: std::collections::HashMap<_, _> = authority
+            .original_versions
+            .iter()
+            .map(|version| (version.version_id.unwrap_or_default(), version))
+            .collect();
+        if sources.len() != self.original_versions.len() || certified.len() != authority.original_versions.len() {
+            return false;
+        }
+        self.requests.iter().all(|request| {
+            let (local, remote) = if request.deleted && request.mark_deleted {
+                (self.original_versions.first(), authority.original_versions.first())
+            } else {
+                let id = request.version_id.unwrap_or_default();
+                (sources.get(&id).copied(), certified.get(&id).copied())
+            };
+            match (local, remote) {
+                (Some(local), Some(remote)) => local == remote,
+                (None, None) => self.original_digest.is_none() && authority.original_digest.is_none(),
+                _ => false,
+            }
+        })
+    }
+}
+
+#[derive(Debug, Clone, PartialEq, Eq, Hash, Serialize, Deserialize)]
+#[serde(deny_unknown_fields)]
+pub struct DeleteRecoveryVersionIdentity {
+    pub version_id: Option<Uuid>,
+    pub mod_time: Option<OffsetDateTime>,
+    pub data_dir: Option<Uuid>,
+    pub deleted: bool,
+    pub metadata_digest: DeleteRecoveryDigest,
+}
+
+impl DeleteRecoveryVersionIdentity {
+    pub(crate) fn from_file_info(info: FileInfo) -> error::Result<Self> {
+        local::delete_recovery::generation(info)
+    }
+}
+
+#[derive(Debug, Clone, Copy, PartialEq, Eq, Serialize, Deserialize)]
+pub enum DeleteRecoveryState {
+    Absent,
+    NewPrepared,
+    QuorumCommitted,
+    CommitFinished,
+    LegacyAmbiguous,
+}
+
+#[derive(Debug, Clone, PartialEq, Serialize, Deserialize)]
+#[serde(deny_unknown_fields)]
+pub struct DeleteRecoveryInspection {
+    pub protocol_version: u32,
+    pub id: DeleteTransactionId,
+    pub state: DeleteRecoveryState,
+    pub journal: Option<DeleteRecoveryJournal>,
+    pub current_digest: Option<DeleteRecoveryDigest>,
+    pub current_versions: Vec<DeleteRecoveryVersionIdentity>,
+    pub backup_digest: Option<DeleteRecoveryDigest>,
+    pub rollback_data_complete: bool,
+    pub commit_data_safe: bool,
+    pub explicit_delete_safe: bool,
+    pub fingerprint: DeleteRecoveryDigest,
+}
+
+#[derive(Debug, Clone, PartialEq, Serialize, Deserialize)]
+pub enum DeleteRecoveryAction {
+    PublishCommit,
+    FinishCommit,
+    FinalizeCommit,
+    Rollback,
+    DiscardLegacy,
+    AdoptCommit { authority: Box<DeleteRecoveryJournal> },
+}
+
+#[derive(Debug, Clone, Serialize, Deserialize)]
+#[serde(deny_unknown_fields)]
+pub enum DeleteRecoveryRpcOperation {
+    Discover {
+        object: Option<String>,
+        cursor: Option<DeleteRecoveryCursor>,
+        limit: u32,
+    },
+    Inspect {
+        object: String,
+        transaction: Uuid,
+    },
+    Settle {
+        object: String,
+        transaction: Uuid,
+        expected_fingerprint: DeleteRecoveryDigest,
+        action: DeleteRecoveryAction,
+    },
+    ApplyVersion {
+        object: String,
+        file_info: Box<FileInfo>,
+        force_del_marker: bool,
+        opts: DeleteOptions,
+    },
+    ApplyVersions {
+        versions: Vec<FileInfoVersions>,
+        opts: DeleteOptions,
+    },
+}
+
+impl DeleteRecoveryRpcOperation {
+    /// Full metadata application and adoption share the existing mutation envelope.
+    pub fn max_wire_payload_bytes(&self) -> usize {
+        match self {
+            Self::ApplyVersion { .. }
+            | Self::ApplyVersions { .. }
+            | Self::Settle {
+                action: DeleteRecoveryAction::AdoptCommit { .. },
+                ..
+            } => rustfs_protos::DELETE_RECOVERY_MAX_PAYLOAD_BYTES,
+            _ => rustfs_protos::DELETE_RECOVERY_MAX_CONTROL_PAYLOAD_BYTES,
+        }
+    }
+}
+
+#[derive(Debug, Clone, Serialize, Deserialize)]
+#[serde(deny_unknown_fields)]
+pub enum DeleteRecoveryRpcReply {
+    Discovered(DeleteRecoveryPage),
+    Inspected(DeleteRecoveryInspection),
+    Settled(DeleteRecoveryInspection),
+    AppliedVersion,
+    AppliedVersions(Vec<Option<u32>>),
 }
 
 #[derive(Debug, Clone, Serialize, Deserialize)]
@@ -1872,6 +2364,8 @@ mod tests {
             undo_delete: false,
             old_data_dir: Some(Uuid::new_v4()),
             expected_delete_marker: None,
+            delete_recovery_bucket_incarnation: None,
+            recoverable_delete: false,
         };
 
         assert!(opts.recursive);
@@ -2044,6 +2538,8 @@ mod tests {
             cleanup_data_dir: Some(uuid),
             sign: Some(signature.clone()),
             old_current_size: Some(OldCurrentSize::Present(42)),
+            old_current_source_checked: false,
+            old_current_source: None,
         };
 
         assert_eq!(resp.old_data_dir, Some(uuid));
@@ -2064,6 +2560,8 @@ mod tests {
                 cleanup_data_dir: Some(Uuid::new_v4()),
                 sign: Some(vec![0x01, 0x02, 0x03]),
                 old_current_size,
+                old_current_source_checked: false,
+                old_current_source: None,
             };
 
             let encoded = rmp_serde::encode::to_vec_named(&resp).expect("named msgpack should encode");
@@ -2101,6 +2599,8 @@ mod tests {
         assert_eq!(decoded.cleanup_data_dir, None);
         assert_eq!(decoded.sign, legacy.sign);
         assert_eq!(decoded.old_current_size, None);
+        assert!(!decoded.old_current_source_checked);
+        assert!(decoded.old_current_source.is_none());
     }
 
     /// Test constants
@@ -2296,3 +2796,183 @@ mod tests {
         }
     }
 }
+
+#[cfg(test)]
+mod delete_recovery_timeout_tests {
+    use super::*;
+    #[test]
+    fn recovery_timeout_defaults_and_validates_range() {
+        assert_eq!(
+            parse_delete_recovery_timeout(None).expect("default timeout"),
+            std::time::Duration::from_secs(900)
+        );
+        for seconds in ["30", "900", "86400"] {
+            assert!(parse_delete_recovery_timeout(Some(seconds)).is_ok());
+        }
+        for value in ["", "invalid", "0", "2", "29", "86401", "-1", "18446744073709551616"] {
+            assert!(parse_delete_recovery_timeout(Some(value)).is_err());
+        }
+    }
+}
+
+#[cfg(test)]
+mod delete_recovery_wire_limit_tests {
+    use super::*;
+
+    #[test]
+    fn boxed_apply_version_preserves_existing_wire_payload() {
+        #[derive(Serialize)]
+        enum PriorOperation<'a> {
+            ApplyVersion {
+                object: &'a str,
+                file_info: &'a FileInfo,
+                force_del_marker: bool,
+                opts: &'a DeleteOptions,
+            },
+        }
+        let file_info = FileInfo {
+            name: "object".into(),
+            version_id: Some(Uuid::new_v4()),
+            metadata: std::collections::HashMap::from([("metadata".into(), "value".into())]),
+            ..Default::default()
+        };
+        let opts = DeleteOptions {
+            old_data_dir: Some(Uuid::new_v4()),
+            recoverable_delete: true,
+            delete_recovery_bucket_incarnation: Some(Uuid::new_v4()),
+            ..Default::default()
+        };
+        let prior = PriorOperation::ApplyVersion {
+            object: "object",
+            file_info: &file_info,
+            force_del_marker: true,
+            opts: &opts,
+        };
+        let encoded = rmp_serde::to_vec_named(&prior).expect("encode the previous unboxed request shape");
+        let operation = DeleteRecoveryRpcOperation::ApplyVersion {
+            object: "object".into(),
+            file_info: Box::new(file_info.clone()),
+            force_del_marker: true,
+            opts: opts.clone(),
+        };
+        assert_eq!(rmp_serde::to_vec_named(&operation).expect("encode boxed request"), encoded);
+        assert_eq!(
+            serde_json::to_value(&operation).expect("boxed request JSON"),
+            serde_json::to_value(prior).expect("previous request JSON")
+        );
+        let decoded: DeleteRecoveryRpcOperation =
+            rustfs_protos::decode_delete_recovery_payload(&encoded).expect("decode the previous payload");
+        assert_eq!(rmp_serde::to_vec_named(&decoded).expect("re-encode decoded prior request"), encoded);
+    }
+    #[test]
+    fn adoption_and_inspection_reply_support_large_version_journals() {
+        let transaction = Uuid::new_v4();
+        let request = FileInfo {
+            metadata: std::collections::HashMap::from([("metadata".into(), "x".repeat(2048))]),
+            ..Default::default()
+        };
+        let journal = DeleteRecoveryJournal {
+            protocol_version: DELETE_RECOVERY_PROTOCOL_VERSION,
+            volume: "bucket".into(),
+            object: "object".into(),
+            transaction,
+            bucket_incarnation: None,
+            original_digest: None,
+            target_digest: None,
+            data_dirs: Vec::new(),
+            deleted_versions: Vec::new(),
+            requests: vec![request; 1000],
+            original_versions: Vec::new(),
+            predecessors: Vec::new(),
+        };
+        let operation = DeleteRecoveryRpcOperation::Settle {
+            object: "object".into(),
+            transaction,
+            expected_fingerprint: [0; 32],
+            action: DeleteRecoveryAction::AdoptCommit {
+                authority: Box::new(journal.clone()),
+            },
+        };
+        let encoded = rmp_serde::to_vec_named(&operation).expect("encode large adoption");
+        assert!(encoded.len() > rustfs_protos::DELETE_RECOVERY_MAX_CONTROL_PAYLOAD_BYTES);
+        assert!(encoded.len() < operation.max_wire_payload_bytes());
+        let decoded: DeleteRecoveryRpcOperation =
+            rustfs_protos::decode_delete_recovery_payload(&encoded).expect("large adoption can decode");
+        assert_eq!(decoded.max_wire_payload_bytes(), rustfs_protos::DELETE_RECOVERY_MAX_PAYLOAD_BYTES);
+        let reply = DeleteRecoveryRpcReply::Inspected(DeleteRecoveryInspection {
+            protocol_version: DELETE_RECOVERY_PROTOCOL_VERSION,
+            id: DeleteTransactionId {
+                object: "object".into(),
+                transaction,
+            },
+            state: DeleteRecoveryState::QuorumCommitted,
+            journal: Some(journal),
+            current_digest: None,
+            current_versions: Vec::new(),
+            backup_digest: None,
+            rollback_data_complete: false,
+            commit_data_safe: true,
+            explicit_delete_safe: false,
+            fingerprint: [0; 32],
+        });
+        let encoded = rmp_serde::to_vec_named(&reply).expect("encode large inspection");
+        assert!(encoded.len() > rustfs_protos::DELETE_RECOVERY_MAX_CONTROL_PAYLOAD_BYTES);
+        assert!(rustfs_protos::decode_delete_recovery_payload::<DeleteRecoveryRpcReply>(&encoded).is_ok());
+        let inspect = DeleteRecoveryRpcOperation::Inspect {
+            object: "object".into(),
+            transaction,
+        };
+        assert_eq!(inspect.max_wire_payload_bytes(), rustfs_protos::DELETE_RECOVERY_MAX_CONTROL_PAYLOAD_BYTES);
+    }
+}
+
+#[cfg(test)]
+mod delete_recovery_adoption_tests {
+    use super::*;
+
+    #[test]
+    fn source_matching_preserves_non_null_absence_and_rejects_foreign_or_missing_null() {
+        let id = Uuid::new_v4();
+        let source = DeleteRecoveryVersionIdentity {
+            version_id: Some(id),
+            mod_time: None,
+            data_dir: None,
+            deleted: false,
+            metadata_digest: [1; 32],
+        };
+        let mut journal = DeleteRecoveryJournal {
+            protocol_version: DELETE_RECOVERY_PROTOCOL_VERSION,
+            volume: "bucket".into(),
+            object: "object".into(),
+            transaction: Uuid::new_v4(),
+            bucket_incarnation: None,
+            original_digest: None,
+            target_digest: None,
+            data_dirs: Vec::new(),
+            deleted_versions: vec![id],
+            requests: vec![FileInfo {
+                version_id: Some(id),
+                ..Default::default()
+            }],
+            original_versions: vec![source.clone()],
+            predecessors: Vec::new(),
+        };
+        assert!(journal.matches_adoption_source(&[]), "certified non-null removal can already be absent");
+        assert!(journal.matches_adoption_source(std::slice::from_ref(&source)));
+        let mut foreign = source.clone();
+        foreign.metadata_digest = [2; 32];
+        assert!(!journal.matches_adoption_source(std::slice::from_ref(&foreign)));
+        journal.original_versions.push(foreign.clone());
+        assert!(
+            journal.matches_adoption_source(&[source.clone(), foreign.clone()]),
+            "retain original first-match semantics"
+        );
+        assert!(!journal.matches_adoption_source(&[foreign, source]));
+        journal.requests[0].version_id = None;
+        journal.original_versions[0].version_id = None;
+        assert!(!journal.matches_adoption_source(&[]), "null absence does not establish its generation");
+        assert!(journal.matches_adoption_source(std::slice::from_ref(&journal.original_versions[0])));
+        journal.requests.clear();
+        assert!(!journal.matches_adoption_source(&journal.original_versions));
+    }
+}

crates/ecstore/src/error/mod.rs

@@ -17,8 +17,8 @@
 use crate::bucket::error::BucketMetadataError;
 use crate::disk::error::DiskError;
 use crate::storage_api_contracts::{error::StorageErrorCode, range::HTTPRangeError};
+use rustfs_s3_types::S3ErrorCode;
 use rustfs_utils::path::decode_dir_object;
-use s3s::S3ErrorCode;
 
 pub type Error = StorageError;
 pub type Result<T> = core::result::Result<T, Error>;

crates/ecstore/src/io_support/bitrot.rs

@@ -174,10 +174,14 @@ async fn open_reader_source(
     metrics_path: Option<&'static str>,
 ) -> disk::error::Result<Option<BoxedObjectReader>> {
     if let Some(data) = inline_data {
+        #[cfg(test)]
+        crate::set_disk::disk_call_counters::record_shard_open(path, true);
         let mut reader = Cursor::new(data);
         reader.set_position(u64::try_from(offset).map_err(|_| DiskError::FileCorrupt)?);
         Ok(Some(ShardReader::InMemory(reader)))
     } else if let Some(disk) = disk {
+        #[cfg(test)]
+        crate::set_disk::disk_call_counters::record_shard_open(path, false);
         open_disk_reader(disk, bucket, path, offset, length, use_mmap_read, metrics_path)
             .await
             .map(Some)

crates/ecstore/src/object_api/mod.rs

@@ -102,10 +102,14 @@ mod body_cache_hook;
 mod encryption;
 mod hook_slot;
 mod object_mutation_hook;
+mod read_condition;
 mod readers;
 mod types;
 pub(crate) mod write_commit_context;
 
+pub use read_condition::{GetObjectReadCondition, GetObjectReadMetadata, with_get_object_read_condition};
+pub(crate) use read_condition::{get_object_read_condition_is_active, get_object_read_condition_is_terminal};
+
 #[cfg(test)]
 mod persisted_metadata_keys_tests;
 

crates/ecstore/src/object_api/object_api_utils.rs

@@ -29,7 +29,7 @@ use crate::object_api::{GetObjectReader, ObjectInfo, ObjectOptions};
 use crate::storage_api_contracts::range::HTTPRangeSpec;
 use rustfs_filemeta::ObjectPartInfo;
 use rustfs_rio::HashReader;
-use s3s::S3ErrorCode;
+use rustfs_s3_types::S3ErrorCode;
 
 //#[derive(Clone)]
 pub struct PutObjReader {
@@ -231,6 +231,45 @@ mod tests {
         assert_eq!(result.2, 1024 * 1024);
     }
 
+    #[test]
+    fn test_new_getobjectreader_rejects_a_range_past_the_object_as_invalid_range() {
+        let oi = multipart_object_info();
+        let past_the_end = Some(HTTPRangeSpec {
+            is_suffix_length: false,
+            start: oi.size,
+            end: oi.size + 10,
+        });
+
+        let err = new_getobjectreader(&past_the_end, &oi, &ObjectOptions::default(), &HeaderMap::new())
+            .err()
+            .expect("a range starting at the object size must be rejected");
+
+        assert_eq!(err.code, S3ErrorCode::InvalidRange);
+        assert_eq!(err.code.as_str(), "InvalidRange");
+        assert_eq!(err.key, None);
+        assert_eq!(err.bucket_name, None);
+    }
+
+    #[test]
+    fn test_new_getobjectreader_rejects_an_unknown_size_without_a_range_as_invalid_range() {
+        let oi = ObjectInfo {
+            bucket: "bucket".to_string(),
+            name: "object".to_string(),
+            size: -1,
+            ..Default::default()
+        };
+
+        let err = new_getobjectreader(&None, &oi, &ObjectOptions::default(), &HeaderMap::new())
+            .err()
+            .expect("an object of unknown size cannot be read without a range");
+
+        assert_eq!(err.code, S3ErrorCode::InvalidRange);
+        assert_eq!(err.code.as_str(), "InvalidRange");
+        assert_eq!(err.message, "Invalid range");
+        assert_eq!(err.key.as_deref(), Some("object"));
+        assert_eq!(err.bucket_name.as_deref(), Some("bucket"));
+    }
+
     #[test]
     fn test_to_s3s_etag() {
         // Test unquoted ETag - should become strong etag

crates/ecstore/src/object_api/read_condition.rs

@@ -0,0 +1,113 @@
+// Copyright 2024 RustFS Team
+//
+// Licensed under the Apache License, Version 2.0 (the "License");
+// you may not use this file except in compliance with the License.
+// You may obtain a copy of the License at
+//
+//     http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing, software
+// distributed under the License is distributed on an "AS IS" BASIS,
+// WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+// See the License for the specific language governing permissions and
+// limitations under the License.
+
+use rustfs_filemeta::{FileInfo, ObjectPartInfo};
+use std::future::Future;
+use std::sync::Arc;
+use time::OffsetDateTime;
+
+/// Borrowed fields needed to decide whether an object read needs a body.
+pub struct GetObjectReadMetadata<'a> {
+    pub bucket: &'a str,
+    pub object: &'a str,
+    pub etag: Option<&'a str>,
+    pub mod_time: Option<OffsetDateTime>,
+    pub parts: &'a [ObjectPartInfo],
+    pub delete_marker: bool,
+}
+
+impl<'a> GetObjectReadMetadata<'a> {
+    pub(crate) fn from_file_info(bucket: &'a str, object: &'a str, info: &'a FileInfo) -> Self {
+        Self {
+            bucket,
+            object,
+            etag: info.metadata.get("etag").map(String::as_str),
+            mod_time: info.mod_time,
+            parts: &info.parts,
+            delete_marker: info.deleted,
+        }
+    }
+}
+
+/// A pure decision on quorum-validated metadata. Authorization belongs to the caller.
+pub trait GetObjectReadCondition: Send + Sync {
+    fn is_terminal(&self, metadata: GetObjectReadMetadata<'_>) -> bool;
+}
+
+tokio::task_local! {
+    static READ_CONDITION: Arc<dyn GetObjectReadCondition>;
+}
+
+/// Applies a condition to this reader call, without inheriting it into shared producers.
+pub async fn with_get_object_read_condition<F: Future>(condition: Arc<dyn GetObjectReadCondition>, read: F) -> F::Output {
+    READ_CONDITION.scope(condition, read).await
+}
+
+pub(crate) fn get_object_read_condition_is_terminal(metadata: GetObjectReadMetadata<'_>) -> bool {
+    READ_CONDITION
+        .try_with(|condition| condition.is_terminal(metadata))
+        .unwrap_or(false)
+}
+
+pub(crate) fn get_object_read_condition_is_active() -> bool {
+    READ_CONDITION.try_with(|_| ()).is_ok()
+}
+
+#[cfg(test)]
+mod tests {
+    use super::*;
+
+    struct Stop;
+    impl GetObjectReadCondition for Stop {
+        fn is_terminal(&self, _: GetObjectReadMetadata<'_>) -> bool {
+            true
+        }
+    }
+    fn metadata() -> GetObjectReadMetadata<'static> {
+        GetObjectReadMetadata {
+            bucket: "bucket",
+            object: "object",
+            etag: Some("etag"),
+            mod_time: None,
+            parts: &[],
+            delete_marker: false,
+        }
+    }
+
+    #[tokio::test]
+    async fn read_condition_scope_does_not_escape_or_enter_shared_producers() {
+        assert!(!get_object_read_condition_is_terminal(metadata()));
+        with_get_object_read_condition(Arc::new(Stop), async {
+            assert!(get_object_read_condition_is_terminal(metadata()));
+            assert!(
+                !tokio::spawn(async { get_object_read_condition_is_terminal(metadata()) })
+                    .await
+                    .expect("shared producer")
+            );
+        })
+        .await;
+        assert!(!get_object_read_condition_is_terminal(metadata()));
+    }
+
+    #[tokio::test]
+    async fn dropping_a_pending_read_removes_its_condition_scope() {
+        let mut read = Box::pin(with_get_object_read_condition(Arc::new(Stop), async {
+            assert!(get_object_read_condition_is_terminal(metadata()));
+            std::future::pending::<()>().await;
+        }));
+        assert!(futures::poll!(&mut read).is_pending());
+        drop(read);
+        assert!(!get_object_read_condition_is_terminal(metadata()));
+    }
+}

crates/ecstore/src/object_api/readers.rs

@@ -528,6 +528,14 @@ pub struct GetObjectReader {
 }
 
 impl GetObjectReader {
+    pub(crate) fn condition_stopped(object_info: ObjectInfo) -> Self {
+        Self {
+            stream: Box::new(ConditionStoppedBody),
+            object_info,
+            buffered_body: None,
+            body_source: GetObjectBodySource::Unprobed,
+        }
+    }
     /// Builds a fully materialized reader from a cache-coordinated body.
     pub fn from_cache_body(mut object_info: ObjectInfo, body: Bytes) -> Result<Self> {
         object_info.size = i64::try_from(body.len()).map_err(|_| Error::other("cached GET body length exceeds i64::MAX"))?;
@@ -552,6 +560,17 @@ impl GetObjectReader {
     }
 }
 
+struct ConditionStoppedBody;
+
+impl AsyncRead for ConditionStoppedBody {
+    fn poll_read(self: Pin<&mut Self>, _cx: &mut Context<'_>, buf: &mut ReadBuf<'_>) -> Poll<std::io::Result<()>> {
+        if buf.remaining() == 0 {
+            return Poll::Ready(Ok(()));
+        }
+        Poll::Ready(Err(std::io::Error::other("object read condition must be handled before body access")))
+    }
+}
+
 enum ReadTransform {
     // Written but never read by production code: the enclosing struct already
     // carries the same pair as `storage_offset`/`storage_length`. They survive

crates/ecstore/src/object_api/types.rs

@@ -938,6 +938,10 @@ pub struct ObjectOptions {
     /// A genuine target write guard; loss fences alone do not authorize a borrowed write.
     #[doc(hidden)]
     pub write_lock_context: Option<WriteLockContext>,
+    /// In-process delete ownership for late filesystem completion. Never
+    /// serialized or copied into object metadata.
+    #[doc(hidden)]
+    pub delete_mutation_owner: Option<Arc<dyn Send + Sync>>,
     /// Internal read-only inspection must not enqueue metadata or payload repairs.
     pub suppress_read_repair: bool,
     /// Control-plane writers that immediately read or CAS the same namespace
@@ -1158,6 +1162,7 @@ impl std::fmt::Debug for ObjectOptions {
             .field("http_preconditions", &self.http_preconditions.is_some())
             .field("delete_replication", &self.delete_replication.is_some())
             .field("delete_replication_config_snapshot", &self.delete_replication_config_snapshot)
+            .field("delete_mutation_owner", &self.delete_mutation_owner.is_some())
             .field("namespace_lock_fence", &self.namespace_lock_fence.is_some())
             .field("bucket_lifecycle_lock_fence", &self.bucket_lifecycle_lock_fence.is_some())
             .field("replication_request", &self.replication_request)

crates/ecstore/src/runtime/sources.rs

@@ -50,6 +50,7 @@ use s3s::dto::BucketLifecycleConfiguration;
 use s3s::region::Region;
 use tokio::sync::{OwnedRwLockReadGuard, RwLock};
 use tokio_util::sync::CancellationToken;
+#[cfg(test)]
 use tonic::transport::Channel;
 use uuid::Uuid;
 
@@ -251,10 +252,6 @@ pub(crate) async fn root_disk_threshold_for_erasure_disk() -> Option<u64> {
     }
 }
 
-pub(crate) async fn cached_node_channel(addr: &str) -> Option<Channel> {
-    rustfs_common::cached_connection(addr).await
-}
-
 #[cfg(test)]
 pub(crate) async fn cache_test_node_channel(addr: String, channel: Channel) {
     rustfs_common::cache_connection(addr, channel).await;

crates/ecstore/src/services/rebalance/entry.rs

@@ -50,7 +50,7 @@ fn ensure_rebalance_entry_active(cancel: &CancellationToken) -> Result<()> {
     Ok(())
 }
 
-#[cfg(test)]
+#[cfg(any(test, feature = "test-util"))]
 tokio::task_local! {
     static REBALANCE_ENTRY_RUN_FENCE_BARRIER: (Arc<tokio::sync::Notify>, Arc<tokio::sync::Notify>);
 }
@@ -268,7 +268,7 @@ impl ECStore {
             self.rebalance_run_guard(rebalance_id.as_ref(), "rebalance entry")
         })
         .await?;
-        #[cfg(test)]
+        #[cfg(any(test, feature = "test-util"))]
         if let Ok((arrived, release)) = REBALANCE_ENTRY_RUN_FENCE_BARRIER.try_with(Clone::clone) {
             arrived.notify_one();
             release.notified().await;
@@ -836,7 +836,8 @@ pub mod test_util {
         _temp_dirs: Vec<tempfile::TempDir>,
         store: Arc<ECStore>,
         cancel: CancellationToken,
-        barrier: crate::data_movement::SourceCleanupDeleteBarrier,
+        entry_arrived: Arc<tokio::sync::Notify>,
+        entry_release: Arc<tokio::sync::Notify>,
         stop_probe: super::super::control::RebalanceStopWaitProbe,
         entry_task: Option<tokio::task::JoinHandle<Result<RebalanceEntryOutcome>>>,
     }
@@ -897,31 +898,36 @@ pub mod test_util {
                 cached: Some(file_meta),
                 reusable: false,
             };
-            let barrier = crate::data_movement::SourceCleanupDeleteBarrier::install(bucket, object);
+            let entry_arrived = Arc::new(tokio::sync::Notify::new());
+            let entry_release = Arc::new(tokio::sync::Notify::new());
             let entry_store = Arc::clone(&store);
             let entry_set = Arc::clone(&set);
             let entry_cancel = cancel.clone();
-            let entry_task = tokio::spawn(async move {
-                entry_store
-                    .rebalance_entry(
-                        RebalanceEntryTarget {
-                            bucket: bucket.to_string(),
-                            pool_index: 0,
-                        },
-                        entry,
-                        entry_set,
-                        Arc::new(RebalanceBucketConfigs::default()),
-                        Arc::from(rebalance_id),
-                        entry_cancel,
-                    )
-                    .await
-            });
+            let entry_task = tokio::spawn(REBALANCE_ENTRY_RUN_FENCE_BARRIER.scope(
+                (Arc::clone(&entry_arrived), Arc::clone(&entry_release)),
+                async move {
+                    entry_store
+                        .rebalance_entry(
+                            RebalanceEntryTarget {
+                                bucket: bucket.to_string(),
+                                pool_index: 0,
+                            },
+                            entry,
+                            entry_set,
+                            Arc::new(RebalanceBucketConfigs::default()),
+                            Arc::from(rebalance_id),
+                            entry_cancel,
+                        )
+                        .await
+                },
+            ));
 
             Self {
                 _temp_dirs,
                 store,
                 cancel,
-                barrier,
+                entry_arrived,
+                entry_release,
                 stop_probe: super::super::control::RebalanceStopWaitProbe::install(rebalance_id),
                 entry_task: Some(entry_task),
             }
@@ -932,7 +938,9 @@ pub mod test_util {
         }
 
         pub async fn wait_until_entry_paused(&self) {
-            self.barrier.wait_until_paused().await;
+            tokio::time::timeout(std::time::Duration::from_secs(30), self.entry_arrived.notified())
+                .await
+                .expect("the entry should acquire its run fence before pausing");
         }
 
         pub async fn wait_until_admission_cancelled(&self) {
@@ -946,7 +954,7 @@ pub mod test_util {
         }
 
         pub fn release_entry(&self) {
-            self.barrier.release();
+            self.entry_release.notify_one();
         }
 
         pub async fn assert_entry_cancelled(&mut self) {
@@ -964,7 +972,7 @@ pub mod test_util {
 
     impl Drop for PausedRebalanceEntryTestFixture {
         fn drop(&mut self) {
-            self.barrier.release();
+            self.entry_release.notify_one();
             if let Some(task) = self.entry_task.take() {
                 task.abort();
             }

crates/ecstore/src/services/tier/tier.rs

@@ -82,8 +82,9 @@ use crate::{
 use rustfs_filemeta::FileInfo;
 use rustfs_rio::HashReader;
 use rustfs_s3_client::{admin_handler_utils::AdminError, provider_versions::ProviderVersionCapabilities};
+use rustfs_s3_types::S3ErrorCode;
 use rustfs_utils::path::{SLASH_SEPARATOR, path_join};
-use s3s::{S3ErrorCode, dto::BucketLifecycleConfiguration};
+use s3s::dto::BucketLifecycleConfiguration;
 
 use super::{
     tier_handlers::{ERR_TIER_BUCKET_NOT_FOUND, ERR_TIER_CONNECT_ERR, ERR_TIER_INVALID_CREDENTIALS, ERR_TIER_PERM_ERR},

crates/ecstore/src/set_disk/conditional_read_bench.rs

@@ -0,0 +1,63 @@
+// Copyright 2024 RustFS Team
+//
+// Licensed under the Apache License, Version 2.0 (the "License");
+// you may not use this file except in compliance with the License.
+// You may obtain a copy of the License at
+//
+//     http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing, software
+// distributed under the License is distributed on an "AS IS" BASIS,
+// WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+// See the License for the specific language governing permissions and
+// limitations under the License.
+
+//! Test-util-only controls for the paired read benchmark's dedicated bucket.
+
+use std::sync::atomic::{AtomicBool, Ordering};
+
+static ACTIVE: AtomicBool = AtomicBool::new(false);
+
+/// Keeps metadata off the cache in the benchmark's isolated bucket so GET
+/// metadata reads exercise the configured slow disk. Body-cache policy
+/// remains intact.
+#[must_use]
+pub struct ConditionalReadBenchmarkMetadataGuard {
+    _private: (),
+}
+
+impl ConditionalReadBenchmarkMetadataGuard {
+    pub fn acquire() -> Self {
+        assert!(
+            ACTIVE
+                .compare_exchange(false, true, Ordering::AcqRel, Ordering::Acquire)
+                .is_ok()
+        );
+        Self { _private: () }
+    }
+}
+
+impl Drop for ConditionalReadBenchmarkMetadataGuard {
+    fn drop(&mut self) {
+        ACTIVE.store(false, Ordering::Release);
+    }
+}
+
+pub(super) fn cache_bypass_applies(bucket: &str) -> bool {
+    bucket == "conditional-bench" && ACTIVE.load(Ordering::Acquire)
+}
+
+#[cfg(test)]
+mod tests {
+    use super::*;
+
+    #[test]
+    fn benchmark_metadata_profile_is_scoped_to_its_bucket_and_owner() {
+        assert!(!cache_bypass_applies("conditional-bench"));
+        let owner = ConditionalReadBenchmarkMetadataGuard::acquire();
+        assert!(cache_bypass_applies("conditional-bench"));
+        assert!(!cache_bypass_applies("other-bucket"));
+        drop(owner);
+        assert!(!cache_bypass_applies("conditional-bench"));
+    }
+}

crates/ecstore/src/set_disk/core/metadata_read.rs

@@ -66,6 +66,7 @@ use tokio::task::JoinSet;
 pub(in crate::set_disk) struct MetadataReadResult {
     pub(in crate::set_disk) slots: Vec<MetadataDiskObservation>,
     pub(in crate::set_disk) diagnostics: MetadataFanoutDiagnostics,
+    pub(in crate::set_disk) condition_stopped: bool,
 }
 
 impl MetadataReadResult {
@@ -575,6 +576,7 @@ impl MetadataFanoutDiagnostics {
 
     pub(in crate::set_disk) fn record_quorum_candidate_latency(&self, path: &'static str, read_quorum: usize) {
         if let Some(latency) = self.quorum_candidate_latency(read_quorum) {
+            rustfs_io_metrics::record_get_object_metadata_quorum_result(path, "reached");
             rustfs_io_metrics::record_get_object_quorum_reached_latency(path, latency.as_secs_f64());
         }
     }
@@ -655,6 +657,8 @@ async fn inline_body_miss_reason_from_observations<'a>(
         return Some(GET_METADATA_EARLY_STOP_REASON_DATA_READ_INLINE_BODY_VERIFY);
     };
 
+    #[cfg(test)]
+    crate::set_disk::disk_call_counters::record(object, crate::set_disk::disk_call_counters::KIND_BODY_DECODE, 0);
     match try_read_inline_data_shards_direct(&mut readers, erasure.data_shards, read_length, object_size).await {
         Some(body) if body.len() == object_size => None,
         _ => Some(GET_METADATA_EARLY_STOP_REASON_DATA_READ_INLINE_BODY_VERIFY),
@@ -1078,7 +1082,11 @@ impl SetDisks {
                 0,
             );
         }
-        Ok(MetadataReadResult { slots, diagnostics })
+        Ok(MetadataReadResult {
+            slots,
+            diagnostics,
+            condition_stopped: false,
+        })
     }
 
     #[allow(clippy::too_many_arguments)]
@@ -1121,6 +1129,7 @@ impl SetDisks {
         let mut next_fanout_index = 0usize;
         let mut scheduled_count = 0usize;
         let mut force_full_wait = false;
+        let read_condition_active = read_data && crate::object_api::get_object_read_condition_is_active();
         let mut final_miss_reason_override = None;
         let mut non_inline_candidate_eligible = None;
         let mut single_pending_hedge_deadline = None;
@@ -1258,12 +1267,28 @@ impl SetDisks {
                 }
             }
 
-            if !force_full_wait
+            if (!force_full_wait || read_condition_active)
                 && let Some(decision) = accumulator
                     .early_stop_decision()
                     .or_else(|| accumulator.version_early_stop_decision())
             {
-                let should_return_early = if read_data {
+                let condition_stopped = read_condition_active
+                    && decision.reason == crate::diagnostics::get::GET_METADATA_EARLY_STOP_REASON_VALID_QUORUM
+                    && accumulator.candidate.as_ref().is_some_and(|candidate| {
+                        !candidate.deleted
+                            && crate::object_api::get_object_read_condition_is_terminal(
+                                crate::object_api::GetObjectReadMetadata::from_file_info(
+                                    bucket.as_ref(),
+                                    object.as_ref(),
+                                    candidate,
+                                ),
+                            )
+                    });
+                let should_return_early = if condition_stopped {
+                    true
+                } else if force_full_wait {
+                    false
+                } else if read_data {
                     match accumulator.candidate.as_ref() {
                         Some(_candidate) if non_inline_candidate_eligible == Some(true) => {
                             accumulator.candidate_has_read_reserve()
@@ -1332,7 +1357,11 @@ impl SetDisks {
                         cancelled_count,
                     );
                     let diagnostics = MetadataFanoutDiagnostics::new(fanout_start.elapsed(), observations);
-                    return Ok(MetadataReadResult { slots, diagnostics });
+                    return Ok(MetadataReadResult {
+                        slots,
+                        diagnostics,
+                        condition_stopped,
+                    });
                 }
             }
 
@@ -1410,7 +1439,11 @@ impl SetDisks {
         rustfs_io_metrics::record_get_object_metadata_early_stop_saved_responses(metrics_path, 0);
         rustfs_io_metrics::record_get_object_metadata_fanout_lifecycle(metrics_path, scheduled_count, scheduled_count, 0);
         let diagnostics = MetadataFanoutDiagnostics::new(fanout_start.elapsed(), observations);
-        Ok(MetadataReadResult { slots, diagnostics })
+        Ok(MetadataReadResult {
+            slots,
+            diagnostics,
+            condition_stopped: false,
+        })
     }
 
     /// Test-only seam that records one per-disk `read_version` metadata RPC for

crates/ecstore/src/set_disk/mod.rs

@@ -166,7 +166,7 @@ use sha2::Sha256;
 use std::hash::{BuildHasher, Hash, Hasher};
 use std::mem::{self};
 use std::pin::Pin;
-use std::sync::atomic::{AtomicU64, Ordering};
+use std::sync::atomic::{AtomicBool, AtomicU64, Ordering};
 use std::sync::{Arc, OnceLock};
 use std::task::{Context, Poll};
 use std::time::{Instant, SystemTime, UNIX_EPOCH};
@@ -410,6 +410,12 @@ impl Drop for ObjectLockDiagGuard {
         }
 
         let hold = self.acquired_at.elapsed();
+        if self.op == "put_object_commit" && self.mode == "write" && rustfs_io_metrics::put_stage_metrics_enabled() {
+            rustfs_io_metrics::record_put_object_stage_duration(
+                rustfs_io_metrics::PUT_STAGE_PUT_OBJECT_COMMIT_NAMESPACE_LOCK_HELD,
+                hold.as_secs_f64() * 1000.0,
+            );
+        }
         record_object_lock_diag_hold_duration(self.op, self.mode, hold);
         let threshold = get_object_lock_diag_slow_hold_threshold();
         if hold >= threshold {
@@ -793,6 +799,11 @@ const DEFAULT_RUSTFS_GET_METADATA_TWO_PHASE_READ_PLAN_ENABLE: bool = false;
 const ENV_RUSTFS_GET_METADATA_EARLY_STOP_BOUNDED_FANOUT: &str = "RUSTFS_GET_METADATA_EARLY_STOP_BOUNDED_FANOUT";
 const DEFAULT_RUSTFS_GET_METADATA_EARLY_STOP_BOUNDED_FANOUT: bool = true;
 
+#[cfg(feature = "test-util")]
+mod conditional_read_bench;
+#[cfg(feature = "test-util")]
+pub use conditional_read_bench::ConditionalReadBenchmarkMetadataGuard;
+
 const ENV_RUSTFS_GET_METADATA_SLOWTAIL_FAULT_DELAY_MS: &str = "RUSTFS_GET_METADATA_SLOWTAIL_FAULT_DELAY_MS";
 const ENV_RUSTFS_GET_METADATA_SLOWTAIL_FAULT_DISKS: &str = "RUSTFS_GET_METADATA_SLOWTAIL_FAULT_DISKS";
 const ENV_RUSTFS_GET_METADATA_SLOWTAIL_FAULT_BUCKET: &str = "RUSTFS_GET_METADATA_SLOWTAIL_FAULT_BUCKET";
@@ -874,6 +885,7 @@ mod ctx;
 mod metadata;
 mod ops;
 pub(crate) use ops::bucket::BucketInfoQuorum;
+pub(crate) use ops::delete_recovery::{DeleteRecoveryContext, DeleteRecoverySummary};
 pub(crate) use ops::heal::AbsenceProofRequest;
 #[cfg(test)]
 pub(crate) use ops::heal::DanglingDeleteFailure;
@@ -918,16 +930,22 @@ pub use ops::heal_walk::HealWalkVersion;
 pub(in crate::set_disk) struct GetObjectFileInfo {
     owned: Option<OwnedGetObjectFileInfo>,
     shared: Option<Arc<GetObjectMetadataCacheEntry>>,
+    condition_stopped: bool,
+    late_metadata_fanout: Option<LateMetadataFanout>,
 }
 
 struct OwnedGetObjectFileInfo {
     fi: FileInfo,
     parts_metadata: Vec<FileInfo>,
     online_disks: Vec<Option<DiskStore>>,
-    late_metadata_fanout_disks: Option<Vec<Option<DiskStore>>>,
 }
 
-type OwnedGetObjectFileInfoParts = (FileInfo, Vec<FileInfo>, Vec<Option<DiskStore>>, Option<Vec<Option<DiskStore>>>);
+enum LateMetadataFanout {
+    CapturedDisks(Vec<Option<DiskStore>>),
+    RefreshCurrentSet,
+}
+
+type OwnedGetObjectFileInfoParts = (FileInfo, Vec<FileInfo>, Vec<Option<DiskStore>>, Option<LateMetadataFanout>);
 
 impl GetObjectFileInfo {
     fn owned(fi: FileInfo, parts_metadata: Vec<FileInfo>, online_disks: Vec<Option<DiskStore>>) -> Self {
@@ -936,9 +954,10 @@ impl GetObjectFileInfo {
                 fi,
                 parts_metadata,
                 online_disks,
-                late_metadata_fanout_disks: None,
             }),
             shared: None,
+            condition_stopped: false,
+            late_metadata_fanout: None,
         }
     }
 
@@ -948,24 +967,27 @@ impl GetObjectFileInfo {
         online_disks: Vec<Option<DiskStore>>,
         late_metadata_fanout_disks: Vec<Option<DiskStore>>,
     ) -> Self {
-        Self {
-            owned: Some(OwnedGetObjectFileInfo {
-                fi,
-                parts_metadata,
-                online_disks,
-                late_metadata_fanout_disks: Some(late_metadata_fanout_disks),
-            }),
-            shared: None,
-        }
+        let mut snapshot = Self::owned(fi, parts_metadata, online_disks);
+        snapshot.late_metadata_fanout = Some(LateMetadataFanout::CapturedDisks(late_metadata_fanout_disks));
+        snapshot
     }
 
     fn shared(entry: Arc<GetObjectMetadataCacheEntry>) -> Self {
+        let condition_stopped = entry.capability == GetObjectMetadataCacheCapability::ConditionOnly;
         Self {
             owned: None,
             shared: Some(entry),
+            condition_stopped,
+            late_metadata_fanout: None,
         }
     }
 
+    fn shared_with_late_metadata_refresh(entry: Arc<GetObjectMetadataCacheEntry>) -> Self {
+        let mut snapshot = Self::shared(entry);
+        snapshot.late_metadata_fanout = Some(LateMetadataFanout::RefreshCurrentSet);
+        snapshot
+    }
+
     fn fi(&self) -> &FileInfo {
         match (&self.owned, &self.shared) {
             (Some(snapshot), None) => &snapshot.fi,
@@ -991,9 +1013,7 @@ impl GetObjectFileInfo {
     }
 
     fn has_late_metadata_fanout(&self) -> bool {
-        self.owned
-            .as_ref()
-            .is_some_and(|snapshot| snapshot.late_metadata_fanout_disks.is_some())
+        self.late_metadata_fanout.is_some()
     }
 
     fn into_owned(self) -> (FileInfo, Vec<FileInfo>, Vec<Option<DiskStore>>) {
@@ -1002,16 +1022,22 @@ impl GetObjectFileInfo {
     }
 
     fn into_owned_with_late_metadata_fanout(self) -> OwnedGetObjectFileInfoParts {
-        match (self.owned, self.shared) {
-            (Some(snapshot), None) => (
-                snapshot.fi,
-                snapshot.parts_metadata,
-                snapshot.online_disks,
-                snapshot.late_metadata_fanout_disks,
-            ),
+        let Self {
+            owned,
+            shared,
+            late_metadata_fanout,
+            ..
+        } = self;
+        match (owned, shared) {
+            (Some(snapshot), None) => (snapshot.fi, snapshot.parts_metadata, snapshot.online_disks, late_metadata_fanout),
             (None, Some(entry)) => match Arc::try_unwrap(entry) {
-                Ok(entry) => (entry.fi, entry.parts_metadata, entry.online_disks, None),
-                Err(entry) => (entry.fi.clone(), entry.parts_metadata.clone(), entry.online_disks.clone(), None),
+                Ok(entry) => (entry.fi, entry.parts_metadata, entry.online_disks, late_metadata_fanout),
+                Err(entry) => (
+                    entry.fi.clone(),
+                    entry.parts_metadata.clone(),
+                    entry.online_disks.clone(),
+                    late_metadata_fanout,
+                ),
             },
             _ => unreachable!("GET metadata snapshot representation must be exclusive"),
         }
@@ -1143,6 +1169,248 @@ mod prepared_get_object_metadata_tests {
         assert!(take_prepared_get_object_metadata().is_none());
     }
 
+    struct StopReadForEtag(String);
+    impl crate::object_api::GetObjectReadCondition for StopReadForEtag {
+        fn is_terminal(&self, metadata: crate::object_api::GetObjectReadMetadata<'_>) -> bool {
+            !metadata.delete_marker && metadata.etag == Some(self.0.as_str())
+        }
+    }
+
+    #[tokio::test]
+    #[serial_test::serial(body_cache_hook)]
+    async fn read_condition_stops_on_quorum_before_slow_disk_and_body_verification() {
+        use crate::object_api::with_get_object_read_condition;
+        for size in [4096, 1_300_000] {
+            let (dirs, set) = make_local_set_disks(4, 2).await;
+            let bucket = "read-condition-quorum";
+            let object = object_with_initial_data_shards(bucket, &format!("read-condition-{size}"));
+            let opts = ObjectOptions {
+                no_lock: true,
+                ..Default::default()
+            };
+            set.make_bucket(bucket, &MakeBucketOptions::default())
+                .await
+                .expect("create bucket");
+            let mut put = PutObjReader::from_vec(vec![b'v'; size]);
+            let info = set.put_object(bucket, &object, &mut put, &opts).await.expect("write object");
+            for dir in &dirs {
+                let object_path = dir.path().join(bucket).join(&object);
+                if size == 4096 {
+                    let path = object_path.join("xl.meta");
+                    let mut bytes = tokio::fs::read(&path).await.expect("read inline metadata");
+                    *bytes.last_mut().expect("inline payload") ^= 1;
+                    tokio::fs::write(path, bytes).await.expect("damage inline data");
+                } else {
+                    tokio::fs::remove_dir_all(object_path.join(info.data_dir.expect("data directory").to_string()))
+                        .await
+                        .expect("remove shards");
+                }
+            }
+            let slow = bounded_initial_parity_disk_index(bucket, &object);
+            let barrier = rename_fanout_barrier::arm(&object, slow, rename_fanout_barrier::PHASE_READ_VERSION);
+            let calls = disk_call_counters::observe(&object);
+            let read_opts = ObjectOptions {
+                metadata_cache_safe: true,
+                ..opts
+            };
+            let followup_opts = read_opts.clone();
+            let condition = Arc::new(StopReadForEtag(info.etag.clone().expect("committed ETag")));
+            let set_for_read = Arc::clone(&set);
+            let object_for_read = object.clone();
+            let mut read = tokio::spawn(async move {
+                with_get_object_read_condition(
+                    condition,
+                    set_for_read.get_object_reader(bucket, &object_for_read, None, HeaderMap::new(), &read_opts),
+                )
+                .await
+            });
+            tokio::time::timeout(READ_VERSION_BARRIER_GUARD, barrier.wait_until_paused())
+                .await
+                .expect("slow metadata read is pending");
+            let mut reader = tokio::time::timeout(READ_VERSION_BARRIER_GUARD, &mut read)
+                .await
+                .expect("condition must not wait for the slow disk or decode damaged data")
+                .expect("reader task")
+                .expect("metadata-only result");
+            assert_eq!(reader.object_info.etag, info.etag);
+            assert_eq!(
+                calls.total(disk_call_counters::KIND_INLINE_SHARD_OPEN),
+                0,
+                "damaged inline data was never opened"
+            );
+            assert_eq!(
+                calls.total(disk_call_counters::KIND_DISK_SHARD_OPEN),
+                0,
+                "missing shards were never opened"
+            );
+            assert_eq!(
+                calls.total(disk_call_counters::KIND_BODY_DECODE),
+                0,
+                "the body verifier and decoder were never entered"
+            );
+            let mut body = Vec::new();
+            assert!(
+                reader.stream.read_to_end(&mut body).await.is_err(),
+                "terminal metadata cannot be served as a successful body"
+            );
+            assert!(body.is_empty());
+            let first_calls = calls.total(disk_call_counters::KIND_READ_VERSION);
+            drop(reader);
+            barrier.release();
+            let snapshot = set
+                .get_object_fileinfo_for_get_object_reader(bucket, &object, &followup_opts, true, true)
+                .await
+                .expect("fresh metadata after terminal scope");
+            assert!(!snapshot.condition_stopped);
+            assert!(
+                calls.total(disk_call_counters::KIND_READ_VERSION) > first_calls,
+                "an incomplete terminal snapshot must not satisfy the next metadata read from cache"
+            );
+        }
+    }
+
+    #[tokio::test]
+    #[serial_test::serial(body_cache_hook)]
+    async fn read_condition_skips_shard_open_and_decode_with_live_positive_controls() {
+        for size in [4096, 1_300_000] {
+            let (_dirs, set) = make_local_set_disks(4, 2).await;
+            let bucket = "read-condition-body-counters";
+            let object = object_with_initial_data_shards(bucket, &format!("body-counters-{size}"));
+            let opts = ObjectOptions {
+                no_lock: true,
+                ..Default::default()
+            };
+            let payload: Vec<_> = (0..size)
+                .map(|index| u8::try_from(index % 251).expect("pattern byte"))
+                .collect();
+            set.make_bucket(bucket, &MakeBucketOptions::default())
+                .await
+                .expect("create bucket");
+            let mut put = PutObjReader::from_vec(payload.clone());
+            let info = set.put_object(bucket, &object, &mut put, &opts).await.expect("write object");
+            let calls = disk_call_counters::observe(&object);
+            let reader = crate::object_api::with_get_object_read_condition(
+                Arc::new(StopReadForEtag(info.etag.expect("committed ETag"))),
+                set.get_object_reader(bucket, &object, None, HeaderMap::new(), &opts),
+            )
+            .await
+            .expect("terminal metadata reader");
+            drop(reader);
+            assert!(calls.total(disk_call_counters::KIND_READ_VERSION) > 0, "metadata really was read");
+            assert_eq!(
+                calls.total(disk_call_counters::KIND_INLINE_SHARD_OPEN),
+                0,
+                "no inline shard source opened"
+            );
+            assert_eq!(calls.total(disk_call_counters::KIND_DISK_SHARD_OPEN), 0, "no disk shard source opened");
+            assert_eq!(
+                calls.total(disk_call_counters::KIND_BODY_DECODE),
+                0,
+                "no body verifier or decoder entered"
+            );
+
+            let mut reader = crate::object_api::with_get_object_read_condition(
+                Arc::new(StopReadForEtag("different-etag".to_owned())),
+                set.get_object_reader(bucket, &object, None, HeaderMap::new(), &opts),
+            )
+            .await
+            .expect("unmatched condition opens the intact object");
+            let mut body = Vec::new();
+            reader.stream.read_to_end(&mut body).await.expect("positive-control body");
+            assert_eq!(body, payload);
+            let source_kind = if size == 4096 {
+                disk_call_counters::KIND_INLINE_SHARD_OPEN
+            } else {
+                disk_call_counters::KIND_DISK_SHARD_OPEN
+            };
+            assert!(
+                calls.total(source_kind) > 0,
+                "the same source counter observes the positive control: size={size}, inline={}, disk={}, decode={}",
+                calls.total(disk_call_counters::KIND_INLINE_SHARD_OPEN),
+                calls.total(disk_call_counters::KIND_DISK_SHARD_OPEN),
+                calls.total(disk_call_counters::KIND_BODY_DECODE),
+            );
+            assert!(
+                calls.total(disk_call_counters::KIND_BODY_DECODE) > 0,
+                "the same decode counter observes the positive control"
+            );
+        }
+    }
+
+    #[tokio::test]
+    #[serial_test::serial(body_cache_hook)]
+    async fn read_condition_rechecks_selected_metadata_before_body_access() {
+        struct Changing(std::sync::atomic::AtomicUsize);
+        impl crate::object_api::GetObjectReadCondition for Changing {
+            fn is_terminal(&self, _: crate::object_api::GetObjectReadMetadata<'_>) -> bool {
+                self.0.fetch_add(1, std::sync::atomic::Ordering::Relaxed) == 0
+            }
+        }
+        let (_dirs, set) = make_local_set_disks(4, 2).await;
+        let bucket = "read-condition-consistency";
+        let object = object_with_initial_data_shards(bucket, "read-condition-consistency");
+        let opts = ObjectOptions {
+            no_lock: true,
+            ..Default::default()
+        };
+        set.make_bucket(bucket, &MakeBucketOptions::default())
+            .await
+            .expect("create bucket");
+        let mut put = PutObjReader::from_vec(vec![b'x'; 4096]);
+        set.put_object(bucket, &object, &mut put, &opts).await.expect("write object");
+        let result = crate::object_api::with_get_object_read_condition(
+            Arc::new(Changing(std::sync::atomic::AtomicUsize::new(0))),
+            set.get_object_reader(bucket, &object, None, HeaderMap::new(), &opts),
+        )
+        .await;
+        assert!(result.is_err(), "an unverified terminal snapshot must never become a 200 reader");
+    }
+
+    #[tokio::test]
+    #[serial_test::serial(body_cache_hook)]
+    async fn read_condition_miss_preserves_inline_early_stop_and_exact_body() {
+        let (_dirs, set) = make_local_set_disks(4, 2).await;
+        let bucket = "read-condition-miss";
+        let object = object_with_initial_data_shards(bucket, "read-condition-miss");
+        let opts = ObjectOptions {
+            no_lock: true,
+            ..Default::default()
+        };
+        let payload: Vec<u8> = (0..8192)
+            .map(|index| u8::try_from(index % 251).expect("pattern byte"))
+            .collect();
+        set.make_bucket(bucket, &MakeBucketOptions::default())
+            .await
+            .expect("create bucket");
+        let mut put = PutObjReader::from_vec(payload.clone());
+        set.put_object(bucket, &object, &mut put, &opts)
+            .await
+            .expect("write inline object");
+        let slow = bounded_initial_parity_disk_index(bucket, &object);
+        let barrier = rename_fanout_barrier::arm(&object, slow, rename_fanout_barrier::PHASE_READ_VERSION);
+        let set_for_read = Arc::clone(&set);
+        let object_for_read = object.clone();
+        let mut read = tokio::spawn(async move {
+            crate::object_api::with_get_object_read_condition(
+                Arc::new(StopReadForEtag("other".to_owned())),
+                set_for_read.get_object_reader(bucket, &object_for_read, None, HeaderMap::new(), &opts),
+            )
+            .await
+        });
+        tokio::time::timeout(READ_VERSION_BARRIER_GUARD, barrier.wait_until_paused())
+            .await
+            .expect("slow metadata read is pending");
+        let mut reader = tokio::time::timeout(READ_VERSION_BARRIER_GUARD, &mut read)
+            .await
+            .expect("condition miss preserves the normal early stop")
+            .expect("reader task")
+            .expect("reader");
+        let mut bytes = Vec::new();
+        reader.stream.read_to_end(&mut bytes).await.expect("normal body");
+        assert_eq!(bytes, payload);
+        barrier.release();
+    }
+
     #[test]
     fn cache_hit_consumers_release_snapshot_at_legacy_ownership() {
         let fi = FileInfo {
@@ -1151,6 +1419,7 @@ mod prepared_get_object_metadata_tests {
         };
         let cached = Arc::new(GetObjectMetadataCacheEntry {
             created_at: Instant::now(),
+            capability: GetObjectMetadataCacheCapability::Full,
             parts_metadata: vec![fi.clone()],
             fi,
             online_disks: vec![None],
@@ -1159,11 +1428,11 @@ mod prepared_get_object_metadata_tests {
         let snapshot = GetObjectFileInfo::shared(Arc::clone(&cached));
 
         assert!(snapshot.has_valid_representation());
-        assert!(std::mem::size_of::<GetObjectFileInfo>() >= std::mem::size_of::<OwnedGetObjectFileInfo>());
-        assert!(
-            std::mem::size_of::<GetObjectFileInfo>()
-                <= std::mem::size_of::<OwnedGetObjectFileInfo>() + 2 * std::mem::size_of::<usize>()
-        );
+        // Keep the original payload budget after moving the captured disks out
+        // of OwnedGetObjectFileInfo.
+        let owned_payload_size = std::mem::size_of::<(OwnedGetObjectFileInfo, Option<Vec<Option<DiskStore>>>)>();
+        assert!(std::mem::size_of::<GetObjectFileInfo>() >= owned_payload_size);
+        assert!(std::mem::size_of::<GetObjectFileInfo>() <= owned_payload_size + 2 * std::mem::size_of::<usize>());
         assert_eq!(Arc::strong_count(&cached), 2, "a cache hit must add one snapshot reference");
         assert_eq!(snapshot.fi().name, "object");
         assert_eq!(snapshot.parts_metadata().len(), 1);
@@ -4083,6 +4352,9 @@ pub struct SetDisks {
     /// writes skip the global registry mutex (backlog#1315). `Arc` so clones of
     /// a set share one generation marker.
     capacity_dirty_generation: Arc<AtomicU64>,
+    /// Per-set proof that successful rename_data peers return the old-current
+    /// source capability needed before PUT may skip its pre-rename lookup.
+    tier_free_version_rename_data_source_capable: Arc<AtomicBool>,
     /// Orphan prefixes whose last purge scan met data that can never be
     /// purged by listing (residue without a committed marker, an in-flight
     /// write), keyed by `bucket/prefix` with the time of that scan. Empty
@@ -4338,6 +4610,14 @@ struct GetObjectMetadataCacheKey {
     hash: u64,
 }
 
+#[derive(Clone, Copy, Debug)]
+struct GetObjectMetadataCacheLookupKey<'a> {
+    bucket: &'a str,
+    object: &'a str,
+    generation: u64,
+    hash: u64,
+}
+
 #[derive(Clone, Copy, Debug, Eq, PartialEq)]
 struct GetObjectMetadataCacheGeneration {
     index: usize,
@@ -4426,6 +4706,33 @@ impl GetObjectMetadataCacheKey {
     }
 }
 
+impl<'a> GetObjectMetadataCacheLookupKey<'a> {
+    fn new(bucket: &'a str, object: &'a str, generation: GetObjectMetadataCacheGeneration) -> Self {
+        Self {
+            bucket,
+            object,
+            generation: generation.value,
+            hash: generation.hash,
+        }
+    }
+}
+
+impl Hash for GetObjectMetadataCacheLookupKey<'_> {
+    fn hash<H: Hasher>(&self, state: &mut H) {
+        self.hash.hash(state);
+        self.generation.hash(state);
+    }
+}
+
+impl moka::Equivalent<GetObjectMetadataCacheKey> for GetObjectMetadataCacheLookupKey<'_> {
+    fn equivalent(&self, key: &GetObjectMetadataCacheKey) -> bool {
+        self.bucket == key.bucket.as_ref()
+            && self.object == key.object.as_ref()
+            && self.generation == key.generation
+            && self.hash == key.hash
+    }
+}
+
 impl Hash for GetObjectMetadataCacheKey {
     fn hash<H: Hasher>(&self, state: &mut H) {
         self.hash.hash(state);
@@ -4433,6 +4740,15 @@ impl Hash for GetObjectMetadataCacheKey {
     }
 }
 
+#[derive(Clone, Copy, Debug, PartialEq, Eq)]
+enum GetObjectMetadataCacheCapability {
+    Full,
+    // A quorum-selected layout can serve body reads, but is not a mutation layout.
+    QuorumSelected,
+    // Quorum-validated metadata used only to evaluate a terminal read condition.
+    ConditionOnly,
+}
+
 #[derive(Debug)]
 struct GetObjectMetadataCacheEntry {
     #[allow(dead_code)] // Kept for debugging; moka handles TTL internally
@@ -4441,6 +4757,7 @@ struct GetObjectMetadataCacheEntry {
     parts_metadata: Vec<FileInfo>,
     online_disks: Vec<Option<DiskStore>>,
     read_quorum: usize,
+    capability: GetObjectMetadataCacheCapability,
 }
 
 #[derive(Clone, Debug)]
@@ -4971,6 +5288,7 @@ impl SetDisks {
             ctx,
             capacity_scope_cache: Arc::new(std::sync::RwLock::new(CapacityScopeCache::default())),
             capacity_dirty_generation: Arc::new(AtomicU64::new(u64::MAX)),
+            tier_free_version_rename_data_source_capable: Arc::new(AtomicBool::new(false)),
             orphan_purge_backoff: Arc::new(std::sync::Mutex::new(HashMap::new())),
             #[cfg(test)]
             storage_class_config_override: Arc::new(std::sync::RwLock::new(None)),
@@ -4985,6 +5303,15 @@ impl SetDisks {
         &self.ctx
     }
 
+    pub(in crate::set_disk) fn tier_free_version_rename_data_source_capable(&self) -> bool {
+        self.tier_free_version_rename_data_source_capable.load(Ordering::Acquire)
+    }
+
+    pub(in crate::set_disk) fn observe_tier_free_version_rename_data_source_capability(&self, capable: bool) {
+        self.tier_free_version_rename_data_source_capable
+            .store(capable, Ordering::Release);
+    }
+
     #[cfg(test)]
     pub(crate) fn set_instance_ctx_for_test(&mut self, ctx: Arc<InstanceContext>) {
         self.local_lock_manager = ctx.lock_manager();

crates/ecstore/src/set_disk/ops/bucket.rs

@@ -123,8 +123,18 @@ impl SetDisks {
     }
 
     pub(crate) async fn list_bucket_for_scanner(&self, _opts: &BucketOptions) -> Result<(Vec<BucketInfo>, bool)> {
+        self.list_bucket_with_quorum(BucketInfoQuorum::Write).await
+    }
+
+    pub(crate) async fn list_bucket_with_quorum(&self, quorum: BucketInfoQuorum) -> Result<(Vec<BucketInfo>, bool)> {
         let disks = self.disk_inventory().await;
         let write_quorum = (disks.len() / 2) + 1;
+        // Discovery uses the same namespace read boundary as stat_bucket.
+        // It grants no authority to mutate buckets or accept object reads.
+        let required = match quorum {
+            BucketInfoQuorum::Read => disks.len().div_ceil(2).max(1),
+            BucketInfoQuorum::Write => write_quorum,
+        };
 
         let mut futures = Vec::with_capacity(disks.len());
         for disk in disks {
@@ -153,7 +163,11 @@ impl SetDisks {
             }
         }
 
-        if let Some(err) = reduce_write_quorum_errs(&errs, BUCKET_OP_IGNORED_ERRS, write_quorum) {
+        let error = match quorum {
+            BucketInfoQuorum::Read => reduce_read_quorum_errs(&errs, BUCKET_OP_IGNORED_ERRS, required),
+            BucketInfoQuorum::Write => reduce_write_quorum_errs(&errs, BUCKET_OP_IGNORED_ERRS, required),
+        };
+        if let Some(err) = error {
             return Err(err.into());
         }
 
@@ -179,7 +193,7 @@ impl SetDisks {
         topology_complete &= counts.values().all(|(count, _)| *count >= write_quorum);
         let mut buckets = counts
             .into_values()
-            .filter_map(|(count, bucket)| (count >= write_quorum).then_some(bucket))
+            .filter_map(|(count, bucket)| (count >= required).then_some(bucket))
             .collect::<Vec<_>>();
         buckets.sort_by(|left, right| left.name.cmp(&right.name));
         Ok((buckets, topology_complete))

crates/ecstore/src/set_disk/ops/mod.rs

@@ -19,6 +19,7 @@
 
 pub(crate) mod bitrot_self_verify;
 pub(crate) mod bucket;
+pub(crate) mod delete_recovery;
 pub(crate) mod heal;
 pub(crate) mod heal_walk;
 pub(crate) mod list;

crates/ecstore/src/store/heal.rs

@@ -270,6 +270,134 @@ impl ECStore {
         .map_err(|error| Error::other_with_context("bucket heal owner task failed", error))?
     }
 
+    /// Recover hidden delete journals through the lifecycle-owned heal boundary.
+    pub async fn recover_delete_transactions(self: &Arc<Self>, bucket: &str, opts: &HealOpts) -> Result<HealResultItem> {
+        Ok(self
+            .recover_delete_transactions_at_incarnation(bucket, None, None, opts)
+            .await?
+            .unwrap_or_else(|| HealResultItem {
+                heal_item_type: HealItemType::Bucket.to_string(),
+                bucket: bucket.to_owned(),
+                detail: "delete recovery: discovered=0".to_owned(),
+                ..Default::default()
+            }))
+    }
+
+    pub async fn recover_delete_transactions_at_incarnation(
+        self: &Arc<Self>,
+        bucket: &str,
+        object: Option<&str>,
+        expected: Option<Uuid>,
+        opts: &HealOpts,
+    ) -> Result<Option<HealResultItem>> {
+        self.recover_delete_transactions_in_scope(bucket, object, None, expected, opts)
+            .await
+    }
+
+    pub async fn recover_delete_prefix_at_incarnation(
+        self: &Arc<Self>,
+        bucket: &str,
+        prefix: &str,
+        expected: Option<Uuid>,
+        opts: &HealOpts,
+    ) -> Result<Option<HealResultItem>> {
+        self.recover_delete_transactions_in_scope(bucket, None, Some(prefix), expected, opts)
+            .await
+    }
+
+    async fn recover_delete_transactions_in_scope(
+        self: &Arc<Self>,
+        bucket: &str,
+        object: Option<&str>,
+        prefix: Option<&str>,
+        expected: Option<Uuid>,
+        opts: &HealOpts,
+    ) -> Result<Option<HealResultItem>> {
+        if opts.no_lock && !opts.dry_run {
+            return Err(StorageError::InvalidArgument(
+                "heal".to_string(),
+                "no_lock".to_string(),
+                "delete recovery requires namespace locking".to_string(),
+            ));
+        }
+        let expected = match expected {
+            Some(expected) => expected,
+            None => self.bucket_incarnation_id_from_disk(bucket).await?,
+        };
+        let object = object.map(str::to_owned);
+        let prefix = prefix.map(str::to_owned);
+        self.run_bucket_heal_at_incarnation(bucket, expected, opts, |store, bucket, opts| async move {
+            let scope = bucket_heal_scope(&bucket).ok_or(StorageError::PreconditionFailed)?;
+            let explicit_delete = opts.remove
+                && metadata_sys::has_authoritative_never_versioned_state_in(&store.ctx, &bucket).await?
+                && matches!(
+                    metadata_sys::get_object_lock_config_state_in(&store.ctx, &bucket).await?,
+                    metadata_sys::ObjectLockConfigState::ConfirmedAbsent
+                );
+            let movement_gate = store.ctx.data_movement_operation_gate();
+            let _movement_guard = movement_gate.read().await;
+            let can_commit = || scope.check().is_ok();
+            let recovery_context = crate::set_disk::DeleteRecoveryContext {
+                dry_run: opts.dry_run,
+                explicit_delete,
+                namespace_owner: (!opts.dry_run)
+                    .then(|| Arc::new((scope.clone(), store.ctx.begin_namespace_commit())) as Arc<dyn Send + Sync>),
+                ..crate::set_disk::DeleteRecoveryContext::new(Some(scope.incarnation), &can_commit)
+            };
+            let mut summary = crate::set_disk::DeleteRecoverySummary::default();
+            for pool in store.get_pools_for_heal_object(&opts)? {
+                if store.is_suspended(pool.pool_idx).await || store.is_pool_rebalancing(pool.pool_idx).await {
+                    return Err(StorageError::SlowDown);
+                }
+                for (set_index, set) in pool.disk_set.iter().enumerate() {
+                    if opts.set.is_some_and(|requested| requested != set_index) {
+                        continue;
+                    }
+                    if let Some(object) = &object {
+                        if !Arc::ptr_eq(set, &pool.get_disks_by_key(object)) {
+                            continue;
+                        }
+                        summary.add(set.recover_delete_object(&bucket, object, recovery_context.clone()).await?);
+                    } else if let Some(prefix) = &prefix {
+                        summary.add(set.recover_delete_prefix(&bucket, prefix, recovery_context.clone()).await?);
+                    } else {
+                        summary.add(set.recover_delete_bucket(&bucket, recovery_context.clone()).await?);
+                    }
+                }
+            }
+            if summary.pending > 0 {
+                return Err(StorageError::ErasureReadQuorum);
+            }
+            if object.is_some() && !opts.dry_run && summary.blocked > 0 {
+                return Err(StorageError::SlowDown);
+            }
+            if summary.discovered == 0 {
+                return Ok(None);
+            }
+            Ok(Some(HealResultItem {
+                heal_item_type: if object.is_some() {
+                    HealItemType::Object
+                } else {
+                    HealItemType::Bucket
+                }
+                .to_string(),
+                object: object.or(prefix).unwrap_or_default(),
+                bucket,
+                detail: format!(
+                    "delete recovery{}: discovered={}, committed={}, rolled_back={}, reclaimed={}, blocked={}",
+                    if opts.dry_run { " (dry run)" } else { "" },
+                    summary.discovered,
+                    summary.committed,
+                    summary.rolled_back,
+                    summary.reclaimed,
+                    summary.blocked
+                ),
+                ..Default::default()
+            }))
+        })
+        .await
+    }
+
     pub async fn heal_bucket_at_incarnation(
         self: &Arc<Self>,
         bucket: &str,