fix(test): compare legacy body wire parity as raw bytes

Lossy UTF-8 comparison concealed a changed binary response byte.
Compare the already captured Vec<u8> values directly.

Validation:
- Binary mutant: migrated payload 0x84 vs legacy 0x83; old comparator
  passed 1/1, raw comparator failed 0/1 (exit 100); restored green.
- Non-Swift body-drain mutant: untouched-body delegation test failed
  with empty vs payload (exit 100); restored handler without a diff.
- cargo nextest run --profile ci -p rustfs-s3-types -p rustfs-protocols
  --features rustfs-protocols/swift --no-fail-fast:
  543 tests run, 543 passed, 0 skipped after both mutants restored.
- cargo fmt --all --check and git diff --check passed.

.agents/skills/security-advisory-lessons/references/advisory-patterns.md

@@ -45,13 +45,11 @@ Update this file only when an advisory adds or changes a reusable lesson, affect
 - `GHSA-6r96-hmgc-726c`: request headers collided with lowercase server-derived condition keys such as `userid`, `groups`, `versionid`, and JWT/LDAP claims. Lesson: never let caller-controlled headers append to or replace server-derived policy context; reserve trusted condition keys and keep intentional request-header keys separate.
 - `GHSA-v9cp-qfw9-9pfp`: quantified negated string conditions applied negation after aggregation, transposing `ForAllValues` and `ForAnyValue` semantics. Lesson: push negation into the per-value predicate for quantified operators and test partially overlapping multi-value sets.
 - `GHSA-5w8r-p896-6vq2`: OPA policy mode skipped `ExistingObjectTag/*` loading, so tagged objects looked untagged to external policies. Lesson: external authorization plugins need the same object-tag and request context as built-in policy evaluation before they decide.
-- `GHSA-9wgh-5r68-49p7`: substituted policy-variable values could recreate `${...}` and loop inside a single resolution pass, never reaching the outer iteration limit. Lesson: bound substitutions within each pass and test cycles formed across otherwise literal resolver values; confirm whether attacker-controlled claims can reach those values before assigning remote impact.
 
 ### S3 object actions, copy, multipart, and upload policy validation
 
 - `GHSA-g8w9-qw9q-fghr` and `GHSA-xm99-m3gq-83g8`: a valid presigned `PutObject` accepted extra `x-amz-*` headers omitted from `SignedHeaders`; `x-amz-copy-source` could turn that upload capability into a cross-bucket read performed as the signer. Lesson: a presigned URL is a bounded capability; inspect all received security-sensitive headers and reject unsigned ones before selecting the S3 operation or reaching storage.
 - `GHSA-3ppv-fx5m-m749`: explicit `versionId` reads and copy sources authorized `s3:GetObject` instead of `s3:GetObjectVersion`. Lesson: version-specific object access must select version-specific actions for direct reads, `CopyObject`, and `UploadPartCopy`, with tests proving the backend is not reached on denial.
-- `GHSA-cg33-pxfp-qcj3` and `GHSA-65wq-gjw8-xr9w`: `HeadObject` and object-tagging paths authorized current-object actions while operating on an explicit historical `versionId`. Lesson: select the corresponding version action for metadata and tag reads and writes too; audit every operation that accepts a version ID.
 - `GHSA-gjph-xcgw-4p7w`: `PutObject` and browser POST accepted a caller-supplied `versionId` and replaced an unlocked historical version with only `s3:PutObject`. Lesson: only replication-authorized writes may choose an existing version ID; ordinary uploads must not rewrite version history.
 - `GHSA-x298-9x87-fvjq`: anonymous `ListObjectVersions` fell back to `ListBucket` and returned before public-access-block gates. Lesson: compatibility fallbacks must converge on the same post-authorization checks as direct grants, especially `RestrictPublicBuckets` and anonymous data-plane denies.
 - `GHSA-mx42-j6wv-px98`: `UploadPartCopy` missed source authorization and allowed cross-bucket object exfiltration. Lesson: multipart copy must enforce the same source and destination contract as `CopyObject`.
@@ -112,7 +110,6 @@ Update this file only when an advisory adds or changes a reusable lesson, affect
 
 - `GHSA-j548-9grx-fh4f`: Object Lock enforcement treated unreadable, fabricated, or unparsable bucket metadata as absent configuration and allowed retained objects to be deleted or expired. Lesson: retention must fail closed unless Object Lock absence is authoritative, and every delete, lifecycle, scanner, force-delete, and default-retention path needs the same state distinction.
 - `GHSA-28w4-rp2p-636w` and `GHSA-c7hx-vr97-45rg`: an authorized `DeleteBucket` with `x-rustfs-force-delete: true` removed active COMPLIANCE-retained versions even though ordinary version deletion was denied. Lesson: bucket force-delete must enforce retention across every version, including against root/owner callers.
-- `GHSA-q23w-fp7p-472v`: `PutObjectRetention` policy conditions read `x-amz-object-lock-*` headers while the handler applied mode and date from the XML body. Lesson: authorize the parsed retention that will be stored, or reject mismatched header/body values; compare retention dates as instants so offsets cannot change the applied bound.
 
 ### Serde deserialization and input validation
 
@@ -146,13 +143,13 @@ rg -n "deny_unknown_fields|serde.default|as u32|as usize|as i32" rustfs crates
 - IAM fixes: include import/update/list service-account cases with attacker-controlled parent, claims, access key, secret key, and policy.
 - Copy/upload fixes: include cross-bucket, cross-user, source-denied, destination-denied, copy-source-condition, and multipart completion cases.
 - Presigned upload fixes: include a valid presign with extra unsigned tagging, redirect, storage-class, and cross-bucket copy-source headers; require rejection before source or destination storage access, and verify explicitly signed equivalents still work.
-- Version-action fixes: include historical UUID, explicit current version, `null`, range, partNumber, presigned, STS/session, service-account, anonymous bucket-policy, copy source, multipart-copy source, `HeadObject`, and tag read/write cases.
+- Version-action fixes: include historical UUID, explicit current version, `null`, range, partNumber, presigned, STS/session, service-account, anonymous bucket-policy, copy source, and multipart-copy source cases.
 - Version-write fixes: a `PutObject`-only principal must not replace an existing version through `PutObject` or browser POST `?versionId=`; verify replication-authorized writes and versioned, suspended, and unversioned buckets separately.
-- Policy-condition fixes: include reserved-key header collisions, missing keys, partially overlapping multi-value sets, cyclic variable substitutions, plugin mode, and built-in policy mode.
+- Policy-condition fixes: include reserved-key header collisions, missing keys, partially overlapping multi-value sets, plugin mode, and built-in policy mode.
 - Path fixes: include encoded traversal, absolute path, nested traversal, archive entries with `..`, valid object keys that resemble traversal text but should be rejected, and canonical bucket/prefix boundary checks; test per-key `DeleteObjects` errors and prefix-scoped denials on versioned buckets.
 - Logging fixes: assert redacted output for structs and response bodies that may contain credentials.
 - IAM export fixes: assert exported archives omit plaintext user and service-account secrets unless the format deliberately encrypts or seals them.
 - RPC auth fixes: include captured metadata replay across two concrete methods, stale timestamps, wrong path, wrong method surrogate, wrong secret, and valid same-method calls.
 - Browser/CORS fixes: assert no credentials on reflected/default origins, correct behavior for explicit allowlists, and no same-origin script execution for previewed object content.
 - SSE fixes: inspect stored bytes and verify API metadata, read-back behavior, and on-disk ciphertext together; for ETags, upload equal plaintext under different keys and assert list-only callers cannot derive or correlate the plaintext digest, including multipart objects.
-- Object Lock fixes: include unreadable metadata, fabricated metadata defaults, unparsable config, confirmed absent config, COMPLIANCE/GOVERNANCE retention, lifecycle expiry, scanner sweeps, and force-delete paths; an authorized force-delete must preserve active COMPLIANCE versions. For `PutObjectRetention`, test conflicting header/body values and offset dates against policy bounds and read-back retention.
+- Object Lock fixes: include unreadable metadata, fabricated metadata defaults, unparsable config, confirmed absent config, COMPLIANCE/GOVERNANCE retention, lifecycle expiry, scanner sweeps, and force-delete paths; an authorized force-delete must preserve active COMPLIANCE versions.

.config/e2e-full-selection.txt

@@ -1,2 +1,2 @@
-sha256-darwin=1520cd3027c3a6924afc6a204eddd5475bc30f0109d4c2847456a66177263244
-sha256-linux=26155b7f1c5bf6bf596ff24a4136e6767ea4ee6461c32db7fbe1400791da2c28
+sha256-darwin=b9438fd1b82558f1fcd7e5a19dc0ceb0b34b5357d4ce99b7033de3ef019333ab
+sha256-linux=d4622517c13283247a022f49962c2cdb69e5ed0e13874fb55e0ab34fb923c044

.config/e2e-nightly-selection.txt

@@ -1,2 +1,2 @@
-sha256-darwin=347a0d4e5e547d5388703062ce66a66167a845b6e57ec591ccb4c732a567c68d
-sha256-linux=4f9efa47194279185bb68249e82c788fa147abe944b6c764050e3d47f31e7926
+sha256-darwin=efe68675f4676a0d2901242c5815b1f7af5cdec8231f7cccd408f6dacf5c75db
+sha256-linux=003ed6480935a0a95680cc061be143722fbd37bd477b8ff98607b1a064f3aef1

.config/e2e-smoke-selection.txt

@@ -1 +1 @@
-sha256=ba8f392b617cf9f2338142233a417e43927b28096635e628233ce8a60cf49d10
+sha256=58df0d240cd74b916ce8f094ed1d8ad3e136f2f9bac157a155192a8786dae65c

.config/ecstore-required-tests.json

@@ -161,11 +161,6 @@
       "suite": "rustfs-ecstore",
       "name": "set_disk::core::metadata_quorum::tests::metadata_observation_all_four_slot_states_and_arrival_orders_preserve_reduction"
     },
-    {
-      "invariant": "metadata-observation-mixed-current-identity-arrivals-preserve-selection",
-      "suite": "rustfs-ecstore",
-      "name": "set_disk::core::metadata_quorum::tests::metadata_observation_mixed_current_identity_arrivals_preserve_selection"
-    },
     {
       "invariant": "metadata-observation-pending-newer-version-and-same-time-different-directory-force-full-wait",
       "suite": "rustfs-ecstore",
@@ -191,11 +186,6 @@
       "suite": "rustfs-ecstore",
       "name": "set_disk::read::metadata_cache_tests::late_metadata_refresh_rejects_changed_generation_and_mapped_shards"
     },
-    {
-      "invariant": "late-full-get-rejects-changed-object-generation",
-      "suite": "rustfs-ecstore",
-      "name": "set_disk::prepared_get_object_metadata_tests::non_inline_two_phase_get_rejects_generation_change_before_late_refresh"
-    },
     {
       "invariant": "list-objects-shared-corpus-static-namespace",
       "suite": "rustfs-ecstore",

.config/make/lint-fmt.mak

@@ -70,6 +70,11 @@ s3s-footprint-check: ## Check the s3s dependency footprint ratchet stays frozen
 	@echo "📦 Checking s3s footprint ratchet..."
 	./scripts/check_s3s_footprint.sh
 
+.PHONY: test-count-ratchet-check
+test-count-ratchet-check: ## Check no workspace crate lost tests or gained #[ignore] against the committed baseline
+	@echo "🧪 Checking per-crate test-count ratchet..."
+	./scripts/check_test_count_ratchet.sh
+
 .PHONY: fips-wording-check
 fips-wording-check: ## Check docs and crates/kms do not over-claim crypto capabilities
 	@echo "📣 Checking cryptographic capability wording guard..."

.config/make/pre-commit.mak

@@ -20,11 +20,11 @@ planning-docs-check: ## Check that no planning-type documents are committed
 	./scripts/check_no_planning_docs.sh
 
 .PHONY: pre-commit
-pre-commit: fmt-check unsafe-code-check architecture-migration-check logging-guardrails-check error-other-ratchet-check tokio-io-uring-check extension-schema-check body-cache-whitelist-check s3s-footprint-check fips-wording-check embedded-secrets-check test-wiring-check doc-paths-check planning-docs-check quick-check ## Run fast pre-commit checks without clippy/full tests
+pre-commit: fmt-check unsafe-code-check architecture-migration-check logging-guardrails-check error-other-ratchet-check tokio-io-uring-check extension-schema-check body-cache-whitelist-check s3s-footprint-check test-count-ratchet-check fips-wording-check embedded-secrets-check test-wiring-check doc-paths-check planning-docs-check quick-check ## Run fast pre-commit checks without clippy/full tests
 	@echo "✅ All pre-commit checks passed!"
 
 .PHONY: pre-pr
-pre-pr: fmt-check unsafe-code-check architecture-migration-check logging-guardrails-check error-other-ratchet-check tokio-io-uring-check extension-schema-check body-cache-whitelist-check s3s-footprint-check fips-wording-check embedded-secrets-check test-wiring-check doc-paths-check planning-docs-check log-analyzer-rules-check offline-enrollment-e2e-check clippy-check test ## Run full pre-PR checks with clippy and tests
+pre-pr: fmt-check unsafe-code-check architecture-migration-check logging-guardrails-check error-other-ratchet-check tokio-io-uring-check extension-schema-check body-cache-whitelist-check s3s-footprint-check test-count-ratchet-check fips-wording-check embedded-secrets-check test-wiring-check doc-paths-check planning-docs-check log-analyzer-rules-check offline-enrollment-e2e-check clippy-check test ## Run full pre-PR checks with clippy and tests
 	@echo "✅ All pre-PR checks passed!"
 
 .PHONY: dev-check

.config/make/tests.mak

@@ -50,11 +50,10 @@ script-tests: ## Run shell script tests
 	./scripts/test_fuzz_runner.sh
 	./scripts/test_python_bin.sh
 	./scripts/check_embedded_secrets.sh --self-test
+	./scripts/check_s3s_footprint.sh --self-test
 	$(RUSTFS_PYTHON_BIN) ./scripts/check_test_wiring.py --self-test
+	$(RUSTFS_PYTHON_BIN) ./scripts/codemods/s3_error_to_s3_types.py --self-test
 	$(RUSTFS_PYTHON_BIN) ./scripts/test_e2e_binary.py
-	$(RUSTFS_PYTHON_BIN) ./scripts/test_release_build_workflow.py
-	$(RUSTFS_PYTHON_BIN) ./scripts/test_release_glibc.py
-	$(RUSTFS_PYTHON_BIN) ./scripts/test_upgrade_workflow.py
 	$(RUSTFS_PYTHON_BIN) ./scripts/test_migration_gate_evidence.py
 	$(RUSTFS_PYTHON_BIN) ./scripts/ci_gate.py --self-test
 	$(RUSTFS_PYTHON_BIN) ./scripts/check_security_coverage.py --self-test

.config/nextest.toml

@@ -88,21 +88,10 @@ threads-required = "num-test-threads"
 filter = 'package(rustfs-ecstore) & test(/^(bucket::lifecycle::bucket_lifecycle_ops::tests::manual_transition_worker_result_recovery_marks_unknown_for_corrupt_marker|services::rebalance::entry::tests::real_rebalance_run_fence_loss_blocks_multipart_publication|store::init::tests::(batch_transitioned_delete_uses_free_version_per_item|decommission_entry_(allows_free_version_consumed_before_source_lock|rejects_subquorum_free_version_conflict_and_retains_source|skips_cleanup_only_marker_when_free_version_is_present)|dispatched_tier_delete_recovery_(checks_later_pool_then_commits_after_source_removal|finds_directory_source_on_encoded_set|retains_journal_on_source_metadata_error)|force_tier_remove_blocks_on_physical_free_version_hidden_by_other_pool|legacy_unknown_transition_delete_falls_back_for_single_batch_and_blocks_prefix|multi_pool_(recursive_prefix_rejects_legacy_or_hidden_merge_loser_before_delete|same_remote_tuple_(batch|single)_delete_waits_for_all_sources|same_tuple_recursive_prefix_uses_one_journal_owner|transitioned_delete_persists_one_free_version_per_remote_tuple)|recursive_prefix_partial_(pool|set)_failure_keeps_prepared_cleanup_owners|restored_transitioned_delete_uses_free_version_as_cleanup_owner|stable_transitioned_recursive_prefix_delete_uses_journal_owners|suspended_null_transition_delete_uses_free_version_as_sole_owner|tier_mutation_peer_handler_applies_prepare_commit_and_abort_idempotently|transition_response_loss_persists_unknown_outcome_for_provider_recovery|transition_transaction_recovery_(drops_record_after_confirmed_local_commit|keeps_cleanup_pending_local_commit)|transitioned_delete_(free_version_replays_after_store_restart|local_quorum_failure_rolls_back_without_cleanup_owner|uses_free_version_as_cleanup_owner)|versioned_delete_marker_keeps_transitioned_source_and_remote_object|versioned_explicit_transition_delete_preserves_other_version_then_allows_bucket_delete))$/)'
 setup = 'ecstore-large-stack'
 
-# These cross-crate real-storage fixtures need the same bounded stack as ECStore tests.
-[[profile.default.scripts]]
-filter = '(package(rustfs) & test(/^(admin::handlers::site_replication::tests::durable_iam_state_apply_iam_item_rejects_a_stale_recreate_after_a_replicated_delete|admin::service::federated_identity::tests::(binding_keeps_explicit_iam_mapping_strict|markerless_legacy_oidc_session_crosses_auth_and_iam_deletion_boundaries)|storage::scanner_dirty_journal::tests::(clear_callback_keeps_unknown_coverage_until_all_pending_work_is_confirmed|owned_journals_keep_peer_records_after_interleaved_writes_and_clear))$/)) | (package(rustfs-heal) & ((binary(heal_b5_versioned_regression_test) & test(/^serial_tests::(test_heal_unversioned_bucket_e2e|test_historical_null_recursive_heal_restores_physical_versions|test_recursive_deep_heal_reports_stale_delete_marker_repair)$/)) | (binary(usage_observation_read_repair_test) & test(=usage_observation_read_repair_survives_snapshot_cleanup)))) | (package(rustfs-test-utils) & test(=data_usage_snapshot_tests::data_usage_snapshot_recovery_wires_primary_backup_and_negative_cache))'
-setup = 'ecstore-base-stack'
-
 [[profile.default.scripts]]
 filter = 'package(rustfs-ecstore) | package(rustfs-s3select-api) | package(rustfs-scanner) | (package(rustfs) & test(/^(app::multipart_usecase::tests::concurrent_completions_share_durable_bucket_quota_reservations|app::object::delete::tests::compressed_delete_requests_update_observed_usage_without_releasing_quota_floor|app::object::internal_put::tests::internal_multipart_roundtrip_completes_and_abort_leaves_nothing|app::object::restore::tests::execute_restore_object_maps_failures_to_typed_s3_errors|storage::access::tests::(delete_object_access_captures_authorized_bucket_incarnation|copy_operations_reject_recreated_source_bucket_after_authorization|request_slot_keeps_bucket_policy_bound_to_its_store))$/))'
 setup = 'ecstore-base-stack'
 
-# These handler/store fixtures and the relocated-pool GET compose deep storage
-# futures in libtest threads; on Linux they overflow the default 2 MiB stack.
-[[profile.default.scripts]]
-filter = 'package(rustfs) & (test(/^admin::handlers::table_catalog::tests::/) | test(/^table_catalog::store::tests::/) | test(=app::object::get::tests::execute_get_object_resumes_from_relocated_pool_without_splicing_body))'
-setup = 'ecstore-base-stack'
-
 [[profile.default.scripts]]
 filter = 'binary(lifecycle_integration_test) | (package(rustfs) & test(/^app::lifecycle_transition_api_test::/))'
 setup = 'lifecycle-large-stack'
@@ -344,19 +333,10 @@ threads-required = "num-test-threads"
 filter = 'package(rustfs-ecstore) & test(/^(bucket::lifecycle::bucket_lifecycle_ops::tests::manual_transition_worker_result_recovery_marks_unknown_for_corrupt_marker|services::rebalance::entry::tests::real_rebalance_run_fence_loss_blocks_multipart_publication|store::init::tests::(batch_transitioned_delete_uses_free_version_per_item|decommission_entry_(allows_free_version_consumed_before_source_lock|rejects_subquorum_free_version_conflict_and_retains_source|skips_cleanup_only_marker_when_free_version_is_present)|dispatched_tier_delete_recovery_(checks_later_pool_then_commits_after_source_removal|finds_directory_source_on_encoded_set|retains_journal_on_source_metadata_error)|force_tier_remove_blocks_on_physical_free_version_hidden_by_other_pool|legacy_unknown_transition_delete_falls_back_for_single_batch_and_blocks_prefix|multi_pool_(recursive_prefix_rejects_legacy_or_hidden_merge_loser_before_delete|same_remote_tuple_(batch|single)_delete_waits_for_all_sources|same_tuple_recursive_prefix_uses_one_journal_owner|transitioned_delete_persists_one_free_version_per_remote_tuple)|recursive_prefix_partial_(pool|set)_failure_keeps_prepared_cleanup_owners|restored_transitioned_delete_uses_free_version_as_cleanup_owner|stable_transitioned_recursive_prefix_delete_uses_journal_owners|suspended_null_transition_delete_uses_free_version_as_sole_owner|tier_mutation_peer_handler_applies_prepare_commit_and_abort_idempotently|transition_response_loss_persists_unknown_outcome_for_provider_recovery|transition_transaction_recovery_(drops_record_after_confirmed_local_commit|keeps_cleanup_pending_local_commit)|transitioned_delete_(free_version_replays_after_store_restart|local_quorum_failure_rolls_back_without_cleanup_owner|uses_free_version_as_cleanup_owner)|versioned_delete_marker_keeps_transitioned_source_and_remote_object|versioned_explicit_transition_delete_preserves_other_version_then_allows_bucket_delete))$/)'
 setup = 'ecstore-large-stack'
 
-# These cross-crate real-storage fixtures need the same bounded stack as ECStore tests.
-[[profile.ci.scripts]]
-filter = '(package(rustfs) & test(/^(admin::handlers::site_replication::tests::durable_iam_state_apply_iam_item_rejects_a_stale_recreate_after_a_replicated_delete|admin::service::federated_identity::tests::(binding_keeps_explicit_iam_mapping_strict|markerless_legacy_oidc_session_crosses_auth_and_iam_deletion_boundaries)|storage::scanner_dirty_journal::tests::(clear_callback_keeps_unknown_coverage_until_all_pending_work_is_confirmed|owned_journals_keep_peer_records_after_interleaved_writes_and_clear))$/)) | (package(rustfs-heal) & ((binary(heal_b5_versioned_regression_test) & test(/^serial_tests::(test_heal_unversioned_bucket_e2e|test_historical_null_recursive_heal_restores_physical_versions|test_recursive_deep_heal_reports_stale_delete_marker_repair)$/)) | (binary(usage_observation_read_repair_test) & test(=usage_observation_read_repair_survives_snapshot_cleanup)))) | (package(rustfs-test-utils) & test(=data_usage_snapshot_tests::data_usage_snapshot_recovery_wires_primary_backup_and_negative_cache))'
-setup = 'ecstore-base-stack'
-
 [[profile.ci.scripts]]
 filter = 'package(rustfs-ecstore) | package(rustfs-s3select-api) | package(rustfs-scanner) | (package(rustfs) & test(/^(app::multipart_usecase::tests::concurrent_completions_share_durable_bucket_quota_reservations|app::object::delete::tests::compressed_delete_requests_update_observed_usage_without_releasing_quota_floor|app::object::internal_put::tests::internal_multipart_roundtrip_completes_and_abort_leaves_nothing|app::object::restore::tests::execute_restore_object_maps_failures_to_typed_s3_errors|storage::access::tests::(delete_object_access_captures_authorized_bucket_incarnation|copy_operations_reject_recreated_source_bucket_after_authorization|request_slot_keeps_bucket_policy_bound_to_its_store))$/))'
 setup = 'ecstore-base-stack'
 
-[[profile.ci.scripts]]
-filter = 'package(rustfs) & (test(/^admin::handlers::table_catalog::tests::/) | test(/^table_catalog::store::tests::/) | test(=app::object::get::tests::execute_get_object_resumes_from_relocated_pool_without_splicing_body))'
-setup = 'ecstore-base-stack'
-
 [[profile.ci.scripts]]
 filter = 'binary(lifecycle_integration_test) | (package(rustfs) & test(/^app::lifecycle_transition_api_test::/))'
 setup = 'lifecycle-large-stack'
@@ -599,6 +579,14 @@ test-group = 'ecstore-serial-flaky'
 # the PR-lane e2e coverage for scanner usage snapshots consumed by quota and
 # admin surfaces. It uses the same single-node, random-port, isolated-temp-dir
 # fixture as the existing smoke modules.
+#
+# Dual-stack differential (rustfs/backlog#2740): `dual_stack_differential::`
+# starts two single-node servers from the same binary, one per
+# RUSTFS_S3_STACK value, on random ports with isolated temp dirs, and runs one
+# SDK script covering every operation of `impl S3 for FS` against both. Its
+# comparator unit tests need no server. It meets the admission criteria
+# unchanged; the module clause admits a new differential test without another
+# filter edit.
 [profile.e2e-smoke]
 default-filter = """
     package(e2e_test) & (
@@ -606,6 +594,7 @@ default-filter = """
         | test(/^replication_extension_test::(test_replication_check_succeeds_with_remote_target|test_replication_check_rejects_target_without_object_lock|test_set_remote_target_rejects_unversioned_source_bucket|test_replication_check_rejects_unversioned_source_bucket|test_replication_check_rejects_missing_replication_config|test_replication_check_rejects_invalid_bucket|test_set_remote_target_rejects_same_bucket_on_same_deployment|test_set_remote_target_rejects_unversioned_target_bucket|test_set_remote_target_update_requires_arn|test_set_remote_target_update_rejects_missing_target|test_set_remote_target_rejects_invalid_target_url|test_set_remote_target_rejects_self_signed_https_target_without_skip_tls_verify|test_set_remote_target_rejects_private_ca_https_target_without_ca_cert_pem|test_list_remote_targets_rejects_empty_bucket|test_list_remote_targets_rejects_invalid_bucket|test_remove_remote_target_rejects_missing_target|test_remove_remote_target_rejects_missing_arn|test_remove_remote_target_rejects_invalid_bucket|test_remove_remote_target_rejects_target_used_by_replication|test_delete_bucket_replication_removes_remote_target)$/)
         | test(/^reliant::lifecycle::/)
         | test(/^reliant::tiering::/)
+        | test(/^dual_stack_differential::/)
         | test(/^upgrade_compatibility_test::upgrade_write_readiness_tests::/)
         | test(/^on_demand_migration::(get_basic_test::(get_miss_pulls_inline_and_serves_locally_afterwards|head_miss_answers_from_the_source_without_persisting)|interaction_test::test_odm_admin_config_is_redacted_and_status_counts_match_the_source)$/)
     )

.config/s3s-edge-allowlist.txt

@@ -0,0 +1,37 @@
+# Files that may still reference s3s (`s3s::` or `s3s_sigv4`) when Phase 1 of
+# the gateway migration ends (rustfs/backlog#2734 task T0.6, rustfs/backlog#2741).
+# Consumed only by `scripts/check_s3s_footprint.sh --mode allowlist`.
+#
+# One entry per line: a repository-relative file path or a git `:(glob)`
+# pathspec. `**` matches across directories, `*` stays within one path
+# component, and a directory entry matches everything under it. Blank lines
+# and lines starting with `#` are ignored; an entry may not contain whitespace,
+# so a note belongs on its own comment line.
+#
+# Every Rust file listed by `git grep -l '\bs3s\(::\|_sigv4\)' -- '*.rs'` that
+# no entry matches fails the allowlist mode. Task T1.9 switches CI to that
+# mode; until then `--dry-run` prints the files still outside this list, which
+# is the Phase 1 tracking list.
+#
+# Never add crates/** other than crates/e2e_test: the engine crates leave s3s
+# entirely, and the script rejects such an entry. Only the s3s-facing edge of
+# the rustfs binary and the e2e oracle keep it.
+
+# rustfs binary: the s3s-facing edge that survives Phase 1
+rustfs/src/storage/ecfs.rs
+rustfs/src/storage/s3_compat/**
+rustfs/src/storage/access.rs
+rustfs/src/auth.rs
+rustfs/src/error.rs
+rustfs/src/storage_api.rs
+rustfs/src/app/storage_api.rs
+rustfs/src/admin/router.rs
+rustfs/src/app/metadata_route.rs
+rustfs/src/server/http.rs
+rustfs/src/server/layer.rs
+rustfs/src/server/hybrid.rs
+rustfs/src/protocols/client.rs
+
+# e2e oracle: the test target that verifies S3 behaviour against s3s
+crates/e2e_test/src/fake_s3_target/**
+crates/e2e_test/src/**

.config/s3s-footprint-baseline.txt

@@ -0,0 +1,17 @@
+# Exact baseline for the direct s3s footprint, consumed only by
+# scripts/check_s3s_footprint.sh (rustfs/backlog#1677; rustfs/backlog#2734 T0.4).
+#
+# Each value is the exact measurement of the tree, not a ceiling. CI fails when a
+# measured counter differs from its line here in either direction:
+#   - a larger measurement means new code widened the s3s surface: route it
+#     through the gateway abstractions instead; never raise a value here;
+#   - a smaller measurement means the PR shrank the surface: lower the matching
+#     value to the measured number in the same PR.
+#
+# files           Rust files referencing s3s paths, repo-wide minus crates/e2e_test
+# s3_error_lines  lines holding s3_error!, same scope, minus those the script
+#                 header's rule clears (rustfs_s3_types's macro); fails closed
+# ecstore_files   Rust files referencing s3s paths under crates/ecstore/src
+files=195
+s3_error_lines=1584
+ecstore_files=36

.config/test-count-baseline.txt

@@ -0,0 +1,63 @@
+# Per-crate test-count baseline consumed by scripts/check_test_count_ratchet.sh
+# (rustfs/backlog#2734 section 1 rule 3; T0.3 in rustfs/backlog#2738).
+#
+# One row per workspace member: <package>, tests, ignored, tab-separated.
+# tests counts #[test], #[tokio::test], #[tokio::test(...)] and #[test_case(...)]
+# attribute lines under the package directory; ignored counts #[ignore] and
+# #[ignore = "..."]. Both columns are exact: CI fails when a crate's tests fall
+# below or rise above its row, or when its ignored count changes. Raise a row in
+# the PR that adds tests; lowering one is a reviewed statement that tests were
+# intentionally removed. Regenerate with:
+#   scripts/check_test_count_ratchet.sh --update
+e2e_test	890	18
+rustfs	5498	62
+rustfs-audit	70	0
+rustfs-checksums	26	0
+rustfs-common	23	0
+rustfs-concurrency	27	0
+rustfs-config	43	0
+rustfs-connect-symbol-catalog	6	0
+rustfs-credentials	24	0
+rustfs-crypto	67	0
+rustfs-data-usage	54	0
+rustfs-ecstore	6222	1
+rustfs-extension-schema	23	0
+rustfs-filemeta	306	0
+rustfs-heal	836	3
+rustfs-heal-contracts	6	0
+rustfs-iam	357	0
+rustfs-io-core	49	0
+rustfs-io-metrics	123	0
+rustfs-keystone	29	0
+rustfs-kms	715	25
+rustfs-license	2	0
+rustfs-lifecycle	152	0
+rustfs-lock	135	0
+rustfs-log-analyzer	87	0
+rustfs-madmin	209	0
+rustfs-notify	136	0
+rustfs-object-capacity	92	0
+rustfs-object-data-cache	140	0
+rustfs-obs	362	0
+rustfs-policy	663	0
+rustfs-protocols	736	1
+rustfs-protos	52	0
+rustfs-replication	200	0
+rustfs-rio	198	0
+rustfs-rio-v2	34	6
+rustfs-s3-client	65	0
+rustfs-s3-ops	8	0
+rustfs-s3-types	97	0
+rustfs-s3select-api	206	0
+rustfs-s3select-query	188	2
+rustfs-scanner	971	20
+rustfs-scanner-metrics	75	0
+rustfs-security-governance	20	0
+rustfs-signer	31	0
+rustfs-storage-api	31	0
+rustfs-targets	603	26
+rustfs-test-utils	1	0
+rustfs-tls-runtime	20	0
+rustfs-trusted-proxies	68	0
+rustfs-utils	216	0
+rustfs-zip	30	0

.docker/observability/grafana/dashboards/rustfs.json

@@ -1748,7 +1748,7 @@
             "uid": "${datasource}"
           },
           "editorMode": "code",
-          "expr": "max by (rustfs_cluster_id, job, bucket) (rustfs_cluster_usage_buckets_objects_count{rustfs_cluster_id=\"$storage_cluster\", job=~\"$job\", bucket=~\"$bucket\"})",
+          "expr": "max by (job, bucket) (rustfs_cluster_usage_buckets_objects_count{job=~\"$job\", bucket=~\"$bucket\"})",
           "legendFormat": "{{bucket}}",
           "range": true,
           "refId": "A"
@@ -1848,7 +1848,7 @@
             "uid": "${datasource}"
           },
           "editorMode": "code",
-          "expr": "max by (rustfs_cluster_id, job, bucket) (rustfs_cluster_usage_buckets_total_bytes{rustfs_cluster_id=\"$storage_cluster\", job=~\"$job\", bucket=~\"$bucket\"})",
+          "expr": "max by (job, bucket) (rustfs_cluster_usage_buckets_total_bytes{job=~\"$job\", bucket=~\"$bucket\"})",
           "legendFormat": "{{bucket}}",
           "range": true,
           "refId": "A"

.docker/observability/prometheus.yml

@@ -61,7 +61,6 @@ scrape_configs:
 otlp:
   promote_resource_attributes:
     - service.instance.id
-    - process.pid
     - service.name
     - service.namespace
     - cloud.availability_zone

.docker/observability/tests/bucket-usage.test.yml

@@ -1,48 +0,0 @@
-evaluation_interval: 1m
-tests:
-- name: bucket usage keeps clusters separate and deduplicates reporting nodes
-  interval: 1m
-  input_series:
-  - series: rustfs_cluster_usage_buckets_total_bytes{rustfs_cluster_id="a",job="rustfs",bucket="shared",service_instance_id="a1"}
-    values: '4096 1024'
-  - series: rustfs_cluster_usage_buckets_total_bytes{rustfs_cluster_id="a",job="rustfs",bucket="shared",service_instance_id="a2"}
-    values: '4096 1024'
-  - series: rustfs_cluster_usage_buckets_total_bytes{rustfs_cluster_id="b",job="rustfs",bucket="shared",service_instance_id="b1"}
-    values: '65536 65536'
-  - series: rustfs_cluster_usage_buckets_total_bytes{rustfs_cluster_id="a",job="rustfs",bucket="empty",service_instance_id="a1"}
-    values: '0 0'
-  - series: rustfs_cluster_usage_buckets_objects_count{rustfs_cluster_id="a",job="rustfs",bucket="shared",service_instance_id="a1"}
-    values: '3 1'
-  - series: rustfs_cluster_usage_buckets_objects_count{rustfs_cluster_id="a",job="rustfs",bucket="shared",service_instance_id="a2"}
-    values: '3 1'
-  - series: rustfs_cluster_usage_buckets_objects_count{rustfs_cluster_id="b",job="rustfs",bucket="shared",service_instance_id="b1"}
-    values: '70 70'
-  - series: rustfs_cluster_usage_buckets_objects_count{rustfs_cluster_id="a",job="rustfs",bucket="empty",service_instance_id="a1"}
-    values: '0 0'
-  promql_expr_test:
-  - expr: max by (rustfs_cluster_id, job, bucket) (rustfs_cluster_usage_buckets_total_bytes{rustfs_cluster_id="a",job=~"rustfs",bucket=~".*"})
-    eval_time: 1m
-    exp_samples:
-    - labels: '{rustfs_cluster_id="a",job="rustfs",bucket="shared"}'
-      value: 1024
-    - labels: '{rustfs_cluster_id="a",job="rustfs",bucket="empty"}'
-      value: 0
-  - expr: max by (rustfs_cluster_id, job, bucket) (rustfs_cluster_usage_buckets_objects_count{rustfs_cluster_id="a",job=~"rustfs",bucket=~".*"})
-    eval_time: 1m
-    exp_samples:
-    - labels: '{rustfs_cluster_id="a",job="rustfs",bucket="shared"}'
-      value: 1
-    - labels: '{rustfs_cluster_id="a",job="rustfs",bucket="empty"}'
-      value: 0
-  - expr: max by (rustfs_cluster_id, job, bucket) (rustfs_cluster_usage_buckets_total_bytes)
-    eval_time: 1m
-    exp_samples:
-    - labels: '{rustfs_cluster_id="a",job="rustfs",bucket="shared"}'
-      value: 1024
-    - labels: '{rustfs_cluster_id="a",job="rustfs",bucket="empty"}'
-      value: 0
-    - labels: '{rustfs_cluster_id="b",job="rustfs",bucket="shared"}'
-      value: 65536
-  - expr: rustfs_cluster_usage_buckets_total_bytes{rustfs_cluster_id="a",bucket="unknown"}
-    eval_time: 1m
-    exp_samples: []

.github/actions/quick-checks/action.yml

@@ -86,6 +86,10 @@ runs:
       shell: bash
       run: ./scripts/check_s3s_footprint.sh
 
+    - name: Check per-crate test-count ratchet
+      shell: bash
+      run: ./scripts/check_test_count_ratchet.sh
+
     - name: Check cryptographic capability wording
       shell: bash
       run: ./scripts/check_fips_wording.sh

.github/actions/setup-docker-compose/action.yml

@@ -1,39 +0,0 @@
-# Copyright 2024 RustFS Team
-#
-# Licensed under the Apache License, Version 2.0 (the "License");
-# you may not use this file except in compliance with the License.
-# You may obtain a copy of the License at
-#
-#     http://www.apache.org/licenses/LICENSE-2.0
-#
-# Unless required by applicable law or agreed to in writing, software
-# distributed under the License is distributed on an "AS IS" BASIS,
-# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
-# See the License for the specific language governing permissions and
-# limitations under the License.
-
-name: Set up Docker Compose
-description: Install the verified Compose CLI used by native Linux x86_64 Connect acceptance
-
-runs:
-  using: composite
-  steps:
-    - name: Install verified Docker Compose CLI plugin
-      shell: bash
-      run: |
-        set -euo pipefail
-        [[ $(uname -s) == Linux && $(uname -m) == x86_64 ]]
-        plugin_dir="${DOCKER_CONFIG:-$HOME/.docker}/cli-plugins"
-        mkdir -p "$plugin_dir"
-        compose_download=$(mktemp "${RUNNER_TEMP:?}/connect-compose.XXXXXX")
-        compose_staged=$(mktemp "$plugin_dir/.docker-compose.XXXXXX")
-        trap 'rm -f "$compose_download" "$compose_staged"' EXIT
-        curl --fail --location --retry 3 \
-          https://github.com/docker/compose/releases/download/v2.40.3/docker-compose-linux-x86_64 \
-          --output "$compose_download"
-        printf '%s  %s\n' \
-          dba9d98e1ba5bfe11d88c99b9bd32fc4a0624a30fafe68eea34d61a3e42fd372 \
-          "$compose_download" | sha256sum --check --strict
-        install -m 0755 "$compose_download" "$compose_staged"
-        mv -f "$compose_staged" "$plugin_dir/docker-compose"
-        [[ $(docker compose version --short) == 2.40.3 ]]

.github/actions/setup/action.yml

@@ -36,16 +36,6 @@ inputs:
     description: "Install cross-compilation tools"
     required: false
     default: "false"
-  zig-version:
-    description: >-
-      Zig version for cargo-zigbuild builds. Pinned so the `<target>.<glibc>`
-      baseline semantics cannot drift with an untested zig release.
-    required: false
-    default: "0.17.0"
-  cargo-zigbuild-version:
-    description: "cargo-zigbuild version installed for cross builds."
-    required: false
-    default: "0.23.4"
   target:
     description: "Target architecture to add"
     required: false
@@ -109,14 +99,10 @@ runs:
     - name: Install Zig
       if: inputs.install-cross-tools == 'true'
       uses: mlugg/setup-zig@d1434d08867e3ee9daa34448df10607b98908d29 # v2
-      with:
-        version: ${{ inputs.zig-version }}
 
     - name: Install cargo-zigbuild
       if: inputs.install-cross-tools == 'true'
       uses: taiki-e/install-action@a21ae4029b089b9ddc45704028756f51ab8abe48 # cargo-zigbuild
-      with:
-        tool: cargo-zigbuild@${{ inputs.cargo-zigbuild-version }}
 
     - name: Install cargo-nextest
       if: inputs.install-test-tools == 'true'
@@ -140,5 +126,5 @@ runs:
         # normalises, so cargo fingerprints stay valid.
         cache-all-crates: false
         cache-on-failure: true
-        shared-key: ${{ inputs.cache-shared-key }}${{ inputs.install-cross-tools == 'true' && format('-zig-{0}-zigbuild-{1}', inputs.zig-version, inputs.cargo-zigbuild-version) || '' }}
+        shared-key: ${{ inputs.cache-shared-key }}
         save-if: ${{ inputs.cache-save-if }}

.github/workflows/build.yml

@@ -213,12 +213,12 @@ jobs:
           fi
 
           all='{"include":[
-            {"target_id":"linux-x86_64-musl","os":"sm-standard-4","target":"x86_64-unknown-linux-musl","cross":false,"platform":"linux","rustflags":"","features":"","glibc":""},
-            {"target_id":"linux-aarch64-musl","os":"sm-standard-4-arm","target":"aarch64-unknown-linux-musl","cross":false,"platform":"linux","rustflags":"","features":"","glibc":""},
-            {"target_id":"linux-x86_64-gnu","os":"sm-standard-4","target":"x86_64-unknown-linux-gnu","cross":true,"platform":"linux","rustflags":"","features":"pyroscope","glibc":"2.31"},
-            {"target_id":"linux-aarch64-gnu","os":"sm-standard-4-arm","target":"aarch64-unknown-linux-gnu","cross":true,"platform":"linux","rustflags":"","features":"pyroscope","glibc":"2.31"},
-            {"target_id":"macos-aarch64","os":"macos-latest","target":"aarch64-apple-darwin","cross":false,"platform":"macos","rustflags":"","features":"pyroscope","glibc":""},
-            {"target_id":"windows-x86_64","os":"windows-latest","target":"x86_64-pc-windows-msvc","cross":false,"platform":"windows","rustflags":"","features":"","glibc":""}
+            {"target_id":"linux-x86_64-musl","os":"sm-standard-4","target":"x86_64-unknown-linux-musl","cross":false,"platform":"linux","rustflags":"","features":""},
+            {"target_id":"linux-aarch64-musl","os":"sm-standard-4-arm","target":"aarch64-unknown-linux-musl","cross":false,"platform":"linux","rustflags":"","features":""},
+            {"target_id":"linux-x86_64-gnu","os":"sm-standard-4","target":"x86_64-unknown-linux-gnu","cross":false,"platform":"linux","rustflags":"","features":"pyroscope"},
+            {"target_id":"linux-aarch64-gnu","os":"sm-standard-4-arm","target":"aarch64-unknown-linux-gnu","cross":false,"platform":"linux","rustflags":"","features":"pyroscope"},
+            {"target_id":"macos-aarch64","os":"macos-latest","target":"aarch64-apple-darwin","cross":false,"platform":"macos","rustflags":"","features":"pyroscope"},
+            {"target_id":"windows-x86_64","os":"windows-latest","target":"x86_64-pc-windows-msvc","cross":false,"platform":"windows","rustflags":"","features":""}
           ]}'
 
           if [[ "${selected}" == "all" ]]; then
@@ -246,8 +246,7 @@ jobs:
           echo "Selected platforms: ${selected}"
 
   build-rustfs:
-    # Acceptance workflows select artifacts by this stable platform name.
-    name: Build RustFS (${{ matrix.target_id }})
+    name: Build RustFS
     needs: [ build-check, prepare-platform-matrix ]
     if: needs.build-check.outputs.should_build == 'true' && needs.prepare-platform-matrix.result == 'success'
     runs-on: ${{ matrix.os }}
@@ -274,7 +273,7 @@ jobs:
         with:
           rust-version: stable
           target: ${{ matrix.target }}
-          cache-shared-key: build-${{ matrix.target }}${{ matrix.glibc && format('-glibc-{0}', matrix.glibc) || '' }}
+          cache-shared-key: build-${{ matrix.target }}
           # main only. A cache saved on refs/tags/X is scoped to that tag: no
           # other tag, no main run and no PR can restore it, so every release
           # cycle wrote up to 12 entries of 1-2GB (preview tag plus final tag,
@@ -411,64 +410,35 @@ jobs:
         shell: bash
         env:
           BUILD_TARGET: ${{ matrix.target }}
-          BUILD_OS: ${{ matrix.platform }}-${{ startsWith(matrix.target, 'aarch64-') && 'aarch64' || 'x86_64' }}
         run: |
           # Force rebuild by touching build.rs
           touch rustfs/build.rs
 
+          BINARY_NAMES=(rustfs)
+          if [[ "${{ matrix.platform }}" == "linux" ]]; then
+            BINARY_NAMES+=(rustfs-cli)
+          fi
           FEATURE_ARGS=()
           if [[ -n "${{ matrix.features }}" ]]; then
             FEATURE_ARGS=(--features "${{ matrix.features }}")
           fi
 
-          # gnu targets pin an explicit glibc baseline (`<target>.<glibc>` for
-          # cargo-zigbuild) so release binaries link no symbols newer than it
-          # and run on older distros such as RHEL/Rocky 9 (glibc 2.34); #8062.
-          TARGET_ARG="$BUILD_TARGET"
-          if [[ -n "${{ matrix.glibc }}" ]]; then
-            TARGET_ARG="${TARGET_ARG}.${{ matrix.glibc }}"
-          fi
-
-          PROFILE_ARGS=()
-          if [[ "$BUILD_TARGET" == *-linux-gnu && "${{ matrix.features }}" == "pyroscope" ]]; then
-            # The in-process profiler resolves names from the final executable.
-            # Retain function symbols without enabling DWARF source information.
-            PROFILE_ARGS=(--config 'profile.release.strip="none"')
-          fi
-          if [[ "${{ matrix.cross }}" == "true" ]]; then
-            cargo zigbuild --release --locked --target "$TARGET_ARG" "${FEATURE_ARGS[@]}" "${PROFILE_ARGS[@]}" -p rustfs --bin rustfs
-          else
-            cargo build --release --locked --target "$TARGET_ARG" "${FEATURE_ARGS[@]}" "${PROFILE_ARGS[@]}" -p rustfs --bin rustfs
-          fi
-
-          # All release targets use same-architecture runners, including GNU
-          # builds that use Zig only to select the glibc ABI floor.
-          bin="target/${{ matrix.target }}/release/rustfs"
-          if [[ -f "${bin}.exe" ]]; then
-            bin="${bin}.exe"
-          fi
-          version_output=$("$bin" --version)
-          if ! grep -Fxq "build target : ${BUILD_TARGET}" <<< "$version_output" || \
-             ! grep -Fxq "build os     : ${BUILD_OS}" <<< "$version_output" || \
-             ! grep -Fxq "rust channel : stable-${BUILD_TARGET}" <<< "$version_output"; then
-            echo "release binary reports an unexpected compile target" >&2
-            exit 1
-          fi
-          if [[ -n "${{ matrix.glibc }}" ]]; then
-            python3 scripts/release/check_glibc.py "${{ matrix.glibc }}" \
-              "target/${{ matrix.target }}/release/rustfs"
-          fi
-
-      - name: Verify macOS binary has no dynamic liblzma dependency
-        if: matrix.platform == 'macos'
-        shell: bash
-        run: |
-          binary="target/${{ matrix.target }}/release/rustfs"
-          if otool -L "$binary" | grep -E 'liblzma.*\.dylib'; then
-            echo "macOS release binary must not depend on a dynamic liblzma library" >&2
-            exit 1
-          fi
-          "$binary" --version
+          # Build one binary at a time so release LTO links cannot exhaust a
+          # self-hosted runner by running concurrently.
+          for binary in "${BINARY_NAMES[@]}"; do
+            PROFILE_ARGS=()
+            if [[ "$binary" == "rustfs" && "$BUILD_TARGET" == *-linux-gnu && "${{ matrix.features }}" == "pyroscope" ]]; then
+              # The in-process profiler resolves names from the final executable.
+              # Retain function symbols without enabling DWARF source information.
+              PROFILE_ARGS=(--config 'profile.release.strip="none"')
+            fi
+            if [[ "${{ matrix.cross }}" == "true" ]]; then
+              # All cross targets in the matrix are Linux; zigbuild handles them.
+              cargo zigbuild --release --target ${{ matrix.target }} "${FEATURE_ARGS[@]}" "${PROFILE_ARGS[@]}" -p rustfs --bin "$binary"
+            else
+              cargo build --release --target ${{ matrix.target }} "${FEATURE_ARGS[@]}" "${PROFILE_ARGS[@]}" -p rustfs --bin "$binary"
+            fi
+          done
 
       - name: Generate the CPU symbol catalog
         if: endsWith(matrix.target, '-linux-gnu') && matrix.features == 'pyroscope'
@@ -573,6 +543,9 @@ jobs:
           fi
 
           BINARY_FILES=("$BINARY_NAME")
+          if [[ "${{ matrix.platform }}" == "linux" ]]; then
+            BINARY_FILES+=("rustfs-cli")
+          fi
 
           if [[ "$TARGET" == *-linux-gnu && "${{ matrix.features }}" == "pyroscope" ]]; then
             BINARY_FILES+=(rustfs.cpu-symbol-catalog.json rustfs.cpu-symbol-catalog.sha256)
@@ -687,7 +660,7 @@ jobs:
           echo "🔧 Build type: ${BUILD_TYPE}"
           echo "📊 Version: ${VERSION}"
 
-      - name: Verify packaged Linux binary
+      - name: Verify packaged Linux binaries
         if: matrix.platform == 'linux'
         shell: bash
         run: |
@@ -695,7 +668,7 @@ jobs:
 
           package_file="${{ steps.package.outputs.package_file }}"
           archive_members="$(unzip -Z1 "$package_file" | sort)"
-          expected_members='rustfs'
+          expected_members=$'rustfs\nrustfs-cli'
           target="${{ matrix.target }}"
           features="${{ matrix.features }}"
           if [[ "$target" == *-linux-gnu && "$features" == "pyroscope" ]]; then
@@ -707,14 +680,13 @@ jobs:
             exit 1
           fi
 
-          # musl targets build natively; the gnu legs are same-arch
-          # cargo-zigbuild builds pinned to the release glibc baseline. Either
-          # way the binaries run on their runner, so executing them here
+          # Every Linux target builds natively, so executing the binaries here
           # proves each artifact matches its runner architecture.
-          binary_dir="$(mktemp -d)"
-          trap 'rm -rf "$binary_dir"' EXIT
-          unzip -q "$package_file" rustfs -d "$binary_dir"
-          "$binary_dir/rustfs" --version
+          cli_dir="$(mktemp -d)"
+          trap 'rm -rf "$cli_dir"' EXIT
+          unzip -q "$package_file" rustfs rustfs-cli -d "$cli_dir"
+          "$cli_dir/rustfs" --version
+          "$cli_dir/rustfs-cli" --help >/dev/null
 
       - name: Verify packaged console
         if: matrix.target == 'x86_64-unknown-linux-gnu' || matrix.target == 'aarch64-unknown-linux-gnu'

.github/workflows/cache-warm.yml

@@ -99,7 +99,7 @@ env:
   RUST_BACKTRACE: 1
 
 jobs:
-  # Readers: test-and-lint (workspace and ILM), build-rustfs-debug-binary,
+  # Readers: test-and-lint, test-ilm-integration-serial, build-rustfs-debug-binary,
   # e2e-tests, e2e-full.
   warm-ci-dev:
     name: Warm ci-dev

.github/workflows/ci.yml

@@ -16,7 +16,9 @@ name: Continuous Integration
 
 on:
   push:
-    branches: [ main ]
+    # gateway/integration receives the daily merge of main as a direct push
+    # (rustfs/backlog#2734), so its pushes must prove the result like main's.
+    branches: [ main, gateway/integration ]
     paths-ignore:
       - "**.md"
       - "docs/**"
@@ -168,10 +170,13 @@ jobs:
             echo "started_at=$(date --utc --iso-8601=seconds)"
           } > artifacts/test-and-lint/run-metadata.txt
 
-      # Keep full Clippy validation on main, merge groups, and scheduled/manual
-      # runs. PRs retain the complete default test suite and compile-free checks.
+      # Clippy runs before the test pass: lint failures are the most common
+      # CI-only breakage and should surface in minutes, not after 20+ minutes
+      # of tests.
+      # Sampled too: clippy is the natural control arm for any CARGO_BUILD_JOBS
+      # experiment, since --all-targets is check-only for workspace members and
+      # never links the ~100 test binaries the limit exists to throttle.
       - name: Run clippy lints
-        if: github.event_name != 'pull_request'
         run: |
           ./scripts/ci/resource_sampler.sh start clippy
           trap './scripts/ci/resource_sampler.sh stop' EXIT
@@ -280,49 +285,6 @@ jobs:
       - name: Verify rebalance/decommission migration proofs
         run: ./scripts/check_migration_gate_count.sh evidence artifacts/test-and-lint/core-test-listing.json target/nextest/ci/junit.xml
 
-      # The workspace already compiled these ignored tests. Keep the same Cargo
-      # package selection to reuse their binaries, then run them serially in
-      # separate processes because they share fixed ports and global state.
-      # test_transition_and_restore_flows was re-enabled by rustfs/backlog#1303:
-      # its "missing xl.meta on disk2" was a test-util bug (open_disk hardcoded
-      # disk_index 0), not an EC metadata-distribution issue.
-      # restore_object_usecase_reports_ongoing_conflict_and_completion was
-      # re-enabled by backlog#1304 (restore accepts serialize on a short CAS
-      # guard; the copy-back no longer holds the #4877 whole-copy-back lock,
-      # so the mid-restore ongoing read and fast 409 rejection it asserts are
-      # the implemented contract).
-      - name: Run ignored ILM integration tests serially
-        if: needs.classify-changes.outputs.mode == 'full'
-        env:
-          # Match the preceding workspace build and reuse its feature graph.
-          CARGO_BUILD_JOBS: ${{ (github.event_name == 'push' || github.event_name == 'workflow_dispatch') && '3' || '2' }}
-        run: |
-          mkdir -p artifacts/ilm-integration
-          set +e
-          NEXTEST_HIDE_PROGRESS_BAR=1 timeout --verbose --signal=TERM --kill-after=30s 80m \
-            cargo nextest run --profile default -j1 --run-ignored ignored-only \
-            --all --exclude e2e_test \
-            -E 'binary(lifecycle_integration_test) or (package(rustfs) and test(lifecycle_transition_api_test))' \
-            --status-level all --final-status-level all \
-            2>&1 | tee artifacts/ilm-integration/nextest.log
-          status=${PIPESTATUS[0]}
-          {
-            echo "exit_status=${status}"
-            echo "finished_at=$(date --utc --iso-8601=seconds)"
-            echo
-            echo "Remaining test-related processes:"
-            pgrep -af 'cargo|nextest|target/.*/deps/' || true
-          } > artifacts/ilm-integration/diagnostics.txt
-          exit "${status}"
-
-      - name: Upload ILM test diagnostics
-        if: always() && needs.classify-changes.outputs.mode == 'full'
-        uses: actions/upload-artifact@043fb46d1a93c77aae656e7c1c64a875d1fc6a0a # v7.0.1
-        with:
-          name: ilm-integration-${{ github.run_number }}-${{ github.run_attempt }}
-          path: artifacts/ilm-integration
-          retention-days: 3
-
   # This gate builds into a fresh target directory to isolate the E2E root.
   # Give its cold build a separate budget from workspace tests and migration proofs.
   offline-enrollment-e2e:
@@ -354,11 +316,83 @@ jobs:
       - name: Check offline enrollment E2E root boundary
         run: ./scripts/check_offline_enrollment_e2e.sh
 
+  # Dedicated serial lane for the ILM / lifecycle integration tests. These tests
+  # drive the object layer through process-global singletons (the GLOBAL_ENV
+  # ECStore, the global tier-config manager, background-expiry workers) and bind
+  # fixed ports (9002 for the scanner suite, 9003 for the app suite), so they are
+  # marked #[ignore] to keep them out of the default parallel `cargo nextest run
+  # --all` pass. Run them here explicitly with `--run-ignored ignored-only` and
+  # `-j1` so no two of them share global state or race for a port at once.
+  # serial_test's #[serial] does NOT serialize across nextest's process-per-test
+  # boundary (see .config/nextest.toml); `-j1` is what actually serializes them.
+  # See rustfs/backlog#1148 (ilm-1) and #1155.
+  test-ilm-integration-serial:
+    name: ILM Integration (serial)
+    if: needs.classify-changes.outputs.mode == 'full' && (github.event_name != 'pull_request' || github.event.action != 'closed')
+    needs: [ quick-checks, classify-changes ]
+    runs-on: sm-standard-4
+    timeout-minutes: 90
+    env:
+      FORCE_JAVASCRIPT_ACTIONS_TO_NODE24: "true"
+    steps:
+      - name: Checkout repository
+        uses: actions/checkout@f548e57e544e1ff5a4c46bf1e1b8685f8e4a348a # v7
+        with:
+          persist-credentials: false
+
+      - name: Setup Rust environment
+        uses: ./.github/actions/setup
+        with:
+          rust-version: stable
+          cache-shared-key: ci-dev
+          cache-save-if: 'false'
+          install-build-packaging-tools: 'false'
+
+      # test_transition_and_restore_flows was re-enabled by rustfs/backlog#1303:
+      # its "missing xl.meta on disk2" was a test-util bug (open_disk hardcoded
+      # disk_index 0), not an EC metadata-distribution issue.
+      # restore_object_usecase_reports_ongoing_conflict_and_completion was
+      # re-enabled by backlog#1304 (restore accepts serialize on a short CAS
+      # guard; the copy-back no longer holds the #4877 whole-copy-back lock,
+      # so the mid-restore ongoing read and fast 409 rejection it asserts are
+      # the implemented contract).
+      - name: Run ignored ILM integration tests serially
+        env:
+          # Match the measured Test and Lint link budget. The default exposed
+          # all 14 pod CPUs and a cold cache spent the full 80m compiling
+          # without starting one ILM test (main run 32982910990).
+          CARGO_BUILD_JOBS: ${{ (github.event_name == 'push' || github.event_name == 'workflow_dispatch') && '3' || '2' }}
+        run: |
+          mkdir -p artifacts/ilm-integration
+          set +e
+          NEXTEST_HIDE_PROGRESS_BAR=1 timeout --verbose --signal=TERM --kill-after=30s 80m \
+            cargo nextest run -j1 --run-ignored ignored-only \
+            -p rustfs-scanner -p rustfs \
+            -E 'binary(lifecycle_integration_test) or (package(rustfs) and test(lifecycle_transition_api_test))' \
+            --status-level all --final-status-level all \
+            2>&1 | tee artifacts/ilm-integration/nextest.log
+          status=${PIPESTATUS[0]}
+          {
+            echo "exit_status=${status}"
+            echo "finished_at=$(date --utc --iso-8601=seconds)"
+            echo
+            echo "Remaining test-related processes:"
+            pgrep -af 'cargo|nextest|target/.*/deps/' || true
+          } > artifacts/ilm-integration/diagnostics.txt
+          exit "${status}"
+
+      - name: Upload ILM test diagnostics
+        if: always()
+        uses: actions/upload-artifact@043fb46d1a93c77aae656e7c1c64a875d1fc6a0a # v7.0.1
+        with:
+          name: ilm-integration-${{ github.run_number }}-${{ github.run_attempt }}
+          path: |
+            artifacts/ilm-integration
+            target/nextest/ci/junit.xml
+
   test-and-lint-rio-v2:
     name: Test and Lint (rio-v2)
-    # Alternate feature configurations run on main, merge groups, and scheduled
-    # or manual validation; the default workspace remains the PR gate.
-    if: needs.classify-changes.outputs.mode == 'full' && github.event_name != 'pull_request'
+    if: needs.classify-changes.outputs.mode == 'full' && (github.event_name != 'pull_request' || github.event.action != 'closed')
     needs: [ quick-checks, classify-changes ]
     runs-on: sm-standard-4
     timeout-minutes: 90
@@ -474,7 +508,7 @@ jobs:
 
   test-and-lint-protocols:
     name: "Test and Lint (${{ matrix.features.name }})"
-    if: needs.classify-changes.outputs.mode == 'full' && github.event_name != 'pull_request'
+    if: needs.classify-changes.outputs.mode == 'full' && (github.event_name != 'pull_request' || github.event.action != 'closed')
     needs: [ quick-checks, classify-changes ]
     runs-on: sm-standard-4
     timeout-minutes: 90
@@ -748,6 +782,8 @@ jobs:
     needs: [ build-rustfs-debug-binary ]
     runs-on: sm-standard-2
     timeout-minutes: 30
+    env:
+      RUSTFS_S3_STACK: legacy
     steps:
       - name: Checkout repository
         uses: actions/checkout@f548e57e544e1ff5a4c46bf1e1b8685f8e4a348a # v7
@@ -866,18 +902,105 @@ jobs:
           path: ${{ runner.temp }}/rustfs-e2e-*/rustfs.log
           retention-days: 3
 
+  # The same e2e-smoke suite against the same debug binary with the gateway
+  # S3 stack selected (rustfs/backlog#2734). Report-only: it is outside the
+  # Test and Lint aggregate and becomes required at T2.10. The archive and run
+  # steps must stay identical to e2e-tests; scripts/ci_gate.py enforces that.
+  e2e-tests-gateway:
+    name: e2e-smoke (gateway, report-only until T2.10)
+    needs: [ build-rustfs-debug-binary ]
+    runs-on: sm-standard-2
+    timeout-minutes: 30
+    continue-on-error: true
+    env:
+      RUSTFS_S3_STACK: gateway
+    steps:
+      - name: Checkout repository
+        uses: actions/checkout@f548e57e544e1ff5a4c46bf1e1b8685f8e4a348a # v7
+        with:
+          persist-credentials: false
+
+      - name: Setup Rust environment
+        uses: ./.github/actions/setup
+        with:
+          rust-version: stable
+          cache-shared-key: ci-dev
+          cache-save-if: 'false'
+          install-build-packaging-tools: 'false'
+
+      - name: Set up Python
+        uses: actions/setup-python@5fda3b95a4ea91299a34e894583c3862153e4b97 # v7.0.0
+        with:
+          python-version: "3.12"
+
+      - name: Install awscurl
+        run: |
+          python3 -m pip install --user --upgrade pip "awscurl==0.44"
+          echo "AWSCURL_PATH=$HOME/.local/bin/awscurl" >> "$GITHUB_ENV"
+
+      - name: Verify awscurl
+        run: test -x "$AWSCURL_PATH"
+
+      - name: Download debug binary
+        uses: actions/download-artifact@3e5f45b2cfb9172054b4087a40e8e0b5a5461e7c # v8.0.1
+        with:
+          name: rustfs-debug-binary
+          path: target/debug
+
+      - name: Make binary executable
+        run: chmod +x ./target/debug/rustfs
+
+      - name: Archive e2e smoke test binaries
+        env:
+          NEXTEST_ARCHIVE: ${{ runner.temp }}/rustfs-e2e-smoke.tar.zst
+          NEXTEST_LISTING: ${{ runner.temp }}/rustfs-e2e-smoke-list.json
+        run: |
+          cargo nextest archive --profile e2e-smoke -p e2e_test --archive-file "${NEXTEST_ARCHIVE}"
+          cargo nextest list --profile e2e-smoke --archive-file "${NEXTEST_ARCHIVE}" --message-format json > "${NEXTEST_LISTING}"
+          python3 ./scripts/check_test_wiring.py --check-profile e2e-smoke "${NEXTEST_LISTING}"
+          ./scripts/check_security_smoke_count.sh check "${NEXTEST_LISTING}"
+
+      - name: Run e2e smoke suite
+        env:
+          NEXTEST_ARCHIVE: ${{ runner.temp }}/rustfs-e2e-smoke.tar.zst
+          RUSTFS_E2E_LOG_DIR: ${{ runner.temp }}/rustfs-e2e-smoke-logs
+          SAMPLER_INTERVAL_SECS: "15"
+        run: |
+          ./scripts/ci/resource_sampler.sh start e2e-smoke
+          trap './scripts/ci/resource_sampler.sh stop' EXIT
+          python3 scripts/e2e_binary.py run --features e2e-test-hooks -- cargo nextest run --profile e2e-smoke --archive-file "${NEXTEST_ARCHIVE}" \
+            --status-level all --final-status-level all --failure-output final
+
+      - name: Upload e2e smoke diagnostics
+        if: always()
+        uses: actions/upload-artifact@043fb46d1a93c77aae656e7c1c64a875d1fc6a0a # v7.0.1
+        with:
+          name: e2e-smoke-gateway-diagnostics-${{ github.run_number }}
+          path: |
+            ${{ runner.temp }}/rustfs-e2e-smoke-logs/
+            ${{ runner.temp }}/rustfs-e2e-smoke-list.json
+          if-no-files-found: warn
+
+      - name: Upload e2e smoke JUnit report
+        if: always()
+        uses: actions/upload-artifact@043fb46d1a93c77aae656e7c1c64a875d1fc6a0a # v7.0.1
+        with:
+          name: e2e-smoke-gateway-junit-${{ github.run_number }}
+          path: target/nextest/e2e-smoke/junit.xml
+          if-no-files-found: warn
+
   e2e-full:
     name: End-to-End Tests (full merge gate)
     # Merge gate only (backlog#1149 ci-5): the never-automated user-visible
     # suites — KMS, object_lock, multipart_auth, quota, checksum, encryption,
     # security-boundary, ... — via the e2e-full nextest profile. Too heavy for
-    # every PR, so it is gated to main pushes, the merge queue, and manual
-    # dispatch. protocols / the 7 cluster suites / replication / #[ignore] are
-    # owned by other lanes (see .config/nextest.toml profile.e2e-full).
+    # every PR, so it is gated to main and gateway/integration pushes, the merge
+    # queue, and manual dispatch. protocols / the 7 cluster suites / replication
+    # / #[ignore] are owned by other lanes (see .config/nextest.toml profile.e2e-full).
     if: >-
       github.event_name == 'workflow_dispatch' ||
       github.event_name == 'merge_group' ||
-      (github.event_name == 'push' && github.ref == 'refs/heads/main')
+      (github.event_name == 'push' && (github.ref == 'refs/heads/main' || github.ref == 'refs/heads/gateway/integration'))
     needs: [ build-rustfs-debug-binary ]
     runs-on: sm-standard-2
     timeout-minutes: 55
@@ -1232,6 +1355,7 @@ jobs:
       - quick-checks
       - test-and-lint
       - offline-enrollment-e2e
+      - test-ilm-integration-serial
       - test-and-lint-rio-v2
       - connect-short-credential-boundary
       - test-and-lint-protocols
@@ -1265,6 +1389,7 @@ jobs:
       - quick-checks
       - test-and-lint
       - offline-enrollment-e2e
+      - test-ilm-integration-serial
       - test-and-lint-rio-v2
       - test-and-lint-protocols
       - build-rustfs-debug-binary

.github/workflows/connect-diagnostic-schedule-acceptance.yml

@@ -57,14 +57,6 @@ jobs:
           ref: ${{ inputs.connect_sha }}
           token: ${{ secrets.PF_TESTING_GH_TOKEN }}
 
-      - name: Set up Docker Compose
-        uses: ./.github/actions/setup-docker-compose
-
-      - name: Set up Python
-        uses: actions/setup-python@5fda3b95a4ea91299a34e894583c3862153e4b97 # v7.0.0
-        with:
-          python-version: "3.14"
-
       - name: Set up Node.js
         uses: actions/setup-node@820762786026740c76f36085b0efc47a31fe5020 # v7.0.0
         with:
@@ -102,7 +94,7 @@ jobs:
           ' <<<"$run" >/dev/null
           jobs=$(gh api --paginate --slurp "repos/${GITHUB_REPOSITORY}/actions/runs/${BUILD_RUN_ID}/jobs?per_page=100")
           jq -e '
-            [.[].jobs[] | select(.name | test("^Build RustFS \\(linux-x86_64-gnu\\)$"))] as $matches
+            [.[].jobs[] | select(.name | test("^Build RustFS \\(linux-x86_64-gnu, [a-z0-9-]+, x86_64-unknown-linux-gnu, false, linux, pyroscope\\)$"))] as $matches
             | ($matches | length) == 1 and $matches[0].conclusion == "success"
           ' <<<"$jobs" >/dev/null
           artifact=$(gh api "repos/${GITHUB_REPOSITORY}/actions/artifacts/${ARTIFACT_ID}")
@@ -145,7 +137,7 @@ jobs:
               assert outer.testzip() is None
               with zipfile.ZipFile(io.BytesIO(outer.read(package))) as inner:
                   entries = inner.infolist()
-                  assert {item.filename for item in entries} == {'rustfs', 'rustfs.cpu-symbol-catalog.json', 'rustfs.cpu-symbol-catalog.sha256'} and len(entries) == 3
+                  assert {item.filename for item in entries} == {'rustfs', 'rustfs-cli'} and len(entries) == 2
                   assert all(regular(item) and 0 < item.file_size <= 1073741824 for item in entries)
                   assert inner.testzip() is None
                   pathlib.Path('binary').mkdir()
@@ -170,7 +162,7 @@ jobs:
           connect-harness/web/node_modules/.bin/playwright install --with-deps chromium
           make -C connect-harness e2e-connected-dispatch-check
           (cd connect-harness && ./tests/connectivity/diagnostic-schedule-evidence.test.sh)
-          TMPDIR="$GITHUB_WORKSPACE" RUSTFS_BINARY="$GITHUB_WORKSPACE/binary/rustfs" \
+          RUSTFS_BINARY="$GITHUB_WORKSPACE/binary/rustfs" \
             RUSTFS_WORKTREE="$GITHUB_WORKSPACE/rustfs-source" \
             CONNECT_E2E_DIAGNOSTIC_SCHEDULE_EVIDENCE="$GITHUB_WORKSPACE/diagnostic-schedule-evidence.json" \
             make -C connect-harness e2e-connected E2E_SCENARIO=diagnostic-schedule

.github/workflows/connect-health-service-acceptance.yml

@@ -57,14 +57,6 @@ jobs:
           ref: ${{ inputs.connect_sha }}
           token: ${{ secrets.PF_TESTING_GH_TOKEN }}
 
-      - name: Set up Docker Compose
-        uses: ./.github/actions/setup-docker-compose
-
-      - name: Set up Python
-        uses: actions/setup-python@5fda3b95a4ea91299a34e894583c3862153e4b97 # v7.0.0
-        with:
-          python-version: "3.14"
-
       - name: Set up Node.js
         uses: actions/setup-node@820762786026740c76f36085b0efc47a31fe5020 # v7.0.0
         with:
@@ -135,7 +127,7 @@ jobs:
           ' <<<"$run" >/dev/null
           jobs=$(gh api --paginate --slurp "repos/rustfs/rustfs/actions/runs/${BUILD_RUN_ID}/jobs?per_page=100")
           jq -e '
-            [.[].jobs[] | select(.name | test("^Build RustFS \\(linux-x86_64-gnu\\)$"))] as $matches
+            [.[].jobs[] | select(.name | test("^Build RustFS \\(linux-x86_64-gnu, [a-z0-9-]+, x86_64-unknown-linux-gnu, false, linux, pyroscope\\)$"))] as $matches
             | ($matches | length) == 1 and $matches[0].conclusion == "success"
           ' <<<"$jobs" >/dev/null
           artifact=$(gh api "repos/rustfs/rustfs/actions/artifacts/${ARTIFACT_ID}")
@@ -180,7 +172,7 @@ jobs:
               assert outer.testzip() is None
               with zipfile.ZipFile(io.BytesIO(outer.read(package))) as inner:
                   entries = inner.infolist()
-                  assert {entry.filename for entry in entries} == {'rustfs', 'rustfs.cpu-symbol-catalog.json', 'rustfs.cpu-symbol-catalog.sha256'} and len(entries) == 3
+                  assert {entry.filename for entry in entries} == {'rustfs', 'rustfs-cli', 'rustfs.cpu-symbol-catalog.json', 'rustfs.cpu-symbol-catalog.sha256'} and len(entries) == 4
                   assert all(regular(entry) and 0 < entry.file_size <= 1073741824 for entry in entries)
                   assert inner.testzip() is None
                   pathlib.Path('binary').mkdir()
@@ -207,7 +199,7 @@ jobs:
           connect-harness/web/node_modules/.bin/playwright install --with-deps chromium
           make -C connect-harness e2e-connected-dispatch-check
           (cd connect-harness && ./tests/connectivity/health-service-evidence.test.sh)
-          TMPDIR="$GITHUB_WORKSPACE" RUSTFS_BINARY="$GITHUB_WORKSPACE/binary/rustfs" \
+          RUSTFS_BINARY="$GITHUB_WORKSPACE/binary/rustfs" \
             RUSTFS_WORKTREE="$GITHUB_WORKSPACE/rustfs-source" \
             CONNECT_E2E_HEALTH_SERVICE_EVIDENCE="$GITHUB_WORKSPACE/health-service-evidence.json" \
             make -C connect-harness e2e-connected E2E_SCENARIO=health-service

.github/workflows/connect-performance-drive-acceptance.yml

@@ -73,14 +73,6 @@ jobs:
           ref: ${{ inputs.connect_sha }}
           token: ${{ secrets.PF_TESTING_GH_TOKEN }}
 
-      - name: Set up Docker Compose
-        uses: ./.github/actions/setup-docker-compose
-
-      - name: Set up Python
-        uses: actions/setup-python@5fda3b95a4ea91299a34e894583c3862153e4b97 # v7.0.0
-        with:
-          python-version: "3.14"
-
       - name: Set up Node.js
         uses: actions/setup-node@820762786026740c76f36085b0efc47a31fe5020 # v7.0.0
         with:
@@ -115,7 +107,7 @@ jobs:
           [[ $(jq -r '.head_repository.full_name' <<<"$run") == "$GITHUB_REPOSITORY" ]]
           [[ $(jq -r '.name' <<<"$run") == "Build and Release" ]]
 
-          expected_job='Build RustFS (linux-x86_64-gnu)'
+          expected_job='Build RustFS (linux-x86_64-gnu, sm-standard-4, x86_64-unknown-linux-gnu, false, linux, pyroscope)'
           jobs=$(gh api --paginate --slurp "repos/${GITHUB_REPOSITORY}/actions/runs/${BUILD_RUN_ID}/jobs?per_page=100")
           jq -e --arg name "$expected_job" '
             [.[].jobs[] | select(.name == $name)] as $matches
@@ -162,7 +154,7 @@ jobs:
           npm --prefix connect-harness/web ci
           connect-harness/web/node_modules/.bin/playwright install --with-deps chromium
           make -C connect-harness e2e-connected-dispatch-check
-          TMPDIR="$GITHUB_WORKSPACE" RUSTFS_BINARY="$GITHUB_WORKSPACE/binary/rustfs" \
+          RUSTFS_BINARY="$GITHUB_WORKSPACE/binary/rustfs" \
             RUSTFS_WORKTREE="$GITHUB_WORKSPACE/rustfs-source" \
             CONNECT_E2E_PERFORMANCE_DRIVE_EVIDENCE="$GITHUB_WORKSPACE/performance-drive-service-job-evidence.json" \
             make -C connect-harness e2e-connected E2E_SCENARIO=performance-drive

.github/workflows/connect-performance-network-acceptance.yml

@@ -73,14 +73,6 @@ jobs:
           ref: ${{ inputs.connect_sha }}
           token: ${{ secrets.PF_TESTING_GH_TOKEN }}
 
-      - name: Set up Docker Compose
-        uses: ./.github/actions/setup-docker-compose
-
-      - name: Set up Python
-        uses: actions/setup-python@5fda3b95a4ea91299a34e894583c3862153e4b97 # v7.0.0
-        with:
-          python-version: "3.14"
-
       - name: Set up Node.js
         uses: actions/setup-node@820762786026740c76f36085b0efc47a31fe5020 # v7.0.0
         with:
@@ -115,7 +107,7 @@ jobs:
           [[ $(jq -r '.head_repository.full_name' <<<"$run") == "$GITHUB_REPOSITORY" ]]
           [[ $(jq -r '.name' <<<"$run") == "Build and Release" ]]
 
-          expected_job='Build RustFS (linux-x86_64-gnu)'
+          expected_job='Build RustFS (linux-x86_64-gnu, sm-standard-4, x86_64-unknown-linux-gnu, false, linux, pyroscope)'
           jobs=$(gh api --paginate --slurp "repos/${GITHUB_REPOSITORY}/actions/runs/${BUILD_RUN_ID}/jobs?per_page=100")
           jq -e --arg name "$expected_job" '
             [.[].jobs[] | select(.name == $name)] as $matches
@@ -162,7 +154,7 @@ jobs:
           npm --prefix connect-harness/web ci
           connect-harness/web/node_modules/.bin/playwright install --with-deps chromium
           make -C connect-harness e2e-connected-dispatch-check
-          TMPDIR="$GITHUB_WORKSPACE" RUSTFS_BINARY="$GITHUB_WORKSPACE/binary/rustfs" \
+          RUSTFS_BINARY="$GITHUB_WORKSPACE/binary/rustfs" \
             RUSTFS_WORKTREE="$GITHUB_WORKSPACE/rustfs-source" \
             CONNECT_E2E_PERFORMANCE_NETWORK_EVIDENCE="$GITHUB_WORKSPACE/performance-network-service-job-evidence.json" \
             make -C connect-harness e2e-connected E2E_SCENARIO=performance-network

.github/workflows/connect-profile-cpu-acceptance.yml

@@ -77,14 +77,6 @@ jobs:
           ref: ${{ inputs.connect_sha }}
           token: ${{ secrets.PF_TESTING_GH_TOKEN }}
 
-      - name: Set up Docker Compose
-        uses: ./.github/actions/setup-docker-compose
-
-      - name: Set up Python
-        uses: actions/setup-python@5fda3b95a4ea91299a34e894583c3862153e4b97 # v7.0.0
-        with:
-          python-version: "3.14"
-
       - name: Set up Node.js
         uses: actions/setup-node@820762786026740c76f36085b0efc47a31fe5020 # v7.0.0
         with:
@@ -126,7 +118,7 @@ jobs:
           [[ $(jq -r '.head_repository.full_name' <<<"$run") == "$GITHUB_REPOSITORY" ]]
           [[ $(jq -r '.name' <<<"$run") == "Build and Release" ]]
 
-          expected_job='Build RustFS (linux-x86_64-gnu)'
+          expected_job='Build RustFS (linux-x86_64-gnu, sm-standard-4, x86_64-unknown-linux-gnu, false, linux, pyroscope)'
           jobs=$(github_api "repos/${GITHUB_REPOSITORY}/actions/runs/${BUILD_RUN_ID}/jobs?per_page=100")
           jq -e --arg name "$expected_job" '
             [.jobs[] | select(.name == $name)] as $matches
@@ -181,7 +173,7 @@ jobs:
           npm --prefix connect-harness/web ci
           connect-harness/web/node_modules/.bin/playwright install --with-deps chromium
           make -C connect-harness e2e-connected-dispatch-check
-          TMPDIR="$GITHUB_WORKSPACE" RUSTFS_BINARY="$GITHUB_WORKSPACE/binary/rustfs" \
+          RUSTFS_BINARY="$GITHUB_WORKSPACE/binary/rustfs" \
             RUSTFS_WORKTREE="$GITHUB_WORKSPACE/rustfs-source" \
             CONNECT_E2E_CPU_CATALOG="$GITHUB_WORKSPACE/binary/rustfs.cpu-symbol-catalog.json" \
             CONNECT_E2E_CPU_CATALOG_SHA256="$CATALOG_SHA256" \

.github/workflows/connect-profile-memory-acceptance.yml

@@ -57,14 +57,6 @@ jobs:
           ref: ${{ inputs.connect_sha }}
           token: ${{ secrets.PF_TESTING_GH_TOKEN }}
 
-      - name: Set up Docker Compose
-        uses: ./.github/actions/setup-docker-compose
-
-      - name: Set up Python
-        uses: actions/setup-python@5fda3b95a4ea91299a34e894583c3862153e4b97 # v7.0.0
-        with:
-          python-version: "3.14"
-
       - name: Set up Node.js
         uses: actions/setup-node@820762786026740c76f36085b0efc47a31fe5020 # v7.0.0
         with:
@@ -135,7 +127,7 @@ jobs:
           ' <<<"$run" >/dev/null
           jobs=$(gh api --paginate --slurp "repos/rustfs/rustfs/actions/runs/${BUILD_RUN_ID}/jobs?per_page=100")
           jq -e '
-            [.[].jobs[] | select(.name | test("^Build RustFS \\(linux-x86_64-gnu\\)$"))] as $matches
+            [.[].jobs[] | select(.name | test("^Build RustFS \\(linux-x86_64-gnu, [a-z0-9-]+, x86_64-unknown-linux-gnu, false, linux, pyroscope\\)$"))] as $matches
             | ($matches | length) == 1 and $matches[0].conclusion == "success"
           ' <<<"$jobs" >/dev/null
           artifact=$(gh api "repos/rustfs/rustfs/actions/artifacts/${ARTIFACT_ID}")
@@ -180,7 +172,7 @@ jobs:
               assert outer.testzip() is None
               with zipfile.ZipFile(io.BytesIO(outer.read(package))) as inner:
                   entries = inner.infolist()
-                  expected = {'rustfs', 'rustfs.cpu-symbol-catalog.json', 'rustfs.cpu-symbol-catalog.sha256'}
+                  expected = {'rustfs', 'rustfs-cli', 'rustfs.cpu-symbol-catalog.json', 'rustfs.cpu-symbol-catalog.sha256'}
                   assert {entry.filename for entry in entries} == expected and len(entries) == len(expected)
                   assert all(regular(entry) and 0 < entry.file_size <= 1073741824 for entry in entries)
                   assert inner.testzip() is None
@@ -206,7 +198,7 @@ jobs:
           connect-harness/web/node_modules/.bin/playwright install --with-deps chromium
           make -C connect-harness e2e-connected-dispatch-check
           (cd connect-harness && ./tests/connectivity/profile-memory-evidence.test.sh)
-          TMPDIR="$GITHUB_WORKSPACE" RUSTFS_BINARY="$GITHUB_WORKSPACE/binary/rustfs" \
+          RUSTFS_BINARY="$GITHUB_WORKSPACE/binary/rustfs" \
             RUSTFS_WORKTREE="$GITHUB_WORKSPACE/rustfs-source" \
             CONNECT_E2E_PROFILE_MEMORY_EVIDENCE="$GITHUB_WORKSPACE/profile-memory-evidence.json" \
             make -C connect-harness e2e-connected E2E_SCENARIO=profile-memory

.github/workflows/connect-profile-threads-acceptance.yml

@@ -81,14 +81,6 @@ jobs:
           ref: ${{ inputs.connect_sha }}
           token: ${{ secrets.PF_TESTING_GH_TOKEN }}
 
-      - name: Set up Docker Compose
-        uses: ./.github/actions/setup-docker-compose
-
-      - name: Set up Python
-        uses: actions/setup-python@5fda3b95a4ea91299a34e894583c3862153e4b97 # v7.0.0
-        with:
-          python-version: "3.14"
-
       - name: Set up Node.js
         uses: actions/setup-node@820762786026740c76f36085b0efc47a31fe5020 # v7.0.0
         with:
@@ -156,7 +148,7 @@ jobs:
           [[ $(jq -r '.head_repository.full_name' <<<"$run") == "$GITHUB_REPOSITORY" ]]
           [[ $(jq -r '.name' <<<"$run") == "Build and Release" ]]
 
-          expected_job='Build RustFS (linux-x86_64-gnu)'
+          expected_job='Build RustFS (linux-x86_64-gnu, sm-standard-4, x86_64-unknown-linux-gnu, false, linux, pyroscope)'
           jobs=$(gh api --paginate --slurp "repos/${GITHUB_REPOSITORY}/actions/runs/${BUILD_RUN_ID}/jobs?per_page=100")
           jq -e --arg name "$expected_job" '
             [.[].jobs[] | select(.name == $name)] as $matches
@@ -209,7 +201,7 @@ jobs:
           npm --prefix connect-harness/web ci
           connect-harness/web/node_modules/.bin/playwright install --with-deps chromium
           make -C connect-harness e2e-connected-dispatch-check
-          TMPDIR="$GITHUB_WORKSPACE" RUSTFS_BINARY="$GITHUB_WORKSPACE/binary/rustfs" \
+          RUSTFS_BINARY="$GITHUB_WORKSPACE/binary/rustfs" \
             RUSTFS_WORKTREE="$GITHUB_WORKSPACE/rustfs-source" \
             CONNECT_E2E_PROFILE_THREADS_EVIDENCE="$GITHUB_WORKSPACE/profile-threads-service-job-evidence.json" \
             make -C connect-harness e2e-connected E2E_SCENARIO=profile-threads

.github/workflows/connect-top-api-acceptance.yml

@@ -73,14 +73,6 @@ jobs:
           ref: ${{ inputs.connect_sha }}
           token: ${{ secrets.PF_TESTING_GH_TOKEN }}
 
-      - name: Set up Docker Compose
-        uses: ./.github/actions/setup-docker-compose
-
-      - name: Set up Python
-        uses: actions/setup-python@5fda3b95a4ea91299a34e894583c3862153e4b97 # v7.0.0
-        with:
-          python-version: "3.14"
-
       - name: Set up Node.js
         uses: actions/setup-node@820762786026740c76f36085b0efc47a31fe5020 # v7.0.0
         with:
@@ -115,7 +107,7 @@ jobs:
           [[ $(jq -r '.head_repository.full_name' <<<"$run") == "$GITHUB_REPOSITORY" ]]
           [[ $(jq -r '.name' <<<"$run") == "Build and Release" ]]
 
-          expected_job='Build RustFS (linux-x86_64-gnu)'
+          expected_job='Build RustFS (linux-x86_64-gnu, sm-standard-4, x86_64-unknown-linux-gnu, false, linux, pyroscope)'
           jobs=$(gh api --paginate --slurp "repos/${GITHUB_REPOSITORY}/actions/runs/${BUILD_RUN_ID}/jobs?per_page=100")
           jq -e --arg name "$expected_job" '
             [.[].jobs[] | select(.name == $name)] as $matches
@@ -162,7 +154,7 @@ jobs:
           npm --prefix connect-harness/web ci
           connect-harness/web/node_modules/.bin/playwright install --with-deps chromium
           make -C connect-harness e2e-connected-dispatch-check
-          TMPDIR="$GITHUB_WORKSPACE" RUSTFS_BINARY="$GITHUB_WORKSPACE/binary/rustfs" \
+          RUSTFS_BINARY="$GITHUB_WORKSPACE/binary/rustfs" \
             RUSTFS_WORKTREE="$GITHUB_WORKSPACE/rustfs-source" \
             CONNECT_E2E_TOP_API_EVIDENCE="$GITHUB_WORKSPACE/top-api-service-job-evidence.json" \
             make -C connect-harness e2e-connected E2E_SCENARIO=top-api

.github/workflows/connect-top-disk-acceptance.yml

@@ -68,14 +68,6 @@ jobs:
           ref: ${{ inputs.connect_sha }}
           token: ${{ secrets.PF_TESTING_GH_TOKEN }}
 
-      - name: Set up Docker Compose
-        uses: ./.github/actions/setup-docker-compose
-
-      - name: Set up Python
-        uses: actions/setup-python@5fda3b95a4ea91299a34e894583c3862153e4b97 # v7.0.0
-        with:
-          python-version: "3.14"
-
       - name: Set up Node.js
         uses: actions/setup-node@820762786026740c76f36085b0efc47a31fe5020 # v7.0.0
         with:
@@ -146,7 +138,7 @@ jobs:
           ' <<<"$run" >/dev/null
           jobs=$(gh api --paginate --slurp "repos/rustfs/rustfs/actions/runs/${BUILD_RUN_ID}/jobs?per_page=100")
           jq -e '
-            [.[].jobs[] | select(.name | test("^Build RustFS \\(linux-x86_64-gnu\\)$"))] as $matches
+            [.[].jobs[] | select(.name | test("^Build RustFS \\(linux-x86_64-gnu, [a-z0-9-]+, x86_64-unknown-linux-gnu, false, linux, pyroscope\\)$"))] as $matches
             | ($matches | length) == 1 and $matches[0].conclusion == "success"
           ' <<<"$jobs" >/dev/null
           artifact=$(gh api "repos/rustfs/rustfs/actions/artifacts/${ARTIFACT_ID}")
@@ -191,7 +183,7 @@ jobs:
               assert outer.testzip() is None
               with zipfile.ZipFile(io.BytesIO(outer.read(package))) as inner:
                   entries = inner.infolist()
-                  expected = {'rustfs', 'rustfs.cpu-symbol-catalog.json', 'rustfs.cpu-symbol-catalog.sha256'}
+                  expected = {'rustfs', 'rustfs-cli', 'rustfs.cpu-symbol-catalog.json', 'rustfs.cpu-symbol-catalog.sha256'}
                   assert {entry.filename for entry in entries} == expected and len(entries) == len(expected)
                   assert all(regular(entry) and 0 < entry.file_size <= 1073741824 for entry in entries)
                   assert inner.testzip() is None
@@ -221,7 +213,7 @@ jobs:
           npm --prefix connect-harness/web ci
           connect-harness/web/node_modules/.bin/playwright install --with-deps chromium
           make -C connect-harness e2e-connected-dispatch-check
-          TMPDIR="$GITHUB_WORKSPACE" RUSTFS_BINARY="$GITHUB_WORKSPACE/binary/rustfs" \
+          RUSTFS_BINARY="$GITHUB_WORKSPACE/binary/rustfs" \
             RUSTFS_WORKTREE="$GITHUB_WORKSPACE/rustfs-source" \
             CONNECT_E2E_TOP_DISK_EVIDENCE="$GITHUB_WORKSPACE/top-disk-service-job-evidence.json" \
             make -C connect-harness e2e-connected E2E_SCENARIO=top-disk

.github/workflows/connect-top-locks-acceptance.yml

@@ -73,14 +73,6 @@ jobs:
           ref: ${{ inputs.connect_sha }}
           token: ${{ secrets.PF_TESTING_GH_TOKEN }}
 
-      - name: Set up Docker Compose
-        uses: ./.github/actions/setup-docker-compose
-
-      - name: Set up Python
-        uses: actions/setup-python@5fda3b95a4ea91299a34e894583c3862153e4b97 # v7.0.0
-        with:
-          python-version: "3.14"
-
       - name: Set up Node.js
         uses: actions/setup-node@820762786026740c76f36085b0efc47a31fe5020 # v7.0.0
         with:
@@ -115,7 +107,7 @@ jobs:
           [[ $(jq -r '.head_repository.full_name' <<<"$run") == "$GITHUB_REPOSITORY" ]]
           [[ $(jq -r '.name' <<<"$run") == "Build and Release" ]]
 
-          expected_job='Build RustFS (linux-x86_64-gnu)'
+          expected_job='Build RustFS (linux-x86_64-gnu, sm-standard-4, x86_64-unknown-linux-gnu, false, linux, pyroscope)'
           jobs=$(gh api --paginate --slurp "repos/${GITHUB_REPOSITORY}/actions/runs/${BUILD_RUN_ID}/jobs?per_page=100")
           jq -e --arg name "$expected_job" '
             [.[].jobs[] | select(.name == $name)] as $matches
@@ -162,7 +154,7 @@ jobs:
           npm --prefix connect-harness/web ci
           connect-harness/web/node_modules/.bin/playwright install --with-deps chromium
           make -C connect-harness e2e-connected-dispatch-check
-          TMPDIR="$GITHUB_WORKSPACE" RUSTFS_BINARY="$GITHUB_WORKSPACE/binary/rustfs" \
+          RUSTFS_BINARY="$GITHUB_WORKSPACE/binary/rustfs" \
             RUSTFS_WORKTREE="$GITHUB_WORKSPACE/rustfs-source" \
             CONNECT_E2E_TOP_LOCKS_EVIDENCE="$GITHUB_WORKSPACE/top-locks-service-job-evidence.json" \
             make -C connect-harness e2e-connected E2E_SCENARIO=top-locks

.github/workflows/connect-top-rpc-acceptance.yml

@@ -73,14 +73,6 @@ jobs:
           ref: ${{ inputs.connect_sha }}
           token: ${{ secrets.PF_TESTING_GH_TOKEN }}
 
-      - name: Set up Docker Compose
-        uses: ./.github/actions/setup-docker-compose
-
-      - name: Set up Python
-        uses: actions/setup-python@5fda3b95a4ea91299a34e894583c3862153e4b97 # v7.0.0
-        with:
-          python-version: "3.14"
-
       - name: Set up Node.js
         uses: actions/setup-node@820762786026740c76f36085b0efc47a31fe5020 # v7.0.0
         with:
@@ -115,7 +107,7 @@ jobs:
           [[ $(jq -r '.head_repository.full_name' <<<"$run") == "$GITHUB_REPOSITORY" ]]
           [[ $(jq -r '.name' <<<"$run") == "Build and Release" ]]
 
-          expected_job='Build RustFS (linux-x86_64-gnu)'
+          expected_job='Build RustFS (linux-x86_64-gnu, sm-standard-4, x86_64-unknown-linux-gnu, false, linux, pyroscope)'
           jobs=$(gh api --paginate --slurp "repos/${GITHUB_REPOSITORY}/actions/runs/${BUILD_RUN_ID}/jobs?per_page=100")
           jq -e --arg name "$expected_job" '
             [.[].jobs[] | select(.name == $name)] as $matches
@@ -162,7 +154,7 @@ jobs:
           npm --prefix connect-harness/web ci
           connect-harness/web/node_modules/.bin/playwright install --with-deps chromium
           make -C connect-harness e2e-connected-dispatch-check
-          TMPDIR="$GITHUB_WORKSPACE" RUSTFS_BINARY="$GITHUB_WORKSPACE/binary/rustfs" \
+          RUSTFS_BINARY="$GITHUB_WORKSPACE/binary/rustfs" \
             RUSTFS_WORKTREE="$GITHUB_WORKSPACE/rustfs-source" \
             CONNECT_E2E_TOP_RPC_EVIDENCE="$GITHUB_WORKSPACE/top-rpc-service-job-evidence.json" \
             make -C connect-harness e2e-connected E2E_SCENARIO=top-rpc

.github/workflows/e2e-distributed.yml

@@ -168,13 +168,11 @@ jobs:
           FILTER: ${{ inputs.filter }}
         run: |
           set -euo pipefail
-          rm -f target/nextest/e2e-distributed/junit.xml
           if [ -n "${FILTER}" ]; then
             python3 scripts/e2e_binary.py run -- cargo nextest run --profile e2e-distributed -p e2e_test -E "${FILTER}"
           else
             python3 scripts/e2e_binary.py run -- cargo nextest run --profile e2e-distributed -p e2e_test --no-tests=fail
           fi
-          test -s target/nextest/e2e-distributed/junit.xml
 
       - name: Upload distributed e2e diagnostics
         if: always()

.github/workflows/e2e-upgrade.yml

@@ -96,8 +96,38 @@ jobs:
           overwrite: true
 
   upgrade:
-    name: Upgrade compatibility
+    name: ${{ matrix.name }}
     needs: build
+    strategy:
+      fail-fast: false
+      matrix:
+        include:
+          # The two `_from_rc2_` tests keep their names: they assert
+          # release-independent object contracts and pass unchanged against the
+          # newer pinned source, so renaming them would only churn history and
+          # the CI required-check names. UPGRADE_SOURCE_VERSION above is the
+          # single source of truth for which release they actually run against.
+          - name: Direct upgrade from the previous release
+            test: direct_upgrade_from_rc2_preserves_object_contracts
+            artifact: direct-upgrade
+          - name: Mixed-version rolling upgrade from the previous release
+            test: rolling_upgrade_from_rc2_preserves_mixed_version_contracts
+            artifact: mixed-version-upgrade
+          - name: Bucket configuration survives the upgrade
+            test: direct_upgrade_from_previous_release_preserves_bucket_configuration
+            artifact: bucket-config-upgrade
+          - name: Rollback reads current bucket metadata
+            test: rollback_to_previous_release_reads_current_bucket_metadata
+            artifact: bucket-config-rollback
+          - name: ODM configuration recovery after rc.5 rollback
+            test: rc5_rollback_requires_restoring_odm_configuration
+            artifact: odm-config-rollback
+          - name: Multipart layouts survive the rc.5 upgrade
+            test: direct_upgrade_from_rc5_preserves_multipart_layouts
+            artifact: multipart-layout-upgrade
+          - name: rc.5 multipart replication baseline
+            test: rc5_baseline_replicates_multipart_layouts
+            artifact: multipart-layout-baseline
     runs-on: sm-standard-2
     timeout-minutes: 60
     env:
@@ -112,7 +142,8 @@ jobs:
         uses: ./.github/actions/setup
         with:
           cache-shared-key: e2e-upgrade-tests
-          cache-save-if: ${{ github.ref == 'refs/heads/main' }}
+          # One main-branch consumer warms the shared test-harness dependencies.
+          cache-save-if: ${{ github.ref == 'refs/heads/main' && matrix.artifact == 'direct-upgrade' }}
           install-build-packaging-tools: "false"
           install-test-tools: "false"
 
@@ -139,49 +170,22 @@ jobs:
           echo "RUSTFS_UPGRADE_SOURCE_BINARY=$SOURCE_DIR/rustfs" >> "$GITHUB_ENV"
           echo "RUSTFS_E2E_LOG_DIR=$RUNNER_TEMP/rustfs-upgrade-logs" >> "$GITHUB_ENV"
 
-      # Reuse one compiled harness, while keeping each case in its own process.
-      # Continue after a failure to retain the former matrix's complete verdict.
-      - name: Run upgrade compatibility tests
-        shell: bash
+      - name: Run upgrade compatibility test
         env:
+          RUSTFS_SCANNER_HEAL_G09_EVIDENCE_DIR: ${{ runner.temp }}/rustfs-upgrade-g09-evidence/${{ matrix.artifact }}
           NO_PROXY: 127.0.0.1,localhost,::1
         run: |
           export no_proxy="$NO_PROXY"
-          cases=(
-            direct-upgrade:direct_upgrade_from_rc2_preserves_object_contracts
-            mixed-version-upgrade:rolling_upgrade_from_rc2_preserves_mixed_version_contracts
-            bucket-config-upgrade:direct_upgrade_from_previous_release_preserves_bucket_configuration
-            bucket-config-rollback:rollback_to_previous_release_reads_current_bucket_metadata
-            odm-config-rollback:rc5_rollback_requires_restoring_odm_configuration
-            multipart-layout-upgrade:direct_upgrade_from_rc5_preserves_multipart_layouts
-            multipart-layout-baseline:rc5_baseline_replicates_multipart_layouts
-          )
-          status=0
-          for entry in "${cases[@]}"; do
-            artifact="${entry%%:*}"
-            test="${entry#*:}"
-            export RUSTFS_SCANNER_HEAL_G09_EVIDENCE_DIR="$RUNNER_TEMP/rustfs-upgrade-g09-evidence/$artifact"
-            export RUSTFS_E2E_LOG_DIR="$RUNNER_TEMP/rustfs-upgrade-logs/$artifact"
-            mkdir -p "$RUSTFS_E2E_LOG_DIR"
-            echo "::group::$test"
-            if python3 scripts/e2e_binary.py run -- cargo test --locked -p e2e_test \
-              "upgrade_compatibility_test::$test" -- --ignored --exact --nocapture \
-              2>&1 | tee "$RUSTFS_E2E_LOG_DIR/test.log"; then
-              echo "PASS: $test"
-            else
-              echo "::error::Upgrade compatibility failed: $test"
-              status=1
-            fi
-            echo "::endgroup::"
-          done
-          exit "$status"
+          python3 scripts/e2e_binary.py run -- cargo test --locked -p e2e_test \
+            "upgrade_compatibility_test::${{ matrix.test }}" \
+            -- --ignored --exact --nocapture
 
       - name: Upload scanner/heal G09 evidence
         if: always()
         uses: actions/upload-artifact@043fb46d1a93c77aae656e7c1c64a875d1fc6a0a # v7.0.1
         with:
-          name: upgrade-g09-evidence-${{ github.run_number }}
-          path: ${{ runner.temp }}/rustfs-upgrade-g09-evidence
+          name: ${{ matrix.artifact }}-g09-evidence-${{ github.run_number }}
+          path: ${{ runner.temp }}/rustfs-upgrade-g09-evidence/${{ matrix.artifact }}
           if-no-files-found: ignore
           retention-days: 14
 
@@ -189,7 +193,7 @@ jobs:
         if: always()
         uses: actions/upload-artifact@043fb46d1a93c77aae656e7c1c64a875d1fc6a0a # v7.0.1
         with:
-          name: upgrade-server-logs-${{ github.run_number }}
+          name: ${{ matrix.artifact }}-server-logs-${{ github.run_number }}
           path: ${{ runner.temp }}/rustfs-upgrade-logs
           if-no-files-found: warn
           retention-days: 14

.github/workflows/nightly-gnu.yml

@@ -80,24 +80,13 @@ jobs:
       - name: Setup Rust environment
         uses: ./.github/actions/setup
         with:
-          cache-shared-key: build-x86_64-unknown-linux-gnu-glibc-2.31
+          cache-shared-key: build-x86_64-unknown-linux-gnu
           cache-save-if: 'false'
-          install-cross-tools: 'true'
-          target: x86_64-unknown-linux-gnu
           install-build-packaging-tools: 'false'
           install-test-tools: 'false'
 
       - name: Build RustFS
-        # Pin the glibc baseline (`<target>.<glibc>` for cargo-zigbuild) so
-        # nightlies run on older distros such as RHEL/Rocky 9 (glibc 2.34); #8062.
-        run: cargo zigbuild --release --locked --target x86_64-unknown-linux-gnu.2.31 -p rustfs --bins
-
-      - name: Verify glibc symbol baseline
-        run: |
-          python3 scripts/release/check_glibc.py 2.31 \
-            target/x86_64-unknown-linux-gnu/release/rustfs \
-            target/x86_64-unknown-linux-gnu/release/rustfs-cli
-          target/x86_64-unknown-linux-gnu/release/rustfs --version | grep -Fx 'build target : x86_64-unknown-linux-gnu'
+        run: cargo build --release --locked --target x86_64-unknown-linux-gnu -p rustfs --bins
 
       - name: Build DEB package
         id: deb

.github/workflows/oidc-keycloak.yml

@@ -20,9 +20,8 @@ on:
       - ".github/workflows/oidc-keycloak.yml"
       - "crates/config/src/constants/oidc.rs"
       - "crates/iam/src/federation/**"
-      - "crates/iam/src/oidc/**"
+      - "crates/iam/src/oidc.rs"
       - "rustfs/src/admin/handlers/oidc.rs"
-      - "rustfs/src/admin/service/oidc_config.rs"
       - "rustfs/src/admin/handlers/sts.rs"
       - "scripts/test/oidc_keycloak_live.sh"
       - "scripts/test/fixtures/keycloak-rustfs-ci-realm.json"
@@ -32,9 +31,8 @@ on:
       - ".github/workflows/oidc-keycloak.yml"
       - "crates/config/src/constants/oidc.rs"
       - "crates/iam/src/federation/**"
-      - "crates/iam/src/oidc/**"
+      - "crates/iam/src/oidc.rs"
       - "rustfs/src/admin/handlers/oidc.rs"
-      - "rustfs/src/admin/service/oidc_config.rs"
       - "rustfs/src/admin/handlers/sts.rs"
       - "scripts/test/oidc_keycloak_live.sh"
       - "scripts/test/fixtures/keycloak-rustfs-ci-realm.json"

.gitignore

@@ -90,4 +90,3 @@ worktrees/*
 # insta scratch files; the accepted .snap files ARE the assertions and are committed
 *.snap.new
 *.pending-snap
-.mimosa

.vscode/launch.json

@@ -7,9 +7,7 @@
             "name": "Debug(only) executable 'rustfs'",
             "env": {
               "RUST_LOG": "rustfs=info,ecstore=info,s3s=info,iam=info",
-              "RUSTFS_SKIP_BACKGROUND_TASK": "on",
-              // Local test volumes share one physical disk.
-              "RUSTFS_UNSAFE_BYPASS_DISK_CHECK": "true"
+              "RUSTFS_SKIP_BACKGROUND_TASK": "on"
               //"RUSTFS_OBS_LOG_DIRECTORY": "./deploy/logs",
               // "RUSTFS_POLICY_PLUGIN_URL":"http://localhost:8181/v1/data/rustfs/authz/allow",
               // "RUSTFS_POLICY_PLUGIN_AUTH_TOKEN":"your-opa-token"
@@ -46,7 +44,6 @@
             "env": {
                 "RUST_LOG": "rustfs=debug,ecstore=info,s3s=debug,iam=debug",
                 "RUSTFS_SKIP_BACKGROUND_TASK": "on",
-                "RUSTFS_UNSAFE_BYPASS_DISK_CHECK": "true",
                 //"RUSTFS_OBS_LOG_DIRECTORY": "./deploy/logs",
                 // "RUSTFS_POLICY_PLUGIN_URL":"http://localhost:8181/v1/data/rustfs/authz/allow",
                 // "RUSTFS_POLICY_PLUGIN_AUTH_TOKEN":"your-opa-token" 
@@ -120,8 +117,7 @@
             "env": {
                 "RUSTFS_ACCESS_KEY": "rustfsadmin",
                 "RUSTFS_SECRET_KEY": "rustfsadmin",
-                "RUSTFS_VOLUMES": "./target/volume/test{1...4}",
-                "RUSTFS_UNSAFE_BYPASS_DISK_CHECK": "true",
+                //"RUSTFS_VOLUMES": "./target/volume/test{1...4}",
                 "RUSTFS_ADDRESS": ":9000",
                 "RUSTFS_CONSOLE_ENABLE": "true",
                 // "RUSTFS_OBS_TRACE_ENDPOINT": "http://127.0.0.1:4318/v1/traces", // jeager otlp http endpoint
@@ -139,31 +135,42 @@
         {
             "type": "lldb",
             "request": "launch",
-            "name": "Debug test_transition_and_restore_flows",
+            "name": "Debug test_lifecycle_transition_basic",
             "cargo": {
                 "args": [
                     "test",
-                    "--no-run",
                     "-p",
                     "rustfs-scanner",
                     "--test",
                     "lifecycle_integration_test",
+                    "serial_tests::test_lifecycle_transition_basic",
                     "-j",
                     "1"
-                ],
-                "filter": {
-                    "name": "lifecycle_integration_test",
-                    "kind": "test"
-                }
+                ]
             },
-            "args": [
-                "serial_tests::test_transition_and_restore_flows",
-                "--exact",
-                "--ignored",
-                "--test-threads=1"
-            ],
+            "args": [],
             "cwd": "${workspaceFolder}"
         },
+        {
+            "name": "Debug executable target/debug/test",
+            "type":  "lldb",
+            "request": "launch",
+            "program": "${workspaceFolder}/target/debug/deps/lifecycle_integration_test-5915cbfcab491b3b",
+            "args": [
+              "--skip",
+              "test_lifecycle_expiry_basic",
+              "--skip",
+              "test_lifecycle_expiry_deletemarker",
+              //"--skip",
+              //"test_lifecycle_transition_basic",
+            ],
+            "cwd": "${workspaceFolder}",
+            //"stopAtEntry": false,
+            //"preLaunchTask": "cargo build",
+            "sourceLanguages": [
+                "rust"
+            ],
+        },
         {
             "name": "Debug executable target/debug/rustfs with sse kms",
             "type":  "lldb",
@@ -278,7 +285,7 @@
                 "RUSTFS_COMPRESSION_ENABLED": "true",
                 "RUSTFS_KMS_ENABLE": "true",
                 "RUSTFS_KMS_BACKEND": "local",
-                "RUSTFS_KMS_KEY_DIR": "${workspaceFolder}/target/rio-v2/kms-key-dir",
+                "RUSTFS_KMS_KEY_DIR": "./target/rio-v2/kms-key-dir",
                 "RUSTFS_KMS_LOCAL_MASTER_KEY": "my-secret-key", // Some Password
                 "RUSTFS_KMS_DEFAULT_KEY_ID": "rustfs-master-key",
             },
@@ -300,7 +307,6 @@
                 "RUSTFS_ACCESS_KEY": "rustfsadmin",
                 "RUSTFS_SECRET_KEY": "rustfsadmin",
                 "RUSTFS_VOLUMES": "./target/e2e-test/test{1...4}",
-                "RUSTFS_UNSAFE_BYPASS_DISK_CHECK": "true",
                 "RUSTFS_REGION": "us-east-1",
                 "RUSTFS_ADDRESS": ":9000",
                 "RUSTFS_CONSOLE_ENABLE": "true",
@@ -309,7 +315,7 @@
 
                 "RUSTFS_KMS_ENABLE": "true",
                 "RUSTFS_KMS_BACKEND": "local",
-                "RUSTFS_KMS_KEY_DIR": "${workspaceFolder}/target/e2e-key-dir",
+                "RUSTFS_KMS_KEY_DIR": "./target/e2e-key-dir",
                 "RUSTFS_KMS_LOCAL_MASTER_KEY": "my-secret-key", // Some Password
                 "RUSTFS_KMS_DEFAULT_KEY_ID": "rustfs-master-key",
             },

ARCHITECTURE.md

@@ -155,7 +155,7 @@ default build (lifecycle:
    - ⚠️ PARTIALLY VIOLATED: serving-side `s3s` references remain in ecstore
      (bucket metadata/replication/lifecycle DTOs and error mapping). The
      count is ratcheted shrink-only by `scripts/check_s3s_footprint.sh`
-     (`S3S_ECSTORE_FILES_BASELINE`; the `object_lock` module was converted to
+     (the `ecstore_files` line of `.config/s3s-footprint-baseline.txt`; the `object_lock` module was converted to
      storage-level types as the first ratchet step). Target state: the
      baseline reaches zero and ecstore's `Cargo.toml` drops `s3s`.
 

Cargo.toml

@@ -48,7 +48,7 @@ members = [
     "crates/replication", # Replication contracts and wire formats
     "crates/concurrency", # Concurrency management for RustFS - timeout, locking, backpressure, and I/O scheduling
     "crates/s3-client", # S3 client for engine-side consumption of remote S3 endpoints (tiering, transition targets)
-    "crates/s3-types", # S3 event type definitions
+    "crates/s3-types", # S3 event and error type definitions
     "crates/s3-ops", # S3 operation definitions and mapping
     "crates/s3select-api", # S3 Select API interface
     "crates/s3select-query", # S3 Select query engine
@@ -74,7 +74,7 @@ edition = "2024"
 license = "Apache-2.0"
 repository = "https://github.com/rustfs/rustfs"
 rust-version = "1.99.0"
-version = "1.0.2"
+version = "1.0.1"
 homepage = "https://rustfs.com"
 description = "RustFS is a high-performance distributed object storage software built using Rust, one of the most popular languages worldwide. "
 keywords = ["RustFS", "Minio", "object-storage", "filesystem", "s3"]
@@ -91,56 +91,56 @@ redundant_clone = "warn"
 
 [workspace.dependencies]
 # RustFS Internal Crates
-rustfs = { path = "./rustfs", version = "1.0.2" }
-rustfs-heal = { path = "crates/heal", version = "1.0.2" }
-rustfs-heal-contracts = { path = "crates/heal-contracts", version = "1.0.2" }
-rustfs-scanner-metrics = { path = "crates/scanner-metrics", version = "1.0.2" }
-rustfs-audit = { path = "crates/audit", version = "1.0.2" }
-rustfs-checksums = { path = "crates/checksums", version = "1.0.2" }
-rustfs-common = { path = "crates/common", version = "1.0.2" }
-rustfs-data-usage = { path = "crates/data-usage", version = "1.0.2" }
-rustfs-config = { path = "./crates/config", version = "1.0.2" }
-rustfs-concurrency = { path = "./crates/concurrency", version = "1.0.2" }
-rustfs-credentials = { path = "crates/credentials", version = "1.0.2" }
-rustfs-crypto = { path = "crates/crypto", version = "1.0.2" }
-rustfs-ecstore = { path = "crates/ecstore", version = "1.0.2" }
-rustfs-filemeta = { path = "crates/filemeta", version = "1.0.2" }
-rustfs-iam = { path = "crates/iam", version = "1.0.2" }
-rustfs-keystone = { path = "crates/keystone", version = "1.0.2" }
-rustfs-license = { path = "crates/license", version = "1.0.2" }
-rustfs-lifecycle = { path = "crates/lifecycle", version = "1.0.2" }
-rustfs-kms = { path = "crates/kms", version = "1.0.2" }
-rustfs-lock = { path = "crates/lock", version = "1.0.2" }
-rustfs-madmin = { path = "crates/madmin", version = "1.0.2" }
-rustfs-notify = { path = "crates/notify", version = "1.0.2" }
-rustfs-io-metrics = { path = "crates/io-metrics", version = "1.0.2" }
-rustfs-io-core = { path = "crates/io-core", version = "1.0.2" }
-rustfs-object-capacity = { path = "crates/object-capacity", version = "1.0.2" }
-rustfs-object-data-cache = { path = "crates/object-data-cache", version = "1.0.2", default-features = false }
-rustfs-log-analyzer = { path = "crates/log-analyzer", version = "1.0.2" }
-rustfs-obs = { path = "crates/obs", version = "1.0.2" }
-rustfs-policy = { path = "crates/policy", version = "1.0.2" }
-rustfs-protos = { path = "crates/protos", version = "1.0.2" }
-rustfs-protocols = { path = "crates/protocols", version = "1.0.2" }
-rustfs-replication = { path = "crates/replication", version = "1.0.2" }
-rustfs-rio = { path = "crates/rio", version = "1.0.2" }
-rustfs-rio-v2 = { path = "crates/rio-v2", version = "1.0.2" }
-rustfs-s3-client = { path = "crates/s3-client", version = "1.0.2" }
-rustfs-s3-types = { path = "crates/s3-types", version = "1.0.2" }
-rustfs-s3-ops = { path = "crates/s3-ops", version = "1.0.2" }
-rustfs-s3select-api = { path = "crates/s3select-api", version = "1.0.2" }
-rustfs-s3select-query = { path = "crates/s3select-query", version = "1.0.2" }
-rustfs-scanner = { path = "crates/scanner", version = "1.0.2" }
-rustfs-security-governance = { path = "crates/security-governance", version = "1.0.2" }
-rustfs-extension-schema = { path = "crates/extension-schema", version = "1.0.2" }
-rustfs-signer = { path = "crates/signer", version = "1.0.2" }
-rustfs-storage-api = { path = "crates/storage-api", version = "1.0.2" }
-rustfs-trusted-proxies = { path = "crates/trusted-proxies", version = "1.0.2" }
-rustfs-targets = { path = "crates/targets", version = "1.0.2" }
-rustfs-test-utils = { path = "crates/test-utils", version = "1.0.2" }
-rustfs-tls-runtime = { path = "crates/tls-runtime", version = "1.0.2" }
-rustfs-utils = { path = "crates/utils", version = "1.0.2" }
-rustfs-zip = { path = "./crates/zip", version = "1.0.2" }
+rustfs = { path = "./rustfs", version = "1.0.1" }
+rustfs-heal = { path = "crates/heal", version = "1.0.1" }
+rustfs-heal-contracts = { path = "crates/heal-contracts", version = "1.0.1" }
+rustfs-scanner-metrics = { path = "crates/scanner-metrics", version = "1.0.1" }
+rustfs-audit = { path = "crates/audit", version = "1.0.1" }
+rustfs-checksums = { path = "crates/checksums", version = "1.0.1" }
+rustfs-common = { path = "crates/common", version = "1.0.1" }
+rustfs-data-usage = { path = "crates/data-usage", version = "1.0.1" }
+rustfs-config = { path = "./crates/config", version = "1.0.1" }
+rustfs-concurrency = { path = "./crates/concurrency", version = "1.0.1" }
+rustfs-credentials = { path = "crates/credentials", version = "1.0.1" }
+rustfs-crypto = { path = "crates/crypto", version = "1.0.1" }
+rustfs-ecstore = { path = "crates/ecstore", version = "1.0.1" }
+rustfs-filemeta = { path = "crates/filemeta", version = "1.0.1" }
+rustfs-iam = { path = "crates/iam", version = "1.0.1" }
+rustfs-keystone = { path = "crates/keystone", version = "1.0.1" }
+rustfs-license = { path = "crates/license", version = "1.0.1" }
+rustfs-lifecycle = { path = "crates/lifecycle", version = "1.0.1" }
+rustfs-kms = { path = "crates/kms", version = "1.0.1" }
+rustfs-lock = { path = "crates/lock", version = "1.0.1" }
+rustfs-madmin = { path = "crates/madmin", version = "1.0.1" }
+rustfs-notify = { path = "crates/notify", version = "1.0.1" }
+rustfs-io-metrics = { path = "crates/io-metrics", version = "1.0.1" }
+rustfs-io-core = { path = "crates/io-core", version = "1.0.1" }
+rustfs-object-capacity = { path = "crates/object-capacity", version = "1.0.1" }
+rustfs-object-data-cache = { path = "crates/object-data-cache", version = "1.0.1", default-features = false }
+rustfs-log-analyzer = { path = "crates/log-analyzer", version = "1.0.1" }
+rustfs-obs = { path = "crates/obs", version = "1.0.1" }
+rustfs-policy = { path = "crates/policy", version = "1.0.1" }
+rustfs-protos = { path = "crates/protos", version = "1.0.1" }
+rustfs-protocols = { path = "crates/protocols", version = "1.0.1" }
+rustfs-replication = { path = "crates/replication", version = "1.0.1" }
+rustfs-rio = { path = "crates/rio", version = "1.0.1" }
+rustfs-rio-v2 = { path = "crates/rio-v2", version = "1.0.1" }
+rustfs-s3-client = { path = "crates/s3-client", version = "1.0.1" }
+rustfs-s3-types = { path = "crates/s3-types", version = "1.0.1" }
+rustfs-s3-ops = { path = "crates/s3-ops", version = "1.0.1" }
+rustfs-s3select-api = { path = "crates/s3select-api", version = "1.0.1" }
+rustfs-s3select-query = { path = "crates/s3select-query", version = "1.0.1" }
+rustfs-scanner = { path = "crates/scanner", version = "1.0.1" }
+rustfs-security-governance = { path = "crates/security-governance", version = "1.0.1" }
+rustfs-extension-schema = { path = "crates/extension-schema", version = "1.0.1" }
+rustfs-signer = { path = "crates/signer", version = "1.0.1" }
+rustfs-storage-api = { path = "crates/storage-api", version = "1.0.1" }
+rustfs-trusted-proxies = { path = "crates/trusted-proxies", version = "1.0.1" }
+rustfs-targets = { path = "crates/targets", version = "1.0.1" }
+rustfs-test-utils = { path = "crates/test-utils", version = "1.0.1" }
+rustfs-tls-runtime = { path = "crates/tls-runtime", version = "1.0.1" }
+rustfs-utils = { path = "crates/utils", version = "1.0.1" }
+rustfs-zip = { path = "./crates/zip", version = "1.0.1" }
 
 # Async Runtime and Networking
 async-channel = "2.5.0"
@@ -156,9 +156,8 @@ futures-lite = "2.6.1"
 futures-util = "0.3.34"
 pollster = "1.0.1"
 pulsar = { default-features = false, version = "6.9.0" }
-lapin = { default-features = false, version = "4.12.2" }
+lapin = { default-features = false, version = "4.12.0" }
 hyper = { version = "1.12.0" }
-h2 = { version = "0.4.20" }
 hyper-rustls = { default-features = false, version = "0.27.10" }
 hyper-util = { version = "0.1.21" }
 http = "1.5.0"
@@ -169,11 +168,11 @@ minlz = "1.2.3"
 reqwest = "0.13.5"
 rustfs-kafka-async = { version = "1.4.0" }
 socket2 = { version = "0.6.5" }
-tokio = { version = "1.53.3" }
+tokio = { version = "1.53.2" }
 tokio-rustls = { default-features = false, version = "0.26.6" }
 tokio-stream = { version = "0.1.19" }
 tokio-test = "0.4.6"
-tokio-util = { version = "0.7.20" }
+tokio-util = { version = "0.7.19" }
 tonic = { version = "0.14.6" }
 tonic-prost = { version = "0.14.6" }
 tonic-prost-build = { version = "0.14.6" }
@@ -181,7 +180,7 @@ tower = { version = "0.5.3" }
 tower-http = { version = "0.7.1" }
 h3 = "0.0.8"
 h3-quinn = "0.0.10"
-quinn = { version = "0.11.12", default-features = false, features = [
+quinn = { version = "0.11.11", default-features = false, features = [
       "futures-io",
       "runtime-tokio",
       "rustls-aws-lc-rs",
@@ -201,7 +200,7 @@ rmp-serde = { version = "1.3.1" }
 serde = { version = "1.0.229" }
 serde_ignored = { version = "0.1" }
 serde_with = { version = "3", default-features = false, features = ["macros", "std"] }
-serde_json = { version = "1.0.154" }
+serde_json = { version = "1.0.151" }
 serde_urlencoded = "0.7.1"
 
 # Cryptography and Security
@@ -225,7 +224,7 @@ rustls-native-certs = "0.8"
 rustls-pki-types = "1.15.1"
 x509-parser = "0.18.1"
 sha1 = "0.11.0"
-sha2 = "0.11.1"
+sha2 = "0.11.0"
 subtle = "2.6"
 zeroize = { version = "1.9.1" }
 proptest = "1"
@@ -269,7 +268,7 @@ crossbeam-queue = "0.3.14"
 crossbeam-channel = "0.5.17"
 crossbeam-deque = "0.8.8"
 crossbeam-utils = "0.8.23"
-datafusion = { default-features = false, version = "55.2.0" }
+datafusion = { default-features = false, version = "55.1.0" }
 derive_builder = "0.20.2"
 ed25519-dalek = "3.0.0"
 enumset = "1.1.14"
@@ -320,18 +319,23 @@ rumqttc = { package = "rumqttc-next", version = "0.34.0" }
 redis = { version = "1.7.1" }
 rustify = { version = "0.7", default-features = false }
 rustix = { version = "1.1.5" }
-rust-embed = { version = "8.13.0" }
+rust-embed = { version = "8.12.0" }
 rustc-hash = { version = "2.1.3" }
 s3s = { git = "https://github.com/s3s-project/s3s", rev = "5761ddfe4c509fcac87e3b99fd5bd9e176b442c0", features = ["minio"] }
 s3s-sigv4 = { git = "https://github.com/s3s-project/s3s", rev = "5761ddfe4c509fcac87e3b99fd5bd9e176b442c0" }
+# The same s3s pin under the name the legacy edge of the gateway migration uses
+# (crates/s3-types, feature `compat-s3s`). Keep `rev` identical to `s3s` above.
+# DELETE BY T4.2 (rustfs/backlog#2784).
+legacy-s3s = { package = "s3s", git = "https://github.com/s3s-project/s3s", rev = "5761ddfe4c509fcac87e3b99fd5bd9e176b442c0", features = ["minio"] }
 serial_test = "4.0.1"
 shadow-rs = { default-features = false, version = "2.0.0" }
 siphasher = "1.0.4"
-smallvec = { version = "1.16.3" }
+smallvec = { version = "1.16.2" }
 compact_str = "0.10.0"
 snap = "1.1.2"
 starshard = { version = "2.4.0" }
 strum = { version = "0.28.0" }
+sync_wrapper = "1.0.2"
 sysinfo = "0.39.6"
 temp-env = "0.3.6"
 tempfile = "3.27.0"
@@ -341,12 +345,12 @@ tracing = { version = "0.1.44" }
 tracing-appender = "0.2.5"
 tracing-core = "0.1.36"
 tracing-error = "0.2.1"
-tracing-opentelemetry = { version = "0.34.0" }
+tracing-opentelemetry = { version = "0.34" }
 tracing-subscriber = { version = "0.3.23" }
 transform-stream = "0.3.1"
 url = "2.5.8"
 urlencoding = "2.1.3"
-uuid = { version = "1.28.0" }
+uuid = { version = "1.27.0" }
 vaultrs = { version = "0.8.0" }
 tar = "0.4.46"
 walkdir = "2.5.0"
@@ -354,26 +358,26 @@ winapi-util = "0.1.11"
 windows = { version = "0.62.2" }
 windows-sys = "0.61.2"
 xxhash-rust = { version = "0.8.19" }
-zip = "9.0.3"
-zstd = "0.14.2"
+zip = "8.6.0"
+zstd = "0.14.0"
 
 # Observability and Metrics
 metrics = "0.24.6"
 metrics-util = "0.20"
-dial9 = { version = "0.5.4", default-features = false }
-opentelemetry = { version = "0.33.1" }
-opentelemetry-appender-tracing = { version = "0.33.1" }
-opentelemetry-otlp = { version = "0.33.1" }
-opentelemetry-proto = { version = "0.33.1", default-features = false, features = ["metrics", "gen-tonic-messages"] }
-opentelemetry_sdk = { version = "0.33.1" }
-opentelemetry-semantic-conventions = { version = "0.33.1" }
-opentelemetry-stdout = { version = "0.33.1" }
-pyroscope = { version = "2.1.2" }
+dial9 = { version = "0.5.3", default-features = false }
+opentelemetry = { version = "0.33.0" }
+opentelemetry-appender-tracing = { version = "0.33.0" }
+opentelemetry-otlp = { version = "0.33.0" }
+opentelemetry-proto = { version = "0.33.0", default-features = false, features = ["metrics", "gen-tonic-messages"] }
+opentelemetry_sdk = { version = "0.33.0" }
+opentelemetry-semantic-conventions = { version = "0.33.0" }
+opentelemetry-stdout = { version = "0.33.0" }
+pyroscope = { version = "2.1.1" }
 
 # FTP and SFTP
 libunftp = { version = "0.23.1" }
 unftp-core = "0.1.0"
-suppaftp = { version = "12.2.0" }
+suppaftp = { version = "12.1.1" }
 rcgen = { version = "0.14.10", default-features = false, features = ["aws_lc_rs", "crypto", "pem"] }
 russh = { version = "0.64.1" }
 russh-sftp = "3.0.1"
@@ -384,7 +388,7 @@ dav-server = "0.11.0"
 # Performance Analysis and Memory Profiling
 rustfs-mimalloc = { version = "0.5.6" }
 # Preserve Unicode focus filters until rustfs/backlog#2302 is resolved.
-hotpath = { version = "0.28.6", default-features = false }
+hotpath = { version = "0.28.5", default-features = false }
 # Snapshot testing for output format regression detection
 insta = { version = "1.49" }
 

_typos.toml

@@ -51,8 +51,6 @@ unparseable = "unparseable"
 # Disaster-recovery objectives: recovery time and recovery point.
 RTO = "RTO"
 rto = "rto"
-# rustix OFlags variant (POSIX O_WRONLY equivalent).
-WRONLY = "WRONLY"
 
 [files]
 extend-exclude = []

crates/common/Cargo.toml

@@ -34,7 +34,6 @@ hotpath-alloc = ["hotpath", "hotpath/hotpath-alloc"]
 hotpath-cpu = ["hotpath", "hotpath/hotpath-cpu"]
 
 [dependencies]
-arc-swap.workspace = true
 hotpath.workspace = true
 tokio = { workspace = true, features = ["fs", "rt-multi-thread"] }
 tonic = { workspace = true, features = ["gzip", "deflate"] }

crates/common/src/globals.rs

@@ -12,10 +12,9 @@
 // See the License for the specific language governing permissions and
 // limitations under the License.
 
-use arc_swap::ArcSwap;
 use std::collections::HashMap;
+use std::sync::LazyLock;
 use std::sync::atomic::{AtomicU64, Ordering};
-use std::sync::{Arc, LazyLock};
 use tokio::sync::RwLock;
 use tonic::transport::Channel;
 
@@ -28,42 +27,6 @@ pub static GLOBAL_ROOT_CERT: LazyLock<RwLock<Option<Vec<u8>>>> = LazyLock::new(|
 pub static GLOBAL_MTLS_IDENTITY: LazyLock<RwLock<Option<MtlsIdentityPem>>> = LazyLock::new(|| RwLock::new(None));
 pub static GLOBAL_OUTBOUND_TLS_GENERATION: LazyLock<AtomicU64> = LazyLock::new(|| AtomicU64::new(0));
 
-#[derive(Clone, Default)]
-pub struct OutboundTlsSnapshot {
-    pub generation: u64,
-    pub root_ca_pem: Option<Vec<u8>>,
-    pub mtls_identity: Option<MtlsIdentityPem>,
-}
-
-static GLOBAL_OUTBOUND_TLS_STATE: LazyLock<ArcSwap<OutboundTlsSnapshot>> =
-    LazyLock::new(|| ArcSwap::from_pointee(OutboundTlsSnapshot::default()));
-
-pub fn get_global_outbound_tls_snapshot() -> Arc<OutboundTlsSnapshot> {
-    GLOBAL_OUTBOUND_TLS_STATE.load_full()
-}
-
-/// Publish identity, trust and generation together; the individual globals are legacy views.
-pub async fn publish_global_outbound_tls_snapshot(snapshot: OutboundTlsSnapshot) -> u64 {
-    GLOBAL_OUTBOUND_TLS_STATE.rcu(|current| {
-        if current.generation != 0
-            && current.root_ca_pem == snapshot.root_ca_pem
-            && current.mtls_identity == snapshot.mtls_identity
-        {
-            return Arc::clone(current);
-        }
-        Arc::new(OutboundTlsSnapshot {
-            generation: snapshot.generation.max(current.generation.saturating_add(1)),
-            root_ca_pem: snapshot.root_ca_pem.clone(),
-            mtls_identity: snapshot.mtls_identity.clone(),
-        })
-    });
-    let accepted = get_global_outbound_tls_snapshot();
-    *GLOBAL_ROOT_CERT.write().await = accepted.root_ca_pem.clone();
-    *GLOBAL_MTLS_IDENTITY.write().await = accepted.mtls_identity.clone();
-    GLOBAL_OUTBOUND_TLS_GENERATION.store(accepted.generation, Ordering::Relaxed);
-    accepted.generation
-}
-
 /// Log level to use when reporting cached gRPC connection eviction.
 #[derive(Clone, Copy, Debug, Eq, PartialEq)]
 pub enum ConnectionEvictionLogLevel {
@@ -122,27 +85,17 @@ pub async fn get_global_addr() -> String {
 /// # Arguments
 /// * `cert` - A vector of bytes representing the PEM-encoded root CA certificate.
 pub async fn set_global_root_cert(cert: Vec<u8>) {
-    GLOBAL_OUTBOUND_TLS_STATE.rcu(|current| {
-        let mut next = (**current).clone();
-        next.root_ca_pem = Some(cert.clone());
-        Arc::new(next)
-    });
     *GLOBAL_ROOT_CERT.write().await = Some(cert);
 }
 
 /// Clear the global root CA certificate for outbound gRPC clients.
 pub async fn clear_global_root_cert() {
-    GLOBAL_OUTBOUND_TLS_STATE.rcu(|current| {
-        let mut next = (**current).clone();
-        next.root_ca_pem = None;
-        Arc::new(next)
-    });
     *GLOBAL_ROOT_CERT.write().await = None;
 }
 
 /// Get the global root CA certificate for outbound gRPC clients.
 pub async fn get_global_root_cert() -> Option<Vec<u8>> {
-    get_global_outbound_tls_snapshot().root_ca_pem.clone()
+    GLOBAL_ROOT_CERT.read().await.clone()
 }
 
 /// Set the global mTLS identity (cert+key PEM) for outbound gRPC clients.
@@ -152,32 +105,22 @@ pub async fn get_global_root_cert() -> Option<Vec<u8>> {
 /// # Arguments
 /// * `identity` - An optional MtlsIdentityPem struct containing the cert and key PEM.
 pub async fn set_global_mtls_identity(identity: Option<MtlsIdentityPem>) {
-    GLOBAL_OUTBOUND_TLS_STATE.rcu(|current| {
-        let mut next = (**current).clone();
-        next.mtls_identity = identity.clone();
-        Arc::new(next)
-    });
     *GLOBAL_MTLS_IDENTITY.write().await = identity;
 }
 
 /// Get the global mTLS identity for outbound gRPC clients.
 pub async fn get_global_mtls_identity() -> Option<MtlsIdentityPem> {
-    get_global_outbound_tls_snapshot().mtls_identity.clone()
+    GLOBAL_MTLS_IDENTITY.read().await.clone()
 }
 
 /// Set the global outbound TLS generation.
 pub fn set_global_outbound_tls_generation(generation: u64) {
-    GLOBAL_OUTBOUND_TLS_STATE.rcu(|current| {
-        let mut next = (**current).clone();
-        next.generation = generation;
-        Arc::new(next)
-    });
     GLOBAL_OUTBOUND_TLS_GENERATION.store(generation, Ordering::Relaxed);
 }
 
 /// Get the global outbound TLS generation.
 pub fn get_global_outbound_tls_generation() -> u64 {
-    get_global_outbound_tls_snapshot().generation
+    GLOBAL_OUTBOUND_TLS_GENERATION.load(Ordering::Relaxed)
 }
 
 /// Evict a stale/dead connection from the global connection cache.
@@ -251,77 +194,8 @@ pub async fn clear_all_connections() {
 ///
 /// When present, gRPC clients will present this identity to servers requesting/requiring mTLS.
 /// When absent, clients proceed with standard server-authenticated TLS.
-#[derive(Clone, PartialEq, Eq)]
+#[derive(Clone, Debug)]
 pub struct MtlsIdentityPem {
     pub cert_pem: Vec<u8>,
     pub key_pem: Vec<u8>,
 }
-
-impl std::fmt::Debug for MtlsIdentityPem {
-    fn fmt(&self, f: &mut std::fmt::Formatter<'_>) -> std::fmt::Result {
-        f.debug_struct("MtlsIdentityPem")
-            .field("cert_bytes", &self.cert_pem.len())
-            .field("key_pem", &"[REDACTED]")
-            .finish()
-    }
-}
-
-#[cfg(test)]
-mod tls_snapshot_tests {
-    use super::*;
-
-    #[tokio::test]
-    async fn outbound_snapshot_publication_is_coherent_and_skips_unchanged_material() {
-        let start = get_global_outbound_tls_generation().saturating_add(1);
-        let snapshot = OutboundTlsSnapshot {
-            generation: start,
-            root_ca_pem: Some(vec![1]),
-            mtls_identity: Some(MtlsIdentityPem {
-                cert_pem: vec![1],
-                key_pem: vec![1],
-            }),
-        };
-        let first = publish_global_outbound_tls_snapshot(snapshot.clone()).await;
-        assert_eq!(
-            publish_global_outbound_tls_snapshot(OutboundTlsSnapshot {
-                generation: first.saturating_add(1),
-                ..snapshot
-            })
-            .await,
-            first
-        );
-        let writer = tokio::spawn(async move {
-            for tag in 2_u8..=255 {
-                publish_global_outbound_tls_snapshot(OutboundTlsSnapshot {
-                    generation: start.saturating_add(u64::from(tag)),
-                    root_ca_pem: Some(vec![tag]),
-                    mtls_identity: Some(MtlsIdentityPem {
-                        cert_pem: vec![tag],
-                        key_pem: vec![tag],
-                    }),
-                })
-                .await;
-                tokio::task::yield_now().await;
-            }
-        });
-        while !writer.is_finished() {
-            let state = get_global_outbound_tls_snapshot();
-            let identity = state.mtls_identity.as_ref().expect("identity in complete snapshot");
-            assert_eq!(state.root_ca_pem.as_ref().expect("roots in complete snapshot"), &identity.cert_pem);
-            assert_eq!(identity.cert_pem, identity.key_pem);
-            tokio::task::yield_now().await;
-        }
-        writer.await.expect("writer completes");
-    }
-
-    #[test]
-    fn mtls_identity_debug_redacts_private_key() {
-        let identity = MtlsIdentityPem {
-            cert_pem: b"public certificate".to_vec(),
-            key_pem: b"PRIVATE-KEY-MARKER".to_vec(),
-        };
-        let debug = format!("{identity:?}");
-        assert!(!debug.contains("PRIVATE-KEY-MARKER"));
-        assert!(debug.contains("[REDACTED]"));
-    }
-}

crates/config/src/constants/app.rs

@@ -18,8 +18,10 @@ use const_str::concat;
 /// Default value: RustFS
 /// Environment variable: RUSTFS_APP_NAME
 pub const APP_NAME: &str = "RustFS";
-/// Cargo package version for this workspace member.
-pub const VERSION: &str = env!("CARGO_PKG_VERSION");
+/// Application version
+/// Default value: 1.0.0
+/// Environment variable: RUSTFS_VERSION
+pub const VERSION: &str = "1.0.0";
 
 /// Default configuration logger level
 /// Default value: error
@@ -37,6 +39,12 @@ pub const SAMPLE_RATIO: f64 = 1.0;
 /// Default value: 30
 pub const METER_INTERVAL: u64 = 30;
 
+/// Default configuration service version
+/// Default value: 1.0.0
+/// Environment variable: RUSTFS_OBS_SERVICE_VERSION
+/// Uses the same value as VERSION constant
+pub const SERVICE_VERSION: &str = "1.0.0";
+
 /// Default configuration environment
 /// Default value: production
 pub const ENVIRONMENT: &str = "production";
@@ -402,6 +410,7 @@ mod tests {
             APP_NAME,
             VERSION,
             DEFAULT_LOG_LEVEL,
+            SERVICE_VERSION,
             ENVIRONMENT,
             RUSTFS_TLS_KEY,
             RUSTFS_TLS_CERT,
@@ -437,7 +446,8 @@ mod tests {
     fn test_configuration_consistency() {
         // Test configuration consistency
 
-        assert_eq!(VERSION, env!("CARGO_PKG_VERSION"));
+        // Version consistency
+        assert_eq!(VERSION, SERVICE_VERSION, "Application version should match service version");
 
         // Port conflict check
         let ports = [DEFAULT_PORT, DEFAULT_CONSOLE_PORT];

crates/config/src/observability/mod.rs

@@ -51,11 +51,6 @@ pub const ENV_OBS_PROFILING_EXPORT_ENABLED: &str = "RUSTFS_OBS_PROFILING_EXPORT_
 /// PUT records multiple timers and histograms when attribution is active.
 pub const ENV_OBS_PUT_STAGE_METRICS_ENABLED: &str = "RUSTFS_OBS_PUT_STAGE_METRICS_ENABLED";
 
-/// Enables sampled in-process GetObject stage summaries without OTLP histograms.
-pub const ENV_OBS_GET_STAGE_LOCAL_SUMMARY_ENABLED: &str = "RUSTFS_OBS_GET_STAGE_LOCAL_SUMMARY_ENABLED";
-pub const ENV_OBS_GET_STAGE_LOCAL_SUMMARY_SAMPLE_RATE: &str = "RUSTFS_OBS_GET_STAGE_LOCAL_SUMMARY_SAMPLE_RATE";
-pub const ENV_OBS_GET_STAGE_LOCAL_SUMMARY_INTERVAL_SECS: &str = "RUSTFS_OBS_GET_STAGE_LOCAL_SUMMARY_INTERVAL_SECS";
-
 pub const ENV_OBS_LOGGER_LEVEL: &str = "RUSTFS_OBS_LOGGER_LEVEL";
 pub const ENV_OBS_LOG_STDOUT_ENABLED: &str = "RUSTFS_OBS_LOG_STDOUT_ENABLED";
 pub const ENV_OBS_LOG_DIRECTORY: &str = "RUSTFS_OBS_LOG_DIRECTORY";
@@ -164,15 +159,6 @@ mod tests {
         assert_eq!(ENV_OBS_LOGS_EXPORT_ENABLED, "RUSTFS_OBS_LOGS_EXPORT_ENABLED");
         assert_eq!(ENV_OBS_PROFILING_EXPORT_ENABLED, "RUSTFS_OBS_PROFILING_EXPORT_ENABLED");
         assert_eq!(ENV_OBS_PUT_STAGE_METRICS_ENABLED, "RUSTFS_OBS_PUT_STAGE_METRICS_ENABLED");
-        assert_eq!(ENV_OBS_GET_STAGE_LOCAL_SUMMARY_ENABLED, "RUSTFS_OBS_GET_STAGE_LOCAL_SUMMARY_ENABLED");
-        assert_eq!(
-            ENV_OBS_GET_STAGE_LOCAL_SUMMARY_SAMPLE_RATE,
-            "RUSTFS_OBS_GET_STAGE_LOCAL_SUMMARY_SAMPLE_RATE"
-        );
-        assert_eq!(
-            ENV_OBS_GET_STAGE_LOCAL_SUMMARY_INTERVAL_SECS,
-            "RUSTFS_OBS_GET_STAGE_LOCAL_SUMMARY_INTERVAL_SECS"
-        );
         // Test log cleanup related env keys
         assert_eq!(ENV_OBS_LOG_MAX_TOTAL_SIZE_BYTES, "RUSTFS_OBS_LOG_MAX_TOTAL_SIZE_BYTES");
         assert_eq!(ENV_OBS_LOG_MAX_SINGLE_FILE_SIZE_BYTES, "RUSTFS_OBS_LOG_MAX_SINGLE_FILE_SIZE_BYTES");

crates/e2e_test/src/common.rs

@@ -1542,15 +1542,6 @@ impl RustFSTestClusterEnvironment {
         self.start_node_from_binary(node_idx, &binary_path).await
     }
 
-    /// Start a process and wait only for TCP, allowing tests to bring up a
-    /// quorum one node at a time before the S3 service can become ready.
-    pub async fn start_node_process(&mut self, node_idx: usize) -> Result<(), Box<dyn std::error::Error + Send + Sync>> {
-        let binary_path = rustfs_binary_path();
-        let volumes_arg = self.build_volumes_arg();
-        self.spawn_node(node_idx, &binary_path, &volumes_arg)?;
-        self.wait_for_node_ready(&self.nodes[node_idx].address, node_idx).await
-    }
-
     /// Start one stopped cluster node with a specific RustFS binary while
     /// preserving the cluster's volume layout and that node's data directory.
     pub async fn start_node_from_binary(

crates/e2e_test/src/distributed/harness.rs

@@ -860,12 +860,10 @@ pub(crate) async fn start_rebalance(cluster: &RustFSTestClusterEnvironment) -> T
         if status.is_success() {
             break response;
         }
-        // The init stage answers 503 + Retry-After while the cluster capability
-        // proof is still missing; older binaries surfaced the same condition as
-        // a 500 carrying the proof marker.
-        let waiting_for_readiness = status == StatusCode::SERVICE_UNAVAILABLE
-            || (status == StatusCode::INTERNAL_SERVER_ERROR && response.contains("requires a live fleet capability proof"));
-        if waiting_for_readiness && Instant::now() < deadline {
+        if status == StatusCode::INTERNAL_SERVER_ERROR
+            && response.contains("requires a live fleet capability proof")
+            && Instant::now() < deadline
+        {
             sleep(Duration::from_secs(1)).await;
             continue;
         }

crates/e2e_test/src/distributed/upgrade_test.rs

@@ -31,7 +31,6 @@ use super::harness::{
 use crate::common::{
     AdminTransport, admin_add_canned_policy_via, admin_attach_user_policy_via, admin_create_user_via, init_logging,
 };
-use crate::upgrade_compatibility_test::wait_for_upgrade_write_readiness;
 use aws_sdk_s3::Client;
 use aws_sdk_s3::error::ProvideErrorMetadata;
 use std::path::{Path, PathBuf};
@@ -401,19 +400,6 @@ async fn four_node_direct_upgrade_preserves_history_and_iam_credentials() -> Tes
     Ok(())
 }
 
-async fn replace_node_and_wait_for_write_readiness(dist: &mut DistCluster, seed: &UpgradeSeed, node_idx: usize) -> TestResult {
-    dist.replace_node_with_current_binary(node_idx).await?;
-    // Node readiness proves reads can succeed; a rolling writer must also
-    // recover before replacing the next peer or checking upgrade contracts.
-    wait_for_upgrade_write_readiness(
-        &dist.clients()?,
-        &seed.history_bucket,
-        &format!("replaced-node-{node_idx}"),
-        CREDENTIAL_TIMEOUT,
-    )
-    .await
-}
-
 #[tokio::test]
 async fn four_node_rolling_upgrade_preserves_history_and_iam_credentials() -> TestResult {
     init_logging();
@@ -424,15 +410,15 @@ async fn four_node_rolling_upgrade_preserves_history_and_iam_credentials() -> Te
 
     let seed = seed_history_and_iam(&dist).await?;
 
-    replace_node_and_wait_for_write_readiness(&mut dist, &seed, 0).await?;
+    dist.replace_node_with_current_binary(0).await?;
     assert_history_and_iam(&dist, &seed, "one-current-node").await?;
 
     for node_idx in [1, 2] {
-        replace_node_and_wait_for_write_readiness(&mut dist, &seed, node_idx).await?;
+        dist.replace_node_with_current_binary(node_idx).await?;
     }
     assert_history_and_iam(&dist, &seed, "one-previous-node").await?;
 
-    replace_node_and_wait_for_write_readiness(&mut dist, &seed, 3).await?;
+    dist.replace_node_with_current_binary(3).await?;
     assert_history_and_iam(&dist, &seed, "homogeneous-current").await?;
     Ok(())
 }

crates/e2e_test/src/dual_stack_differential/compare.rs

@@ -0,0 +1,512 @@
+// Copyright 2024 RustFS Team
+//
+// Licensed under the Apache License, Version 2.0 (the "License");
+// you may not use this file except in compliance with the License.
+// You may obtain a copy of the License at
+//
+//     http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing, software
+// distributed under the License is distributed on an "AS IS" BASIS,
+// WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+// See the License for the specific language governing permissions and
+// limitations under the License.
+
+//! What one stack answered, and how two answers are compared.
+//!
+//! An [`Observation`] keeps only what a client acts on: the HTTP status, the S3 error code, the
+//! allowlisted response headers and a few typed fields decoded from the response. The error
+//! message is kept for diagnostics and never compared: message wording is judged by the wire
+//! corpus, not here. The comparator is pure so it can be tested without a server.
+
+use serde::Serialize;
+use std::collections::{BTreeMap, BTreeSet};
+use std::path::{Path, PathBuf};
+
+/// Response headers compared exactly (after [`normalized_header_value`]).
+const HEADER_ALLOWLIST: [&str; 7] = [
+    "content-type",
+    "x-amz-version-id",
+    "etag",
+    "x-amz-delete-marker",
+    "content-range",
+    "accept-ranges",
+    "x-amz-request-charged",
+];
+
+/// Every response header with this prefix is compared as well.
+const HEADER_ALLOWLIST_PREFIX: &str = "x-amz-server-side-encryption";
+
+/// Placeholder for a field one side did not send.
+const ABSENT: &str = "<absent>";
+
+/// The lowercase header name if the comparator looks at this header.
+pub(crate) fn compared_header(name: &str) -> Option<String> {
+    let name = name.to_ascii_lowercase();
+    (HEADER_ALLOWLIST.contains(&name.as_str()) || name.starts_with(HEADER_ALLOWLIST_PREFIX)).then_some(name)
+}
+
+/// A version ID is minted per process, so its value never matches across two servers; only
+/// whether it is the literal `null` (an unversioned write) carries meaning.
+fn normalized_header_value(name: &str, value: &str) -> String {
+    if name == "x-amz-version-id" && value != "null" {
+        "<version-id>".to_owned()
+    } else {
+        value.to_owned()
+    }
+}
+
+/// One request's outcome as a client sees it.
+#[derive(Clone, Debug, Serialize)]
+pub(crate) struct Observation {
+    /// The S3 operation, named as in the `s3s::S3` trait (`GetObject`).
+    pub op: &'static str,
+    /// Which request of the script this was (`normal`, `negative`, or a setup label).
+    pub case: &'static str,
+    /// HTTP status; 0 when no response arrived.
+    pub status: u16,
+    /// The S3 error code, or the SDK's error kind when there was no service error.
+    pub error_code: Option<String>,
+    /// Allowlisted headers only, lowercase names, normalized values.
+    pub headers: BTreeMap<String, String>,
+    /// Typed values decoded from the response body, never free text.
+    pub fields: BTreeMap<String, String>,
+    /// Diagnostic only; never compared.
+    #[serde(skip)]
+    pub message: Option<String>,
+}
+
+impl Observation {
+    /// Builds an observation, keeping only the allowlisted headers.
+    pub(crate) fn new<'h>(
+        op: &'static str,
+        case: &'static str,
+        status: u16,
+        error_code: Option<String>,
+        headers: impl IntoIterator<Item = (&'h str, &'h str)>,
+    ) -> Self {
+        let headers = headers
+            .into_iter()
+            .filter_map(|(name, value)| compared_header(name).map(|name| (name, value)))
+            .map(|(name, value)| {
+                let value = normalized_header_value(&name, value);
+                (name, value)
+            })
+            .collect();
+        Self {
+            op,
+            case,
+            status,
+            error_code,
+            headers,
+            fields: BTreeMap::new(),
+            message: None,
+        }
+    }
+
+    pub(crate) fn with_field(mut self, name: &str, value: impl Into<String>) -> Self {
+        self.fields.insert(name.to_owned(), value.into());
+        self
+    }
+
+    pub(crate) fn is_success(&self) -> bool {
+        (200..300).contains(&self.status)
+    }
+}
+
+/// One field on which the two stacks answered differently.
+#[derive(Clone, Debug, PartialEq, Eq, Serialize)]
+pub(crate) struct Difference {
+    pub op: String,
+    pub case: String,
+    pub field: String,
+    pub a: String,
+    pub b: String,
+}
+
+/// Every difference between two transcripts of the same script, matched by `(op, case)`.
+pub(crate) fn compare(a: &[Observation], b: &[Observation]) -> Vec<Difference> {
+    let mut differences = Vec::new();
+    let (index_a, duplicates_a) = index(a);
+    let (index_b, duplicates_b) = index(b);
+    for (op, case) in duplicates_a.union(&duplicates_b) {
+        let count = |side: &[Observation]| {
+            side.iter()
+                .filter(|observation| (observation.op, observation.case) == (*op, *case))
+                .count()
+                .to_string()
+        };
+        differences.push(difference(op, case, "duplicate_case", count(a), count(b)));
+    }
+    let keys: BTreeSet<_> = index_a.keys().chain(index_b.keys()).copied().collect();
+    for key @ (op, case) in keys {
+        match (index_a.get(&key), index_b.get(&key)) {
+            (Some(a), Some(b)) => compare_one(a, b, &mut differences),
+            (a, b) => {
+                let presence = |side: Option<&&Observation>| if side.is_some() { "present" } else { ABSENT }.to_owned();
+                differences.push(difference(op, case, "observation", presence(a), presence(b)));
+            }
+        }
+    }
+    differences
+}
+
+type Key = (&'static str, &'static str);
+
+fn index(side: &[Observation]) -> (BTreeMap<Key, &Observation>, BTreeSet<Key>) {
+    let mut index = BTreeMap::new();
+    let mut duplicates = BTreeSet::new();
+    for observation in side {
+        let key = (observation.op, observation.case);
+        if index.insert(key, observation).is_some() {
+            duplicates.insert(key);
+        }
+    }
+    (index, duplicates)
+}
+
+fn difference(op: &str, case: &str, field: &str, a: String, b: String) -> Difference {
+    Difference {
+        op: op.to_owned(),
+        case: case.to_owned(),
+        field: field.to_owned(),
+        a,
+        b,
+    }
+}
+
+fn compare_one(a: &Observation, b: &Observation, differences: &mut Vec<Difference>) {
+    let mut field = |name: String, value_a: Option<&str>, value_b: Option<&str>| {
+        if value_a != value_b {
+            let show = |value: Option<&str>| value.unwrap_or(ABSENT).to_owned();
+            differences.push(difference(a.op, a.case, &name, show(value_a), show(value_b)));
+        }
+    };
+    field(
+        "status".to_owned(),
+        Some(a.status.to_string().as_str()),
+        Some(b.status.to_string().as_str()),
+    );
+    field("error_code".to_owned(), a.error_code.as_deref(), b.error_code.as_deref());
+    let headers: BTreeSet<_> = a.headers.keys().chain(b.headers.keys()).collect();
+    for name in headers {
+        field(
+            format!("header.{name}"),
+            a.headers.get(name).map(String::as_str),
+            b.headers.get(name).map(String::as_str),
+        );
+    }
+    let fields: BTreeSet<_> = a.fields.keys().chain(b.fields.keys()).collect();
+    for name in fields {
+        field(
+            format!("body.{name}"),
+            a.fields.get(name).map(String::as_str),
+            b.fields.get(name).map(String::as_str),
+        );
+    }
+}
+
+/// Operations observed on both sides.
+pub(crate) fn compared_ops(a: &[Observation], b: &[Observation]) -> BTreeSet<&'static str> {
+    let ops_b: BTreeSet<_> = b.iter().map(|observation| observation.op).collect();
+    a.iter()
+        .map(|observation| observation.op)
+        .filter(|op| ops_b.contains(op))
+        .collect()
+}
+
+/// `(op, case)` pairs observed on both sides.
+pub(crate) fn compared_cases(a: &[Observation], b: &[Observation]) -> usize {
+    let cases_b: BTreeSet<_> = b.iter().map(|observation| (observation.op, observation.case)).collect();
+    a.iter()
+        .map(|observation| (observation.op, observation.case))
+        .collect::<BTreeSet<_>>()
+        .intersection(&cases_b)
+        .count()
+}
+
+/// The JSON report written for every run.
+#[derive(Debug, Serialize)]
+pub(crate) struct Report {
+    pub schema: u32,
+    pub run: String,
+    pub stack_a: String,
+    pub stack_b: String,
+    pub region_a: Option<String>,
+    pub region_b: Option<String>,
+    pub compared_ops: usize,
+    pub compared_cases: usize,
+    pub differences: Vec<Difference>,
+}
+
+impl Report {
+    /// Writes the report as `<directory>/<run>.json` and returns its path.
+    pub(crate) fn write_to(&self, directory: &Path) -> std::io::Result<PathBuf> {
+        std::fs::create_dir_all(directory)?;
+        let path = directory.join(format!("{}.json", self.run));
+        let json = serde_json::to_vec_pretty(self).map_err(std::io::Error::other)?;
+        std::fs::write(&path, json)?;
+        Ok(path)
+    }
+}
+
+/// One line per difference, for a failing assertion.
+pub(crate) fn render(differences: &[Difference]) -> String {
+    differences
+        .iter()
+        .map(|d| format!("({}, {}, {}, a={:?}, b={:?})", d.op, d.case, d.field, d.a, d.b))
+        .collect::<Vec<_>>()
+        .join("\n")
+}
+
+#[cfg(test)]
+mod tests {
+    use super::*;
+
+    fn ok(op: &'static str) -> Observation {
+        Observation::new(op, "normal", 200, None, [("content-type", "application/xml")])
+    }
+
+    fn fields(differences: &[Difference]) -> Vec<&str> {
+        differences.iter().map(|d| d.field.as_str()).collect()
+    }
+
+    #[test]
+    fn identical_transcripts_have_no_difference() {
+        let a = vec![
+            ok("HeadBucket"),
+            ok("GetBucketLocation").with_field("location_constraint", "us-east-1"),
+        ];
+        let b = a.clone();
+        assert_eq!(compare(&a, &b), Vec::new());
+        assert_eq!(compared_ops(&a, &b).len(), 2);
+    }
+
+    #[test]
+    fn a_status_difference_is_reported() {
+        let a = vec![ok("PutBucketAcl")];
+        let b = vec![Observation::new(
+            "PutBucketAcl",
+            "normal",
+            403,
+            None,
+            [("content-type", "application/xml")],
+        )];
+        assert_eq!(
+            compare(&a, &b),
+            vec![Difference {
+                op: "PutBucketAcl".into(),
+                case: "normal".into(),
+                field: "status".into(),
+                a: "200".into(),
+                b: "403".into(),
+            }]
+        );
+    }
+
+    #[test]
+    fn an_error_code_difference_is_reported() {
+        let a = vec![Observation::new("GetObject", "negative", 404, Some("NoSuchKey".into()), [])];
+        let b = vec![Observation::new(
+            "GetObject",
+            "negative",
+            404,
+            Some("NoSuchBucket".into()),
+            [],
+        )];
+        let differences = compare(&a, &b);
+        assert_eq!(fields(&differences), ["error_code"]);
+        assert_eq!((differences[0].a.as_str(), differences[0].b.as_str()), ("NoSuchKey", "NoSuchBucket"));
+    }
+
+    #[test]
+    fn an_error_code_on_one_side_only_is_reported() {
+        let a = vec![Observation::new("HeadBucket", "negative", 404, Some("NotFound".into()), [])];
+        let b = vec![Observation::new("HeadBucket", "negative", 404, None, [])];
+        let differences = compare(&a, &b);
+        assert_eq!(fields(&differences), ["error_code"]);
+        assert_eq!(differences[0].b, ABSENT);
+    }
+
+    #[test]
+    fn an_allowlisted_header_difference_is_reported() {
+        let a = vec![Observation::new("HeadObject", "normal", 200, None, [("ETag", "\"aa\"")])];
+        let b = vec![Observation::new("HeadObject", "normal", 200, None, [("etag", "\"bb\"")])];
+        assert_eq!(fields(&compare(&a, &b)), ["header.etag"]);
+    }
+
+    #[test]
+    fn a_header_sent_by_one_side_only_is_reported() {
+        let a = vec![Observation::new(
+            "DeleteObject",
+            "normal",
+            204,
+            None,
+            [("x-amz-delete-marker", "true")],
+        )];
+        let b = vec![Observation::new("DeleteObject", "normal", 204, None, [])];
+        let differences = compare(&a, &b);
+        assert_eq!(fields(&differences), ["header.x-amz-delete-marker"]);
+        assert_eq!((differences[0].a.as_str(), differences[0].b.as_str()), ("true", ABSENT));
+    }
+
+    #[test]
+    fn every_server_side_encryption_header_is_compared() {
+        let header = "x-amz-server-side-encryption-customer-key-md5";
+        let a = vec![Observation::new("PutObject", "sse-c", 200, None, [(header, "one")])];
+        let b = vec![Observation::new("PutObject", "sse-c", 200, None, [(header, "two")])];
+        assert_eq!(fields(&compare(&a, &b)), [format!("header.{header}")]);
+    }
+
+    #[test]
+    fn headers_outside_the_allowlist_are_not_compared() {
+        let a = vec![Observation::new(
+            "ListBuckets",
+            "normal",
+            200,
+            None,
+            [
+                ("date", "Mon"),
+                ("x-amz-request-id", "1"),
+                ("server", "a"),
+                ("content-length", "10"),
+            ],
+        )];
+        let b = vec![Observation::new(
+            "ListBuckets",
+            "normal",
+            200,
+            None,
+            [
+                ("date", "Tue"),
+                ("x-amz-request-id", "2"),
+                ("server", "b"),
+                ("content-length", "11"),
+            ],
+        )];
+        assert_eq!(compare(&a, &b), Vec::new());
+        assert!(a[0].headers.is_empty(), "non-allowlisted headers were kept: {:?}", a[0].headers);
+    }
+
+    #[test]
+    fn distinct_version_ids_are_not_a_difference() {
+        let a = vec![Observation::new(
+            "PutObject",
+            "normal",
+            200,
+            None,
+            [("x-amz-version-id", "0b1c")],
+        )];
+        let b = vec![Observation::new(
+            "PutObject",
+            "normal",
+            200,
+            None,
+            [("x-amz-version-id", "9f8e")],
+        )];
+        assert_eq!(compare(&a, &b), Vec::new());
+    }
+
+    #[test]
+    fn a_null_version_id_against_a_real_one_is_reported() {
+        let a = vec![Observation::new(
+            "PutObject",
+            "normal",
+            200,
+            None,
+            [("x-amz-version-id", "null")],
+        )];
+        let b = vec![Observation::new(
+            "PutObject",
+            "normal",
+            200,
+            None,
+            [("x-amz-version-id", "9f8e")],
+        )];
+        assert_eq!(fields(&compare(&a, &b)), ["header.x-amz-version-id"]);
+    }
+
+    #[test]
+    fn a_body_field_difference_is_reported() {
+        let a = vec![ok("GetBucketLocation").with_field("location_constraint", "us-east-1")];
+        let b = vec![ok("GetBucketLocation").with_field("location_constraint", "eu-west-1")];
+        let differences = compare(&a, &b);
+        assert_eq!(fields(&differences), ["body.location_constraint"]);
+        assert_eq!((differences[0].a.as_str(), differences[0].b.as_str()), ("us-east-1", "eu-west-1"));
+    }
+
+    #[test]
+    fn a_body_field_on_one_side_only_is_reported() {
+        let a = vec![ok("ListObjectsV2").with_field("keys", "[]")];
+        let b = vec![ok("ListObjectsV2")];
+        assert_eq!(fields(&compare(&a, &b)), ["body.keys"]);
+    }
+
+    #[test]
+    fn error_messages_are_never_compared() {
+        let mut a = Observation::new("GetObject", "negative", 404, Some("NoSuchKey".into()), []);
+        let mut b = a.clone();
+        a.message = Some("The specified key does not exist.".into());
+        b.message = Some("Object not found".into());
+        assert_eq!(compare(&[a], &[b]), Vec::new());
+    }
+
+    #[test]
+    fn a_case_observed_on_one_side_only_is_reported() {
+        let a = vec![ok("HeadBucket"), ok("ListBuckets")];
+        let b = vec![ok("HeadBucket")];
+        let differences = compare(&a, &b);
+        assert_eq!(fields(&differences), ["observation"]);
+        assert_eq!(differences[0].op, "ListBuckets");
+        assert_eq!(compared_ops(&a, &b).into_iter().collect::<Vec<_>>(), ["HeadBucket"]);
+    }
+
+    #[test]
+    fn a_case_recorded_twice_is_reported() {
+        let a = vec![ok("HeadBucket"), ok("HeadBucket")];
+        let b = vec![ok("HeadBucket")];
+        assert_eq!(fields(&compare(&a, &b)), ["duplicate_case"]);
+    }
+
+    #[test]
+    fn compared_ops_counts_operations_not_cases() {
+        let negative = Observation::new("HeadBucket", "negative", 404, Some("NotFound".into()), []);
+        let a = vec![ok("HeadBucket"), negative, ok("ListBuckets")];
+        let b = a.clone();
+        assert_eq!(compared_ops(&a, &b).len(), 2);
+    }
+
+    #[test]
+    fn the_report_lists_each_difference_as_op_field_a_b() {
+        let directory = std::env::temp_dir().join(format!("dual-stack-report-{}", uuid::Uuid::new_v4()));
+        let report = Report {
+            schema: 1,
+            run: "unit".into(),
+            stack_a: "legacy".into(),
+            stack_b: "legacy".into(),
+            region_a: None,
+            region_b: Some("eu-west-1".into()),
+            compared_ops: 1,
+            compared_cases: 1,
+            differences: compare(
+                &[ok("GetBucketLocation").with_field("location_constraint", "us-east-1")],
+                &[ok("GetBucketLocation").with_field("location_constraint", "eu-west-1")],
+            ),
+        };
+        let path = report.write_to(&directory).expect("write the report");
+        let json: serde_json::Value = serde_json::from_slice(&std::fs::read(&path).expect("read the report")).expect("parse");
+        let _ = std::fs::remove_dir_all(&directory);
+        assert_eq!(path.file_name().and_then(|name| name.to_str()), Some("unit.json"));
+        assert_eq!(json["compared_ops"], 1);
+        assert_eq!(
+            json["differences"],
+            serde_json::json!([{
+                "op": "GetBucketLocation",
+                "case": "normal",
+                "field": "body.location_constraint",
+                "a": "us-east-1",
+                "b": "eu-west-1",
+            }])
+        );
+    }
+}

crates/e2e_test/src/dual_stack_differential/mod.rs

@@ -0,0 +1,197 @@
+// Copyright 2024 RustFS Team
+//
+// Licensed under the Apache License, Version 2.0 (the "License");
+// you may not use this file except in compliance with the License.
+// You may obtain a copy of the License at
+//
+//     http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing, software
+// distributed under the License is distributed on an "AS IS" BASIS,
+// WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+// See the License for the specific language governing permissions and
+// limitations under the License.
+
+//! Dual-stack differential: one `rustfs` binary, two processes, one SDK script (rustfs/backlog#2740).
+//!
+//! Stack A always runs `RUSTFS_S3_STACK=legacy`. Stack B runs the value of `RUSTFS_E2E_STACK_B`
+//! (default `legacy` until the gateway stack is assembled), and `RUSTFS_E2E_STACK_B_REGION`, when
+//! set, starts B in another region so that a difference can be injected by hand. Both servers
+//! answer the script in `scenarios`, which covers every operation of `impl S3 for FS`; `compare`
+//! reports each `(op, field, a, b)` on which the answers differ, and every run writes its report
+//! to `<target>/dual-stack/<run>.json`. Two processes, not two embedded servers: bucket metadata
+//! is process-global, so two stacks in one process would read each other's buckets.
+
+mod compare;
+mod scenarios;
+
+use crate::common::{RustFSTestEnvironment, init_logging, workspace_root};
+use compare::{Difference, Observation, Report, compare, compared_cases, compared_ops, render};
+use std::path::PathBuf;
+use std::time::{SystemTime, UNIX_EPOCH};
+
+type BoxError = Box<dyn std::error::Error + Send + Sync>;
+
+const STACK_A: &str = "legacy";
+const ENV_STACK_B: &str = "RUSTFS_E2E_STACK_B";
+const ENV_STACK_B_REGION: &str = "RUSTFS_E2E_STACK_B_REGION";
+/// Becomes `gateway` once rustfs/backlog#2734 T2.5 assembles the gateway stack.
+const DEFAULT_STACK_B: &str = "legacy";
+/// The region the negative control gives stack A explicitly, so an inherited `RUSTFS_REGION`
+/// cannot erase the injected difference.
+const CONTROL_REGION_A: &str = "us-east-1";
+const CONTROL_REGION_B: &str = "eu-west-1";
+/// `normal` cases the legacy stack must answer with a 2xx. The comparison alone cannot tell a
+/// script that exercises real behavior from one in which both stacks fail the same way. Five of
+/// the 73 fail on legacy by design of this script: PutBucketReplication (no registered remote
+/// target), GetBucketReplication (so no configuration), PutBucketNotificationConfiguration (no
+/// notification target configured), RestoreObject (the object is not archived) and
+/// GetObjectTorrent (legacy answers `NoSuchKey`).
+const MIN_NORMAL_SUCCESSES: usize = 68;
+
+struct PairConfig {
+    stack_b: String,
+    region_a: Option<String>,
+    region_b: Option<String>,
+}
+
+impl PairConfig {
+    fn from_env() -> Self {
+        Self {
+            stack_b: std::env::var(ENV_STACK_B).unwrap_or_else(|_| DEFAULT_STACK_B.to_owned()),
+            region_a: None,
+            region_b: std::env::var(ENV_STACK_B_REGION).ok(),
+        }
+    }
+}
+
+/// Starts one server on `stack`, overriding its region only when asked.
+async fn start_stack(stack: &str, region: Option<&str>) -> Result<RustFSTestEnvironment, BoxError> {
+    let mut env = RustFSTestEnvironment::new().await?;
+    let mut vars = vec![("RUSTFS_S3_STACK", stack)];
+    if let Some(region) = region {
+        vars.push(("RUSTFS_REGION", region));
+    }
+    env.start_rustfs_server_without_cleanup_with_env(&vars).await?;
+    Ok(env)
+}
+
+/// Starts stack A and stack B side by side, each with its own port and data directory.
+async fn spawn_pair(config: &PairConfig) -> Result<(RustFSTestEnvironment, RustFSTestEnvironment), BoxError> {
+    tokio::try_join!(
+        start_stack(STACK_A, config.region_a.as_deref()),
+        start_stack(&config.stack_b, config.region_b.as_deref())
+    )
+}
+
+fn report_directory() -> PathBuf {
+    let target = std::env::var_os("CARGO_TARGET_DIR").map_or_else(|| PathBuf::from("target"), PathBuf::from);
+    let target = if target.is_absolute() {
+        target
+    } else {
+        workspace_root().join(target)
+    };
+    target.join("dual-stack")
+}
+
+struct Run {
+    report: Report,
+    path: PathBuf,
+    a: Vec<Observation>,
+}
+
+/// Runs the script against both stacks concurrently and writes the report.
+async fn run_differential(label: &str, config: &PairConfig) -> Run {
+    let (server_a, server_b) = spawn_pair(config).await.expect("start stack A and stack B");
+    let (a, b) = tokio::join!(Box::pin(scenarios::run(&server_a)), Box::pin(scenarios::run(&server_b)));
+    let millis = SystemTime::now()
+        .duration_since(UNIX_EPOCH)
+        .map_or(0, |elapsed| elapsed.as_millis());
+    let report = Report {
+        schema: 1,
+        run: format!("{label}-{millis}-{}", std::process::id()),
+        stack_a: STACK_A.to_owned(),
+        stack_b: config.stack_b.clone(),
+        region_a: config.region_a.clone(),
+        region_b: config.region_b.clone(),
+        compared_ops: compared_ops(&a, &b).len(),
+        compared_cases: compared_cases(&a, &b),
+        differences: compare(&a, &b),
+    };
+    let path = report.write_to(&report_directory()).expect("write the dual-stack report");
+    Run { report, path, a }
+}
+
+#[cfg(test)]
+mod tests {
+    use super::*;
+
+    #[tokio::test]
+    async fn both_stacks_answer_every_operation_identically() {
+        init_logging();
+        let config = PairConfig::from_env();
+        let run = run_differential("self-comparison", &config).await;
+        let report = &run.report;
+
+        let mut observed: Vec<_> = run.a.iter().map(|observation| observation.op).collect();
+        observed.sort_unstable();
+        observed.dedup();
+        assert_eq!(
+            observed,
+            scenarios::OPERATIONS,
+            "the script must drive exactly the operations of impl S3 for FS"
+        );
+        assert_eq!(report.compared_ops, scenarios::OPERATIONS.len(), "report: {}", run.path.display());
+        assert!(
+            report.differences.is_empty(),
+            "stack {} and stack {} differ on {} of {} cases (report: {}):\n{}",
+            report.stack_a,
+            report.stack_b,
+            report.differences.len(),
+            report.compared_cases,
+            run.path.display(),
+            render(&report.differences)
+        );
+
+        let not_4xx: Vec<_> = run
+            .a
+            .iter()
+            .filter(|observation| observation.case == "negative" && !(400..500).contains(&observation.status))
+            .collect();
+        assert!(not_4xx.is_empty(), "negative cases must fail with a 4xx on {STACK_A}: {not_4xx:#?}");
+        let successes = run
+            .a
+            .iter()
+            .filter(|observation| observation.case == "normal" && observation.is_success())
+            .count();
+        assert!(
+            successes >= MIN_NORMAL_SUCCESSES,
+            "only {successes} normal cases succeeded on {STACK_A}: {:#?}",
+            run.a
+        );
+    }
+
+    #[tokio::test]
+    async fn differential_detects_an_injected_difference() {
+        init_logging();
+        let config = PairConfig {
+            stack_b: STACK_A.to_owned(),
+            region_a: Some(CONTROL_REGION_A.to_owned()),
+            region_b: Some(CONTROL_REGION_B.to_owned()),
+        };
+        let run = run_differential("injected-region", &config).await;
+        let expected = Difference {
+            op: "GetBucketLocation".to_owned(),
+            case: "normal".to_owned(),
+            field: "body.location_constraint".to_owned(),
+            a: CONTROL_REGION_A.to_owned(),
+            b: CONTROL_REGION_B.to_owned(),
+        };
+        assert!(
+            run.report.differences.contains(&expected),
+            "B started in {CONTROL_REGION_B} but GetBucketLocation was not reported (report: {}):\n{}",
+            run.path.display(),
+            render(&run.report.differences)
+        );
+    }
+}

crates/e2e_test/src/heal_erasure_disk_rebuild_test.rs

@@ -1191,59 +1191,11 @@ mod tests {
         .into())
     }
 
-    fn scanner_cycle_completion_budget(
-        cycle_end: u64,
-        interval: Option<u64>,
-        now: u64,
-        active: bool,
-    ) -> Result<Duration, &'static str> {
-        if cycle_end > now {
-            return Err("scanner cycle timestamp is in the future");
-        }
-        if active {
-            return Ok(Duration::from_secs(60));
-        }
-        let interval = interval
-            .filter(|interval| *interval != 0)
-            .ok_or("scanner schedule has no positive interval")?;
-        let next_cycle = cycle_end
-            .checked_add(interval)
-            .ok_or("scanner schedule timestamp overflows")?;
-        let budget = next_cycle
-            .saturating_sub(now)
-            .checked_add(60)
-            .ok_or("scanner completion budget overflows")?;
-        Ok(Duration::from_secs(budget))
-    }
-
-    #[test]
-    fn scanner_cycle_completion_budget_accounts_for_captured_retry_schedule() {
-        assert_eq!(
-            scanner_cycle_completion_budget(1_791_815_390, Some(256), 1_791_815_447, false),
-            Ok(Duration::from_secs(259)),
-        );
-        assert_eq!(scanner_cycle_completion_budget(100, Some(1), 110, false), Ok(Duration::from_secs(60)));
-    }
-
-    #[test]
-    fn scanner_cycle_completion_budget_does_not_grant_planned_sleep_to_active_cycle() {
-        assert_eq!(scanner_cycle_completion_budget(100, Some(256), 110, true), Ok(Duration::from_secs(60)));
-        assert_eq!(scanner_cycle_completion_budget(0, None, 110, true), Ok(Duration::from_secs(60)));
-    }
-
-    #[test]
-    fn scanner_cycle_completion_budget_rejects_invalid_schedule() {
-        for (end, interval, now) in [(101, 1, 100), (100, 0, 100), (u64::MAX, 1, u64::MAX), (1, u64::MAX - 1, 1)] {
-            assert!(scanner_cycle_completion_budget(end, Some(interval), now, false).is_err());
-        }
-    }
-
     async fn wait_for_scanner_cycle_after(
         cluster: &RustFSTestClusterEnvironment,
         previous_cycle_end: u64,
     ) -> Result<u64, Box<dyn Error + Send + Sync>> {
-        let mut deadline = Instant::now() + Duration::from_secs(60);
-        let mut schedule_accounted = false;
+        let deadline = Instant::now() + Duration::from_secs(60);
         loop {
             let mut latest_cycle_end = 0;
             let mut versions_observed = false;
@@ -1285,36 +1237,14 @@ mod tests {
                 let versions_scanned = metrics["versions_scanned"]
                     .as_u64()
                     .ok_or("scanner status is missing its version-coverage counter")?;
-                let now = std::time::SystemTime::now().duration_since(std::time::UNIX_EPOCH)?.as_secs();
-                if cycle_end > now {
-                    return Err("scanner cycle timestamp is in the future".into());
-                }
-                let schedule = &status["cycle_schedule"];
-                if !schedule_accounted && schedule["execution_role"].as_str() == Some("leader") {
-                    let active = metrics["current_cycle_active"]
-                        .as_bool()
-                        .ok_or("scanner status is missing its active-cycle flag")?;
-                    if active || (cycle_end != 0 && schedule["effective_interval_available"].as_bool() == Some(true)) {
-                        let interval = schedule["effective_interval_seconds"].as_u64();
-                        let budget = scanner_cycle_completion_budget(cycle_end, interval, now, active)?;
-                        deadline = Instant::now().checked_add(budget).ok_or("scanner deadline overflows")?;
-                        schedule_accounted = true;
-                    }
-                }
-                // Superseded walks complete all sets while withholding quota authority.
-                // Deferred and partial attempts do not certify completed scanning.
-                if matches!(metrics["last_cycle_result"].as_str(), Some("success" | "superseded")) {
-                    latest_cycle_end = latest_cycle_end.max(cycle_end);
-                }
+                latest_cycle_end = latest_cycle_end.max(cycle_end);
                 versions_observed |= versions_scanned > 0;
                 observations.push(format!(
-                    "node{node_index}: end={cycle_end}, versions={versions_scanned}, cycle={}, active={}, leader={}, result={}, partial_reason={}, schedule={}",
+                    "node{node_index}: end={cycle_end}, versions={versions_scanned}, cycle={}, active={}, leader={}, result={}",
                     metrics["current_cycle"],
                     metrics["current_cycle_active"],
                     metrics["leader_lock_state"],
                     metrics["last_cycle_result"],
-                    metrics["last_cycle_partial_reason"],
-                    schedule,
                 ));
             }
             // The coordinator records cycle completion, but remote workers
@@ -1625,8 +1555,7 @@ mod tests {
         }
 
         if background_enabled {
-            let baseline_cycle_floor = std::time::SystemTime::now().duration_since(std::time::UNIX_EPOCH)?.as_secs();
-            wait_for_scanner_cycle_after(&cluster, baseline_cycle_floor).await?;
+            wait_for_scanner_cycle_after(&cluster, 0).await?;
         }
 
         cluster.stop_node(1)?;
@@ -1969,16 +1898,11 @@ mod tests {
             let status_body = signed_admin_post(&status_url, None, &cluster.access_key, &cluster.secret_key).await?;
             let status: serde_json::Value = serde_json::from_str(&status_body)
                 .map_err(|err| format!("pre-interrupt background heal status is not JSON ({err}): {status_body}"))?;
-            let admin_active = background_rejoin_heal_evidence
-                || (status["state"].as_str() == Some("active")
-                    && status["healOperations"]["activeBySource"]["admin"].as_u64() == Some(1));
-            if admin_active && (background_enabled || only_admin_heal_is_active(&status)) {
+            if background_enabled || only_admin_heal_is_active(&status) {
                 break status;
             }
             if Instant::now() >= partial_deadline {
-                return Err(
-                    format!("heal tasks did not reach the required interruption state before the deadline: {status}").into(),
-                );
+                return Err(format!("non-admin heal did not drain before the interruption deadline: {status}").into());
             }
             sleep(Duration::from_millis(50)).await;
         };
@@ -2638,157 +2562,4 @@ mod tests {
         };
         assert!(!outage_candidate_targets_replacement(&insufficient_peer_shards, 12, replacement_set_slot));
     }
-    async fn two_node_heal_status(
-        cluster: &RustFSTestClusterEnvironment,
-        bucket: Option<&str>,
-        recreate: bool,
-        scan_mode: u8,
-    ) -> Result<serde_json::Value, Box<dyn Error + Send + Sync>> {
-        let url = format!("{}/rustfs/admin/v3/heal/{}", cluster.nodes[0].url, bucket.unwrap_or_default());
-        let options = serde_json::json!({
-            "recursive": true, "dryRun": false, "remove": false,
-            "recreate": recreate, "scanMode": scan_mode, "updateParity": false, "nolock": false,
-        });
-        let started =
-            start_root_heal_when_control_ready(&url, &options.to_string(), &cluster.access_key, &cluster.secret_key).await?;
-        let started: serde_json::Value = serde_json::from_str(&started)?;
-        let token = started["clientToken"]
-            .as_str()
-            .ok_or_else(|| format!("missing heal token: {started}"))?;
-        let status_url = format!("{url}?clientToken={token}");
-        let deadline = Instant::now() + Duration::from_secs(240);
-        loop {
-            let body = timeout(
-                Duration::from_secs(20),
-                signed_admin_post(&status_url, None, &cluster.access_key, &cluster.secret_key),
-            )
-            .await??;
-            let status: serde_json::Value = serde_json::from_str(&body)?;
-            match status["summary"].as_str() {
-                Some("finished" | "stopped") => return Ok(status),
-                Some("running") if Instant::now() < deadline => sleep(Duration::from_millis(250)).await,
-                _ => return Err(format!("heal did not reach a terminal state: {status}").into()),
-            }
-        }
-    }
-
-    async fn verify_two_node_directory_recovery(
-        remove_bucket: bool,
-        scan_mode: u8,
-        bucket_scoped: bool,
-        recreate: bool,
-    ) -> Result<(), Box<dyn Error + Send + Sync>> {
-        init_logging();
-        let mut cluster = RustFSTestClusterEnvironment::new(2).await?;
-        // Isolate explicit administrator recovery from scanner, MRF, and disk healing.
-        for key in [
-            "RUSTFS_HEAL_AUTO_HEAL_ENABLE",
-            "RUSTFS_HEAL_MRF_ENABLE",
-            "RUSTFS_SCANNER_ENABLED",
-            "RUSTFS_PUT_RENAME_EARLY_ACK_ENABLE",
-        ] {
-            cluster.set_env(key, "false");
-        }
-        cluster.start().await?;
-        let source = cluster.create_s3_client(0)?;
-        let target = cluster.create_s3_client(1)?;
-        let bucket = "two-node-directory-recovery";
-        source.create_bucket().bucket(bucket).send().await?;
-        let objects = [
-            ("empty", Vec::new()),
-            ("nested/inline", vec![0x3c; 1024]),
-            ("nested/external", (0_u8..=255).cycle().take(1024 * 1024).collect::<Vec<_>>()),
-        ];
-        for (key, bytes) in &objects {
-            source
-                .put_object()
-                .bucket(bucket)
-                .key(*key)
-                .body(ByteStream::from(bytes.clone()))
-                .send()
-                .await?;
-        }
-        let target_bucket = Path::new(&cluster.nodes[1].data_dir).join(bucket);
-        let original = objects
-            .iter()
-            .map(|(key, _)| census_object_version_on_disk(Path::new(&cluster.nodes[1].data_dir), bucket, key, None))
-            .collect::<Result<Vec<_>, _>>()?;
-        for census in &original {
-            assert!(census.has_xl_meta, "fixture must have a target metadata copy: {census:?}");
-        }
-        assert!(
-            !original[2].present_part_fingerprints.is_empty(),
-            "large fixture must exercise external parts"
-        );
-        let removed = Path::new(&cluster.temp_dir).join("removed");
-        tokio::fs::create_dir(&removed).await?;
-        if remove_bucket {
-            tokio::fs::rename(&target_bucket, removed.join(bucket)).await?;
-        } else {
-            for (index, (key, _)) in objects.iter().enumerate() {
-                tokio::fs::rename(target_bucket.join(key), removed.join(index.to_string())).await?;
-            }
-        }
-        let status = two_node_heal_status(&cluster, bucket_scoped.then_some(bucket), recreate, scan_mode).await?;
-        assert_eq!(status["summary"], "finished", "administrator recovery should complete: {status}");
-        assert_eq!(status["outcome"]["execution"]["state"], "completed");
-        assert_eq!(status["outcome"]["coverage"], "complete");
-        for ((key, _), before) in objects.iter().zip(&original) {
-            let after = census_object_version_on_disk(Path::new(&cluster.nodes[1].data_dir), bucket, key, None)?;
-            assert!(after.has_xl_meta, "metadata must be restored before GET: {key}");
-            assert_eq!(
-                after.present_part_fingerprints, before.present_part_fingerprints,
-                "external shard bytes must match: {key}"
-            );
-            assert_eq!(
-                after.inline_data_fingerprint, before.inline_data_fingerprint,
-                "inline shard bytes must match: {key}"
-            );
-        }
-        cluster.stop_node(0)?;
-        for (key, expected) in &objects {
-            let actual = timeout(Duration::from_secs(30), async {
-                target
-                    .get_object()
-                    .bucket(bucket)
-                    .key(*key)
-                    .send()
-                    .await?
-                    .body
-                    .collect()
-                    .await
-                    .map(|body| body.into_bytes())
-                    .map_err(Box::<dyn Error + Send + Sync>::from)
-            })
-            .await??;
-            assert_eq!(actual.len(), expected.len(), "target-only read length: {key}");
-            assert_eq!(sha256_hex(&actual), sha256_hex(expected), "target-only content: {key}");
-        }
-        Ok(())
-    }
-
-    #[tokio::test]
-    async fn test_two_node_bucket_discovery_and_explicit_recreate() -> Result<(), Box<dyn Error + Send + Sync>> {
-        verify_two_node_directory_recovery(true, 2, true, true).await
-    }
-
-    #[tokio::test]
-    async fn test_two_node_missing_bucket_default_cluster_recovery() -> Result<(), Box<dyn Error + Send + Sync>> {
-        verify_two_node_directory_recovery(true, 2, false, false).await
-    }
-
-    #[tokio::test]
-    async fn test_two_node_missing_bucket_default_scoped_recovery() -> Result<(), Box<dyn Error + Send + Sync>> {
-        verify_two_node_directory_recovery(true, 2, true, false).await
-    }
-
-    #[tokio::test]
-    async fn test_two_node_object_directory_recovery_normal() -> Result<(), Box<dyn Error + Send + Sync>> {
-        verify_two_node_directory_recovery(false, 1, false, false).await
-    }
-
-    #[tokio::test]
-    async fn test_two_node_object_directory_recovery_deep() -> Result<(), Box<dyn Error + Send + Sync>> {
-        verify_two_node_directory_recovery(false, 2, false, false).await
-    }
 }

crates/e2e_test/src/internode_rpc_signature_e2e_test.rs

@@ -264,7 +264,7 @@ async fn call_make_volume_response(
     request: MakeVolumeRequest,
     headers: HeaderMap,
 ) -> Result<Response<MakeVolumeResponse>, Status> {
-    let mut client = node_service_time_out_client_no_auth(url)
+    let mut client = node_service_time_out_client_no_auth(&url.to_string())
         .await
         .map_err(|err| Status::unavailable(format!("cannot reach the node service: {err}")))?;
     let mut rpc_request = Request::new(request);
@@ -273,7 +273,7 @@ async fn call_make_volume_response(
 }
 
 async fn call_ping_response(url: &str, headers: HeaderMap) -> Result<Response<PingResponse>, Status> {
-    let mut client = node_service_time_out_client_no_auth(url)
+    let mut client = node_service_time_out_client_no_auth(&url.to_string())
         .await
         .map_err(|err| Status::unavailable(format!("cannot reach the node service: {err}")))?;
     let mut rpc_request = Request::new(PingRequest {

crates/e2e_test/src/lib.rs

@@ -31,6 +31,12 @@ pub mod chaos;
 #[cfg(test)]
 pub mod fake_s3_target;
 
+// Dual-stack differential (rustfs/backlog#2740): one binary, two processes on
+// different RUSTFS_S3_STACK values, the same SDK script for every S3 operation,
+// compared on status, error code, allowlisted headers and typed body fields.
+#[cfg(test)]
+mod dual_stack_differential;
+
 // On-demand migration (backlog#2147): shared two-server environment, admin
 // wrappers, and the harness self-test (backlog#2151). Behavior scenarios are
 // added by later ODM tasks.

crates/e2e_test/src/namespace_lock_quorum_test.rs

@@ -12,7 +12,7 @@
 // See the License for the specific language governing permissions and
 // limitations under the License.
 
-use crate::common::{ClusterTopology, RustFSTestClusterEnvironment};
+use crate::common::RustFSTestClusterEnvironment;
 use aws_sdk_s3::Client;
 use aws_sdk_s3::error::SdkError;
 use bytes::Bytes;
@@ -185,113 +185,21 @@ async fn wait_for_quorum_read_admission(clients: &[Client], bucket: &str) -> Tes
             return Ok(());
         }
         if tokio::time::Instant::now() >= deadline {
-            let diagnostics = clients[0].get_object().bucket(bucket).key("warm-small").send().await;
-            return Err(format!(
-                "read quorum did not become available after lease convergence for {bucket}: GET={diagnostics:?}"
-            )
-            .into());
+            return Err(format!("read quorum did not become available after lease convergence for {bucket}").into());
         }
         tokio::time::sleep(std::time::Duration::from_millis(100)).await;
     }
 }
 
-#[tokio::test]
-async fn test_cold_degraded_object_mutations_require_only_storage_quorum() -> TestResult {
-    crate::common::init_logging();
-    let mut cluster = RustFSTestClusterEnvironment::new(4).await?;
-    cluster.set_env("RUSTFS_STORAGE_CLASS_STANDARD", "EC:2");
-    cluster.set_env("RUSTFS_PUT_RENAME_EARLY_ACK_ENABLE", "false");
-    cluster.set_env("RUSTFS_OBS_METRICS_EXPORT_ENABLED", "false");
-    cluster.set_env("RUSTFS_GET_OBJECT_METADATA_CACHE_DISTRIBUTED_ENABLE", "false");
-    cluster.start().await?;
-    let clients = cluster
-        .create_all_clients()?
-        .into_iter()
-        .map(|client| {
-            Client::from_conf(
-                client
-                    .config()
-                    .to_builder()
-                    .retry_config(aws_sdk_s3::config::retry::RetryConfig::standard().with_max_attempts(1))
-                    .build(),
-            )
-        })
-        .collect::<Vec<_>>();
-    let bucket = "cold-degraded-mutations";
-    clients[0].create_bucket().bucket(bucket).send().await?;
-    // No object mutation may warm a process-wide configuration probe before
-    // the missing peer is introduced. Three shards still satisfy EC:2 writes.
-    cluster.stop_node(3)?;
-    let admission_deadline = Instant::now() + Duration::from_secs(30);
-    for client in &clients[..3] {
-        loop {
-            match client
-                .get_object()
-                .bucket(bucket)
-                .key("never-created-readiness-probe")
-                .send()
-                .await
-            {
-                Err(error) if error.as_service_error().and_then(|error| error.meta().code()) == Some("NoSuchKey") => break,
-                Err(error)
-                    if error.raw_response().is_some_and(|response| response.status().as_u16() == 503)
-                        && Instant::now() < admission_deadline =>
-                {
-                    tokio::time::sleep(Duration::from_millis(100)).await;
-                }
-                Err(error) => return Err(error.into()),
-                Ok(_) => panic!("a cold readiness probe must not create an object"),
-            }
-        }
-    }
-    let body = b"first write while one peer is offline";
-    clients[0]
-        .put_object()
-        .bucket(bucket)
-        .key("object")
-        .body(Bytes::from_static(body).into())
-        .send()
-        .await?;
-    for client in &clients[..3] {
-        assert_quorum_object_body(client, bucket, "object", body).await?;
-    }
-    // This coordinator has not previously committed an object mutation either.
-    clients[1].delete_object().bucket(bucket).key("object").send().await?;
-    let error = clients[2]
-        .get_object()
-        .bucket(bucket)
-        .key("object")
-        .send()
-        .await
-        .expect_err("acknowledged deletion must be visible at another entrypoint");
-    assert_eq!(error.as_service_error().and_then(|error| error.meta().code()), Some("NoSuchKey"));
-
-    cluster.stop_node(2)?;
-    let error = clients[0]
-        .put_object()
-        .bucket(bucket)
-        .key("below-quorum")
-        .body(Bytes::from_static(body).into())
-        .send()
-        .await
-        .expect_err("removing the cache gate must not weaken storage write quorum");
-    assert_eq!(error.as_service_error().and_then(|error| error.meta().code()), Some("ServiceUnavailable"));
-    cluster.stop();
-    Ok(())
-}
-
 #[tokio::test]
 async fn test_degraded_cluster_read_quorum_follows_erasure_layout() -> TestResult {
     crate::common::init_logging();
 
-    for (node_count, parity) in [(3, 1), (4, 2), (6, 3), (6, 2), (4, 0)] {
+    for (node_count, parity) in [(4, 2), (6, 3), (6, 2)] {
         let read_quorum = node_count - parity;
         let write_quorum = read_quorum + usize::from(read_quorum == parity);
         let mut cluster = RustFSTestClusterEnvironment::new(node_count).await?;
         cluster.set_env("RUSTFS_STORAGE_CLASS_STANDARD", format!("EC:{parity}"));
-        if parity == 0 {
-            cluster.set_env("RUSTFS_STORAGE_CLASS_RRS", "EC:0");
-        }
         // Wait for every seed fanout before removing any physical shard.
         cluster.set_env("RUSTFS_PUT_RENAME_EARLY_ACK_ENABLE", "false");
         cluster.set_env("RUSTFS_OBS_METRICS_EXPORT_ENABLED", "false");
@@ -430,277 +338,6 @@ async fn test_degraded_cluster_read_quorum_follows_erasure_layout() -> TestResul
     Ok(())
 }
 
-#[tokio::test]
-async fn test_cold_start_read_quorum_follows_erasure_layout() -> TestResult {
-    crate::common::init_logging();
-    for (node_count, parity) in [(3, 1), (4, 2), (6, 3), (6, 2), (4, 0)] {
-        let read_quorum = node_count - parity;
-        let write_quorum = read_quorum + usize::from(read_quorum == parity);
-        let mut cluster = RustFSTestClusterEnvironment::new(node_count).await?;
-        cluster.set_env("RUSTFS_STORAGE_CLASS_STANDARD", format!("EC:{parity}"));
-        if parity == 0 {
-            cluster.set_env("RUSTFS_STORAGE_CLASS_RRS", "EC:0");
-        }
-        cluster.set_env("RUSTFS_PUT_RENAME_EARLY_ACK_ENABLE", "false");
-        cluster.set_env("RUSTFS_OBS_METRICS_EXPORT_ENABLED", "false");
-        cluster.set_env("RUST_LOG", "warn");
-        cluster.start().await?;
-        let clients = cluster
-            .create_all_clients()?
-            .into_iter()
-            .map(|client| {
-                Client::from_conf(
-                    client
-                        .config()
-                        .to_builder()
-                        .retry_config(aws_sdk_s3::config::retry::RetryConfig::standard().with_max_attempts(1))
-                        .build(),
-                )
-            })
-            .collect::<Vec<_>>();
-        let bucket = format!("cold-read-quorum-{node_count}-{parity}");
-        clients[0].create_bucket().bucket(&bucket).send().await?;
-        let small = b"cold startup follows the stored EC layout".to_vec();
-        let large = vec![73; 256 * 1024];
-        for (key, body) in [
-            ("warm-small", &small),
-            ("warm-large", &large),
-            ("cold-large", &large),
-            ("below-quorum", &large),
-        ] {
-            clients[0]
-                .put_object()
-                .bucket(&bucket)
-                .key(key)
-                .body(Bytes::copy_from_slice(body).into())
-                .send()
-                .await?;
-            for node in &cluster.nodes {
-                let census =
-                    crate::chaos::census_object_version_on_disk(std::path::Path::new(&node.data_dir), &bucket, key, None)?;
-                assert!(census.is_complete(), "seed shard must exist before cold startup: {census:?}");
-                assert_eq!((census.data_blocks, census.parity_blocks), (Some(read_quorum), Some(parity)));
-            }
-        }
-        for node in 0..node_count {
-            cluster.stop_node(node)?;
-        }
-        // Rebuild every process's in-memory state below read quorum, then
-        // cross the EC boundary. A format bootstrap must not admit S3 reads
-        // merely because half the peers are online (e.g. six disks, EC:2).
-        for node in 0..read_quorum - 1 {
-            cluster.start_node_process(node).await?;
-        }
-        let error = clients[0]
-            .get_object()
-            .bucket(&bucket)
-            .key("below-quorum")
-            .send()
-            .await
-            .expect_err("cold startup below the configured EC read quorum must stay unavailable");
-        assert_eq!(error.raw_response().map(|response| response.status().as_u16()), Some(503));
-        cluster.start_node_process(read_quorum - 1).await?;
-        wait_for_quorum_read_admission(&clients[..read_quorum], &bucket).await?;
-        for client in clients.iter().take(read_quorum) {
-            assert_quorum_object_body(client, &bucket, "warm-small", &small).await?;
-            assert_quorum_object_body(client, &bucket, "cold-large", &large).await?;
-            let write = client
-                .put_object()
-                .bucket(&bucket)
-                .key("cold-quorum-write")
-                .body(Bytes::copy_from_slice(&small).into())
-                .send()
-                .await;
-            if read_quorum >= write_quorum {
-                write?;
-                assert_quorum_object_body(client, &bucket, "cold-quorum-write", &small).await?;
-            } else {
-                let error = write.expect_err("cold startup at read quorum must preserve the EC write quorum");
-                assert_eq!(error.raw_response().map(|response| response.status().as_u16()), Some(503));
-            }
-        }
-        for node in read_quorum..node_count {
-            cluster.start_node(node).await?;
-        }
-        for client in &clients {
-            assert_quorum_object_body(client, &bucket, "cold-large", &large).await?;
-        }
-    }
-    Ok(())
-}
-
-#[tokio::test]
-async fn test_four_by_six_cold_start_at_read_quorum() -> TestResult {
-    crate::common::init_logging();
-    let mut cluster = RustFSTestClusterEnvironment::with_topology(ClusterTopology::single_pool_multidrive(4, 6)).await?;
-    // Explicit URLs are consumed in order. Place one drive from each node in
-    // every set, matching a four-host, six-drive-per-host deployment.
-    let volumes = (0..6)
-        .flat_map(|drive| {
-            cluster
-                .nodes
-                .iter()
-                .map(move |node| format!("http://{}{}", node.address, node.data_dirs[drive]))
-        })
-        .collect::<Vec<_>>()
-        .join(" ");
-    cluster.set_env("RUSTFS_VOLUMES", volumes);
-    cluster.set_env("RUSTFS_ERASURE_SET_DRIVE_COUNT", "4");
-    cluster.set_env("RUSTFS_STORAGE_CLASS_STANDARD", "EC:2");
-    cluster.set_env("RUSTFS_PUT_RENAME_EARLY_ACK_ENABLE", "false");
-    cluster.set_env("RUSTFS_OBS_METRICS_EXPORT_ENABLED", "false");
-    cluster.set_env("RUST_LOG", "error");
-    cluster.start().await?;
-    let clients = cluster
-        .create_all_clients()?
-        .into_iter()
-        .map(|client| {
-            Client::from_conf(
-                client
-                    .config()
-                    .to_builder()
-                    .retry_config(aws_sdk_s3::config::retry::RetryConfig::standard().with_max_attempts(1))
-                    .build(),
-            )
-        })
-        .collect::<Vec<_>>();
-    let bucket = "cold-start-read-quorum";
-    clients[0].create_bucket().bucket(bucket).send().await?;
-    let mut fixtures = Vec::new();
-    let mut covered = [[false; 2]; 6];
-    for index in 0..128 {
-        let key = match index {
-            0 => "warm-small".to_string(),
-            1 => "warm-large".to_string(),
-            _ => format!("cold-{index}"),
-        };
-        let size_class = index % 2;
-        let size = if size_class == 0 { 71 } else { 256 * 1024 };
-        let body = (0..size)
-            .map(|offset| u8::try_from(offset % 251).expect("bounded fixture byte"))
-            .collect::<Vec<_>>();
-        clients[0]
-            .put_object()
-            .bucket(bucket)
-            .key(&key)
-            .body(Bytes::copy_from_slice(&body).into())
-            .send()
-            .await?;
-        for node in &cluster.nodes {
-            let mut complete_shards = 0;
-            for (drive, path) in node.data_dirs.iter().enumerate() {
-                let census = crate::chaos::census_object_version_on_disk(std::path::Path::new(path), bucket, &key, None)?;
-                if census.has_xl_meta {
-                    assert!(census.is_complete(), "seed shard must be complete before shutdown: {census:?}");
-                    assert_eq!((census.data_blocks, census.parity_blocks), (Some(2), Some(2)));
-                    covered[drive][size_class] = true;
-                    complete_shards += 1;
-                }
-            }
-            assert_eq!(complete_shards, 1, "each node must own one shard of every seed object");
-        }
-        fixtures.push((key, body));
-        if covered.iter().flatten().all(|covered| *covered) {
-            break;
-        }
-    }
-    assert!(
-        covered.iter().flatten().all(|covered| *covered),
-        "fixtures must cover all six sets with inline and large objects"
-    );
-
-    for first in 0..4 {
-        for second in first + 1..4 {
-            for node in 0..4 {
-                cluster.stop_node(node)?;
-            }
-            cluster.start_node_process(first).await?;
-            let error = clients[first]
-                .get_object()
-                .bucket(bucket)
-                .key("warm-large")
-                .send()
-                .await
-                .expect_err("one node cannot satisfy read quorum after a cold start");
-            assert_eq!(error.raw_response().map(|response| response.status().as_u16()), Some(503));
-            cluster.start_node_process(second).await?;
-            wait_for_quorum_read_admission(&[clients[first].clone(), clients[second].clone()], bucket).await?;
-            for node in [first, second] {
-                for (key, body) in &fixtures {
-                    assert_quorum_object_body(&clients[node], bucket, key, body).await?;
-                }
-                let range = clients[node]
-                    .get_object()
-                    .bucket(bucket)
-                    .key("warm-large")
-                    .range("bytes=17-1023")
-                    .send()
-                    .await?;
-                assert_eq!(range.body.collect().await?.into_bytes().as_ref(), &fixtures[1].1[17..1024]);
-                let listing = clients[node].list_objects_v2().bucket(bucket).send().await?;
-                for (key, _) in &fixtures {
-                    assert!(
-                        listing.contents().iter().any(|object| object.key() == Some(key.as_str())),
-                        "listing omitted {key}"
-                    );
-                }
-                let error = clients[node]
-                    .put_object()
-                    .bucket(bucket)
-                    .key("recovery-write")
-                    .body(Bytes::from_static(b"after recovery").into())
-                    .send()
-                    .await
-                    .expect_err("two nodes must not authorize a write");
-                assert_eq!(error.raw_response().map(|response| response.status().as_u16()), Some(503));
-            }
-            let error = clients[first]
-                .delete_object()
-                .bucket(bucket)
-                .key("warm-large")
-                .send()
-                .await
-                .expect_err("two nodes must not authorize a delete");
-            assert_eq!(error.raw_response().map(|response| response.status().as_u16()), Some(503));
-            assert_quorum_object_body(&clients[first], bucket, "warm-large", &fixtures[1].1).await?;
-
-            let remaining = (0..4).filter(|node| *node != first && *node != second).collect::<Vec<_>>();
-            cluster.start_node(remaining[0]).await?;
-            let deadline = tokio::time::Instant::now() + Duration::from_secs(90);
-            loop {
-                match clients[first]
-                    .put_object()
-                    .bucket(bucket)
-                    .key("recovery-write")
-                    .body(Bytes::from_static(b"after recovery").into())
-                    .send()
-                    .await
-                {
-                    Ok(_) => break,
-                    Err(error)
-                        if error.raw_response().is_some_and(|response| response.status().as_u16() == 503)
-                            && tokio::time::Instant::now() < deadline =>
-                    {
-                        tokio::time::sleep(Duration::from_millis(200)).await
-                    }
-                    Err(error) => return Err(error.into()),
-                }
-            }
-            for node in [first, second, remaining[0]] {
-                assert_quorum_object_body(&clients[node], bucket, "recovery-write", b"after recovery").await?;
-            }
-            cluster.start_node(remaining[1]).await?;
-            for client in &clients {
-                for (key, body) in &fixtures {
-                    assert_quorum_object_body(client, bucket, key, body).await?;
-                }
-                assert_quorum_object_body(client, bucket, "recovery-write", b"after recovery").await?;
-            }
-        }
-    }
-    Ok(())
-}
-
 async fn put_object(client: Client, payload: Vec<u8>, writer_id: usize) -> Result<(), String> {
     client
         .put_object()

crates/e2e_test/src/reliant/grpc_lock_server.rs

@@ -465,13 +465,6 @@ impl NodeService for MinimalLockNodeService {
         Err(Status::unimplemented("lock-only test server"))
     }
 
-    async fn recover_delete_transaction(
-        &self,
-        _request: Request<rustfs_protos::proto_gen::node_service::RecoverDeleteTransactionRequest>,
-    ) -> Result<Response<rustfs_protos::proto_gen::node_service::RecoverDeleteTransactionResponse>, Status> {
-        Err(Status::unimplemented("delete recovery is not supported by the lock fixture"))
-    }
-
     async fn settle_part_transaction(
         &self,
         _request: Request<rustfs_protos::proto_gen::node_service::SettlePartTransactionRequest>,

crates/e2e_test/src/replication_extension_test.rs

@@ -3945,15 +3945,7 @@ async fn test_bucket_replication_version_purge_of_non_inline_object_releases_sou
         (sse_key, sse_put.version_id().ok_or("SSE PUT omitted version ID")?.to_string()),
         (large_key, large_put.version_id().ok_or("large PUT omitted version ID")?.to_string()),
     ];
-    // Managed-SSE listings may omit ETags. Check the replicated version's body
-    // before deleting it rather than comparing keyless listing fingerprints.
-    for (key, version_id) in &purged {
-        let source_body = get_version_body(&source_client, &source_bucket, key, version_id).await?;
-        let expected_sha256: [u8; 32] = Sha256::digest(&source_body).into();
-        wait_for_replicated_sha256(&target_client, &target_bucket, key, expected_sha256).await?;
-        let target_body = get_version_body(&target_client, &target_bucket, key, version_id).await?;
-        assert_eq!(target_body, source_body, "replicated version body differs for {key}/{version_id}");
-    }
+    assert_replication_converged(&source_client, &source_bucket, &target_client, &target_bucket).await?;
 
     for (key, version_id) in &purged {
         source_client
@@ -3964,6 +3956,10 @@ async fn test_bucket_replication_version_purge_of_non_inline_object_releases_sou
             .send()
             .await?;
     }
+    assert_replication_converged(&source_client, &source_bucket, &target_client, &target_bucket).await?;
+    let target_state = list_replication_state(&target_client, &target_bucket).await?;
+    assert!(target_state.is_empty(), "target retained an explicitly purged version: {target_state:?}");
+
     // The purge state is applied on the source asynchronously after the target
     // acknowledges the delete; only then does the retained version go away and
     // the bucket become deletable. A listing that is empty while DeleteBucket
@@ -3988,9 +3984,6 @@ async fn test_bucket_replication_version_purge_of_non_inline_object_releases_sou
         }
     }
 
-    let target_state = list_replication_state(&target_client, &target_bucket).await?;
-    assert!(target_state.is_empty(), "target retained an explicitly purged version: {target_state:?}");
-
     Ok(())
 }
 

crates/e2e_test/src/sts_query_compat_test.rs

@@ -740,7 +740,7 @@ async fn test_bucket_tags_opa_contract() -> TestResult {
         .version_id(version)
         .send()
         .await?;
-    opa.expect_bucket_tags("s3:GetObjectVersion", source, Some("finance")).await?;
+    opa.expect_bucket_tags("s3:GetObject", source, Some("finance")).await?;
     client.list_objects_v2().bucket(source).send().await?;
     opa.expect_bucket_tags("s3:ListBucket", source, Some("finance")).await?;
     client.list_object_versions().bucket(source).send().await?;

crates/e2e_test/src/upgrade_compatibility_test.rs

@@ -578,7 +578,7 @@ async fn exercise_mixed_cluster(
     previous_node: usize,
 ) -> TestResult {
     let clients = cluster.create_all_clients()?;
-    wait_for_upgrade_write_readiness(&clients, MIXED_BUCKET, phase, LISTING_CONVERGENCE_TIMEOUT).await?;
+    wait_for_upgrade_write_readiness(&clients, phase, LISTING_CONVERGENCE_TIMEOUT).await?;
     let current_client = &clients[current_node];
     let previous_client = &clients[previous_node];
 
@@ -646,12 +646,7 @@ fn upgrade_probe_client(client: &Client) -> Client {
     )
 }
 
-pub(crate) async fn wait_for_upgrade_write_readiness(
-    clients: &[Client],
-    bucket: &str,
-    phase: &str,
-    budget: Duration,
-) -> TestResult {
+async fn wait_for_upgrade_write_readiness(clients: &[Client], phase: &str, budget: Duration) -> TestResult {
     // ListBuckets can succeed before peers recover a restarted disk. Cluster
     // health also accepts Returning disks whose write health is still FAULTY.
     // Probe every writer outside the asserted phase prefix; compatibility
@@ -671,7 +666,7 @@ pub(crate) async fn wait_for_upgrade_write_readiness(
                 deadline,
                 client
                     .put_object()
-                    .bucket(bucket)
+                    .bucket(MIXED_BUCKET)
                     .key(&key)
                     .body(ByteStream::from_static(UPGRADE_READINESS_BODY))
                     .send(),
@@ -706,20 +701,19 @@ async fn prepare_previous_release_baseline(cluster: &mut RustFSTestClusterEnviro
     // waits for pool metadata (#7473). First prove the elected writer can
     // persist data, then restart each old process once with three peers still
     // readable. This preparation ends before any current binary is started.
-    wait_for_upgrade_write_readiness(&clients[..1], MIXED_BUCKET, "previous-seed", LISTING_CONVERGENCE_TIMEOUT).await?;
+    wait_for_upgrade_write_readiness(&clients[..1], "previous-seed", LISTING_CONVERGENCE_TIMEOUT).await?;
     for node in [1, 2, 3, 0] {
         cluster.stop_node(node)?;
         cluster.start_node_from_binary(node, previous_binary).await?;
         wait_for_upgrade_write_readiness(
             std::slice::from_ref(&clients[node]),
-            MIXED_BUCKET,
             &format!("previous-restart-{node}"),
             LISTING_CONVERGENCE_TIMEOUT,
         )
         .await?;
     }
 
-    wait_for_upgrade_write_readiness(&clients, MIXED_BUCKET, "previous-baseline", LISTING_CONVERGENCE_TIMEOUT).await?;
+    wait_for_upgrade_write_readiness(&clients, "previous-baseline", LISTING_CONVERGENCE_TIMEOUT).await?;
     for (reader, client) in clients.iter().enumerate() {
         assert_eq!(
             read_object(client, MIXED_BUCKET, PREVIOUS_RELEASE_SEED_KEY, None).await?.1,
@@ -746,10 +740,9 @@ mod upgrade_write_readiness_tests {
     #[tokio::test]
     async fn waits_for_each_writer_after_metadata_is_ready() -> TestResult {
         let target = FakeS3Target::start().await?;
-        let bucket = "distributed-upgrade-history";
-        target.create_bucket(bucket);
+        target.create_bucket(MIXED_BUCKET);
         let client = fake_source_client(&target);
-        client.head_bucket().bucket(bucket).send().await?;
+        client.head_bucket().bucket(MIXED_BUCKET).send().await?;
 
         let phase = "one-previous-node";
         let first_key = format!(".upgrade-readiness/{phase}/node-0");
@@ -769,7 +762,7 @@ mod upgrade_write_readiness_tests {
         // Metadata readiness does not prove that a data write can succeed.
         let premature = client
             .put_object()
-            .bucket(bucket)
+            .bucket(MIXED_BUCKET)
             .key(&first_key)
             .body(ByteStream::from_static(b"upgrade write readiness"))
             .send()
@@ -777,15 +770,15 @@ mod upgrade_write_readiness_tests {
             .expect_err("metadata readiness does not prove write readiness");
         assert_eq!(premature.raw_response().map(|response| response.status().as_u16()), Some(503));
 
-        wait_for_upgrade_write_readiness(&[client.clone(), client.clone()], bucket, phase, Duration::from_secs(5)).await?;
+        wait_for_upgrade_write_readiness(&[client.clone(), client.clone()], phase, Duration::from_secs(5)).await?;
         assert_eq!(target.count_requests(FakeTargetOperation::PutObject, &first_key), 3);
         assert_eq!(target.count_requests(FakeTargetOperation::PutObject, &second_key), 2);
-        assert!(target.has_object(bucket, &first_key));
-        assert!(target.has_object(bucket, &second_key));
+        assert!(target.has_object(MIXED_BUCKET, &first_key));
+        assert!(target.has_object(MIXED_BUCKET, &second_key));
         assert!(
             client
                 .list_objects_v2()
-                .bucket(bucket)
+                .bucket(MIXED_BUCKET)
                 .prefix(format!("{phase}/"))
                 .send()
                 .await?
@@ -815,10 +808,9 @@ mod upgrade_write_readiness_tests {
             let phase = format!("permanent-{}", status.as_u16());
             let key = format!(".upgrade-readiness/{phase}/node-0");
             target.inject_for_key(FakeTargetOperation::PutObject, &key, FaultAction::Status(status), 1);
-            let error =
-                wait_for_upgrade_write_readiness(std::slice::from_ref(&client), MIXED_BUCKET, &phase, Duration::from_secs(5))
-                    .await
-                    .expect_err("a permanent error must not be retried into success");
+            let error = wait_for_upgrade_write_readiness(std::slice::from_ref(&client), &phase, Duration::from_secs(5))
+                .await
+                .expect_err("a permanent error must not be retried into success");
             assert!(error.to_string().contains("node 0 write readiness failed"), "{error}");
             assert_eq!(target.count_requests(FakeTargetOperation::PutObject, &key), 1);
             assert!(!target.has_object(MIXED_BUCKET, &key));
@@ -841,7 +833,7 @@ mod upgrade_write_readiness_tests {
             FaultAction::Status(StatusCode::SERVICE_UNAVAILABLE),
             10,
         );
-        let error = wait_for_upgrade_write_readiness(&[client], MIXED_BUCKET, phase, Duration::from_secs(1))
+        let error = wait_for_upgrade_write_readiness(&[client], phase, Duration::from_secs(1))
             .await
             .expect_err("persistent unavailability must exhaust the shared deadline");
         // Transport scheduling consumes the same budget; do not require a
@@ -860,7 +852,7 @@ mod upgrade_write_readiness_tests {
         let phase = "stalled";
         let key = format!(".upgrade-readiness/{phase}/node-0");
         target.inject_for_key(FakeTargetOperation::PutObject, &key, FaultAction::Stall(Duration::from_secs(30)), 1);
-        let error = wait_for_upgrade_write_readiness(std::slice::from_ref(&client), MIXED_BUCKET, phase, Duration::from_secs(1))
+        let error = wait_for_upgrade_write_readiness(std::slice::from_ref(&client), phase, Duration::from_secs(1))
             .await
             .expect_err("a stalled request must not outlive the readiness deadline");
         assert!(error.to_string().contains("deadline exceeded during PutObject"), "{error}");
@@ -2345,7 +2337,7 @@ fn layout_reported_parts(attributes: &aws_sdk_s3::operation::get_object_attribut
     attributes.object_parts().map(|parts| parts.parts().len())
 }
 
-async fn assert_layout_full_body(client: &Client, case: &LayoutCase, context: &str) -> TestResult {
+async fn assert_layout_readable(client: &Client, case: &LayoutCase, context: &str) -> TestResult {
     let label = case.label();
     let head = layout_head(client, case).await?;
     assert_eq!(
@@ -2362,12 +2354,7 @@ async fn assert_layout_full_body(client: &Client, case: &LayoutCase, context: &s
     let full = layout_get(client, case, None, None).await?.body.collect().await?.into_bytes();
     assert_eq!(full.len(), case.body.len(), "{context}: {label}: full GET length");
     assert!(full == case.body, "{context}: {label}: full GET body must equal the rc.5 upload");
-    Ok(())
-}
 
-async fn assert_layout_readable(client: &Client, case: &LayoutCase, context: &str) -> TestResult {
-    assert_layout_full_body(client, case, context).await?;
-    let label = case.label();
     if case.is_multipart_layout() {
         let first = case.part_sizes[0];
         let range = format!("bytes={}-{}", first - 32, first + 31);
@@ -2646,11 +2633,10 @@ async fn direct_upgrade_from_rc5_preserves_multipart_layouts() -> TestResult {
             "rc.5 wrote a legacy layout"
         );
     }
-    // Prove the rc.5 writer persisted every original byte before upgrading.
-    // Its compressed cross-part range reader has a legacy limitation; the
-    // upgraded reader must still satisfy every range and partNumber assertion.
+    // The rc.5 writer must itself still read what it wrote, so a later
+    // failure is attributable to the upgrade rather than to the fixture.
     for case in &cases {
-        assert_layout_full_body(&old_client, case, "rc.5").await?;
+        assert_layout_readable(&old_client, case, "rc.5").await?;
     }
 
     // Upgrade in place.

crates/ecstore/Cargo.toml

@@ -183,7 +183,6 @@ sha2 = { workspace = true }
 hex-simd = { workspace = true }
 tempfile.workspace = true
 hyper = { workspace = true, features = ["http2", "http1", "server"] }
-h2 = { workspace = true }
 hyper-util = { workspace = true, features = ["tokio", "server-auto", "server-graceful", "tracing"] }
 hyper-rustls = { workspace = true, default-features = false, features = ["native-tokio", "http1", "tls12", "logging", "http2", "aws-lc-rs"] }
 hostname.workspace = true
@@ -201,8 +200,8 @@ md-5.workspace = true
 memmap2 = { workspace = true }
 libc.workspace = true
 # "process" adds getrlimit for the io_uring fd-cache RLIMIT_NOFILE gate
-# (backlog#1178); "thread" provides cleanup affinity; "fs" is a workspace default.
-rustix = { workspace = true, features = ["process", "fs", "thread"] }
+# (backlog#1178); "fs" comes from the workspace default.
+rustix = { workspace = true, features = ["process", "fs"] }
 rustfs-madmin.workspace = true
 reqwest = { workspace = true }
 aws-sdk-s3 = { workspace = true, default-features = false, features = ["sigv4a", "default-https-client", "rt-tokio"] }

crates/ecstore/src/api/mod.rs

@@ -199,10 +199,10 @@ pub mod bucket {
             get_object_lock_config, get_object_lock_config_state, get_on_demand_migration_config,
             get_on_demand_migration_config_in, get_public_access_block_config, get_quota_config, get_replication_config,
             get_request_payment_config, get_sse_config, get_tagging_config, get_versioning_config, get_website_config,
-            init_bucket_metadata_sys, list_bucket_targets, reload_bucket_metadata, remove_bucket_metadata,
-            remove_bucket_metadata_if_absent, set_bucket_metadata, update, update_bucket_targets_under_transaction_lock,
-            update_config_with, update_if_incarnation, update_if_incarnation_at, update_quota_if_incarnation,
-            update_quota_if_incarnation_at, update_under_transaction_lock, update_under_transaction_lock_at,
+            init_bucket_metadata_sys, list_bucket_targets, reload_bucket_metadata, remove_bucket_metadata, set_bucket_metadata,
+            update, update_bucket_targets_under_transaction_lock, update_config_with, update_if_incarnation,
+            update_if_incarnation_at, update_quota_if_incarnation, update_quota_if_incarnation_at, update_under_transaction_lock,
+            update_under_transaction_lock_at,
         };
         #[cfg(feature = "test-util")]
         pub use crate::bucket::metadata_sys::{ConfigWriteLockProbe, test_support};
@@ -210,8 +210,7 @@ pub mod bucket {
 
     pub mod migration {
         pub use crate::bucket::migration::{
-            LegacyBlobDecryptFn, migration_startup_error, try_migrate_bucket_metadata, try_migrate_bucket_metadata_for_buckets,
-            try_migrate_iam_config,
+            LegacyBlobDecryptFn, migration_startup_error, try_migrate_bucket_metadata, try_migrate_iam_config,
         };
     }
 
@@ -241,10 +240,7 @@ pub mod bucket {
     }
 
     pub mod quota {
-        pub use crate::bucket::quota::{
-            BucketQuota, QUOTA_RESERVATION_PROTOCOL_V1, QUOTA_RESERVATION_PROTOCOL_V2, QuotaCheckResult, QuotaError,
-            QuotaOperation,
-        };
+        pub use crate::bucket::quota::{BucketQuota, QuotaCheckResult, QuotaError, QuotaOperation};
 
         pub mod checker {
             pub use crate::bucket::quota::checker::QuotaChecker;
@@ -366,11 +362,10 @@ pub mod config {
             ServerConfigCorruptError, ServerConfigSaveResult, ServerConfigSnapshot, delete_config, delete_config_no_lock,
             is_server_config_corrupt_error, lookup_configs, read_config, read_config_no_lock, read_config_with_metadata,
             read_config_without_migrate, read_config_without_migrate_no_lock, read_existing_server_config_no_lock,
-            read_server_config_snapshot, save_config, save_config_no_lock, save_config_with_opts, save_config_with_opts_quiet,
-            save_server_config, save_server_config_no_lock, save_server_config_snapshot,
-            save_server_config_snapshot_with_generation, server_config_path, try_migrate_server_config,
-            with_config_object_read_lock, with_config_object_write_lock, with_server_config_read_lock,
-            with_server_config_write_lock,
+            read_server_config_snapshot, save_config, save_config_no_lock, save_config_with_opts, save_server_config,
+            save_server_config_no_lock, save_server_config_snapshot, save_server_config_snapshot_with_generation,
+            server_config_path, try_migrate_server_config, with_config_object_read_lock, with_config_object_write_lock,
+            with_server_config_read_lock, with_server_config_write_lock,
         };
     }
 
@@ -423,9 +418,7 @@ pub mod disk {
     pub use crate::disk::os::{LocalPublicationPause, LocalPublicationStage};
     pub use crate::disk::{
         BATCH_READ_VERSION_MAX_ITEMS, BUCKET_META_PREFIX, BatchReadVersionItem, BatchReadVersionReq, BatchReadVersionResp,
-        CheckPartsResp, ConditionalFileUpdate, DELETE_RECOVERY_PROTOCOL_VERSION, DeleteOptions, DeleteRecoveryAction,
-        DeleteRecoveryCursor, DeleteRecoveryDigest, DeleteRecoveryInspection, DeleteRecoveryPage, DeleteRecoveryRpcOperation,
-        DeleteRecoveryRpcReply, DeleteRecoveryState, Disk, DiskAPI, DiskInfo, DiskInfoOptions, DiskLocation, DiskOption,
+        CheckPartsResp, ConditionalFileUpdate, DeleteOptions, Disk, DiskAPI, DiskInfo, DiskInfoOptions, DiskLocation, DiskOption,
         DiskStore, FileInfoVersions, FileReader, FileWriter, HEALING_MARKER_PATH, NsScannerOpenRequest, OldCurrentSize,
         PartTransactionAction, RUSTFS_META_BUCKET, ReadMultipleReq, ReadMultipleResp, ReadOptions, RenameDataResp,
         STORAGE_FORMAT_FILE, SnapshotLeaseToken, UpdateMetadataOpts, VolumeInfo, WalkDirOptions, new_disk,
@@ -475,7 +468,7 @@ pub mod event {
 pub mod global {
     pub use crate::runtime::global::{
         set_global_endpoints, set_global_region, set_global_rustfs_port, set_object_store_resolver, shutdown_background_services,
-        update_erasure_type, wait_for_detached_mutations,
+        update_erasure_type,
     };
 }
 
@@ -522,14 +515,13 @@ pub mod integrity {
 pub mod object {
     pub use crate::object_api::{
         BLOCK_SIZE_V2, ERASURE_ALGORITHM, EncryptionResolutionError, EncryptionResolutionErrorKind, GetObjectBodyCacheHook,
-        GetObjectBodyCacheHookLookup, GetObjectBodySource, GetObjectReadCondition, GetObjectReadMetadata, GetObjectReader,
-        NamespaceLockFence, ObjectEncryptionResolver, ObjectInfo, ObjectLockConfigSnapshot, ObjectMutationHook, ObjectOptions,
-        PutObjReader, QuotaAdmission, RangedDecompressReader, ReadEncryptionMaterial, ReadEncryptionMode, ReadEncryptionRequest,
+        GetObjectBodyCacheHookLookup, GetObjectBodySource, GetObjectReader, NamespaceLockFence, ObjectEncryptionResolver,
+        ObjectInfo, ObjectLockConfigSnapshot, ObjectMutationHook, ObjectOptions, PutObjReader, QuotaAdmission,
+        RangedDecompressReader, ReadEncryptionMaterial, ReadEncryptionMode, ReadEncryptionRequest,
         SCANNER_PUBLICATION_LEASE_FENCE_METADATA_KEY, ScannerPublicationCommitScope, ScannerPublicationCommitStartError,
         ScannerPublicationCommitState, ShardIntegrityWriteMode, StreamConsumer, WriteCommitGuard, WriteCompletion,
         get_object_body_cache_plaintext_len, lookup_get_object_body_cache_hook, register_get_object_body_cache_hook,
         register_object_mutation_hook, unregister_get_object_body_cache_hook, unregister_object_mutation_hook,
-        with_get_object_read_condition,
     };
     pub use crate::store::{
         PrepareSelectObjectSnapshotError, PreparedGetObjectReader, SelectObjectSnapshot, SelectObjectSnapshotReadError,
@@ -594,7 +586,6 @@ pub mod set_disk {
     #[cfg(feature = "test-util")]
     pub mod test_util {
         pub use crate::bucket::quota::reservation::fail_next_quota_ledger_save_for_test;
-        pub use crate::set_disk::ConditionalReadBenchmarkMetadataGuard;
         pub use crate::set_disk::{MultipartCommitBarrier, MultipartCommitPause, PutObjectCommitBarrier, PutObjectCommitPause};
 
         /// Keep a namespace commit pending until the returned owner is dropped.

crates/ecstore/src/bucket/lifecycle/bucket_lifecycle_ops.rs

@@ -87,7 +87,6 @@ use rustfs_filemeta::{
     FileInfo, FileInfoOpts, NULL_VERSION_ID, RestoreStatus, RestoreStatusOps, TRANSITION_COMPLETE, get_file_info,
     is_restored_object_on_disk,
 };
-use rustfs_lock::{NamespaceLockGuard, error::LockError};
 use rustfs_scanner_metrics::metrics::{
     IlmAction, Metrics, ScannerLifecycleExpiryStateUpdate, ScannerLifecycleTransitionStateUpdate, global_metrics,
 };
@@ -191,8 +190,6 @@ const TIER_FREE_VERSION_RECOVERY_BASE_INTERVAL: StdDuration = StdDuration::from_
 // before another full sweep can discover work persisted by another node.
 const TIER_FREE_VERSION_RECOVERY_MAX_IDLE_INTERVAL: StdDuration = StdDuration::from_secs(10 * 60);
 const TIER_FREE_VERSION_RECOVERY_JITTER_PERCENT: u64 = 10;
-const TIER_FREE_VERSION_RECOVERY_LEADER_LOCK: &str = "tier-free-version-recovery/leader.lock";
-const TIER_FREE_VERSION_RECOVERY_LOCK_TIMEOUT: StdDuration = StdDuration::from_secs(1);
 const DATE_EXPIRY_EXISTING_OBJECTS_GRACE_SECS: i64 = 5;
 const EXPIRY_WORKER_QUEUE_CAPACITY: usize = 1000;
 const DEFAULT_MANUAL_TRANSITION_JOB_RECOVERY_LIMIT: usize = 100;
@@ -1444,24 +1441,13 @@ pub(crate) async fn enqueue_committed_free_versions(api: &ECStore, free_versions
     queued
 }
 
-async fn enqueue_recovered_free_version_with_state(
-    state: &Arc<RwLock<ExpiryState>>,
-    oi: ObjectInfo,
-    cancel: &CancellationToken,
-) -> bool {
+async fn enqueue_recovered_free_version_with_state(state: &Arc<RwLock<ExpiryState>>, oi: ObjectInfo) -> bool {
     let task = FreeVersionTask(oi);
     let hash = task.op_hash();
     let (wrkr, stats) = {
-        let state = select! {
-            biased;
-            _ = cancel.cancelled() => return false,
-            state = state.read() => state,
-        };
+        let state = state.read().await;
         (state.get_worker_ch(hash), Arc::clone(&state.stats))
     };
-    if cancel.is_cancelled() {
-        return false;
-    }
     let Some(wrkr) = wrkr else {
         stats.increment_missed_freevers_tasks();
         stats.record_scanner_expiry_state();
@@ -1511,13 +1497,9 @@ fn set_recovered_free_version_enqueue_observer(
     RecoveredFreeVersionEnqueueObserverGuard
 }
 
-pub(super) async fn enqueue_recovered_free_version_with_cancel(
-    api: &ECStore,
-    oi: ObjectInfo,
-    cancel: &CancellationToken,
-) -> bool {
+pub async fn enqueue_recovered_free_version(api: &ECStore, oi: ObjectInfo) -> bool {
     let expiry_state = api.ctx.expiry_state();
-    let queued = enqueue_recovered_free_version_with_state(&expiry_state, oi, cancel).await;
+    let queued = enqueue_recovered_free_version_with_state(&expiry_state, oi).await;
 
     #[cfg(test)]
     if let Some(observer) = RECOVERED_FREE_VERSION_ENQUEUE_OBSERVER
@@ -2417,16 +2399,6 @@ fn is_slow_down(err: &Error) -> bool {
     matches!(err, Error::SlowDown)
 }
 
-/// Lifecycle expiry is idempotent when the exact object or version has already
-/// disappeared between the scanner listing and the delete admission.
-///
-/// Keep this predicate narrower than bucket/volume "not found" helpers: an
-/// unavailable bucket or disk is an operational failure and must not be
-/// reported as a completed expiry.
-fn is_lifecycle_expiry_already_gone(err: &Error) -> bool {
-    is_err_object_not_found(err) || is_err_version_not_found(err)
-}
-
 /// Resolves the expiry worker count from the single documented knob,
 /// `RUSTFS_MAX_EXPIRY_WORKERS`: a set, parsable, non-zero value wins;
 /// anything else falls back to `min(cpus, 16)`. The historical
@@ -2949,20 +2921,6 @@ fn resolve_tier_free_version_recovery_enabled(value: Result<String, env::VarErro
 }
 
 fn spawn_tier_free_version_recovery_once(api: Arc<ECStore>, started: &OnceLock<()>) -> Option<JoinHandle<()>> {
-    spawn_tier_free_version_recovery_once_with_options(
-        api,
-        started,
-        TIER_FREE_VERSION_RECOVERY_LEADER_LOCK.to_string(),
-        tier_free_version_recovery_leader_retry_delay,
-    )
-}
-
-fn spawn_tier_free_version_recovery_once_with_options(
-    api: Arc<ECStore>,
-    started: &OnceLock<()>,
-    leader_lock_name: String,
-    retry_delay: fn() -> StdDuration,
-) -> Option<JoinHandle<()>> {
     if !tier_free_version_recovery_enabled() {
         warn!(
             event = EVENT_LIFECYCLE_WORKER_STATE,
@@ -2979,43 +2937,20 @@ fn spawn_tier_free_version_recovery_once_with_options(
 
     Some(tokio::spawn(async move {
         let cancel_token = api.ctx.background_cancel_token().unwrap_or_default();
-        let acquire_api = Arc::clone(&api);
-        let acquire_lock_name = Arc::<str>::from(leader_lock_name);
-        run_tier_free_version_recovery_coordinator(
+        let expiry_state = api.ctx.expiry_state();
+        run_tier_free_version_recovery_loop(
             cancel_token,
-            retry_delay,
-            move || {
-                let api = Arc::clone(&acquire_api);
-                let lock_name = Arc::clone(&acquire_lock_name);
-                async move {
-                    let lock = api.new_ns_lock(RUSTFS_META_BUCKET, &lock_name).await?;
-                    match lock.get_write_lock_quiet(TIER_FREE_VERSION_RECOVERY_LOCK_TIMEOUT).await {
-                        Ok(guard) => Ok(Some(guard)),
-                        Err(LockError::Timeout { .. }) => Ok(None),
-                        Err(err) => Err(err.into()),
-                    }
-                }
-            },
-            move |leader_cancel| {
+            expiry_state,
+            jitter_tier_free_version_recovery_delay,
+            move |bucket_marker, object_marker, recovery_cancel| {
                 let api = Arc::clone(&api);
                 async move {
-                    run_tier_free_version_recovery_loop(
-                        leader_cancel,
-                        api.ctx.expiry_state(),
-                        jitter_tier_free_version_recovery_delay,
-                        move |bucket_marker, object_marker, recovery_cancel| {
-                            let api = Arc::clone(&api);
-                            async move {
-                                recover_tier_free_versions_with_cancel(
-                                    api,
-                                    DEFAULT_FREE_VERSION_RECOVERY_LIMIT,
-                                    bucket_marker,
-                                    object_marker,
-                                    recovery_cancel,
-                                )
-                                .await
-                            }
-                        },
+                    recover_tier_free_versions_with_cancel(
+                        api,
+                        DEFAULT_FREE_VERSION_RECOVERY_LIMIT,
+                        bucket_marker,
+                        object_marker,
+                        recovery_cancel,
                     )
                     .await
                 }
@@ -3025,86 +2960,6 @@ fn spawn_tier_free_version_recovery_once_with_options(
     }))
 }
 
-fn tier_free_version_recovery_leader_retry_delay() -> StdDuration {
-    StdDuration::from_secs(rand::rng().random_range(15..=30))
-}
-
-async fn run_tier_free_version_recovery_coordinator<A, AF, R, RF>(
-    cancel: CancellationToken,
-    retry_delay: fn() -> StdDuration,
-    mut acquire: A,
-    mut recover: R,
-) where
-    A: FnMut() -> AF,
-    AF: Future<Output = crate::error::Result<Option<NamespaceLockGuard>>>,
-    R: FnMut(CancellationToken) -> RF,
-    RF: Future<Output = ()>,
-{
-    loop {
-        let acquired = select! {
-            biased;
-            _ = cancel.cancelled() => return,
-            result = acquire() => result,
-        };
-        match acquired {
-            Ok(Some(mut guard)) => {
-                let leader_cancel = cancel.child_token();
-                // Hold ownership across every page, cooldown and idle wait. A
-                // follower taking over starts a fresh sweep of persisted markers.
-                let recovery = recover(leader_cancel.clone());
-                tokio::pin!(recovery);
-                debug!(
-                    event = EVENT_LIFECYCLE_WORKER_STATE,
-                    component = LOG_COMPONENT_ECSTORE,
-                    subsystem = LOG_SUBSYSTEM_LIFECYCLE,
-                    state = "free_version_recovery_leader_acquired",
-                    lock_name = TIER_FREE_VERSION_RECOVERY_LEADER_LOCK,
-                    "Tier free-version recovery leader acquired"
-                );
-                select! {
-                    biased;
-                    _ = cancel.cancelled() => {
-                        leader_cancel.cancel();
-                        recovery.await;
-                    }
-                    _ = guard.lock_lost_notified() => {
-                        debug!(
-                            event = EVENT_LIFECYCLE_WORKER_STATE,
-                            component = LOG_COMPONENT_ECSTORE,
-                            subsystem = LOG_SUBSYSTEM_LIFECYCLE,
-                            state = "free_version_recovery_leader_lost",
-                            lock_name = TIER_FREE_VERSION_RECOVERY_LEADER_LOCK,
-                            "Tier free-version recovery leader lost"
-                        );
-                        // Dropping the recovery future would detach its active
-                        // walk. Cancel it and poll the same future through cleanup.
-                        leader_cancel.cancel();
-                        recovery.await;
-                    }
-                    _ = &mut recovery => {}
-                }
-                leader_cancel.cancel();
-                guard.release();
-            }
-            Ok(None) => {}
-            Err(err) => warn!(
-                event = EVENT_LIFECYCLE_WORKER_STATE,
-                component = LOG_COMPONENT_ECSTORE,
-                subsystem = LOG_SUBSYSTEM_LIFECYCLE,
-                state = "free_version_recovery_leader_acquire_failed",
-                lock_name = TIER_FREE_VERSION_RECOVERY_LEADER_LOCK,
-                error = %err,
-                "Tier free-version recovery leader acquisition failed"
-            ),
-        }
-        select! {
-            biased;
-            _ = cancel.cancelled() => return,
-            _ = tokio::time::sleep(retry_delay()) => {}
-        }
-    }
-}
-
 async fn run_tier_free_version_recovery_loop<F, Fut>(
     cancel_token: CancellationToken,
     expiry_state: Arc<RwLock<ExpiryState>>,
@@ -3114,11 +2969,7 @@ async fn run_tier_free_version_recovery_loop<F, Fut>(
     F: FnMut(Option<String>, Option<String>, CancellationToken) -> Fut,
     Fut: Future<Output = crate::error::Result<FreeVersionRecoveryStats>>,
 {
-    let recovery_notify = select! {
-        biased;
-        _ = cancel_token.cancelled() => return,
-        state = expiry_state.read() => Arc::clone(&state.recovery_notify),
-    };
+    let recovery_notify = Arc::clone(&expiry_state.read().await.recovery_notify);
     let mut schedule = TierFreeVersionRecoverySchedule::default();
 
     loop {
@@ -3141,11 +2992,7 @@ async fn run_tier_free_version_recovery_loop<F, Fut>(
                 schedule.record_success(&stats);
                 rustfs_io_metrics::record_stage_duration("lifecycle_free_version_recovery", elapsed.as_secs_f64() * 1000.0);
                 let (pending_tasks, active_tasks) = {
-                    let state = select! {
-                        biased;
-                        _ = cancel_token.cancelled() => return,
-                        state = expiry_state.read() => state,
-                    };
+                    let state = expiry_state.read().await;
                     (state.pending_tasks(), state.stats.active_tasks())
                 };
                 debug!(
@@ -4930,20 +4777,6 @@ async fn expire_transitioned_object_with_lock_lost_signal(
                 //audit_log_lifecycle(*oi, ILMExpiry, tags, traceFn);
                 Ok(dobj)
             }
-            Err(err) if is_lifecycle_expiry_already_gone(&err) => {
-                debug!(
-                    event = EVENT_LIFECYCLE_DELETE_COMPLETED,
-                    component = LOG_COMPONENT_ECSTORE,
-                    subsystem = LOG_SUBSYSTEM_LIFECYCLE,
-                    bucket = %oi.bucket,
-                    object = %oi.name,
-                    operation = "delete_restored_object",
-                    reason = "already_gone",
-                    "Lifecycle delete target was already absent"
-                );
-                crate::object_api::notify_object_mutation(&oi.bucket, &oi.name).await;
-                Ok(oi.clone())
-            }
             Err(err) => Err(std::io::Error::other(err)),
         };
     }
@@ -4954,8 +4787,6 @@ async fn expire_transitioned_object_with_lock_lost_signal(
     // pointing to a remote tier version that has already been removed. If this
     // only creates a delete marker, remote cleanup must be driven by persisted
     // free-version recovery rather than the visible delete result.
-    // Keep remote transitioned not-found errors visible: local metadata
-    // absence alone does not prove that the tier object was cleaned up.
     let dobj = match api.delete_object(&oi.bucket, &oi.name, opts).await {
         Ok(obj) => obj,
         Err(e) => {
@@ -5821,27 +5652,11 @@ async fn apply_expiry_on_non_transitioned_objects_with_lock_lost_signal(
     } else {
         None
     };
-    // The delete owns the publication guard from here on, so a cancelled
-    // expiry worker cannot let table-bucket publication overtake it.
     let mut dobj = match api
-        .delete_object_with_tier_delete_journal_and_guards(&oi.bucket, &encode_dir_object(&oi.name), opts, [publication_guard])
+        .delete_object_with_tier_delete_journal(&oi.bucket, &encode_dir_object(&oi.name), opts)
         .await
     {
         Ok(dobj) => dobj,
-        Err(err) if is_lifecycle_expiry_already_gone(&err) => {
-            debug!(
-                event = EVENT_LIFECYCLE_DELETE_COMPLETED,
-                component = LOG_COMPONENT_ECSTORE,
-                subsystem = LOG_SUBSYSTEM_LIFECYCLE,
-                bucket = %oi.bucket,
-                object = %oi.name,
-                operation = "delete_object",
-                reason = "already_gone",
-                "Lifecycle delete target was already absent"
-            );
-            crate::object_api::notify_object_mutation(&oi.bucket, &oi.name).await;
-            return true;
-        }
         Err(e) => {
             error!(
                 event = EVENT_LIFECYCLE_DELETE_FAILED,
@@ -6081,16 +5896,15 @@ mod tests {
         cleanup_empty_multipart_sha_dirs_on_local_disks, cleanup_stale_multipart_uploads_once_at,
         enqueue_recovered_free_version_with_state, enqueue_transition_for_existing_objects_scoped,
         enqueue_transition_with_lifecycle, enqueue_transition_with_lifecycle_report, eval_action_from_lifecycle,
-        get_lock_acquire_timeout, is_lifecycle_expiry_already_gone, jitter_tier_free_version_recovery_delay,
-        lifecycle_action_blocked_by_replication, lifecycle_delete_all_versions_replication_scan, lifecycle_deleted_object,
-        lifecycle_replication_blocks_action, lifecycle_rule_has_date_expiration, lifecycle_version_delete_target,
-        manual_transition_duration_elapsed, manual_transition_has_more_after_limit, manual_transition_recovery_progress_sink,
-        manual_transition_version_marker, manual_transition_worker_failure_reason,
-        mark_delete_opts_skip_decommissioned_on_remote_success, merge_stale_multipart_candidate,
-        persist_manual_transition_job_progress_if_owned, persist_manual_transition_page_checkpoint,
-        recover_manual_transition_job, recover_manual_transition_jobs, resolve_tier_free_version_recovery_enabled,
-        resolve_transition_queue_capacity, resolve_transition_queue_send_timeout, resolve_transition_worker_count,
-        resolve_transition_workers_absolute_max, run_tier_free_version_recovery_coordinator, run_tier_free_version_recovery_loop,
+        get_lock_acquire_timeout, jitter_tier_free_version_recovery_delay, lifecycle_action_blocked_by_replication,
+        lifecycle_delete_all_versions_replication_scan, lifecycle_deleted_object, lifecycle_replication_blocks_action,
+        lifecycle_rule_has_date_expiration, lifecycle_version_delete_target, manual_transition_duration_elapsed,
+        manual_transition_has_more_after_limit, manual_transition_recovery_progress_sink, manual_transition_version_marker,
+        manual_transition_worker_failure_reason, mark_delete_opts_skip_decommissioned_on_remote_success,
+        merge_stale_multipart_candidate, persist_manual_transition_job_progress_if_owned,
+        persist_manual_transition_page_checkpoint, recover_manual_transition_job, recover_manual_transition_jobs,
+        resolve_tier_free_version_recovery_enabled, resolve_transition_queue_capacity, resolve_transition_queue_send_timeout,
+        resolve_transition_worker_count, resolve_transition_workers_absolute_max, run_tier_free_version_recovery_loop,
         select_restore_s3_location, set_lifecycle_observability_observer, set_recovered_free_version_enqueue_observer,
         should_defer_date_expiry_for_recent_config_update, transitioned_cleanup_tuple, transitioned_object_delete_opts,
         wait_for_tier_free_version_recovery,
@@ -6187,231 +6001,6 @@ mod tests {
     use tokio_util::sync::CancellationToken;
     use uuid::Uuid;
 
-    fn recovery_test_lock() -> (Arc<rustfs_lock::NamespaceLock>, Vec<Arc<dyn rustfs_lock::LockClient>>) {
-        let clients: Vec<Arc<dyn rustfs_lock::LockClient>> = (0..3)
-            .map(|_| {
-                Arc::new(rustfs_lock::LocalClient::with_manager(Arc::new(rustfs_lock::GlobalLockManager::new())))
-                    as Arc<dyn rustfs_lock::LockClient>
-            })
-            .collect();
-        let lock =
-            rustfs_lock::NamespaceLock::with_clients_and_quorum(format!("recovery-test-{}", Uuid::new_v4()), clients.clone(), 2);
-        (Arc::new(lock), clients)
-    }
-
-    async fn acquire_recovery_test_lock(
-        lock: Arc<rustfs_lock::NamespaceLock>,
-        owner: String,
-    ) -> crate::error::Result<Option<rustfs_lock::NamespaceLockGuard>> {
-        // Keep this lease longer than the virtual-time cooldown/idle advances;
-        // production uses the namespace wrapper's renewable 30-second lease.
-        lock.lock_guard(
-            rustfs_lock::ObjectKey::new(crate::disk::RUSTFS_META_BUCKET, super::TIER_FREE_VERSION_RECOVERY_LEADER_LOCK),
-            &owner,
-            StdDuration::from_millis(5),
-            StdDuration::from_secs(3600),
-        )
-        .await
-        .map_err(Into::into)
-    }
-
-    fn recovery_test_retry_delay() -> StdDuration {
-        StdDuration::from_secs(15)
-    }
-
-    #[tokio::test(start_paused = true)]
-    #[serial]
-    async fn tier_free_version_recovery_eight_coordinators_hold_one_leader_and_take_over() {
-        let (lock, _clients) = recovery_test_lock();
-        let (pages_tx, mut pages_rx) = tokio::sync::mpsc::unbounded_channel();
-        let mut tasks = Vec::new();
-        let mut cancels = Vec::new();
-        for node in 0..8 {
-            let cancel = CancellationToken::new();
-            cancels.push(cancel.clone());
-            let lock = Arc::clone(&lock);
-            let pages_tx = pages_tx.clone();
-            tasks.push(tokio::spawn(async move {
-                run_tier_free_version_recovery_coordinator(
-                    cancel,
-                    recovery_test_retry_delay,
-                    move || acquire_recovery_test_lock(Arc::clone(&lock), format!("node-{node}")),
-                    move |leader_cancel| {
-                        let pages_tx = pages_tx.clone();
-                        async move {
-                            let mut pages = 0;
-                            run_tier_free_version_recovery_loop(
-                                leader_cancel,
-                                ExpiryState::new(),
-                                std::convert::identity,
-                                move |bucket, object, _| {
-                                    pages += 1;
-                                    pages_tx.send((node, bucket, object)).expect("page observer should be open");
-                                    std::future::ready(Ok(free_version_recovery_stats(0, 0, pages == 1)))
-                                },
-                            )
-                            .await;
-                        }
-                    },
-                )
-                .await;
-            }));
-        }
-        let (leader, bucket, object) = pages_rx.recv().await.expect("one coordinator should start recovery");
-        assert!(bucket.is_none() && object.is_none());
-        // The same leader must survive both a truncated page's cooldown and
-        // complete idle sweeps. All other coordinators keep probing the lock.
-        let mut observed_pages = 1;
-        for _ in 0..27 {
-            tokio::time::advance(StdDuration::from_secs(60)).await;
-            for _ in 0..100 {
-                tokio::task::yield_now().await;
-            }
-            while let Ok((node, bucket, object)) = pages_rx.try_recv() {
-                assert_eq!(node, leader, "followers must not repeat a namespace sweep");
-                if observed_pages == 1 {
-                    assert_eq!(bucket.as_deref(), Some("bucket"));
-                    assert_eq!(object.as_deref(), Some("object"));
-                } else {
-                    assert!(bucket.is_none() && object.is_none(), "complete sweeps should reset their cursors");
-                }
-                observed_pages += 1;
-            }
-        }
-        assert!(observed_pages >= 7, "the leader should finish a full ten-minute idle wait");
-        cancels[leader].cancel();
-        let (successor, bucket, object) = tokio::time::timeout(StdDuration::from_secs(60), pages_rx.recv())
-            .await
-            .expect("a follower should take over after leader shutdown")
-            .expect("page observer should remain open");
-        assert_ne!(successor, leader);
-        assert!(bucket.is_none() && object.is_none(), "a new leader must restart discovery safely");
-        for cancel in cancels {
-            cancel.cancel();
-        }
-        for task in tasks {
-            task.await.expect("all coordinators should shut down cleanly");
-        }
-    }
-
-    #[tokio::test(start_paused = true)]
-    #[serial]
-    async fn tier_free_version_recovery_lock_loss_waits_for_active_page_cleanup() {
-        let (lock, _clients) = recovery_test_lock();
-        let cancel = CancellationToken::new();
-        let loop_cancel = cancel.clone();
-        let (page_tx, mut page_rx) = tokio::sync::mpsc::unbounded_channel();
-        let cleanup_started = Arc::new(tokio::sync::Notify::new());
-        let finish_cleanup = Arc::new(tokio::sync::Notify::new());
-        let observed_cleanup = Arc::clone(&cleanup_started);
-        let release_cleanup = Arc::clone(&finish_cleanup);
-        let task = tokio::spawn(async move {
-            run_tier_free_version_recovery_coordinator(
-                loop_cancel,
-                recovery_test_retry_delay,
-                move || acquire_recovery_test_lock(Arc::clone(&lock), "old-leader".to_string()),
-                move |leader_cancel| {
-                    let page_tx = page_tx.clone();
-                    let cleanup_started = Arc::clone(&cleanup_started);
-                    let finish_cleanup = Arc::clone(&finish_cleanup);
-                    async move {
-                        run_tier_free_version_recovery_loop(
-                            leader_cancel,
-                            ExpiryState::new(),
-                            std::convert::identity,
-                            move |_, _, page_cancel| {
-                                let cleanup_started = Arc::clone(&cleanup_started);
-                                let finish_cleanup = Arc::clone(&finish_cleanup);
-                                page_tx.send(()).expect("page observer should remain open");
-                                async move {
-                                    page_cancel.cancelled().await;
-                                    cleanup_started.notify_one();
-                                    finish_cleanup.notified().await;
-                                    Err(std::io::Error::new(std::io::ErrorKind::Interrupted, "cancelled page").into())
-                                }
-                            },
-                        )
-                        .await;
-                    }
-                },
-            )
-            .await;
-        });
-        page_rx.recv().await.expect("leader should start an active page");
-        // Let authoritative backend leases expire before the next heartbeat
-        // can renew them. A failed refresh must cancel the active page.
-        tokio::time::advance(StdDuration::from_secs(3600)).await;
-        tokio::time::timeout(StdDuration::from_secs(1), observed_cleanup.notified())
-            .await
-            .expect("expired backend leases must trigger recovery cancellation");
-        assert!(!task.is_finished(), "lock loss must wait for the active page's cleanup");
-        assert!(page_rx.try_recv().is_err(), "the old leader must not start another page");
-        cancel.cancel();
-        release_cleanup.notify_one();
-        task.await.expect("coordinator should finish cleanup before exiting");
-    }
-
-    #[tokio::test(start_paused = true)]
-    async fn tier_free_version_recovery_coordinator_retries_without_scanning_after_lock_error() {
-        let cancel = CancellationToken::new();
-        let loop_cancel = cancel.clone();
-        let attempts = Arc::new(AtomicUsize::new(0));
-        let observed_attempts = Arc::clone(&attempts);
-        let task = tokio::spawn(async move {
-            run_tier_free_version_recovery_coordinator(
-                loop_cancel,
-                recovery_test_retry_delay,
-                move || {
-                    observed_attempts.fetch_add(1, Ordering::SeqCst);
-                    std::future::ready(Err(rustfs_lock::LockError::InsufficientNodes {
-                        required: 2,
-                        available: 1,
-                    }
-                    .into()))
-                },
-                |_| async { panic!("a node without ownership must never start discovery") },
-            )
-            .await;
-        });
-        tokio::task::yield_now().await;
-        assert_eq!(attempts.load(Ordering::SeqCst), 1);
-        tokio::time::advance(StdDuration::from_secs(14)).await;
-        assert_eq!(attempts.load(Ordering::SeqCst), 1, "lock errors must not cause a tight retry loop");
-        tokio::time::advance(StdDuration::from_secs(1)).await;
-        tokio::task::yield_now().await;
-        assert_eq!(attempts.load(Ordering::SeqCst), 2);
-        cancel.cancel();
-        task.await.expect("a follower should respond to shutdown during retry wait");
-    }
-
-    #[tokio::test]
-    async fn tier_free_version_recovery_enqueue_cancels_while_waiting_for_state_lock() {
-        let state = ExpiryState::new();
-        let (tx, mut rx) = tokio::sync::mpsc::channel(1);
-        let mut held = state.write().await;
-        held.tasks_tx.push(tx);
-        let cancel = CancellationToken::new();
-        let enqueue_cancel = cancel.clone();
-        let enqueue_state = Arc::clone(&state);
-        let enqueue = tokio::spawn(async move {
-            enqueue_recovered_free_version_with_state(&enqueue_state, ObjectInfo::default(), &enqueue_cancel).await
-        });
-        tokio::task::yield_now().await;
-        assert!(!enqueue.is_finished(), "enqueue should be blocked on the state lock");
-        cancel.cancel();
-        let queued = tokio::time::timeout(StdDuration::from_secs(1), enqueue)
-            .await
-            .expect("cancellation must not wait for the state write guard")
-            .expect("enqueue should return cleanly");
-        assert!(!queued);
-        assert_eq!(held.pending_tasks(), 0);
-        assert_eq!(held.stats.missed_free_vers_tasks(), 0);
-        assert!(rx.try_recv().is_err(), "a cancelled recovery must not enqueue a task");
-        drop(held);
-        assert!(!enqueue_recovered_free_version_with_state(&state, ObjectInfo::default(), &cancel).await);
-        assert!(rx.try_recv().is_err(), "cancellation must win even when the lock is available");
-    }
-
     fn free_version_recovery_stats(enqueued: usize, failed: usize, truncated: bool) -> FreeVersionRecoveryStats {
         FreeVersionRecoveryStats {
             scanned: enqueued.saturating_add(failed),
@@ -8091,7 +7680,7 @@ mod tests {
         assert_eq!(state.read().await.stats.pending_tasks(), 1);
 
         // The single-slot queue is full for both enqueue paths.
-        assert!(!enqueue_recovered_free_version_with_state(&state, oi.clone(), &CancellationToken::new()).await);
+        assert!(!enqueue_recovered_free_version_with_state(&state, oi.clone()).await);
         assert_eq!(state.read().await.stats.pending_tasks(), 1);
         assert_eq!(state.read().await.stats.missed_free_vers_tasks(), 1);
 
@@ -8123,7 +7712,7 @@ mod tests {
             ..Default::default()
         };
 
-        let queued = enqueue_recovered_free_version_with_state(&state, oi, &CancellationToken::new()).await;
+        let queued = enqueue_recovered_free_version_with_state(&state, oi).await;
         let state = state.read().await;
 
         assert!(!queued);
@@ -8252,8 +7841,8 @@ mod tests {
             ..Default::default()
         };
 
-        let first = enqueue_recovered_free_version_with_state(&state, oi.clone(), &CancellationToken::new()).await;
-        let second = enqueue_recovered_free_version_with_state(&state, oi, &CancellationToken::new()).await;
+        let first = enqueue_recovered_free_version_with_state(&state, oi.clone()).await;
+        let second = enqueue_recovered_free_version_with_state(&state, oi).await;
         let state = state.read().await;
 
         assert!(first);
@@ -8354,7 +7943,7 @@ mod tests {
         };
 
         assert!(
-            super::enqueue_recovered_free_version_with_cancel(&ecstore, oi, &CancellationToken::new()).await,
+            super::enqueue_recovered_free_version(&ecstore, oi).await,
             "the resized production worker queue should accept the task"
         );
         stop_tx.send(None).await.expect("worker stop signal should be delivered");
@@ -10552,26 +10141,6 @@ mod tests {
         );
     }
 
-    #[test]
-    fn lifecycle_expiry_already_gone_only_accepts_object_or_version_absence() {
-        assert!(is_lifecycle_expiry_already_gone(&Error::FileNotFound));
-        assert!(is_lifecycle_expiry_already_gone(&Error::ObjectNotFound(
-            "bucket".to_string(),
-            "object".to_string(),
-        )));
-        assert!(is_lifecycle_expiry_already_gone(&Error::FileVersionNotFound));
-        assert!(is_lifecycle_expiry_already_gone(&Error::VersionNotFound(
-            "bucket".to_string(),
-            "object".to_string(),
-            "version".to_string(),
-        )));
-
-        assert!(!is_lifecycle_expiry_already_gone(&Error::VolumeNotFound));
-        assert!(!is_lifecycle_expiry_already_gone(&Error::DiskNotFound));
-        assert!(!is_lifecycle_expiry_already_gone(&Error::ErasureReadQuorum));
-        assert!(!is_lifecycle_expiry_already_gone(&Error::MethodNotAllowed));
-    }
-
     #[tokio::test]
     async fn manual_transition_counts_already_transitioned_object() {
         let lc = latest_transition_lifecycle();
@@ -13446,9 +13015,7 @@ mod tests {
                 .bucket_incarnation_id_from_disk(&bucket)
                 .await
                 .expect("bucket incarnation");
-            // Replaying the same scanner snapshot after the delete is an
-            // already-gone success and must not be reported as a failure.
-            for (attempt_incarnation, expected) in [(Uuid::new_v4(), false), (incarnation, true), (incarnation, true)] {
+            for (attempt_incarnation, expected) in [(Uuid::new_v4(), false), (incarnation, true), (incarnation, false)] {
                 assert_eq!(
                     super::apply_expiry_on_non_transitioned_objects(
                         ecstore.clone(),
@@ -14270,10 +13837,6 @@ mod tests {
         .await;
     }
 
-    fn fast_tier_free_version_recovery_leader_retry_delay() -> StdDuration {
-        StdDuration::from_millis(10)
-    }
-
     #[tokio::test]
     #[serial]
     async fn tier_free_version_recovery_production_entrypoint_enqueues_seeded_item() {
@@ -14289,13 +13852,8 @@ mod tests {
         seed_recoverable_free_version(&disk_paths, &bucket, object, None, None).await;
 
         let started = OnceLock::new();
-        let recovery = super::spawn_tier_free_version_recovery_once_with_options(
-            Arc::clone(&ecstore),
-            &started,
-            format!("tier-free-version-recovery/test/{}", Uuid::new_v4()),
-            fast_tier_free_version_recovery_leader_retry_delay,
-        )
-        .expect("production recovery entrypoint should start once");
+        let recovery = super::spawn_tier_free_version_recovery_once(Arc::clone(&ecstore), &started)
+            .expect("production recovery entrypoint should start once");
         let task = tokio::time::timeout(StdDuration::from_secs(30), async { recovery_rx.lock().await.recv().await })
             .await
             .expect("production recovery entrypoint should enqueue the seeded item")
@@ -14338,14 +13896,13 @@ mod tests {
         };
         let mut recovery_rx = recovery_rx.lock().await;
         assert!(
-            super::enqueue_recovered_free_version_with_cancel(
+            super::enqueue_recovered_free_version(
                 &ecstore,
                 ObjectInfo {
                     bucket: "prefill".to_string(),
                     name: "prefill".to_string(),
                     ..Default::default()
                 },
-                &CancellationToken::new(),
             )
             .await,
             "the production recovery queue should accept its first task"

crates/ecstore/src/bucket/lifecycle/tier_free_version_recovery.rs

@@ -20,7 +20,7 @@ use std::time::Duration;
 use tokio::sync::mpsc;
 use tokio_util::sync::CancellationToken;
 
-use crate::bucket::lifecycle::bucket_lifecycle_ops::enqueue_recovered_free_version_with_cancel;
+use crate::bucket::lifecycle::bucket_lifecycle_ops::enqueue_recovered_free_version;
 use crate::disk::RUSTFS_META_BUCKET;
 use crate::error::Result;
 use crate::object_api::ObjectInfo;
@@ -201,11 +201,7 @@ pub(super) async fn recover_tier_free_versions_with_cancel(
             return Err(tier_free_version_recovery_cancelled());
         }
         retry_cursor.visit(&oi);
-        let queued = enqueue_recovered_free_version_with_cancel(&api, oi, &cancel_token).await;
-        if cancel_token.is_cancelled() {
-            return Err(tier_free_version_recovery_cancelled());
-        }
-        if !record_recovered_free_version_enqueue(&mut stats, queued) {
+        if !record_recovered_free_version_enqueue(&mut stats, enqueue_recovered_free_version(&api, oi).await) {
             let (bucket_marker, object_marker) = retry_cursor.retry_markers();
             stats.truncated = true;
             stats.next_bucket_marker = bucket_marker;

crates/ecstore/src/bucket/metadata_sys.rs

@@ -178,13 +178,6 @@ pub(crate) fn object_lock_config_state_from_authoritative_metadata(bm: &BucketMe
     Ok(ObjectLockConfigState::ConfirmedAbsent)
 }
 
-#[derive(Debug, Clone, PartialEq)]
-pub(crate) struct CachedQuotaMetadataSnapshot {
-    pub(crate) quota: Option<BucketQuota>,
-    pub(crate) bucket_incarnation: Uuid,
-    pub(crate) quota_revision: OffsetDateTime,
-}
-
 /// Convert the persisted serving-layer configuration into the storage-level
 /// [`DefaultRetention`](crate::bucket::object_lock::types::DefaultRetention)
 /// the WORM evaluation code consumes (rustfs/backlog#1842). A rule without a
@@ -321,33 +314,6 @@ pub async fn remove_bucket_metadata(bucket: &str) -> Result<bool> {
     Ok(lock.remove(bucket).await)
 }
 
-/// Remove a peer's cached metadata only when the bucket is absent under the
-/// namespace read fence. A delayed delete notification must not withdraw the
-/// metadata of a same-name bucket that has already been recreated.
-pub async fn remove_bucket_metadata_if_absent(api: Arc<ECStore>, bucket: &str) -> Result<bool> {
-    if is_meta_bucketname(bucket) {
-        return Ok(false);
-    }
-    // Lock order: bucket namespace read -> metadata system read (released)
-    // -> metadata publish, matching peer reloads.
-    let namespace_lock = api.new_ns_lock(bucket, bucket).await?;
-    let namespace_guard = namespace_lock
-        .get_read_lock(crate::set_disk::get_lock_acquire_timeout())
-        .await?;
-    let sys = bucket_metadata_sys_of(&api.ctx)?;
-    let sys = sys.read().await.clone();
-    if sys
-        .bucket_exists(bucket, &namespace_guard, "peer bucket metadata removal existence check")
-        .await?
-    {
-        return Ok(false);
-    }
-    await_bucket_namespace_operation(Some(&namespace_guard), bucket, "peer bucket metadata removal", async {
-        Ok(sys.remove(bucket).await)
-    })
-    .await
-}
-
 fn start_refresh_buckets_metadata_loop(sys: Arc<RwLock<BucketMetadataSys>>) {
     let Some(cancel_token) = runtime_sources::background_services_cancel_token() else {
         warn!("bucket metadata refresh loop skipped because background cancellation token is not initialized");
@@ -525,36 +491,6 @@ pub(crate) async fn get_bucket_incarnation_id_in(ctx: &crate::runtime::instance:
     sys.get_bucket_incarnation_id_from_disk(bucket).await
 }
 
-/// Directory loss is recoverable only for a committed, non-retired generation.
-/// The caller retains the bucket lifecycle fence through the ensuing reconstruction.
-pub(crate) async fn validate_bucket_reconstruction_in(
-    ctx: &crate::runtime::instance::InstanceContext,
-    bucket: &str,
-    expected: Uuid,
-) -> Result<()> {
-    let api = bucket_metadata_sys_of(ctx)?.read().await.object_store();
-    let lock = api
-        .new_ns_lock(RUSTFS_META_BUCKET, &bucket_metadata_transaction_lock_key(bucket))
-        .await?;
-    let guard = lock.get_read_lock(crate::set_disk::get_lock_acquire_timeout()).await?;
-    let (metadata, persisted) = load_bucket_metadata_parse_with_presence(api.clone(), bucket, true).await?;
-    if !persisted
-        || !metadata.bucket_incarnation_sidecar
-        || expected.is_nil()
-        || metadata.bucket_incarnation_id != expected
-        || !metadata.bucket_creation_committed
-    {
-        return Err(Error::PreconditionFailed);
-    }
-    if crate::bucket::retirement::is_retired(api, bucket, expected).await? {
-        return Err(Error::BucketNotFound(bucket.to_owned()));
-    }
-    if guard.is_lock_lost() {
-        return Err(Error::PreconditionFailed);
-    }
-    Ok(())
-}
-
 /// Validate against disk while retaining an already-held Object Lock fence.
 pub(crate) async fn get_bucket_incarnation_id_for_options_in(
     ctx: &crate::runtime::instance::InstanceContext,
@@ -1217,7 +1153,7 @@ async fn acquire_transaction_lock_with_sys(
 /// agree on, so renaming it would leave a mixed-version cluster with two
 /// disjoint keys — and old and new nodes would stop excluding each other on
 /// the very writes that are serialized today.
-pub(crate) fn bucket_metadata_transaction_lock_key(bucket: &str) -> String {
+fn bucket_metadata_transaction_lock_key(bucket: &str) -> String {
     format!("bucket-targets/{bucket}/transaction.lock")
 }
 
@@ -1356,34 +1292,14 @@ pub(crate) async fn get_object_lock_config_state_in(
 pub(crate) async fn get_object_lock_config_and_incarnation_from_disk_in(
     ctx: &crate::runtime::instance::InstanceContext,
     bucket: &str,
-) -> Result<(ObjectLockConfigState, Uuid, OffsetDateTime)> {
-    get_object_lock_config_and_incarnation_from_disk_with_guard_in(ctx, bucket, None).await
-}
-
-/// Bucket deletion already owns the namespace/lifecycle fences. Reuse its
-/// live guard rather than recursively acquiring the bucket namespace lock.
-/// The caller must also hold the metadata transaction read lock.
-pub(crate) async fn get_object_lock_config_and_incarnation_from_disk_with_guard_in(
-    ctx: &crate::runtime::instance::InstanceContext,
-    bucket: &str,
-    guard: Option<&rustfs_lock::NamespaceLockGuard>,
 ) -> Result<(ObjectLockConfigState, Uuid, OffsetDateTime)> {
     let bucket_meta_sys_lock = bucket_metadata_sys_of(ctx)?;
     let bucket_meta_sys = bucket_meta_sys_lock.read().await.clone();
 
-    let authority = match guard {
-        Some(guard) => {
-            bucket_meta_sys
-                .read_authoritative_metadata_from_disk_under_guard(bucket, guard)
-                .await?
-        }
-        None => {
-            bucket_meta_sys
-                .read_authoritative_metadata_from_disk_under_transaction_lock(bucket)
-                .await?
-        }
-    };
-    match authority {
+    match bucket_meta_sys
+        .read_authoritative_metadata_from_disk_under_transaction_lock(bucket)
+        .await?
+    {
         BucketMetadataAuthority::Authoritative(metadata)
             if metadata.bucket_incarnation_sidecar && !metadata.bucket_incarnation_id.is_nil() =>
         {
@@ -1487,9 +1403,6 @@ fn quota_config_and_incarnation_from_authority(
         BucketMetadataAuthority::Authoritative(metadata)
             if metadata.bucket_incarnation_sidecar && !metadata.bucket_incarnation_id.is_nil() =>
         {
-            if !metadata.quota_config_json.is_empty() && metadata.quota_config.is_none() {
-                return Err(Error::other("persisted bucket quota configuration is invalid"));
-            }
             Ok((
                 metadata.quota_config.clone(),
                 metadata.bucket_incarnation_id,
@@ -1504,15 +1417,6 @@ fn quota_config_and_incarnation_from_authority(
     }
 }
 
-pub(crate) async fn get_cached_quota_config_and_incarnation_in(
-    ctx: &crate::runtime::instance::InstanceContext,
-    bucket: &str,
-) -> Result<Option<CachedQuotaMetadataSnapshot>> {
-    let bucket_meta_sys_lock = bucket_metadata_sys_of(ctx)?;
-    let bucket_meta_sys = bucket_meta_sys_lock.read().await.clone();
-    bucket_meta_sys.cached_quota_config_and_incarnation(bucket).await
-}
-
 pub async fn get_replication_config(bucket: &str) -> Result<(ReplicationConfiguration, OffsetDateTime)> {
     let bucket_meta_sys_lock = get_bucket_metadata_sys()?;
     let bucket_meta_sys = bucket_meta_sys_lock.read().await;
@@ -3028,27 +2932,6 @@ impl BucketMetadataSys {
         }
     }
 
-    async fn cached_quota_config_and_incarnation(&self, bucket: &str) -> Result<Option<CachedQuotaMetadataSnapshot>> {
-        let metadata = {
-            let metadata_map = self.metadata_map.read().await;
-            metadata_map.get(bucket).cloned()
-        };
-        let Some(metadata) = metadata else {
-            return Ok(None);
-        };
-        if !metadata.bucket_incarnation_sidecar || metadata.bucket_incarnation_id.is_nil() {
-            return Ok(None);
-        }
-        if !metadata.quota_config_json.is_empty() && metadata.quota_config.is_none() {
-            return Err(Error::other("persisted bucket quota configuration is invalid"));
-        }
-        Ok(Some(CachedQuotaMetadataSnapshot {
-            quota: metadata.quota_config.clone(),
-            bucket_incarnation: metadata.bucket_incarnation_id,
-            quota_revision: metadata.quota_config_updated_at,
-        }))
-    }
-
     pub async fn get_replication_config(&self, bucket: &str) -> Result<(ReplicationConfiguration, OffsetDateTime)> {
         let (bm, _) = self.get_config(bucket).await?;
 
@@ -4953,154 +4836,6 @@ mod tests {
         );
     }
 
-    #[tokio::test]
-    #[serial]
-    async fn quota_admission_rejects_stale_cached_no_quota() {
-        let (_dirs, store) = isolated_store_over_temp_disks().await;
-        init_bucket_metadata_sys(store.clone(), Vec::new()).await;
-        let sys = bucket_metadata_sys_of(&store.ctx).expect("scoped metadata system");
-        let bucket = "stale-no-quota-admission";
-        store
-            .make_bucket(bucket, &MakeBucketOptions::default())
-            .await
-            .expect("create bucket");
-        let (stale, _) = sys.read().await.get_config(bucket).await.expect("cache initial metadata");
-        store
-            .update_bucket_metadata_config(bucket, "quota.json", br#"{"quota":1}"#.to_vec())
-            .await
-            .expect("publish quota to authoritative storage");
-        // Model a peer that has not received the newer configuration snapshot.
-        sys.read().await.set(bucket.to_owned(), stale).await;
-        assert!(
-            get_cached_quota_config_and_incarnation_in(&store.ctx, bucket)
-                .await
-                .expect("read stale cache")
-                .expect("cached bucket identity")
-                .quota
-                .is_none()
-        );
-
-        temp_env::async_with_vars([("RUSTFS_QUOTA_BEGIN_SAFE_NO_QUOTA_CACHE_FAST_PATH", Some("true"))], async {
-            let error = crate::bucket::quota::reservation::begin(&store.ctx, bucket, "object", &ObjectOptions::default(), 0, 0)
-                .await
-                .err()
-                .expect("persisted quota must reject admission without its snapshot");
-            assert!(matches!(error, Error::PartMissingOrCorrupt), "{error:?}");
-        })
-        .await;
-    }
-
-    #[tokio::test]
-    #[serial]
-    async fn peer_reload_clears_negative_cache_after_remote_bucket_creation() {
-        let (dirs, node_a, node_b) = two_nodes_over_one_store().await;
-        let bucket = "cross-node-negative-cache-refresh";
-        let node_b_sys = node_b.read().await.clone();
-
-        assert_eq!(
-            node_b_sys.get_bucket_incarnation_id(bucket).await.unwrap_err(),
-            Error::BucketNotFound(bucket.to_string()),
-            "the probe must populate node B's missing-bucket cache"
-        );
-
-        for dir in &dirs {
-            std::fs::create_dir_all(dir.path().join(bucket)).expect("physical bucket should exist before creation metadata");
-        }
-        let created = BucketMetadata::new(bucket);
-        let created_incarnation = created.bucket_incarnation_id;
-        node_a
-            .read()
-            .await
-            .persist_new_and_set(created)
-            .await
-            .expect("node A should publish the created bucket metadata");
-
-        node_b_sys
-            .reload_from_store(bucket)
-            .await
-            .expect("peer reload should clear the stale negative cache");
-        assert_eq!(
-            node_b_sys.get_bucket_incarnation_id(bucket).await.unwrap(),
-            created_incarnation,
-            "node B should resolve the newly created bucket immediately"
-        );
-    }
-
-    #[tokio::test(flavor = "multi_thread", worker_threads = 4)]
-    #[serial]
-    async fn peer_delete_clears_metadata_for_absent_bucket() {
-        let (_dirs, store) = isolated_store_over_temp_disks().await;
-        init_bucket_metadata_sys(store.clone(), Vec::new()).await;
-        let sys = bucket_metadata_sys_of(&store.ctx).expect("metadata owner");
-        let bucket = "peer-delete-absent-bucket";
-
-        store
-            .make_bucket(bucket, &MakeBucketOptions::default())
-            .await
-            .expect("initial bucket creation");
-        let cached = sys.read().await.get(bucket).await.expect("initial cached metadata");
-        store
-            .delete_bucket(bucket, &DeleteBucketOptions::default())
-            .await
-            .expect("bucket deletion");
-
-        // Model a peer that still caches the metadata after the physical delete.
-        sys.read().await.set(bucket.to_string(), cached).await;
-        assert!(
-            remove_bucket_metadata_if_absent(store.clone(), bucket)
-                .await
-                .expect("conditional peer delete"),
-            "the peer must evict the deleted bucket's cached metadata"
-        );
-        let sys = sys.read().await;
-        assert!(matches!(sys.get(bucket).await, Err(Error::ConfigNotFound)));
-        assert_eq!(
-            sys.get_bucket_incarnation_id(bucket)
-                .await
-                .expect_err("deleted bucket lookup"),
-            Error::BucketNotFound(bucket.to_string()),
-        );
-    }
-
-    #[tokio::test(flavor = "multi_thread", worker_threads = 4)]
-    #[serial]
-    async fn delayed_peer_delete_preserves_recreated_bucket_metadata() {
-        let (_dirs, store) = isolated_store_over_temp_disks().await;
-        init_bucket_metadata_sys(store.clone(), Vec::new()).await;
-        let sys = bucket_metadata_sys_of(&store.ctx).expect("metadata owner");
-        let bucket = "delayed-peer-delete-recreated-bucket";
-
-        store
-            .make_bucket(bucket, &MakeBucketOptions::default())
-            .await
-            .expect("initial bucket creation");
-        let old_incarnation = store.bucket_incarnation_id_from_disk(bucket).await.expect("old incarnation");
-        store
-            .delete_bucket(bucket, &DeleteBucketOptions::default())
-            .await
-            .expect("bucket deletion");
-        store
-            .make_bucket(bucket, &MakeBucketOptions::default())
-            .await
-            .expect("bucket recreation");
-        let new_incarnation = store.bucket_incarnation_id_from_disk(bucket).await.expect("new incarnation");
-        assert_ne!(old_incarnation, new_incarnation);
-
-        let removed = remove_bucket_metadata_if_absent(store.clone(), bucket)
-            .await
-            .expect("conditional peer delete");
-        assert!(!removed, "a delayed delete must not remove metadata for a live recreated bucket");
-        assert_eq!(
-            sys.read()
-                .await
-                .get(bucket)
-                .await
-                .expect("recreated metadata should remain cached")
-                .bucket_incarnation_id,
-            new_incarnation,
-        );
-    }
-
     #[tokio::test(flavor = "multi_thread", worker_threads = 4)]
     #[serial]
     async fn stale_config_request_cannot_mutate_a_recreated_bucket() {

crates/ecstore/src/bucket/migration.rs

@@ -273,30 +273,6 @@ where
 
     let buckets: Vec<String> = buckets_list.into_iter().map(|b| b.name).collect();
 
-    try_migrate_bucket_metadata_for_buckets(store, &buckets).await
-}
-
-/// Migrate the already discovered startup candidates without repeating a
-/// majority-only namespace listing. Every migration write retains its quorum.
-pub async fn try_migrate_bucket_metadata_for_buckets<S>(store: Arc<S>, buckets: &[String]) -> Result<()>
-where
-    S: ObjectIO<
-            Error = Error,
-            RangeSpec = HTTPRangeSpec,
-            HeaderMap = HeaderMap,
-            ObjectOptions = ObjectOptions,
-            ObjectInfo = ObjectInfo,
-            GetObjectReader = GetObjectReader,
-            PutObjectReader = PutObjReader,
-        > + ObjectOperations<
-            Error = Error,
-            ObjectInfo = ObjectInfo,
-            ObjectOptions = ObjectOptions,
-            FileInfo = FileInfo,
-            ObjectToDelete = ObjectToDelete,
-            DeletedObject = DeletedObject,
-        >,
-{
     if buckets.is_empty() {
         debug!("No migrating bucket metadata found");
         return Ok(());

crates/ecstore/src/bucket/quota/mod.rs

@@ -32,10 +32,7 @@ pub enum QuotaType {
     Hard,
 }
 
-pub const QUOTA_RESERVATION_PROTOCOL_V1: u32 = 1;
-/// Sharded durable reservations. Older nodes reject this protocol instead of
-/// silently falling back to the single-ledger protocol.
-pub const QUOTA_RESERVATION_PROTOCOL_V2: u32 = 2;
+pub(crate) const QUOTA_RESERVATION_PROTOCOL_V1: u32 = 1;
 
 /// Bucket quota configuration. quota_type defaults to Hard when omitted.
 #[derive(Debug, Default, Clone, PartialEq)]
@@ -92,10 +89,7 @@ impl<'de> Deserialize<'de> for BucketQuota {
         D: Deserializer<'de>,
     {
         let wire = BucketQuotaWire::deserialize(deserializer)?;
-        let quota = if matches!(
-            wire.reservation_protocol,
-            Some(QUOTA_RESERVATION_PROTOCOL_V1) | Some(QUOTA_RESERVATION_PROTOCOL_V2)
-        ) {
+        let quota = if wire.reservation_protocol == Some(QUOTA_RESERVATION_PROTOCOL_V1) {
             Some(
                 wire.reservation_quota
                     .ok_or_else(|| D::Error::custom("reservation_quota is required for reservation protocol v1"))?,
@@ -135,33 +129,17 @@ impl BucketQuota {
         }
     }
 
-    /// Creates a quota using the sharded durable reservation protocol.
-    /// Deployment must complete the mixed-version rollout before enabling it;
-    /// older nodes fail closed when they see protocol v2.
-    pub fn new_sharded(quota: u64) -> Self {
-        let mut config = Self::new(Some(quota));
-        config.reservation_protocol = Some(QUOTA_RESERVATION_PROTOCOL_V2);
-        config
-    }
-
     pub fn get_quota_limit(&self) -> Option<u64> {
         self.quota
     }
 
     pub fn uses_durable_reservations(&self) -> bool {
-        matches!(
-            self.reservation_protocol,
-            Some(QUOTA_RESERVATION_PROTOCOL_V1) | Some(QUOTA_RESERVATION_PROTOCOL_V2)
-        )
-    }
-
-    pub fn uses_sharded_reservations(&self) -> bool {
-        self.reservation_protocol == Some(QUOTA_RESERVATION_PROTOCOL_V2)
+        self.reservation_protocol == Some(QUOTA_RESERVATION_PROTOCOL_V1)
     }
 
     pub fn has_unsupported_reservation_protocol(&self) -> bool {
         self.reservation_protocol
-            .is_some_and(|version| !matches!(version, QUOTA_RESERVATION_PROTOCOL_V1 | QUOTA_RESERVATION_PROTOCOL_V2))
+            .is_some_and(|version| version != QUOTA_RESERVATION_PROTOCOL_V1)
     }
 
     pub fn check_operation_allowed(&self, current_usage: u64, operation_size: u64) -> bool {
@@ -344,21 +322,9 @@ mod tests {
     }
 
     #[test]
-    fn sharded_quota_roundtrips_with_authoritative_reservation_quota() {
-        let quota = BucketQuota::new_sharded(4096);
-        let json = serde_json::to_value(&quota).expect("sharded quota should serialize");
-        assert_eq!(json["quota"], 0);
-        assert_eq!(json["reservation_protocol"], QUOTA_RESERVATION_PROTOCOL_V2);
-        assert_eq!(json["reservation_quota"], 4096);
-        let restored: BucketQuota = serde_json::from_value(json).expect("sharded quota should parse");
-        assert_eq!(restored.quota, Some(4096));
-        assert!(restored.uses_sharded_reservations());
-    }
-
-    #[test]
-    fn unknown_reservation_protocol_does_not_activate_any_known_protocol() {
+    fn unknown_reservation_protocol_does_not_activate_v1() {
         let quota: BucketQuota =
-            serde_json::from_str(r#"{"quota":0,"quota_type":"Hard","reservation_protocol":3,"reservation_quota":2048}"#)
+            serde_json::from_str(r#"{"quota":0,"quota_type":"Hard","reservation_protocol":2,"reservation_quota":2048}"#)
                 .expect("future protocol should remain parseable");
 
         assert!(!quota.uses_durable_reservations());

crates/ecstore/src/bucket/replication/replication_resyncer.rs

@@ -65,9 +65,9 @@ use super::replication_target_boundary::{
     SsecPassthroughGate, TargetClient, VersionIdentityCapability, is_replication_target_offline_error,
     replication_action_for_target_head, replication_complete_multipart_options, replication_delete_marker_purge_remove_options,
     replication_delete_remove_options, replication_force_delete_remove_options, replication_object_is_ssec_encrypted,
-    replication_put_object_header_size, replication_put_object_options, replication_source_etag_is_comparable,
-    replication_target_head_is_newer_null_version, resolve_read_api_version_id, ssec_passthrough_evidence_present,
-    ssec_passthrough_gate, version_identity_capability_from_put, version_identity_drifted,
+    replication_put_object_header_size, replication_put_object_options, replication_target_head_is_newer_null_version,
+    resolve_read_api_version_id, ssec_passthrough_evidence_present, ssec_passthrough_gate, version_identity_capability_from_put,
+    version_identity_drifted,
 };
 use super::replication_versioning_boundary::ReplicationVersioningStore;
 use super::runtime_boundary as runtime_sources;
@@ -349,22 +349,15 @@ fn replica_etag_verification_enabled() -> bool {
 /// target that stores a transformed payload (e.g. undecoded `aws-chunked`
 /// frames, #6853) returns the ETag of what it actually wrote. Reporting
 /// COMPLETED over such a replica is silent corruption, so a decidable
-/// mismatch fails the replication instead. Encrypted sources are exempt: an
-/// SSE-C ciphertext passthrough sends ciphertext while the source ETag is the
-/// plaintext MD5 (that path has its own HEAD-back audit), and a managed-SSE
-/// (SSE-S3 / SSE-KMS) source stores a keyed fingerprint, or a legacy plaintext
-/// MD5, that the ETag of the target's re-encrypted replica can neither confirm
-/// nor refute. The checksum header on the replication PUT stays the integrity
-/// check for those objects.
+/// mismatch fails the replication instead. An SSE-C ciphertext passthrough
+/// transfer is exempt: the wire bytes are ciphertext while the source ETag is
+/// the plaintext MD5, and that path has its own HEAD-back audit.
 fn verify_single_part_replica(
     object_info: &ObjectInfo,
     response: &RemotePutObjectResponse,
     ciphertext_passthrough: bool,
 ) -> std::result::Result<(), std::io::Error> {
-    if ciphertext_passthrough
-        || !replication_source_etag_is_comparable(&object_info.user_defined)
-        || !replica_etag_verification_enabled()
-    {
+    if ciphertext_passthrough || !replica_etag_verification_enabled() {
         return Ok(());
     }
     if single_part_replica_etag_mismatch(object_info.etag.as_deref(), response.etag.as_deref()) {
@@ -455,26 +448,6 @@ fn is_version_id_format_mismatch(err: &SdkError<HeadObjectError>) -> bool {
     is_version_id_mismatch(code, raw_status)
 }
 
-/// A version-addressed HEAD rejected because the target cannot address the
-/// source version id (AWS S3: bodiless 400). Narrower than
-/// [`is_version_id_format_mismatch`]: a bodiless 403 may be a genuine
-/// `s3:GetObjectVersion` denial, so it is not accepted where a mutation or a
-/// completion verdict follows instead of another non-destructive HEAD.
-fn rejects_source_version_id(err: &SdkError<HeadObjectError>) -> bool {
-    is_version_id_format_mismatch(err) && !has_raw_status(err, 403)
-}
-
-/// Whether the target's latest version of `object` is a delete marker. For
-/// targets that mint their own version ids, which cannot be asked about the
-/// source marker id: S3 answers a versionless HEAD on such a key with 404
-/// (405 on some targets) plus `x-amz-delete-marker: true`.
-async fn latest_target_version_is_delete_marker(tgt_client: &TargetClient, object: &str) -> bool {
-    match head_object_for_worker(tgt_client, &tgt_client.bucket, object, None).await {
-        Ok(head) => head.delete_marker == Some(true),
-        Err(e) => raw_header_value(e.as_ref(), "x-amz-delete-marker").as_deref() == Some("true"),
-    }
-}
-
 async fn mark_replication_target_offline_if_needed(target_client: &Arc<TargetClient>, err: &(impl Display + ?Sized)) {
     if is_replication_target_offline_error(err) {
         ReplicationTargetStore::mark_target_offline(target_client).await;
@@ -1873,17 +1846,7 @@ async fn verify_resync_head_result(
             // only absence proves completion. All other outcomes remain
             // failed (backlog#862 / #799 B13).
             let retryable = if roi.version_purge_status.is_empty() {
-                match classify_delete_marker_head_error(&err) {
-                    // A target that mints its own version ids never holds the
-                    // source marker id; verify the marker on the latest version.
-                    DeleteMarkerHeadOutcome::Ambiguous => {
-                        let mints_own = rejects_source_version_id(&err)
-                            || ReplicationTargetStore::version_identity_capability(&target_client.arn)
-                                .version_addressing_unreliable();
-                        !(mints_own && latest_target_version_is_delete_marker(target_client, &roi.name).await)
-                    }
-                    DeleteMarkerHeadOutcome::Exists | DeleteMarkerHeadOutcome::Missing => false,
-                }
+                matches!(classify_delete_marker_head_error(&err), DeleteMarkerHeadOutcome::Ambiguous)
             } else {
                 !matches!(classify_delete_marker_head_error(&err), DeleteMarkerHeadOutcome::Missing)
             };
@@ -3604,16 +3567,7 @@ async fn replicate_delete_to_target(
         }
     }
 
-    // A target that mints its own version ids never holds the source marker
-    // id, so a version-addressed pre-check cannot succeed there; the marker
-    // is created by the versionless DELETE below.
-    let version_addressable =
-        !ReplicationTargetStore::version_identity_capability(&tgt_client.arn).version_addressing_unreliable();
-    if version_addressable
-        && !is_version_purge
-        && dobj.delete_object.delete_marker
-        && dobj.delete_object.delete_marker_version_id.is_some()
-    {
+    if !is_version_purge && dobj.delete_object.delete_marker && dobj.delete_object.delete_marker_version_id.is_some() {
         match head_object_for_worker(
             tgt_client.as_ref(),
             &tgt_client.bucket,
@@ -3622,11 +3576,6 @@ async fn replicate_delete_to_target(
         )
         .await
         {
-            // The verdict is not known yet (e.g. no PUT since restart) and the
-            // target rejected the source id: learn it as the PUT path does.
-            Err(e) if rejects_source_version_id(e.as_ref()) => {
-                ReplicationTargetStore::record_version_identity_capability(&tgt_client.arn, VersionIdentityCapability::MintsOwn);
-            }
             Ok(head) if head.delete_marker == Some(true) => {
                 // Some targets expose an existing marker as a successful HEAD
                 // with the marker bit instead of the usual 405. Treat that as
@@ -5877,57 +5826,6 @@ mod tests {
     use super::super::replication_filemeta_boundary::ReplicateTargetDecision;
     use super::super::replication_object_decision_boundary::ReplicationMultipartPlanError;
 
-    #[test]
-    fn verify_single_part_replica_exempts_managed_sse_sources() {
-        // A new encrypted write stores a keyed fingerprint; a target that
-        // computes its own ETag answers with the MD5 of the plaintext it
-        // received. Both are 32 hex characters, so without the exemption the
-        // audit would fail every SSE-S3 / SSE-KMS replica (rustfs/backlog).
-        let fingerprint = "00112233445566778899aabbccddeeff";
-        let replica_md5 = "ffeeddccbbaa99887766554433221100";
-        let response = RemotePutObjectResponse {
-            version_id: None,
-            etag: Some(replica_md5.to_string()),
-        };
-        for sse in ["AES256", "aws:kms"] {
-            let object_info = ObjectInfo {
-                bucket: "source".to_string(),
-                name: "encrypted".to_string(),
-                etag: Some(fingerprint.to_string()),
-                user_defined: HashMap::from([("x-amz-server-side-encryption".to_string(), sse.to_string())]).into(),
-                ..Default::default()
-            };
-            verify_single_part_replica(&object_info, &response, false)
-                .unwrap_or_else(|err| panic!("{sse}: a managed-SSE source must not be audited against the replica MD5: {err}"));
-        }
-    }
-
-    #[test]
-    fn verify_single_part_replica_still_fails_plaintext_mismatch() {
-        if !replica_etag_verification_enabled() {
-            return;
-        }
-        let object_info = ObjectInfo {
-            bucket: "source".to_string(),
-            name: "plain".to_string(),
-            etag: Some("00112233445566778899aabbccddeeff".to_string()),
-            ..Default::default()
-        };
-        let mismatch = RemotePutObjectResponse {
-            version_id: None,
-            etag: Some("ffeeddccbbaa99887766554433221100".to_string()),
-        };
-        let err = verify_single_part_replica(&object_info, &mismatch, false)
-            .expect_err("a plaintext source whose replica MD5 differs must fail the replication");
-        assert_eq!(err.to_string(), REPLICA_ETAG_MISMATCH_ERROR);
-
-        let matching = RemotePutObjectResponse {
-            version_id: None,
-            etag: object_info.etag.clone(),
-        };
-        verify_single_part_replica(&object_info, &matching, false).expect("matching plaintext MD5s pass the audit");
-    }
-
     #[test]
     fn multipart_read_plan_preserves_legacy_plain_part_ranges() {
         const MIB: usize = 1024 * 1024;
@@ -6777,93 +6675,6 @@ mod tests {
         assert!(seen[0].starts_with("HEAD "), "{seen:?}");
     }
 
-    /// AWS S3 answers a HEAD for a version id it did not mint with a bodiless
-    /// 400. On a target with no verdict yet, that teaches `MintsOwn` and the
-    /// marker is still created with one versionless DELETE.
-    #[tokio::test]
-    async fn live_delete_marker_worker_learns_mints_own_from_bodiless_400() {
-        let (endpoint, server) = spawn_scripted_target_server(2, |line| {
-            if line.starts_with("HEAD ") {
-                empty_response("400 Bad Request")
-            } else if line.starts_with("DELETE ") {
-                empty_response("204 No Content")
-            } else {
-                empty_response("500 Unexpected")
-            }
-        });
-        let target = test_target_client(endpoint);
-        register_test_target(&target).await;
-        let dobj = marker_creation_dobj(&target.arn);
-
-        let rinfo = replicate_delete_to_target(&dobj, target.clone(), None).await;
-
-        assert_eq!(rinfo.replication_status, ReplicationStatusType::Completed, "{rinfo:?}");
-        assert_eq!(
-            ReplicationTargetStore::version_identity_capability(&target.arn),
-            VersionIdentityCapability::MintsOwn
-        );
-        let seen = server.join().expect("test HTTP server should finish");
-        assert_eq!(seen.len(), 2, "{seen:?}");
-        assert!(seen[0].starts_with("HEAD "), "{seen:?}");
-        assert!(seen[1].starts_with("DELETE "), "{seen:?}");
-        assert!(!seen[1].contains("versionId="), "marker creation must remain versionless: {seen:?}");
-    }
-
-    /// A bodiless 403 on the version-addressed HEAD may be a genuine
-    /// GetObjectVersion denial on an id-adopting target. It must fail closed:
-    /// no DELETE, and no `MintsOwn` verdict cached for the ARN.
-    #[tokio::test]
-    async fn live_delete_marker_worker_keeps_bodiless_403_failure_closed() {
-        let (endpoint, server) = spawn_scripted_target_server(1, |line| {
-            if line.starts_with("HEAD ") {
-                empty_response("403 Forbidden")
-            } else if line.starts_with("DELETE ") {
-                empty_response("204 No Content")
-            } else {
-                empty_response("500 Unexpected")
-            }
-        });
-        let target = test_target_client(endpoint);
-        register_test_target(&target).await;
-        let dobj = marker_creation_dobj(&target.arn);
-
-        let rinfo = replicate_delete_to_target(&dobj, target.clone(), None).await;
-
-        assert_eq!(rinfo.replication_status, ReplicationStatusType::Failed, "{rinfo:?}");
-        assert_ne!(
-            ReplicationTargetStore::version_identity_capability(&target.arn),
-            VersionIdentityCapability::MintsOwn
-        );
-        let seen = server.join().expect("test HTTP server should finish");
-        assert_eq!(seen.len(), 1, "a denied HEAD must not trigger DELETE: {seen:?}");
-        assert!(seen[0].starts_with("HEAD "), "{seen:?}");
-    }
-
-    /// A target already known to mint its own version ids is never asked for
-    /// the source marker id: the marker is created by the versionless DELETE.
-    #[tokio::test]
-    async fn live_delete_marker_worker_skips_version_head_on_mints_own_target() {
-        let (endpoint, server) = spawn_scripted_target_server(1, |line| {
-            if line.starts_with("DELETE ") {
-                empty_response("204 No Content")
-            } else {
-                empty_response("500 Unexpected")
-            }
-        });
-        let target = test_target_client(endpoint);
-        register_test_target(&target).await;
-        ReplicationTargetStore::record_version_identity_capability(&target.arn, VersionIdentityCapability::MintsOwn);
-        let dobj = marker_creation_dobj(&target.arn);
-
-        let rinfo = replicate_delete_to_target(&dobj, target.clone(), None).await;
-
-        assert_eq!(rinfo.replication_status, ReplicationStatusType::Completed, "{rinfo:?}");
-        let seen = server.join().expect("test HTTP server should finish");
-        assert_eq!(seen.len(), 1, "no version-addressed HEAD on a MintsOwn target: {seen:?}");
-        assert!(seen[0].starts_with("DELETE "), "{seen:?}");
-        assert!(!seen[0].contains("versionId="), "marker creation must remain versionless: {seen:?}");
-    }
-
     /// A target answers `HEAD ?versionId=<delete marker>` with a bodiless
     /// 405, so the SDK reports no error code. That is the marker having
     /// propagated, not a failure (rustfs/backlog#2479).
@@ -6954,76 +6765,6 @@ mod tests {
         server.join().expect("test HTTP server should finish");
     }
 
-    /// AWS answers `HEAD ?versionId=<source marker id>` with a bodiless 400.
-    /// The marker is verified through the latest version instead: a
-    /// versionless 404 with `x-amz-delete-marker: true` proves it propagated.
-    #[tokio::test]
-    async fn resync_verification_accepts_latest_delete_marker_when_target_rejects_source_id() {
-        let (endpoint, server) = spawn_scripted_target_server(2, |line| {
-            if line.contains("versionId=") {
-                empty_response("400 Bad Request")
-            } else {
-                "HTTP/1.1 404 Not Found\r\nX-Amz-Delete-Marker: true\r\nX-Amz-Version-Id: target-marker\r\nContent-Length: 0\r\nConnection: close\r\n\r\n"
-                    .to_string()
-            }
-        });
-        let target = test_target_client(endpoint);
-        let roi = delete_marker_roi();
-        let mut st = TargetReplicationResyncStatus::default();
-
-        let head_result =
-            head_object_for_worker(target.as_ref(), &target.bucket, &roi.name, roi.version_id.map(|v| v.to_string())).await;
-        let (size, err) = verify_resync_head_result(head_result, &roi, &mut st, &target).await;
-
-        assert!(err.is_none(), "{err:?}");
-        assert_eq!((size, st.replicated_count, st.failed_count), (0, 1, 0));
-        let seen = server.join().expect("test HTTP server should finish");
-        assert!(!seen[1].contains("versionId="), "the re-check must address the latest version: {seen:?}");
-    }
-
-    /// A live latest version on a target that rejects the source id means the
-    /// marker has not propagated: the verification stays failed.
-    #[tokio::test]
-    async fn resync_verification_fails_delete_marker_when_latest_target_version_is_live() {
-        let (endpoint, server) = spawn_scripted_target_server(2, |line| {
-            if line.contains("versionId=") {
-                empty_response("400 Bad Request")
-            } else {
-                empty_response("200 OK")
-            }
-        });
-        let target = test_target_client(endpoint);
-        let roi = delete_marker_roi();
-        let mut st = TargetReplicationResyncStatus::default();
-
-        let head_result =
-            head_object_for_worker(target.as_ref(), &target.bucket, &roi.name, roi.version_id.map(|v| v.to_string())).await;
-        let (size, err) = verify_resync_head_result(head_result, &roi, &mut st, &target).await;
-
-        assert!(err.is_some(), "a live latest version must not verify the marker");
-        assert_eq!((size, st.replicated_count, st.failed_count), (0, 0, 1));
-        server.join().expect("test HTTP server should finish");
-    }
-
-    /// A bodiless 403 may be a genuine permission denial: it is not evidence
-    /// that the target mints its own ids, so no latest-version re-check runs.
-    #[tokio::test]
-    async fn resync_verification_keeps_delete_marker_failed_on_bodiless_403() {
-        let (endpoint, server) = spawn_scripted_target_server(1, |_| empty_response("403 Forbidden"));
-        let target = test_target_client(endpoint);
-        let roi = delete_marker_roi();
-        let mut st = TargetReplicationResyncStatus::default();
-
-        let head_result =
-            head_object_for_worker(target.as_ref(), &target.bucket, &roi.name, roi.version_id.map(|v| v.to_string())).await;
-        let (size, err) = verify_resync_head_result(head_result, &roi, &mut st, &target).await;
-
-        assert!(err.is_some(), "a 403 must not count as a propagated delete marker");
-        assert_eq!((size, st.replicated_count, st.failed_count), (0, 0, 1));
-        let seen = server.join().expect("test HTTP server should finish");
-        assert_eq!(seen.len(), 1, "{seen:?}");
-    }
-
     /// Serves exactly `requests` connections, answering each from
     /// `respond(request_line)`, and returns the request lines it saw. Reads
     /// the whole request (headers plus `Content-Length` body, honoring

crates/ecstore/src/bucket/replication/replication_target_boundary.rs

@@ -157,19 +157,6 @@ pub(crate) fn replication_object_is_ssec_encrypted(user_defined: &HashMap<String
     rustfs_replication::is_ssec_encrypted(user_defined)
 }
 
-/// Whether the stored ETag of a replication source can be compared with the
-/// ETag a target returns for the bytes it received. Only an unencrypted source
-/// stores the MD5 of the bytes that go on the wire. A managed-SSE (SSE-S3 /
-/// SSE-KMS) or SSE-C source stores a keyed fingerprint (new writes) or a legacy
-/// plaintext MD5, and the target re-encrypts what it receives, so neither
-/// equality nor inequality of the two ETags proves anything about the replica.
-pub(crate) fn replication_source_etag_is_comparable(user_defined: &HashMap<String, String>) -> bool {
-    matches!(
-        classify_replication_source_encryption(user_defined),
-        ReplicationSourceEncryption::Plaintext
-    )
-}
-
 /// HeadObjectOutput adapter over the pure SSE-C passthrough evidence
 /// judgment owned by `rustfs-replication`: extract the echoed
 /// customer-algorithm header and let the crate-owned policy decide.
@@ -591,22 +578,6 @@ mod tests {
     use time::Duration;
     use uuid::Uuid;
 
-    #[test]
-    fn replication_source_etag_is_comparable_only_for_plaintext_sources() {
-        assert!(replication_source_etag_is_comparable(&HashMap::new()));
-        for (key, value) in [
-            (metadata_keys::SERVER_SIDE_ENCRYPTION, "AES256"),
-            (metadata_keys::SERVER_SIDE_ENCRYPTION, "aws:kms"),
-            (SSEC_ALGORITHM_HEADER, "AES256"),
-        ] {
-            let metadata = HashMap::from([(key.to_string(), value.to_string())]);
-            assert!(
-                !replication_source_etag_is_comparable(&metadata),
-                "{key}={value} must not expose its stored ETag to the replica audit"
-            );
-        }
-    }
-
     /// backlog#1735 A3b SSE key-case finding. Every RustFS writer persists the
     /// SSE intent as lowercase `x-amz-server-side-encryption`
     /// (`encryption_material_to_metadata`; header-derived keys come from a

crates/ecstore/src/cluster/rpc/client.rs

@@ -24,11 +24,10 @@ use crate::cluster::rpc::{gen_tonic_replay_scope_headers, gen_tonic_signature_he
 #[cfg(test)]
 use crate::cluster::rpc::{tonic_boot_epoch_challenge, tonic_boot_epoch_response_headers};
 use crate::disk::error::{DiskError, Error as DiskErrorType, RpcStatusError};
-#[cfg(test)]
 use crate::runtime::sources as runtime_sources;
 use http::{Request as HttpRequest, Response as HttpResponse, Uri};
 use rustfs_protos::{
-    ChannelClass, get_channel_for_class,
+    ChannelClass, create_new_channel, get_channel_for_class,
     proto_gen::node_service::{
         heal_control_service_client::HealControlServiceClient, node_service_client::NodeServiceClient,
         scanner_control_service_client::ScannerControlServiceClient,
@@ -54,7 +53,7 @@ use super::context_propagation::{inject_request_id_into_metadata, inject_trace_c
 /// 3. Subsequent calls will attempt fresh connections
 /// 4. If node is still down, connection will fail fast (3s timeout)
 pub async fn node_service_time_out_client(
-    addr: &str,
+    addr: &String,
     interceptor: TonicInterceptor,
 ) -> Result<NodeServiceClient<InterceptedService<AuthenticatedChannel, TonicInterceptor>>, Box<dyn Error>> {
     // Default to the latency-sensitive control channel; bulk `bytes` RPCs opt in via the
@@ -67,9 +66,12 @@ pub(crate) async fn scanner_control_time_out_client(
     interceptor: TonicInterceptor,
 ) -> crate::error::Result<ScannerControlServiceClient<InterceptedService<AuthenticatedChannel, TonicInterceptor>>> {
     let interceptor = interceptor.with_rpc_audience(addr)?;
-    let channel = get_channel_for_class(addr, ChannelClass::Control)
-        .await
-        .map_err(|err| crate::error::Error::other(err.to_string()))?;
+    let channel = match runtime_sources::cached_node_channel(addr).await {
+        Some(channel) => channel,
+        None => create_new_channel(addr)
+            .await
+            .map_err(|err| crate::error::Error::other(err.to_string()))?,
+    };
     let channel = ReplayScopeChannel::new(channel, interceptor.replay_scope_audience());
     let limit = rustfs_protos::scoped_dirty_usage::SCOPED_DIRTY_USAGE_MAX_REQUEST_BYTES as usize;
     Ok(ScannerControlServiceClient::with_interceptor(channel, interceptor)
@@ -82,7 +84,10 @@ pub async fn heal_control_time_out_client(
     interceptor: TonicInterceptor,
 ) -> Result<HealControlServiceClient<InterceptedService<AuthenticatedChannel, TonicInterceptor>>, Box<dyn Error>> {
     let interceptor = interceptor.with_rpc_audience(addr)?;
-    let channel = get_channel_for_class(addr, ChannelClass::Control).await?;
+    let channel = match runtime_sources::cached_node_channel(addr).await {
+        Some(channel) => channel,
+        None => create_new_channel(addr).await?,
+    };
     let max_message_size = rustfs_protos::HEAL_CONTROL_RPC_MAX_MESSAGE_SIZE;
     let channel = ReplayScopeChannel::new(channel, interceptor.replay_scope_audience());
     Ok(HealControlServiceClient::with_interceptor(channel, interceptor)
@@ -95,7 +100,10 @@ pub async fn tier_mutation_control_time_out_client(
     interceptor: TonicInterceptor,
 ) -> Result<TierMutationControlServiceClient<InterceptedService<AuthenticatedChannel, TonicInterceptor>>, Box<dyn Error>> {
     let interceptor = interceptor.with_rpc_audience(addr)?;
-    let channel = get_channel_for_class(addr, ChannelClass::Control).await?;
+    let channel = match runtime_sources::cached_node_channel(addr).await {
+        Some(channel) => channel,
+        None => create_new_channel(addr).await?,
+    };
     let max_message_size = rustfs_protos::TIER_MUTATION_RPC_MAX_MESSAGE_SIZE;
     let channel = ReplayScopeChannel::new(channel, interceptor.replay_scope_audience());
     Ok(TierMutationControlServiceClient::with_interceptor(channel, interceptor)
@@ -110,12 +118,22 @@ pub async fn tier_mutation_control_time_out_client(
 /// from lock/health RPCs; everything else uses [`ChannelClass::Control`]. When isolation is
 /// disabled the two classes resolve to the same cached channel, i.e. legacy behavior.
 pub async fn node_service_time_out_client_for_class(
-    addr: &str,
+    addr: &String,
     interceptor: TonicInterceptor,
     class: ChannelClass,
 ) -> Result<NodeServiceClient<InterceptedService<AuthenticatedChannel, TonicInterceptor>>, Box<dyn Error>> {
     let interceptor = interceptor.with_rpc_audience(addr)?;
-    let channel = get_channel_for_class(addr, class).await?;
+    let channel = match class {
+        ChannelClass::Control => match runtime_sources::cached_node_channel(addr).await {
+            Some(channel) => {
+                debug!("Using cached gRPC channel for: {}", addr);
+                channel
+            }
+            // No cached connection, create new one.
+            None => create_new_channel(addr).await?,
+        },
+        ChannelClass::Bulk => get_channel_for_class(addr, ChannelClass::Bulk).await?,
+    };
 
     let max_message_size = rustfs_protos::internode_rpc_max_message_size();
     let channel = ReplayScopeChannel::new(channel, interceptor.replay_scope_audience());
@@ -124,24 +142,8 @@ pub async fn node_service_time_out_client_for_class(
         .max_encoding_message_size(max_message_size))
 }
 
-pub(crate) async fn node_service_delete_recovery_client(
-    addr: &str,
-    interceptor: TonicInterceptor,
-    timeout: std::time::Duration,
-) -> Result<NodeServiceClient<InterceptedService<AuthenticatedChannel, TonicInterceptor>>, DiskError> {
-    let interceptor = interceptor.with_rpc_audience(addr).map_err(DiskError::from)?;
-    let channel = rustfs_protos::create_channel_with_timeout(addr, timeout)
-        .await
-        .map_err(|error| DiskError::RemoteClientUnavailable(error.to_string()))?;
-    let channel = ReplayScopeChannel::new(channel, interceptor.replay_scope_audience());
-    let max_message_size = rustfs_protos::internode_rpc_max_message_size();
-    Ok(NodeServiceClient::with_interceptor(channel, interceptor)
-        .max_decoding_message_size(max_message_size)
-        .max_encoding_message_size(max_message_size))
-}
-
 pub async fn node_service_time_out_client_no_auth(
-    addr: &str,
+    addr: &String,
 ) -> Result<NodeServiceClient<InterceptedService<AuthenticatedChannel, TonicInterceptor>>, Box<dyn Error>> {
     node_service_time_out_client(addr, TonicInterceptor::NoOp(NoOpInterceptor)).await
 }
@@ -353,13 +355,6 @@ fn apply_peer_replay_response(
     }
 }
 
-fn rename_data_grpc_stage(path: &str) -> bool {
-    matches!(
-        path,
-        "/node_service.NodeService/RenameData" | "/node_service.NodeService/RenameDataAtIncarnation"
-    )
-}
-
 impl<S, ReqBody, ResBody> Service<HttpRequest<ReqBody>> for ReplayScopeChannel<S>
 where
     S: Service<HttpRequest<ReqBody>, Response = HttpResponse<ResBody>>,
@@ -377,12 +372,6 @@ where
     }
 
     fn call(&mut self, mut request: HttpRequest<ReqBody>) -> Self::Future {
-        let observe_rename_data = rustfs_io_metrics::put_stage_metrics_enabled() && rename_data_grpc_stage(request.uri().path());
-        let request_scope_started = if observe_rename_data {
-            rustfs_io_metrics::put_stage_timer()
-        } else {
-            None
-        };
         let authenticated = self.audience.as_ref().is_some_and(|_| {
             request
                 .headers()
@@ -417,30 +406,11 @@ where
                 }
             }
         }
-        rustfs_io_metrics::record_put_object_stage_duration_from(
-            rustfs_io_metrics::PUT_STAGE_SET_DISK_RENAME_REMOTE_CLIENT_REQUEST_SCOPE,
-            request_scope_started,
-        );
 
         let audience = self.audience.clone();
         let future = self.inner.call(request);
         Box::pin(async move {
-            let response = if observe_rename_data {
-                rustfs_io_metrics::observe_put_stage_future(
-                    future,
-                    rustfs_io_metrics::PUT_STAGE_SET_DISK_RENAME_REMOTE_CLIENT_TRANSPORT_CALL,
-                    rustfs_io_metrics::PUT_STAGE_SET_DISK_RENAME_REMOTE_CLIENT_TRANSPORT_CALL_POLL_PENDING_COUNT,
-                    rustfs_io_metrics::PUT_STAGE_SET_DISK_RENAME_REMOTE_CLIENT_TRANSPORT_CALL_FIRST_PENDING_TO_READY,
-                )
-                .await?
-            } else {
-                future.await?
-            };
-            let replay_response_started = if observe_rename_data {
-                rustfs_io_metrics::put_stage_timer()
-            } else {
-                None
-            };
+            let response = future.await?;
             if let (Some(audience), Some(challenge)) = (audience, challenge) {
                 let response_state = verify_tonic_peer_replay_capabilities_response(&audience, challenge, response.headers());
                 if let Err(error) = &response_state
@@ -458,10 +428,6 @@ where
                 }
                 apply_peer_replay_response(audience, sent_state, response_state);
             }
-            rustfs_io_metrics::record_put_object_stage_duration_from(
-                rustfs_io_metrics::PUT_STAGE_SET_DISK_RENAME_REMOTE_CLIENT_REPLAY_RESPONSE,
-                replay_response_started,
-            );
             Ok(response)
         })
     }
@@ -580,126 +546,6 @@ mod tests {
     use tracing_opentelemetry::OpenTelemetrySpanExt;
     use tracing_subscriber::{Registry, layer::SubscriberExt};
 
-    #[tokio::test]
-    async fn delete_recovery_channel_uses_its_scoped_deadline() {
-        use bytes::Bytes;
-        use http_body_util::StreamBody;
-        use hyper::{body::Frame, service::service_fn};
-        use hyper_util::rt::{TokioExecutor, TokioIo};
-        use rustfs_protos::proto_gen::node_service::{RecoverDeleteTransactionRequest, RecoverDeleteTransactionResponse};
-        use tonic_prost::prost::Message;
-        let listener = tokio::net::TcpListener::bind("127.0.0.1:0")
-            .await
-            .expect("bind recovery peer");
-        let addr = format!("http://{}", listener.local_addr().expect("peer address"));
-        let server = tokio::spawn(async move {
-            loop {
-                let (stream, _) = listener.accept().await.expect("accept recovery connection");
-                tokio::spawn(async move {
-                    let service = service_fn(|_request: HttpRequest<hyper::body::Incoming>| async {
-                        tokio::time::sleep(std::time::Duration::from_millis(500)).await;
-                        let payload = RecoverDeleteTransactionResponse {
-                            success: true,
-                            protocol_version: 1,
-                            result: Bytes::new(),
-                            error: None,
-                        }
-                        .encode_to_vec();
-                        let mut data = vec![0];
-                        data.extend_from_slice(&u32::try_from(payload.len()).expect("small response").to_be_bytes());
-                        data.extend_from_slice(&payload);
-                        let mut trailers = http::HeaderMap::new();
-                        trailers.insert("grpc-status", "0".parse().expect("status"));
-                        let frames: Vec<Result<Frame<Bytes>, std::convert::Infallible>> =
-                            vec![Ok(Frame::data(data.into())), Ok(Frame::trailers(trailers))];
-                        let mut response = HttpResponse::new(StreamBody::new(futures::stream::iter(frames)));
-                        response
-                            .headers_mut()
-                            .insert("content-type", "application/grpc".parse().expect("content type"));
-                        Ok::<_, std::convert::Infallible>(response)
-                    });
-                    let _ = hyper::server::conn::http2::Builder::new(TokioExecutor::new())
-                        .serve_connection(TokioIo::new(stream), service)
-                        .await;
-                });
-            }
-        });
-        let request = || RecoverDeleteTransactionRequest {
-            disk: "disk".into(),
-            volume: "bucket".into(),
-            protocol_version: 1,
-            operation: Bytes::new(),
-        };
-        // Seed an existing short-lived control channel to reproduce the inherited deadline.
-        let control = tonic::transport::Endpoint::from_shared(addr.clone())
-            .expect("test endpoint")
-            .timeout(std::time::Duration::from_millis(50))
-            .connect()
-            .await
-            .expect("control connect");
-        rustfs_common::cache_connection(addr.clone(), control).await;
-        let mut foreground = node_service_time_out_client(&addr, TonicInterceptor::NoOp(NoOpInterceptor))
-            .await
-            .expect("control client");
-        let error = foreground
-            .recover_delete_transaction(request())
-            .await
-            .expect_err("short control deadline");
-        assert!(matches!(error.code(), tonic::Code::Cancelled | tonic::Code::DeadlineExceeded));
-        // The 2-second test override bypasses the production range parser, which rejects 2.
-        let mut recovery = node_service_delete_recovery_client(
-            &addr,
-            TonicInterceptor::NoOp(NoOpInterceptor),
-            std::time::Duration::from_secs(2),
-        )
-        .await
-        .expect("scoped recovery client");
-        assert!(
-            recovery
-                .recover_delete_transaction(request())
-                .await
-                .expect("scoped deadline must survive control deadline")
-                .into_inner()
-                .success
-        );
-        server.abort();
-        rustfs_protos::evict_failed_connection(&addr).await;
-    }
-
-    #[tokio::test]
-    async fn rpc_client_factories_do_not_reuse_https_channels_without_tls_generation() {
-        let listener = std::net::TcpListener::bind("127.0.0.1:0").expect("reserve local address");
-        let addr = format!("https://{}", listener.local_addr().expect("local address"));
-        drop(listener);
-        for factory in 0..4 {
-            let placeholder = tonic::transport::Endpoint::from_static("http://127.0.0.1:1").connect_lazy();
-            runtime_sources::cache_test_node_channel(addr.clone(), placeholder).await;
-            let interceptor = TonicInterceptor::NoOp(NoOpInterceptor);
-            let _result = match factory {
-                0 => scanner_control_time_out_client(&addr, interceptor)
-                    .await
-                    .map(|_| ())
-                    .map_err(|e| e.to_string()),
-                1 => heal_control_time_out_client(&addr, interceptor)
-                    .await
-                    .map(|_| ())
-                    .map_err(|e| e.to_string()),
-                2 => tier_mutation_control_time_out_client(&addr, interceptor)
-                    .await
-                    .map(|_| ())
-                    .map_err(|e| e.to_string()),
-                _ => node_service_time_out_client(&addr, interceptor)
-                    .await
-                    .map(|_| ())
-                    .map_err(|e| e.to_string()),
-            };
-            assert!(
-                !runtime_sources::test_node_channel_is_cached(&addr).await,
-                "factory {factory} must evict an unbound HTTPS cache entry before attempting a fresh dial"
-            );
-        }
-    }
-
     #[derive(Clone)]
     struct EpochProofService {
         audience: String,

crates/ecstore/src/cluster/rpc/network_probe.rs

@@ -289,9 +289,13 @@ async fn client(
     >,
     NetworkPeerProbeError,
 > {
-    node_service_time_out_client_for_class(address, TonicInterceptor::Signature(gen_tonic_signature_interceptor()), class)
-        .await
-        .map_err(|_| NetworkPeerProbeError::Unreachable)
+    node_service_time_out_client_for_class(
+        &address.to_owned(),
+        TonicInterceptor::Signature(gen_tonic_signature_interceptor()),
+        class,
+    )
+    .await
+    .map_err(|_| NetworkPeerProbeError::Unreachable)
 }
 
 fn ping_request(payload: &[u8]) -> PingRequest {
@@ -325,24 +329,6 @@ fn map_status(status: tonic::Status) -> NetworkPeerProbeError {
     match status.code() {
         Code::Unavailable | Code::Unknown => NetworkPeerProbeError::Unreachable,
         Code::DeadlineExceeded => NetworkPeerProbeError::TimedOut,
-        Code::Cancelled | Code::Internal => {
-            let mut source = std::error::Error::source(&status);
-            while let Some(error) = source {
-                if error
-                    .downcast_ref::<h2::Error>()
-                    .is_some_and(|error| matches!(error.reason(), Some(h2::Reason::CANCEL | h2::Reason::NO_ERROR)))
-                {
-                    return NetworkPeerProbeError::Unreachable;
-                }
-                source = if let Some(error) = error.downcast_ref::<std::io::Error>() {
-                    // io::Error::source skips its immediate custom error.
-                    error.get_ref().map(|inner| inner as &(dyn std::error::Error + 'static))
-                } else {
-                    error.source()
-                };
-            }
-            NetworkPeerProbeError::ProtocolFailure
-        }
         _ => NetworkPeerProbeError::ProtocolFailure,
     }
 }
@@ -353,35 +339,6 @@ mod tests {
     use crate::layout::endpoint::Endpoint;
     use crate::layout::endpoints::{Endpoints, PoolEndpoints};
 
-    #[test]
-    fn peer_transport_resets_are_unreachable_without_reclassifying_application_errors() {
-        for reason in [h2::Reason::CANCEL, h2::Reason::NO_ERROR] {
-            let status = tonic::Status::from_error(Box::new(h2::Error::from(reason)));
-            assert!(matches!(status.code(), Code::Cancelled | Code::Internal));
-            let code = status.code();
-            assert_eq!(map_status(status), NetworkPeerProbeError::Unreachable);
-            let mut wrapped = tonic::Status::new(code, "wrapped transport reset");
-            wrapped.set_source(Arc::new(std::io::Error::other(h2::Error::from(reason))));
-            assert_eq!(map_status(wrapped), NetworkPeerProbeError::Unreachable);
-        }
-        for reason in [h2::Reason::PROTOCOL_ERROR, h2::Reason::COMPRESSION_ERROR] {
-            let status = tonic::Status::from_error(Box::new(h2::Error::from(reason)));
-            assert_eq!(map_status(status), NetworkPeerProbeError::ProtocolFailure);
-        }
-        for status in [
-            tonic::Status::cancelled("peer cancelled the application request"),
-            tonic::Status::internal("peer rejected the probe"),
-            tonic::Status::unauthenticated("authentication rejected"),
-            tonic::Status::permission_denied("authorization rejected"),
-        ] {
-            assert_eq!(map_status(status), NetworkPeerProbeError::ProtocolFailure);
-        }
-        assert_eq!(
-            map_status(tonic::Status::deadline_exceeded("probe deadline elapsed")),
-            NetworkPeerProbeError::TimedOut
-        );
-    }
-
     fn pacing(duration: Duration) -> Arc<DiagnosticPacing> {
         NetworkPeerProbeClient::from_endpoint_pools(&topology())
             .with_diagnostic_pacing(Instant::now(), duration)

crates/ecstore/src/cluster/rpc/remote_disk.rs

@@ -14,8 +14,7 @@
 
 use crate::cluster::rpc::client::{
     AuthenticatedChannel, TonicInterceptor, gen_tonic_signature_interceptor, is_network_like_disk_error,
-    node_service_delete_recovery_client, node_service_time_out_client, node_service_time_out_client_for_class,
-    node_service_time_out_client_no_auth,
+    node_service_time_out_client, node_service_time_out_client_for_class, node_service_time_out_client_no_auth,
 };
 use crate::cluster::rpc::internode_data_transport::{
     InternodeDataTransport, NsScannerCapabilityRequest, NsScannerStreamRequest, ReadStreamRequest, WalkDirStreamRequest,
@@ -23,11 +22,10 @@ use crate::cluster::rpc::internode_data_transport::{
 };
 use crate::disk::error::{Error, Result};
 use crate::disk::{
-    BatchReadVersionReq, BatchReadVersionResp, CheckPartsResp, ConditionalFileUpdate, DELETE_RECOVERY_PROTOCOL_VERSION,
-    DeleteOptions, DeleteRecoveryAction, DeleteRecoveryCursor, DeleteRecoveryDigest, DeleteRecoveryInspection,
-    DeleteRecoveryPage, DeleteRecoveryRpcOperation, DeleteRecoveryRpcReply, DiskAPI, DiskInfo, DiskInfoOptions, DiskLocation,
-    DiskOption, FileInfoVersions, FileReader, FileWriter, PartTransactionAction, ReadMultipleReq, ReadMultipleResp, ReadOptions,
-    RenameDataResp, SnapshotLeaseToken, UpdateMetadataOpts, VolumeInfo, WalkDirOptions, batch_read_version_one_by_one,
+    BatchReadVersionReq, BatchReadVersionResp, CheckPartsResp, ConditionalFileUpdate, DeleteOptions, DiskAPI, DiskInfo,
+    DiskInfoOptions, DiskLocation, DiskOption, FileInfoVersions, FileReader, FileWriter, PartTransactionAction, ReadMultipleReq,
+    ReadMultipleResp, ReadOptions, RenameDataResp, SnapshotLeaseToken, UpdateMetadataOpts, VolumeInfo, WalkDirOptions,
+    batch_read_version_one_by_one,
     disk_store::{
         DEFAULT_RUSTFS_DRIVE_ACTIVE_MONITORING, ENV_RUSTFS_DRIVE_ACTIVE_MONITORING, SKIP_IF_SUCCESS_BEFORE,
         get_drive_active_check_interval, get_drive_active_check_timeout, get_drive_disk_info_timeout, get_drive_list_dir_timeout,
@@ -61,10 +59,10 @@ use rustfs_protos::proto_gen::node_service::{
     CompareAndUpdateFileRequest, DeletePathsRequest, DeleteRequest, DeleteVersionRequest, DeleteVersionsRequest,
     DeleteVersionsResponse, DeleteVolumeRequest, DiskInfoRequest, ListDirRequest, ListVolumesRequest, MakeVolumeRequest,
     MakeVolumesRequest, PreparePartTransactionRequest, ReadAllRequest, ReadMetadataRequest, ReadMultipleRequest,
-    ReadMultipleResponse, ReadPartsRequest, ReadVersionRequest, ReadXlRequest, RecoverDeleteTransactionRequest,
-    RenameDataRequest, RenameFileRequest, SettlePartTransactionRequest, SnapshotLeaseReleaseRequest, SnapshotLeaseRenewRequest,
-    SnapshotLeaseRequest, SnapshotLeaseResponse, StatVolumeRequest, UpdateMetadataRequest, VerifyFileRequest, WriteAllRequest,
-    WriteMetadataRequest, node_service_client::NodeServiceClient,
+    ReadMultipleResponse, ReadPartsRequest, ReadVersionRequest, ReadXlRequest, RenameDataRequest, RenameFileRequest,
+    SettlePartTransactionRequest, SnapshotLeaseReleaseRequest, SnapshotLeaseRenewRequest, SnapshotLeaseRequest,
+    SnapshotLeaseResponse, StatVolumeRequest, UpdateMetadataRequest, VerifyFileRequest, WriteAllRequest, WriteMetadataRequest,
+    node_service_client::NodeServiceClient,
 };
 use serde::{Serialize, de::DeserializeOwned};
 use std::{
@@ -144,45 +142,6 @@ fn decode_delete_versions_errors(response: DeleteVersionsResponse, expected_len:
         .collect()
 }
 
-fn decode_delete_recovery_batch_errors(codes: Vec<Option<u32>>, expected_len: usize) -> Result<Vec<Option<DiskError>>> {
-    if codes.len() != expected_len {
-        return Err(Error::other("malformed recoverable batch delete response length"));
-    }
-    codes
-        .into_iter()
-        .map(|code| match code {
-            None => Ok(None),
-            Some(code) => DiskError::from_u32(code)
-                .map(Some)
-                .ok_or_else(|| Error::other("invalid recoverable batch delete error code")),
-        })
-        .collect()
-}
-
-fn validate_delete_recovery_inspection(
-    volume: &str,
-    object: &str,
-    transaction: Uuid,
-    inspection: &DeleteRecoveryInspection,
-) -> Result<()> {
-    if inspection.protocol_version != DELETE_RECOVERY_PROTOCOL_VERSION
-        || inspection.id.object != object
-        || inspection.id.transaction != transaction
-        || transaction.is_nil()
-    {
-        return Err(Error::other("delete recovery inspection does not match request"));
-    }
-    if let Some(journal) = &inspection.journal
-        && (journal.protocol_version != DELETE_RECOVERY_PROTOCOL_VERSION
-            || journal.volume != volume
-            || journal.object != object
-            || journal.transaction != transaction)
-    {
-        return Err(Error::other("delete recovery journal does not match request"));
-    }
-    Ok(())
-}
-
 fn snapshot_lease_token_from_response(response: SnapshotLeaseResponse) -> Result<SnapshotLeaseToken> {
     if !response.success {
         return Err(response.error.unwrap_or_default().into());
@@ -899,75 +858,6 @@ fn spawn_control_channel_prewarm(addr: String) {
 }
 
 impl RemoteDisk {
-    async fn recover_delete_transaction(
-        &self,
-        volume: &str,
-        operation: DeleteRecoveryRpcOperation,
-    ) -> Result<DeleteRecoveryRpcReply> {
-        let payload = rmp_serde::to_vec_named(&operation)?;
-        if payload.len() > operation.max_wire_payload_bytes() {
-            return Err(Error::other("delete recovery operation exceeds payload limit"));
-        }
-        let settle_timeout = if matches!(&operation, DeleteRecoveryRpcOperation::Settle { .. }) {
-            Some(crate::disk::get_delete_recovery_timeout()?)
-        } else {
-            None
-        };
-        self.execute_with_timeout(
-            || async {
-                let mut client = if let Some(timeout) = settle_timeout {
-                    if let Some(error) = self.offline_bypass_error() {
-                        return Err(error);
-                    }
-                    node_service_delete_recovery_client(
-                        &self.addr,
-                        TonicInterceptor::Signature(gen_tonic_signature_interceptor()),
-                        timeout,
-                    )
-                    .await?
-                } else {
-                    self.get_client().await?
-                };
-                let mut request = Request::new(RecoverDeleteTransactionRequest {
-                    disk: self.endpoint.to_string(),
-                    volume: volume.to_owned(),
-                    protocol_version: DELETE_RECOVERY_PROTOCOL_VERSION,
-                    operation: payload.into(),
-                });
-                let canonical_body = rustfs_protos::canonical_recover_delete_transaction_request_body(request.get_ref())
-                    .map_err(|_| Error::other("recover_delete_transaction request length cannot be represented"))?;
-                crate::cluster::rpc::set_tonic_canonical_body_digest(&mut request, &canonical_body).map_err(Error::other)?;
-                // UNIMPLEMENTED is deliberately propagated: older peers cannot settle transactions.
-                let response = client
-                    .recover_delete_transaction(request)
-                    .await
-                    .map_err(|status| {
-                        if status.code() == Code::Unimplemented {
-                            DiskError::UnsupportedDisk
-                        } else {
-                            status.into()
-                        }
-                    })?
-                    .into_inner();
-                if response.protocol_version != DELETE_RECOVERY_PROTOCOL_VERSION {
-                    return Err(DiskError::UnsupportedDisk);
-                }
-                if !response.success {
-                    return Err(response
-                        .error
-                        .ok_or_else(|| Error::other("missing delete recovery error"))?
-                        .into());
-                }
-                if response.error.is_some() {
-                    return Err(Error::other("malformed delete recovery success"));
-                }
-                rustfs_protos::decode_delete_recovery_payload(&response.result).map_err(Into::into)
-            },
-            settle_timeout.unwrap_or_else(get_max_timeout_duration),
-        )
-        .await
-    }
-
     async fn rename_file_with_durability(
         &self,
         src_volume: &str,
@@ -2030,12 +1920,6 @@ fn encode_msgpack<T: Serialize>(value: &T) -> Result<Vec<u8>> {
     encode_msgpack_with_capacity(value, MSGPACK_ENCODE_CAPACITY_HINT)
 }
 
-// Optional fields are omitted by serde; named maps keep rollback flags and
-// transaction UUIDs independent of field order across peer versions.
-fn encode_delete_options_msgpack(value: &DeleteOptions) -> Result<Vec<u8>> {
-    Ok(rmp_serde::to_vec_named(value)?)
-}
-
 fn encode_file_info_msgpack(value: &FileInfo) -> Result<Vec<u8>> {
     encode_msgpack_with_capacity(value, FILE_INFO_MSGPACK_ENCODE_CAPACITY_HINT)
 }
@@ -2279,7 +2163,6 @@ impl RemoteDisk {
         self.execute_with_timeout_for_op(
             "rename_data",
             || async {
-                let prepare_started = rustfs_io_metrics::put_stage_timer();
                 let file_info = compat_json(fi)?;
                 let file_info_bin = encode_file_info_msgpack(fi)?;
                 let mut client = self.get_client().await?;
@@ -2307,67 +2190,24 @@ impl RemoteDisk {
                 } else {
                     attach_mutation_body_digest(&mut request, canonical_body, "rename_data")?;
                 }
-                rustfs_io_metrics::record_put_object_stage_duration_from(
-                    rustfs_io_metrics::PUT_STAGE_SET_DISK_RENAME_REMOTE_CLIENT_PREPARE,
-                    prepare_started,
-                );
 
-                let rpc_started = rustfs_io_metrics::put_stage_timer();
-                let response = rustfs_io_metrics::observe_put_stage_future(
-                    async {
-                        if incarnation_bound {
-                            // Older peers return Unimplemented before mutation; never downgrade.
-                            client.rename_data_at_incarnation(request).await
-                        } else {
-                            client.rename_data(request).await
-                        }
-                    },
-                    rustfs_io_metrics::PUT_STAGE_SET_DISK_RENAME_REMOTE_CLIENT_RPC_AWAIT,
-                    rustfs_io_metrics::PUT_STAGE_SET_DISK_RENAME_REMOTE_CLIENT_RPC_AWAIT_POLL_PENDING_COUNT,
-                    rustfs_io_metrics::PUT_STAGE_SET_DISK_RENAME_REMOTE_CLIENT_RPC_AWAIT_FIRST_PENDING_TO_READY,
-                )
-                .await?;
-                let response_process_started = rustfs_io_metrics::put_stage_timer();
-                let response_headers_started = rustfs_io_metrics::put_stage_timer();
-                let _ = response.metadata().len();
-                rustfs_io_metrics::record_put_object_stage_duration_from(
-                    rustfs_io_metrics::PUT_STAGE_SET_DISK_RENAME_REMOTE_CLIENT_RESPONSE_HEADERS,
-                    response_headers_started,
-                );
-                let into_inner_started = rustfs_io_metrics::put_stage_timer();
-                let response = response.into_inner();
-                rustfs_io_metrics::record_put_object_stage_duration_from(
-                    rustfs_io_metrics::PUT_STAGE_SET_DISK_RENAME_REMOTE_CLIENT_INTO_INNER,
-                    into_inner_started,
-                );
-                rustfs_io_metrics::record_put_object_stage_duration_from(
-                    rustfs_io_metrics::PUT_STAGE_SET_DISK_RENAME_REMOTE_CLIENT_RPC,
-                    rpc_started,
-                );
+                let response = if incarnation_bound {
+                    // Older peers return Unimplemented before mutation; never downgrade.
+                    client.rename_data_at_incarnation(request).await?
+                } else {
+                    client.rename_data(request).await?
+                }
+                .into_inner();
 
-                let success_check_started = rustfs_io_metrics::put_stage_timer();
                 if !response.success {
                     return Err(response.error.unwrap_or_default().into());
                 }
-                rustfs_io_metrics::record_put_object_stage_duration_from(
-                    rustfs_io_metrics::PUT_STAGE_SET_DISK_RENAME_REMOTE_CLIENT_SUCCESS_CHECK,
-                    success_check_started,
-                );
 
-                let decode_started = rustfs_io_metrics::put_stage_timer();
                 let rename_data_resp = decode_msgpack_or_json::<RenameDataResp>(
                     &response.rename_data_resp_bin,
                     &response.rename_data_resp,
                     "RenameDataResp",
                 )?;
-                rustfs_io_metrics::record_put_object_stage_duration_from(
-                    rustfs_io_metrics::PUT_STAGE_SET_DISK_RENAME_REMOTE_CLIENT_DECODE,
-                    decode_started,
-                );
-                rustfs_io_metrics::record_put_object_stage_duration_from(
-                    rustfs_io_metrics::PUT_STAGE_SET_DISK_RENAME_REMOTE_CLIENT_RESPONSE_PROCESS,
-                    response_process_started,
-                );
 
                 Ok(rename_data_resp)
             },
@@ -2726,7 +2566,7 @@ impl DiskAPI for RemoteDisk {
                 // `_bin` support for DeleteVersion is new (grpc-optimization P2); always dual-write
                 // JSON + msgpack until its fallback counter has read zero across a release window.
                 let file_info_bin = encode_file_info_msgpack(&fi)?;
-                let opts_bin = encode_delete_options_msgpack(&opts)?;
+                let opts_bin = encode_msgpack(&opts)?;
                 let conditional_marker = opts.expected_delete_marker.is_some();
                 let file_info = serde_json::to_string(&fi)?;
                 let opts = serde_json::to_string(&opts)?;
@@ -2798,7 +2638,7 @@ impl DiskAPI for RemoteDisk {
 
         // `_bin` support for DeleteVersions is new (grpc-optimization P2); always dual-write JSON +
         // msgpack until its fallback counter has read zero across a release window.
-        let opts_bin = match encode_delete_options_msgpack(&opts) {
+        let opts_bin = match encode_msgpack(&opts) {
             Ok(opts_bin) => opts_bin,
             Err(err) => {
                 let mut errors = Vec::with_capacity(versions.len());
@@ -3777,138 +3617,6 @@ impl DiskAPI for RemoteDisk {
     }
 
     #[tracing::instrument(level = "trace", skip_all)]
-    async fn delete_version_recoverable(
-        &self,
-        volume: &str,
-        object: &str,
-        file_info: FileInfo,
-        force_del_marker: bool,
-        opts: DeleteOptions,
-    ) -> Result<()> {
-        match self
-            .recover_delete_transaction(
-                volume,
-                DeleteRecoveryRpcOperation::ApplyVersion {
-                    object: object.to_owned(),
-                    file_info: Box::new(file_info),
-                    force_del_marker,
-                    opts,
-                },
-            )
-            .await?
-        {
-            DeleteRecoveryRpcReply::AppliedVersion => Ok(()),
-            _ => Err(Error::other("unexpected recoverable delete reply")),
-        }
-    }
-
-    async fn delete_versions_recoverable(
-        &self,
-        volume: &str,
-        versions: Vec<FileInfoVersions>,
-        opts: DeleteOptions,
-    ) -> Vec<Option<DiskError>> {
-        let expected_len = versions.len();
-        let result = match self
-            .recover_delete_transaction(volume, DeleteRecoveryRpcOperation::ApplyVersions { versions, opts })
-            .await
-        {
-            Ok(DeleteRecoveryRpcReply::AppliedVersions(codes)) => decode_delete_recovery_batch_errors(codes, expected_len),
-            Ok(_) => Err(Error::other("unexpected recoverable batch delete reply")),
-            Err(error) => Err(error),
-        };
-        match result {
-            Ok(errors) => errors,
-            Err(error) => vec![Some(error); expected_len],
-        }
-    }
-
-    async fn discover_delete_transactions(
-        &self,
-        volume: &str,
-        object: Option<&str>,
-        cursor: Option<DeleteRecoveryCursor>,
-        limit: u32,
-    ) -> Result<DeleteRecoveryPage> {
-        match self
-            .recover_delete_transaction(
-                volume,
-                DeleteRecoveryRpcOperation::Discover {
-                    object: object.map(str::to_owned),
-                    cursor,
-                    limit,
-                },
-            )
-            .await?
-        {
-            DeleteRecoveryRpcReply::Discovered(page) => {
-                if object.is_some_and(|object| page.transactions.iter().any(|id| id.object != object)) {
-                    return Err(Error::other("delete recovery discovery escaped requested object"));
-                }
-                if page.transactions.len() > usize::try_from(limit).map_err(|_| Error::other("invalid discovery limit"))?
-                    || page
-                        .transactions
-                        .iter()
-                        .any(|id| id.transaction.is_nil() || id.object.is_empty())
-                {
-                    return Err(Error::other("malformed delete discovery page"));
-                }
-                Ok(page)
-            }
-            _ => Err(Error::other("unexpected delete discovery reply")),
-        }
-    }
-    async fn inspect_delete_transaction(
-        &self,
-        volume: &str,
-        object: &str,
-        transaction: Uuid,
-    ) -> Result<DeleteRecoveryInspection> {
-        match self
-            .recover_delete_transaction(
-                volume,
-                DeleteRecoveryRpcOperation::Inspect {
-                    object: object.to_owned(),
-                    transaction,
-                },
-            )
-            .await?
-        {
-            DeleteRecoveryRpcReply::Inspected(inspection) => {
-                validate_delete_recovery_inspection(volume, object, transaction, &inspection)?;
-                Ok(inspection)
-            }
-            _ => Err(Error::other("unexpected delete inspection reply")),
-        }
-    }
-    async fn settle_delete_transaction(
-        &self,
-        volume: &str,
-        object: &str,
-        transaction: Uuid,
-        expected_fingerprint: DeleteRecoveryDigest,
-        action: DeleteRecoveryAction,
-    ) -> Result<DeleteRecoveryInspection> {
-        match self
-            .recover_delete_transaction(
-                volume,
-                DeleteRecoveryRpcOperation::Settle {
-                    object: object.to_owned(),
-                    transaction,
-                    expected_fingerprint,
-                    action,
-                },
-            )
-            .await?
-        {
-            DeleteRecoveryRpcReply::Settled(inspection) => {
-                validate_delete_recovery_inspection(volume, object, transaction, &inspection)?;
-                Ok(inspection)
-            }
-            _ => Err(Error::other("unexpected delete settlement reply")),
-        }
-    }
-
     async fn settle_part_transaction(&self, volume: &str, path: &str, action: PartTransactionAction) -> Result<()> {
         self.execute_with_timeout(
             || async {
@@ -4322,45 +4030,6 @@ mod tests {
 
     static INIT: Once = Once::new();
 
-    #[test]
-    fn recoverable_batch_delete_rejects_invalid_error_codes_and_length() {
-        let errors = decode_delete_recovery_batch_errors(vec![None, Some(DiskError::FileNotFound.to_u32())], 2)
-            .expect("valid per-item errors");
-        assert!(errors[0].is_none());
-        assert_eq!(errors[1], Some(DiskError::FileNotFound));
-        assert!(decode_delete_recovery_batch_errors(vec![None], 2).is_err());
-        assert!(decode_delete_recovery_batch_errors(vec![Some(0)], 1).is_err());
-        assert!(decode_delete_recovery_batch_errors(vec![Some(u32::MAX)], 1).is_err());
-    }
-
-    #[test]
-    fn delete_recovery_inspection_rejects_wrong_transaction_object_and_protocol() {
-        let transaction = Uuid::new_v4();
-        let inspection = DeleteRecoveryInspection {
-            protocol_version: DELETE_RECOVERY_PROTOCOL_VERSION,
-            id: crate::disk::DeleteTransactionId {
-                object: "object".into(),
-                transaction,
-            },
-            state: crate::disk::DeleteRecoveryState::Absent,
-            journal: None,
-            current_digest: None,
-            current_versions: Vec::new(),
-            backup_digest: None,
-            rollback_data_complete: false,
-            commit_data_safe: false,
-            explicit_delete_safe: false,
-            fingerprint: [0; 32],
-        };
-        validate_delete_recovery_inspection("bucket", "object", transaction, &inspection).expect("matching inspection");
-        assert!(validate_delete_recovery_inspection("bucket", "other", transaction, &inspection).is_err());
-        assert!(validate_delete_recovery_inspection("bucket", "object", Uuid::new_v4(), &inspection).is_err());
-        assert!(validate_delete_recovery_inspection("bucket", "object", Uuid::nil(), &inspection).is_err());
-        let mut incompatible = inspection;
-        incompatible.protocol_version += 1;
-        assert!(validate_delete_recovery_inspection("bucket", "object", transaction, &incompatible).is_err());
-    }
-
     #[test]
     fn compare_and_update_wire_outcome_fails_closed() {
         assert_eq!(
@@ -4394,19 +4063,32 @@ mod tests {
     }
 
     #[test]
-    fn delete_options_named_wire_preserves_rollback_and_legacy_marker_fields() {
+    fn retired_marker_options_preserve_legacy_positional_wire_shape() {
         #[derive(Debug, serde::Serialize, serde::Deserialize, PartialEq)]
         struct LegacyDeleteOptions {
             recursive: bool,
             immediate: bool,
             undo_write: bool,
-            #[serde(default)]
             undo_delete: bool,
             old_data_dir: Option<Uuid>,
-            #[serde(default, skip_serializing_if = "Option::is_none")]
-            expected_delete_marker: Option<rustfs_filemeta::MetaDeleteMarker>,
         }
-        let transaction = Uuid::new_v4();
+        let legacy = LegacyDeleteOptions {
+            recursive: false,
+            immediate: false,
+            undo_write: false,
+            undo_delete: false,
+            old_data_dir: None,
+        };
+        let original = encode_msgpack(&legacy).unwrap();
+        let current = encode_msgpack(&DeleteOptions::default()).unwrap();
+        assert_eq!(current, original, "ordinary deletes must retain the older peer's positional payload");
+        assert_eq!(rmp_serde::from_slice::<LegacyDeleteOptions>(&current).unwrap(), legacy);
+        assert!(
+            rmp_serde::from_slice::<DeleteOptions>(&original)
+                .unwrap()
+                .expected_delete_marker
+                .is_none()
+        );
         let mut marker = FileInfo {
             deleted: true,
             version_id: Some(Uuid::new_v4()),
@@ -4415,69 +4097,12 @@ mod tests {
         };
         marker.set_delete_marker_incarnation(Uuid::new_v4());
         let marker = rustfs_filemeta::MetaDeleteMarker::from(marker);
-        for expected_delete_marker in [None, Some(marker)] {
-            for (undo_write, undo_delete) in [(true, false), (false, true)] {
-                let legacy = LegacyDeleteOptions {
-                    recursive: false,
-                    immediate: false,
-                    undo_write,
-                    undo_delete,
-                    old_data_dir: Some(transaction),
-                    expected_delete_marker: expected_delete_marker.clone(),
-                };
-                // Older senders use positional arrays. Their unchanged field prefix and
-                // missing optional fields must also decode on the named-map sender's peer.
-                for (encoding, payload) in [
-                    ("positional", rmp_serde::to_vec(&legacy).expect("encode legacy positional options")),
-                    ("named", rmp_serde::to_vec_named(&legacy).expect("encode legacy named options")),
-                ] {
-                    let decoded: DeleteOptions = rmp_serde::from_slice(&payload).expect("decode legacy options on current peer");
-                    assert_eq!(decoded.undo_write, undo_write, "{encoding} undo-write flag");
-                    assert_eq!(decoded.undo_delete, undo_delete, "{encoding} undo-delete flag");
-                    assert_eq!(decoded.old_data_dir, Some(transaction), "{encoding} transaction UUID");
-                    assert_eq!(decoded.expected_delete_marker, expected_delete_marker, "{encoding} marker precondition");
-                    assert!(decoded.delete_recovery_bucket_incarnation.is_none(), "{encoding} omitted incarnation");
-                    assert!(!decoded.recoverable_delete, "{encoding} omitted recovery flag");
-                }
-                for incarnation in [None, Some(Uuid::new_v4())] {
-                    let current = DeleteOptions {
-                        undo_write,
-                        undo_delete,
-                        old_data_dir: Some(transaction),
-                        expected_delete_marker: expected_delete_marker.clone(),
-                        delete_recovery_bucket_incarnation: incarnation,
-                        recoverable_delete: true,
-                        ..Default::default()
-                    };
-                    let payload = encode_delete_options_msgpack(&current).expect("encode current named options");
-                    let decoded: DeleteOptions = rmp_serde::from_slice(&payload).expect("decode current options on current peer");
-                    assert_eq!(decoded.undo_write, undo_write);
-                    assert_eq!(decoded.undo_delete, undo_delete);
-                    assert_eq!(decoded.old_data_dir, Some(transaction));
-                    assert_eq!(decoded.expected_delete_marker, expected_delete_marker);
-                    assert_eq!(decoded.delete_recovery_bucket_incarnation, incarnation);
-                    assert!(decoded.recoverable_delete);
-                    let decoded: LegacyDeleteOptions =
-                        rmp_serde::from_slice(&payload).expect("decode current options on legacy peer");
-                    assert_eq!(decoded, legacy, "unknown recovery fields cannot shift rollback or marker values");
-                }
-            }
-        }
-    }
-
-    #[test]
-    fn malformed_nonempty_delete_options_do_not_fall_back_to_json() {
         let options = DeleteOptions {
-            undo_write: true,
-            old_data_dir: Some(Uuid::new_v4()),
+            expected_delete_marker: Some(marker.clone()),
             ..Default::default()
         };
-        let json = serde_json::to_string(&options).expect("valid compatibility JSON");
-        assert!(decode_msgpack_or_json::<DeleteOptions>(&[0xc1], &json, "DeleteOptions").is_err());
-        let decoded: DeleteOptions =
-            decode_msgpack_or_json(&[], &json, "DeleteOptions").expect("absent MessagePack permits JSON compatibility");
-        assert!(decoded.undo_write);
-        assert_eq!(decoded.old_data_dir, options.old_data_dir);
+        let decoded: DeleteOptions = rmp_serde::from_slice(&encode_msgpack(&options).unwrap()).unwrap();
+        assert_eq!(decoded.expected_delete_marker, Some(marker));
     }
 
     #[test]
@@ -5160,8 +4785,6 @@ mod tests {
             cleanup_data_dir: Some(Uuid::new_v4()),
             sign: Some(vec![0x14, 0x35]),
             old_current_size: Some(crate::disk::OldCurrentSize::Present(64 * 1024)),
-            old_current_source_checked: false,
-            old_current_source: None,
         };
         let json = serde_json::to_string(&response).expect("legacy rename_data JSON response should encode");
         let decode_before = rustfs_io_metrics::internode_metrics::global_internode_metrics().msgpack_json_decode_total_for_test();
@@ -5564,8 +5187,6 @@ mod tests {
             cleanup_data_dir: Some(Uuid::new_v4()),
             sign: Some(vec![1_u8; 32]),
             old_current_size: Some(crate::disk::OldCurrentSize::Present(4096)),
-            old_current_source_checked: false,
-            old_current_source: None,
         };
         let json = serde_json::to_vec(&response).expect("rename data response json should encode");
         let named_msgpack = rmp_serde::encode::to_vec_named(&response).expect("rename data response named msgpack should encode");

crates/ecstore/src/core/pools.rs

@@ -3691,55 +3691,6 @@ fn activation_pool_meta_lock_error(err: rustfs_lock::LockError) -> Error {
     }
 }
 
-fn rebalance_worker_read_lock_error(err: rustfs_lock::LockError, object: &str) -> Error {
-    match err {
-        rustfs_lock::LockError::QuorumNotReached { required, achieved } => Error::NamespaceLockQuorumUnavailable {
-            mode: "read",
-            bucket: RUSTFS_META_BUCKET.to_string(),
-            object: object.to_string(),
-            required,
-            achieved,
-        },
-        other => Error::Lock(other),
-    }
-}
-
-/// Read-only admission to an already durable rebalance run. Keeping this
-/// separate from the activation writer prevents admission from authorizing a
-/// pool or rebalance metadata write under shared namespace locks.
-pub(crate) struct PoolRebalanceWorkerFence {
-    pool_meta_guard: rustfs_lock::NamespaceLockGuard,
-    rebalance_meta_guard: rustfs_lock::NamespaceLockGuard,
-    fleet_proof: Option<crate::services::notification_sys::CrossPoolFenceFleetProofToken>,
-}
-
-impl PoolRebalanceWorkerFence {
-    pub(crate) fn set_fleet_proof(
-        &mut self,
-        fleet_proof: Option<crate::services::notification_sys::CrossPoolFenceFleetProofToken>,
-    ) {
-        self.fleet_proof = fleet_proof;
-    }
-
-    pub(crate) fn ensure_held(&self) -> Result<()> {
-        if self.pool_meta_guard.is_lock_lost()
-            || self.pool_meta_guard.is_released()
-            || self.rebalance_meta_guard.is_lock_lost()
-            || self.rebalance_meta_guard.is_released()
-        {
-            return Err(Error::other("worker admission lock lost before rebalance activation"));
-        }
-        if self
-            .fleet_proof
-            .as_ref()
-            .is_some_and(|proof| !crate::services::notification_sys::cross_pool_fence_fleet_proof_matches(proof))
-        {
-            return Err(Error::other(POOL_ACTIVATION_FLEET_PROOF_EXPIRED));
-        }
-        Ok(())
-    }
-}
-
 pub(crate) struct PoolRebalanceActivationFence {
     pool_meta_guard: Arc<rustfs_lock::NamespaceLockGuard>,
     rebalance_meta_guard: Arc<rustfs_lock::NamespaceLockGuard>,
@@ -3811,35 +3762,6 @@ tokio::task_local! {
     pub(crate) static REBALANCE_ACTIVATION_LOCK_ATTEMPT: Arc<tokio::sync::Notify>;
 }
 
-pub(crate) async fn acquire_pool_rebalance_worker_locks<S>(pool: Arc<S>) -> Result<PoolRebalanceWorkerFence>
-where
-    S: crate::storage_api_contracts::namespace::NamespaceLocking<
-            Error = Error,
-            NamespaceLock = rustfs_lock::NamespaceLockWrapper,
-        >,
-{
-    // Match activation and entry admission: rebalance.bin -> pool.bin. Joining
-    // the same durable run must coexist with entries retaining their run fence.
-    let rebalance_meta_lock = pool.new_ns_lock(RUSTFS_META_BUCKET, REBAL_META_NAME).await?;
-    #[cfg(test)]
-    let _ = REBALANCE_ACTIVATION_LOCK_ATTEMPT.try_with(|attempted| attempted.notify_one());
-    let rebalance_meta_guard = rebalance_meta_lock
-        .get_read_lock(get_lock_acquire_timeout())
-        .await
-        .map_err(|err| rebalance_worker_read_lock_error(err, REBAL_META_NAME))?;
-    let pool_meta_lock = pool.new_ns_lock(RUSTFS_META_BUCKET, POOL_META_NAME).await?;
-    let pool_meta_guard = pool_meta_lock
-        .get_read_lock(get_lock_acquire_timeout())
-        .await
-        .map_err(|err| rebalance_worker_read_lock_error(err, POOL_META_NAME))?;
-
-    Ok(PoolRebalanceWorkerFence {
-        pool_meta_guard,
-        rebalance_meta_guard,
-        fleet_proof: None,
-    })
-}
-
 pub(crate) async fn acquire_pool_rebalance_activation_locks<S>(
     pool: Arc<S>,
     fleet_proof: Option<crate::services::notification_sys::CrossPoolFenceFleetProofToken>,
@@ -12508,18 +12430,6 @@ impl ECStore {
         Ok(pool_meta)
     }
 
-    pub(crate) async fn load_runtime_pool_meta_under_worker_fence(
-        &self,
-        write_state: &mut PoolMetaWriteState,
-        worker_fence: &PoolRebalanceWorkerFence,
-        operation: &str,
-    ) -> Result<PoolMeta> {
-        worker_fence.ensure_held()?;
-        let pool_meta = self.load_runtime_pool_meta_observing(write_state, operation).await?;
-        worker_fence.ensure_held()?;
-        Ok(pool_meta)
-    }
-
     pub(crate) async fn load_runtime_pool_meta_under_activation_fence(
         &self,
         write_state: &mut PoolMetaWriteState,

crates/ecstore/src/data_usage/mod.rs

@@ -738,12 +738,7 @@ where
         .filter(|etag| !etag.is_empty())
         .map(str::to_owned)
         .ok_or_else(|| Error::other("data usage snapshot has no ETag"))?;
-    let mut data_usage_info = parse_usage_snapshot(
-        &reader
-            .read_all()
-            .await
-            .map_err(|err| map_data_usage_metadata_read_error(err, object))?,
-    )?;
+    let mut data_usage_info = parse_usage_snapshot(&reader.read_all().await?)?;
     populate_backward_compatible_usage_maps(&mut data_usage_info);
     validate_complete_usage_snapshot(&mut data_usage_info);
     Ok(Some((data_usage_info, revision)))
@@ -3002,61 +2997,6 @@ mod tests {
         }
     }
 
-    #[derive(Debug)]
-    struct UsageSnapshotReadFailureStore;
-
-    struct UsageSnapshotFailureReader;
-
-    impl tokio::io::AsyncRead for UsageSnapshotFailureReader {
-        fn poll_read(
-            self: std::pin::Pin<&mut Self>,
-            _cx: &mut std::task::Context<'_>,
-            _buf: &mut tokio::io::ReadBuf<'_>,
-        ) -> std::task::Poll<std::io::Result<()>> {
-            std::task::Poll::Ready(Err(std::io::Error::other(Error::VolumeNotFound)))
-        }
-    }
-
-    #[async_trait::async_trait]
-    impl crate::storage_api_contracts::object::ObjectIO for UsageSnapshotReadFailureStore {
-        type Error = Error;
-        type RangeSpec = crate::storage_api_contracts::range::HTTPRangeSpec;
-        type HeaderMap = http::HeaderMap;
-        type ObjectOptions = ObjectOptions;
-        type ObjectInfo = ObjectInfo;
-        type GetObjectReader = crate::object_api::GetObjectReader;
-        type PutObjectReader = PutObjReader;
-
-        async fn get_object_reader(
-            &self,
-            _bucket: &str,
-            _object: &str,
-            _range: Option<Self::RangeSpec>,
-            _headers: Self::HeaderMap,
-            _opts: &Self::ObjectOptions,
-        ) -> Result<Self::GetObjectReader, Self::Error> {
-            Ok(crate::object_api::GetObjectReader {
-                stream: Box::new(UsageSnapshotFailureReader),
-                object_info: ObjectInfo {
-                    etag: Some("usage-revision".to_string()),
-                    ..Default::default()
-                },
-                buffered_body: None,
-                body_source: Default::default(),
-            })
-        }
-
-        async fn put_object(
-            &self,
-            _bucket: &str,
-            _object: &str,
-            _data: &mut Self::PutObjectReader,
-            _opts: &Self::ObjectOptions,
-        ) -> Result<Self::ObjectInfo, Self::Error> {
-            unimplemented!("the read-failure fixture does not write")
-        }
-    }
-
     /// Minimal ObjectIO backing `load_data_usage_cache` tests: records the keys
     /// read and fails the first N reads with a transient (non-absence) error.
     #[derive(Debug, Default)]
@@ -3221,20 +3161,6 @@ mod tests {
         }
     }
 
-    #[tokio::test]
-    async fn data_usage_removal_maps_missing_system_volume_during_snapshot_stream_read() {
-        let store = UsageSnapshotReadFailureStore;
-
-        let error = load_data_usage_for_bucket_removal(&store, "bucket-metadata/.usage.json")
-            .await
-            .expect_err("a system volume disappearing during body read must remain a retryable storage error");
-
-        assert_eq!(
-            error,
-            Error::InsufficientReadQuorum(RUSTFS_META_BUCKET.to_string(), "bucket-metadata/.usage.json".to_string())
-        );
-    }
-
     #[tokio::test]
     async fn load_data_usage_cache_retries_a_transient_failure() {
         let name = "usage-cache";

crates/ecstore/src/diagnostics/admin_server_info.rs

@@ -15,9 +15,7 @@
 use crate::cluster::rpc::{
     ScannerBucketListing, TonicInterceptor, gen_tonic_signature_interceptor, node_service_time_out_client,
 };
-use crate::data_usage::{
-    DATA_USAGE_CACHE_NAME, DATA_USAGE_ROOT, load_admin_data_usage_from_backend_cached as load_server_info_usage_snapshot,
-};
+use crate::data_usage::{DATA_USAGE_CACHE_NAME, DATA_USAGE_ROOT, load_data_usage_from_backend_cached};
 use crate::error::{Error, Result};
 use crate::{
     disk::endpoint::{Endpoint, EndpointType},
@@ -44,11 +42,7 @@ use std::{
 use time::OffsetDateTime;
 use tokio::time::timeout;
 use tonic::Request;
-use tracing::{error, trace, warn};
-
-const EVENT_SERVER_PING: &str = "server_ping";
-const LOG_COMPONENT_ECSTORE: &str = "ecstore";
-const LOG_SUBSYSTEM_DIAGNOSTICS: &str = "diagnostics";
+use tracing::warn;
 
 use shadow_rs::shadow;
 
@@ -163,22 +157,11 @@ async fn is_server_resolvable(endpoint: &Endpoint) -> Result<()> {
 
         let response: PingResponse = client.ping(request).await?.into_inner();
 
-        if flatbuffers::root::<PingBody>(&response.body).is_err() {
-            error!(
-                event = EVENT_SERVER_PING,
-                component = LOG_COMPONENT_ECSTORE,
-                subsystem = LOG_SUBSYSTEM_DIAGNOSTICS,
-                result = "invalid_response",
-                "Server ping response decoding failed"
-            );
+        let ping_response_body = flatbuffers::root::<PingBody>(&response.body);
+        if let Err(e) = ping_response_body {
+            eprintln!("{e}");
         } else {
-            trace!(
-                event = EVENT_SERVER_PING,
-                component = LOG_COMPONENT_ECSTORE,
-                subsystem = LOG_SUBSYSTEM_DIAGNOSTICS,
-                result = "success",
-                "Server ping completed"
-            );
+            println!("ping_resp:body(flatbuffer): {ping_response_body:?}");
         }
 
         Ok(())
@@ -311,7 +294,7 @@ pub async fn get_server_info(get_pools: bool) -> InfoMessage {
     if let Some(store) = runtime_sources::object_store_handle() {
         mode = ITEM_ONLINE;
         apply_data_usage_result(
-            load_server_info_usage_snapshot(store.clone()).await,
+            load_data_usage_from_backend_cached(store.clone()).await,
             &mut buckets,
             &mut objects,
             &mut versions,
@@ -984,93 +967,6 @@ mod tests {
         assert_eq!(usage.size, 6);
     }
 
-    #[tokio::test]
-    #[serial]
-    async fn server_info_reads_complete_observation_without_promoting_authority() {
-        use crate::bucket::metadata_sys::test_support::isolated_store_over_temp_disks;
-        use crate::config::com::{read_config, save_config};
-        use crate::data_usage::{
-            DATA_USAGE_OBJ_NAME_PATH, DATA_USAGE_OBSERVED_OBJ_NAME_PATH, invalidate_data_usage_snapshot_cache,
-            load_data_usage_from_backend_cached,
-        };
-        use rustfs_data_usage::{BucketUsageInfo, DataUsageInfo};
-        use std::time::{Duration, SystemTime};
-
-        let (_dirs, store) = isolated_store_over_temp_disks().await;
-        let primary = DataUsageInfo {
-            last_update: Some(SystemTime::UNIX_EPOCH + Duration::from_secs(1)),
-            scanner_epoch: Some(1),
-            usage_snapshot_bootstrap_pending: true,
-            usage_snapshot_converged: Some(false),
-            ..Default::default()
-        };
-        let observed = DataUsageInfo {
-            last_update: Some(SystemTime::UNIX_EPOCH + Duration::from_secs(2)),
-            scanner_epoch: Some(1),
-            scanner_cycle: Some(1),
-            buckets_count: 1,
-            objects_total_count: 3,
-            versions_total_count: 4,
-            delete_markers_total_count: 1,
-            objects_total_size: 6,
-            buckets_usage: [(
-                "bucket".to_string(),
-                BucketUsageInfo {
-                    objects_count: 3,
-                    versions_count: 4,
-                    delete_markers_count: 1,
-                    size: 6,
-                    ..Default::default()
-                },
-            )]
-            .into_iter()
-            .collect(),
-            usage_snapshot_complete: true,
-            usage_snapshot_converged: Some(false),
-            usage_snapshot_authoritative_baseline: Some(primary.snapshot_identity()),
-            ..Default::default()
-        };
-        let primary_bytes = serde_json::to_vec(&primary).expect("bootstrap should encode");
-        save_config(store.clone(), DATA_USAGE_OBJ_NAME_PATH.as_str(), primary_bytes.clone())
-            .await
-            .expect("bootstrap should persist");
-        save_config(
-            store.clone(),
-            DATA_USAGE_OBSERVED_OBJ_NAME_PATH.as_str(),
-            serde_json::to_vec(&observed).expect("observation should encode"),
-        )
-        .await
-        .expect("observation should persist");
-        invalidate_data_usage_snapshot_cache().await;
-
-        let displayed = super::load_server_info_usage_snapshot(store.clone())
-            .await
-            .expect("server info should load the complete observation");
-        assert_eq!(displayed.usage_snapshot_converged, Some(false));
-        let mut buckets = rustfs_madmin::Buckets::default();
-        let mut objects = rustfs_madmin::Objects::default();
-        let mut versions = rustfs_madmin::Versions::default();
-        let mut delete_markers = rustfs_madmin::DeleteMarkers::default();
-        let mut usage = rustfs_madmin::Usage::default();
-        apply_data_usage_result(Ok(displayed), &mut buckets, &mut objects, &mut versions, &mut delete_markers, &mut usage);
-        assert!(objects.error.is_none());
-        assert_eq!(
-            (buckets.count, objects.count, versions.count, delete_markers.count, usage.size),
-            (1, 3, 4, 1, 6)
-        );
-        let quota = load_data_usage_from_backend_cached(store.clone())
-            .await
-            .expect("quota should retain its bootstrap state");
-        assert!(!quota.is_complete_bucket_usage_snapshot());
-        assert_eq!(
-            read_config(store, DATA_USAGE_OBJ_NAME_PATH.as_str())
-                .await
-                .expect("primary should remain readable"),
-            primary_bytes
-        );
-        invalidate_data_usage_snapshot_cache().await;
-    }
-
     #[test]
     fn incomplete_data_usage_is_unavailable_in_server_info() {
         let mut buckets = rustfs_madmin::Buckets::default();

crates/ecstore/src/diagnostics/get.rs

@@ -182,7 +182,6 @@ pub(crate) const GET_METADATA_CACHE_REASON_DELETE_MARKER: &str = "delete_marker"
 pub(crate) const GET_METADATA_CACHE_REASON_DIST_ERASURE: &str = "dist_erasure";
 pub(crate) const GET_METADATA_CACHE_REASON_INCL_FREE_VERSIONS: &str = "incl_free_versions";
 pub(crate) const GET_METADATA_CACHE_REASON_INSUFFICIENT_CACHED_QUORUM: &str = "insufficient_cached_quorum";
-pub(crate) const GET_METADATA_CACHE_REASON_BODY_LAYOUT_REQUIRED: &str = "body_layout_required";
 pub(crate) const GET_METADATA_CACHE_REASON_META_BUCKET: &str = "meta_bucket";
 pub(crate) const GET_METADATA_CACHE_REASON_NO_LOCK: &str = "no_lock";
 pub(crate) const GET_METADATA_CACHE_REASON_NOT_FOUND_OR_EXPIRED: &str = "not_found_or_expired";

crates/ecstore/src/diagnostics/object_lock.rs

@@ -12,7 +12,6 @@
 // See the License for the specific language governing permissions and
 // limitations under the License.
 
-use rustfs_io_metrics::record_object_lock_acquire_failure;
 use rustfs_lock::{LockError, NamespaceLockGuard, NamespaceLockWrapper};
 use std::time::{Duration, Instant};
 use tracing::Span;
@@ -22,26 +21,6 @@ const EVENT_OBJECT_NAMESPACE_LOCK: &str = "object_namespace_lock";
 const LOG_COMPONENT_ECSTORE: &str = "ecstore";
 const LOG_SUBSYSTEM_OBJECT_LOCK: &str = "object_lock";
 
-pub(crate) fn record_lock_acquire_failure(op: &'static str, bucket: &str, object: &str, mode: &'static str, error: &LockError) {
-    let resource_class = if bucket == crate::disk::RUSTFS_META_MULTIPART_BUCKET {
-        "multipart_metadata"
-    } else if bucket == crate::disk::RUSTFS_META_BUCKET || bucket == ".minio.sys" {
-        "system_metadata"
-    } else if object.is_empty() || object == bucket {
-        "bucket_namespace"
-    } else {
-        "object_namespace"
-    };
-    let reason = match error {
-        LockError::Timeout { .. } => "timeout",
-        LockError::AlreadyLocked { .. } => "contention",
-        LockError::QuorumNotReached { .. } | LockError::InsufficientNodes { .. } => "quorum",
-        LockError::Network { .. } | LockError::Internal { .. } => "transport",
-        _ => "other",
-    };
-    record_object_lock_acquire_failure(op, mode, resource_class, reason);
-}
-
 /// Diagnostic lifetime only; it never owns or changes a namespace lock.
 /// Keep it alongside the real guard when ownership moves to a commit tail.
 #[derive(Default)]

crates/ecstore/src/disk/error_reduce.rs

@@ -131,15 +131,6 @@ pub fn build_write_quorum_failure_summary(
 
 fn dominant_error_label(errors: &[Option<Error>], ignored_errs: &[Error], dominant_error: Option<&Error>) -> &'static str {
     let Some(dominant_error) = dominant_error else {
-        // The reducer also returns None when every failure is ignored for
-        // quorum voting. That case contains no successful nil entry.
-        if !errors.is_empty() && errors.iter().all(Option::is_some) {
-            return if errors.iter().all(|error| error.as_ref() == Some(&Error::DiskNotFound)) {
-                "disk_not_found"
-            } else {
-                "all_ignored"
-            };
-        }
         return "nil_dominated";
     };
 
@@ -396,9 +387,9 @@ mod tests {
     }
 
     #[test]
-    fn test_build_write_quorum_failure_summary_all_missing_is_disk_not_found() {
-        // Ignoring unavailable disks for quorum reduction must not imply a
-        // successful nil entry in the diagnostic label.
+    fn test_build_write_quorum_failure_summary_all_ignored_is_nil_dominated() {
+        // All disks offline (all DiskNotFound). The dominant error must be None and
+        // the label must be "nil_dominated" rather than falling back to "other_error".
         let errors = vec![
             Some(Error::DiskNotFound),
             Some(Error::DiskNotFound),
@@ -406,21 +397,7 @@ mod tests {
         ];
         let summary = build_write_quorum_failure_summary(&errors, OBJECT_OP_IGNORED_ERRS, 2);
         assert_eq!(summary.dominant_error, None);
-        assert_eq!(summary.achieved, 0);
-        assert_eq!(summary.dominant_error_label, "disk_not_found");
-    }
-
-    #[test]
-    fn test_build_write_quorum_failure_summary_all_ignored_has_no_success() {
-        let errors = [Some(Error::FaultyDisk), Some(Error::DiskNotFound)];
-        let summary = build_write_quorum_failure_summary(&errors, OBJECT_OP_IGNORED_ERRS, 2);
-        assert_eq!(summary.dominant_error, None);
-        assert_eq!(summary.achieved, 0);
-        assert_eq!(summary.dominant_error_label, "all_ignored");
-        assert_eq!(
-            reduce_write_quorum_errs(&errors, OBJECT_OP_IGNORED_ERRS, 2),
-            Some(Error::ErasureWriteQuorum)
-        );
+        assert_eq!(summary.dominant_error_label, "nil_dominated");
     }
 
     #[test]

crates/ecstore/src/disk/fs.rs

@@ -223,6 +223,12 @@ pub fn rename_std(from: impl AsRef<Path>, to: impl AsRef<Path>) -> io::Result<()
     std::fs::rename(from, to)
 }
 
+#[cfg(any(not(windows), test))]
+#[tracing::instrument(level = "debug", skip_all)]
+pub async fn read_file(path: impl AsRef<Path>) -> io::Result<Vec<u8>> {
+    fs::read(path.as_ref()).await
+}
+
 // Bucket existence cache - reduces statx syscalls for repeated bucket checks
 
 /// Cache for bucket directory existence checks.
@@ -608,6 +614,28 @@ mod tests {
         assert_eq!(content, "test content");
     }
 
+    #[tokio::test]
+    async fn test_read_file() {
+        let temp_dir = TempDir::new().unwrap();
+        let file_path = temp_dir.path().join("test_read.txt");
+
+        let test_content = b"This is test content for reading";
+        tokio::fs::write(&file_path, test_content).await.unwrap();
+
+        // Read file
+        let read_content = read_file(&file_path).await.unwrap();
+        assert_eq!(read_content, test_content);
+    }
+
+    #[tokio::test]
+    async fn test_read_file_nonexistent() {
+        let temp_dir = TempDir::new().unwrap();
+        let file_path = temp_dir.path().join("nonexistent.txt");
+
+        // Should fail for non-existent file
+        assert!(read_file(&file_path).await.is_err());
+    }
+
     #[tokio::test]
     async fn test_same_file() {
         let temp_dir = TempDir::new().unwrap();

crates/ecstore/src/disk/local/commit.rs

@@ -47,33 +47,6 @@ use tokio::fs;
 use tracing::{info, warn};
 use uuid::Uuid;
 
-fn transitioned_old_current_source(
-    dst_meta_existed: bool,
-    dst_meta_unparsable: bool,
-    xlmeta: &FileMeta,
-    volume: &str,
-    path: &str,
-    version_id: Uuid,
-) -> (bool, Option<FileInfo>) {
-    if dst_meta_unparsable {
-        return (false, None);
-    }
-    if !dst_meta_existed {
-        return (true, None);
-    }
-    let Ok(old_current) = xlmeta.into_fileinfo(volume, path, "", false, false, true) else {
-        return (false, None);
-    };
-    if old_current.version_id.unwrap_or_default() != version_id {
-        return (true, None);
-    }
-    if old_current.transition_status == rustfs_filemeta::TRANSITION_COMPLETE {
-        (true, Some(old_current))
-    } else {
-        (true, None)
-    }
-}
-
 /// Hold later repair publications after admitting one baseline object. The
 /// fixture arms this on one replacement disk before rejoining the cluster.
 #[cfg(feature = "e2e-test-hooks")]
@@ -241,7 +214,7 @@ pub(super) async fn lock_rename_commit_directories(
     publication_root: &os::PublicationRoot,
     mutation_lease: Arc<os::NamespaceMutationLease>,
 ) -> Result<os::RenameCommitGuard> {
-    #[cfg(any(windows, target_os = "linux"))]
+    #[cfg(windows)]
     let result = {
         let source_parent = source_parent.to_path_buf();
         let destination_parent = destination_parent.to_path_buf();
@@ -249,7 +222,7 @@ pub(super) async fn lock_rename_commit_directories(
         let publication_root = publication_root.clone();
         os::run_blocking_namespace_operation(mutation_lease, move || {
             let result = os::prepare_rename_commit_guard(&source_parent, &destination_parent, &base_dir, &publication_root);
-            #[cfg(all(test, windows))]
+            #[cfg(test)]
             if result.is_ok() {
                 run_destination_commit_directory_preparation(&destination_parent);
             }
@@ -257,7 +230,7 @@ pub(super) async fn lock_rename_commit_directories(
         })
         .await
     };
-    #[cfg(not(any(windows, target_os = "linux")))]
+    #[cfg(not(windows))]
     let result = {
         let _ = mutation_lease;
         os::prepare_rename_commit_guard(source_parent, destination_parent, base_dir, publication_root)
@@ -271,12 +244,12 @@ pub(super) async fn lock_rename_commit_directories(
     result.map_err(to_file_error).map_err(DiskError::from)
 }
 
-async fn prepare_rename_destination_metadata(
+async fn read_rename_destination_metadata(
     file_path: &Path,
     rename_commit_guard: &os::RenameCommitGuard,
     mutation_lease: Arc<os::NamespaceMutationLease>,
 ) -> Result<Option<Bytes>> {
-    #[cfg(any(windows, target_os = "linux"))]
+    #[cfg(windows)]
     let result = {
         let file_path = file_path.to_path_buf();
         let rename_commit_guard = rename_commit_guard.clone();
@@ -285,10 +258,10 @@ async fn prepare_rename_destination_metadata(
         })
         .await
     };
-    #[cfg(not(any(windows, target_os = "linux")))]
+    #[cfg(not(windows))]
     let _ = (rename_commit_guard, mutation_lease);
-    #[cfg(not(any(windows, target_os = "linux")))]
-    let result = match fs::read(file_path).await {
+    #[cfg(not(windows))]
+    let result = match super::super::fs::read_file(file_path).await {
         Ok(data) => Ok(Some(data)),
         Err(err) if err.kind() == ErrorKind::NotFound => Ok(None),
         Err(err) => Err(err),
@@ -425,7 +398,7 @@ impl LocalDisk {
 
         // xl.meta path
         let src_file_path = self.io_get_object_path(src_volume, format!("{}/{}", src_path, STORAGE_FORMAT_FILE).as_str())?;
-        let dst_file_path = self.io_get_object_open_path(dst_volume, format!("{}/{}", dst_path, STORAGE_FORMAT_FILE).as_str())?;
+        let dst_file_path = self.io_get_object_path(dst_volume, format!("{}/{}", dst_path, STORAGE_FORMAT_FILE).as_str())?;
 
         // data_dir path
         let has_data_dir_path = {
@@ -480,7 +453,8 @@ impl LocalDisk {
             fs::create_dir_all(src_file_parent).await.map_err(to_file_error)?;
         }
         // Acquire the common trees before reading destination metadata. On
-        // Windows this pins the object directory identity across publication.
+        // Windows this pins the object directory identity across metadata
+        // preparation, data publication, rollback backup, and final commit.
         let rename_commit_guard = lock_rename_commit_directories(
             src_file_parent,
             dst_file_parent,
@@ -498,12 +472,9 @@ impl LocalDisk {
             source_volume_dir: &src_volume_dir,
             data_paths: has_data_dir_path.as_ref(),
         };
-        let has_dst_buf = prepare_rename_destination_metadata(
-            commit.destination_metadata,
-            &commit.directory_guard,
-            commit.mutation_lease.clone(),
-        )
-        .await?;
+        let has_dst_buf =
+            read_rename_destination_metadata(commit.destination_metadata, &commit.directory_guard, commit.mutation_lease.clone())
+                .await?;
 
         if no_inline {
             // Non-inline: read xl.meta, parse, write, rename data dir, rename xl.meta
@@ -537,14 +508,6 @@ impl LocalDisk {
             }
 
             let version_id = fi.version_id.unwrap_or_default();
-            let (old_current_source_checked, old_current_source) = transitioned_old_current_source(
-                has_dst_buf.is_some(),
-                dst_meta_unparsable,
-                &xlmeta,
-                dst_volume,
-                dst_path,
-                version_id,
-            );
             let has_old_data_dir = xlmeta.find_unshared_data_dir_for_version(Some(version_id));
             let old_version_exists = xlmeta.find_version(Some(version_id)).is_ok();
             let rollback_data_dir = has_old_data_dir.or_else(|| {
@@ -835,9 +798,7 @@ impl LocalDisk {
                 && let Some(parent) = dst_file_path.parent()
             {
                 let fsync_started = rustfs_io_metrics::put_stage_timer();
-                if let Err(err) =
-                    os::fsync_commit_directory(parent, &commit.directory_guard, commit.mutation_lease.clone(), None).await
-                {
+                if let Err(err) = os::fsync_dst_dir_group_commit(parent, Some(commit.mutation_lease.clone())).await {
                     rustfs_io_metrics::record_put_object_stage_duration_from(
                         rustfs_io_metrics::PUT_STAGE_SET_DISK_RENAME_DST_DIR_FSYNC,
                         fsync_started,
@@ -936,8 +897,6 @@ impl LocalDisk {
                 cleanup_data_dir: has_old_data_dir,
                 sign: version_signature,
                 old_current_size,
-                old_current_source_checked,
-                old_current_source,
             })
         } else {
             // Inline metadata preparation is blocking. The transaction lease is
@@ -951,8 +910,6 @@ impl LocalDisk {
                 None
             };
             let dst_path_for_failpoint = dst_path.to_string();
-            let dst_volume_for_source = dst_volume.to_string();
-            let dst_path_for_source = dst_path.to_string();
             #[cfg(windows)]
             let source_parent = src_file_parent.to_path_buf();
             let rename_commit_guard_for_preparation = commit.directory_guard.clone();
@@ -997,14 +954,6 @@ impl LocalDisk {
                 };
 
                 let version_id = fi.version_id.unwrap_or_default();
-                let (old_current_source_checked, old_current_source) = transitioned_old_current_source(
-                    has_dst_buf.is_some(),
-                    dst_meta_unparsable,
-                    &xlmeta,
-                    &dst_volume_for_source,
-                    &dst_path_for_source,
-                    version_id,
-                );
                 let old_data_dir = xlmeta.find_unshared_data_dir_for_version(Some(version_id));
                 let old_version_exists = xlmeta.find_version(Some(version_id)).is_ok();
                 let rollback_data_dir = old_data_dir.or_else(|| {
@@ -1054,8 +1003,6 @@ impl LocalDisk {
                     old_data_dir,
                     version_signature,
                     old_current_size,
-                    old_current_source_checked,
-                    old_current_source,
                     staged_rollback_path,
                     has_dst_buf.is_none(),
                     prepared_metadata_source,
@@ -1075,8 +1022,6 @@ impl LocalDisk {
                 cleanup_data_dir,
                 version_signature,
                 old_current_size,
-                old_current_source_checked,
-                old_current_source,
                 mut local_rollback_path,
                 destination_was_absent,
                 prepared_metadata_source,
@@ -1202,11 +1147,10 @@ impl LocalDisk {
                     && let Some(dst_parent) = dst_file_path.parent()
                 {
                     let fsync_started = rustfs_io_metrics::put_stage_timer();
-                    if let Err(err) = os::fsync_commit_directory(
+                    if let Err(err) = os::fsync_dst_dir_group_commit_or_namespace_file_sync_limit(
                         dst_parent,
-                        &commit.directory_guard,
                         commit.mutation_lease.clone(),
-                        Some(admission),
+                        admission,
                     )
                     .await
                     {
@@ -1318,8 +1262,6 @@ impl LocalDisk {
                 cleanup_data_dir,
                 sign: version_signature,
                 old_current_size,
-                old_current_source_checked,
-                old_current_source,
             })
         }
     }

crates/ecstore/src/disk/mod.rs

@@ -14,7 +14,6 @@
 
 // #730: disk abstractions still carry staged health and direct-I/O migration paths.
 
-pub(crate) mod cleanup_runtime;
 pub mod disk_store;
 pub mod endpoint;
 pub mod error;
@@ -231,105 +230,6 @@ pub enum Disk {
 }
 
 impl Disk {
-    pub(crate) async fn delete_versions_with_namespace_owner(
-        &self,
-        volume: &str,
-        versions: Vec<FileInfoVersions>,
-        opts: DeleteOptions,
-        namespace_owner: Option<Arc<dyn Send + Sync>>,
-    ) -> Vec<Option<Error>> {
-        match self {
-            Self::Local(disk) => {
-                disk.delete_versions_with_namespace_owner(volume, versions, opts, namespace_owner)
-                    .await
-            }
-            Self::Remote(disk) => {
-                let _owner = namespace_owner;
-                disk.delete_versions(volume, versions, opts).await
-            }
-        }
-    }
-
-    pub(crate) async fn delete_versions_recoverable_with_namespace_owner(
-        &self,
-        volume: &str,
-        versions: Vec<FileInfoVersions>,
-        mut opts: DeleteOptions,
-        namespace_owner: Option<Arc<dyn Send + Sync>>,
-    ) -> Vec<Option<Error>> {
-        if let Err(error) = validate_delete_recovery_options(&opts) {
-            return vec![Some(error); versions.len()];
-        }
-        if versions.iter().any(|version| version.name.starts_with('/')) {
-            return vec![Some(DiskError::InvalidPath); versions.len()];
-        }
-        opts.recoverable_delete = true;
-        match self {
-            Self::Local(disk) => {
-                disk.delete_versions_with_namespace_owner(volume, versions, opts, namespace_owner)
-                    .await
-            }
-            Self::Remote(disk) => {
-                let _owner = namespace_owner;
-                disk.delete_versions_recoverable(volume, versions, opts).await
-            }
-        }
-    }
-
-    pub(crate) async fn settle_delete_transaction_with_namespace_owner(
-        &self,
-        volume: &str,
-        object: &str,
-        transaction: Uuid,
-        expected_fingerprint: DeleteRecoveryDigest,
-        action: DeleteRecoveryAction,
-        namespace_owner: Option<Arc<dyn Send + Sync>>,
-    ) -> Result<DeleteRecoveryInspection> {
-        match self {
-            Self::Local(disk) => {
-                disk.settle_delete_transaction_with_namespace_owner(
-                    volume,
-                    object,
-                    transaction,
-                    expected_fingerprint,
-                    action,
-                    namespace_owner,
-                )
-                .await
-            }
-            Self::Remote(disk) => {
-                let _owner = namespace_owner;
-                disk.settle_delete_transaction(volume, object, transaction, expected_fingerprint, action)
-                    .await
-            }
-        }
-    }
-
-    pub(crate) async fn delete_version_recoverable_with_namespace_owner(
-        &self,
-        volume: &str,
-        path: &str,
-        fi: FileInfo,
-        force_del_marker: bool,
-        mut opts: DeleteOptions,
-        namespace_owner: Option<Arc<dyn Send + Sync>>,
-    ) -> Result<()> {
-        validate_delete_recovery_options(&opts)?;
-        if path.starts_with('/') {
-            return Err(DiskError::InvalidPath);
-        }
-        opts.recoverable_delete = true;
-        match self {
-            Self::Local(disk) => {
-                disk.delete_version_with_namespace_owner(volume, path, fi, force_del_marker, opts, namespace_owner)
-                    .await
-            }
-            Self::Remote(disk) => {
-                disk.delete_version_recoverable(volume, path, fi, force_del_marker, opts)
-                    .await
-            }
-        }
-    }
     pub(crate) async fn set_disk_id_state(&self, id: Option<Uuid>) -> Result<()> {
         match self {
             Disk::Local(local_disk) => {
@@ -357,70 +257,6 @@ impl Disk {
 
 #[async_trait::async_trait]
 impl DiskAPI for Disk {
-    async fn delete_version_recoverable(
-        &self,
-        volume: &str,
-        object: &str,
-        file_info: FileInfo,
-        force_del_marker: bool,
-        opts: DeleteOptions,
-    ) -> Result<()> {
-        self.delete_version_recoverable_with_namespace_owner(volume, object, file_info, force_del_marker, opts, None)
-            .await
-    }
-    async fn delete_versions_recoverable(
-        &self,
-        volume: &str,
-        versions: Vec<FileInfoVersions>,
-        opts: DeleteOptions,
-    ) -> Vec<Option<DiskError>> {
-        match self {
-            Self::Local(disk) => disk.delete_versions_recoverable(volume, versions, opts).await,
-            Self::Remote(disk) => disk.delete_versions_recoverable(volume, versions, opts).await,
-        }
-    }
-    async fn discover_delete_transactions(
-        &self,
-        volume: &str,
-        object: Option<&str>,
-        cursor: Option<DeleteRecoveryCursor>,
-        limit: u32,
-    ) -> Result<DeleteRecoveryPage> {
-        match self {
-            Self::Local(disk) => disk.discover_delete_transactions(volume, object, cursor, limit).await,
-            Self::Remote(disk) => disk.discover_delete_transactions(volume, object, cursor, limit).await,
-        }
-    }
-    async fn inspect_delete_transaction(
-        &self,
-        volume: &str,
-        object: &str,
-        transaction: Uuid,
-    ) -> Result<DeleteRecoveryInspection> {
-        match self {
-            Self::Local(disk) => disk.inspect_delete_transaction(volume, object, transaction).await,
-            Self::Remote(disk) => disk.inspect_delete_transaction(volume, object, transaction).await,
-        }
-    }
-    async fn settle_delete_transaction(
-        &self,
-        volume: &str,
-        object: &str,
-        transaction: Uuid,
-        expected_fingerprint: DeleteRecoveryDigest,
-        action: DeleteRecoveryAction,
-    ) -> Result<DeleteRecoveryInspection> {
-        match self {
-            Self::Local(disk) => {
-                disk.settle_delete_transaction(volume, object, transaction, expected_fingerprint, action)
-                    .await
-            }
-            Self::Remote(disk) => {
-                disk.settle_delete_transaction(volume, object, transaction, expected_fingerprint, action)
-                    .await
-            }
-        }
-    }
     fn to_string(&self) -> String {
         match self {
             Disk::Local(local_disk) => local_disk.to_string(),
@@ -1410,51 +1246,6 @@ pub trait DiskAPI: Debug + Send + Sync + 'static {
         Err(DiskError::MethodNotAllowed)
     }
     async fn delete(&self, volume: &str, path: &str, opt: DeleteOptions) -> Result<()>;
-    async fn discover_delete_transactions(
-        &self,
-        _volume: &str,
-        _object: Option<&str>,
-        _cursor: Option<DeleteRecoveryCursor>,
-        _limit: u32,
-    ) -> Result<DeleteRecoveryPage> {
-        Err(DiskError::MethodNotAllowed)
-    }
-    async fn delete_version_recoverable(
-        &self,
-        _volume: &str,
-        _object: &str,
-        _file_info: FileInfo,
-        _force_del_marker: bool,
-        _opts: DeleteOptions,
-    ) -> Result<()> {
-        Err(DiskError::MethodNotAllowed)
-    }
-    async fn delete_versions_recoverable(
-        &self,
-        _volume: &str,
-        versions: Vec<FileInfoVersions>,
-        _opts: DeleteOptions,
-    ) -> Vec<Option<DiskError>> {
-        vec![Some(DiskError::MethodNotAllowed); versions.len()]
-    }
-    async fn inspect_delete_transaction(
-        &self,
-        _volume: &str,
-        _object: &str,
-        _transaction: Uuid,
-    ) -> Result<DeleteRecoveryInspection> {
-        Err(DiskError::MethodNotAllowed)
-    }
-    async fn settle_delete_transaction(
-        &self,
-        _volume: &str,
-        _object: &str,
-        _transaction: Uuid,
-        _expected_fingerprint: DeleteRecoveryDigest,
-        _action: DeleteRecoveryAction,
-    ) -> Result<DeleteRecoveryInspection> {
-        Err(DiskError::MethodNotAllowed)
-    }
     // VerifyFile
     async fn verify_file(&self, volume: &str, path: &str, fi: &FileInfo) -> Result<CheckPartsResp>;
     // CheckParts
@@ -1777,14 +1568,6 @@ pub struct RenameDataResp {
     /// "cannot vote", never as `Absent`.
     #[serde(default)]
     pub old_current_size: Option<OldCurrentSize>,
-    /// Whether this disk decoded enough of the overwritten current version to
-    /// decide if a transitioned source has to be returned.
-    #[serde(default)]
-    pub old_current_source_checked: bool,
-    /// Transitioned old-current source used by PUT free-version cleanup receipt.
-    /// Empty when the old current is not transitioned or no source was known.
-    #[serde(default)]
-    pub old_current_source: Option<FileInfo>,
 }
 
 #[derive(Debug, Clone, Default, Serialize, Deserialize)]
@@ -1799,269 +1582,6 @@ pub struct DeleteOptions {
     /// Remote calls carrying it must use DeleteRetiredMarker, never DeleteVersion.
     #[serde(default, skip_serializing_if = "Option::is_none")]
     pub expected_delete_marker: Option<rustfs_filemeta::MetaDeleteMarker>,
-    #[serde(default, skip_serializing_if = "Option::is_none")]
-    pub delete_recovery_bucket_incarnation: Option<Uuid>,
-    #[serde(default)]
-    pub recoverable_delete: bool,
-}
-
-pub const DELETE_RECOVERY_PROTOCOL_VERSION: u32 = 1;
-pub(crate) fn get_delete_recovery_timeout() -> Result<std::time::Duration> {
-    let value = std::env::var("RUSTFS_DELETE_RECOVERY_TIMEOUT_SECS");
-    match value {
-        Ok(value) => parse_delete_recovery_timeout(Some(&value)),
-        Err(std::env::VarError::NotPresent) => parse_delete_recovery_timeout(None),
-        Err(std::env::VarError::NotUnicode(_)) => Err(DiskError::other("invalid delete recovery timeout configuration")),
-    }
-}
-
-fn parse_delete_recovery_timeout(value: Option<&str>) -> Result<std::time::Duration> {
-    let seconds = match value {
-        None => 900,
-        Some(value) => value
-            .parse::<u64>()
-            .ok()
-            .filter(|seconds| (30..=86400).contains(seconds))
-            .ok_or_else(|| DiskError::other("delete recovery timeout must be an integer between 30 and 86400 seconds"))?,
-    };
-    Ok(std::time::Duration::from_secs(seconds))
-}
-
-pub(crate) fn validate_delete_recovery_options(opts: &DeleteOptions) -> Result<()> {
-    if opts.old_data_dir.is_none_or(|id| id.is_nil())
-        || opts.undo_write
-        || opts.undo_delete
-        || opts.recursive
-        || opts.immediate
-        || opts.expected_delete_marker.is_some()
-        || opts.delete_recovery_bucket_incarnation.is_some_and(|id| id.is_nil())
-    {
-        return Err(DiskError::FileCorrupt);
-    }
-    Ok(())
-}
-pub type DeleteRecoveryDigest = [u8; 32];
-
-#[derive(Debug, Clone, PartialEq, Eq, Serialize, Deserialize)]
-#[serde(deny_unknown_fields)]
-pub struct DeleteTransactionId {
-    pub object: String,
-    pub transaction: Uuid,
-}
-
-#[derive(Debug, Clone, PartialEq, Eq, Serialize, Deserialize)]
-#[serde(deny_unknown_fields)]
-pub struct DeleteRecoveryCursor {
-    pub token: Uuid,
-}
-
-#[derive(Debug, Clone, PartialEq, Eq, Serialize, Deserialize)]
-#[serde(deny_unknown_fields)]
-pub struct DeleteRecoveryPage {
-    pub transactions: Vec<DeleteTransactionId>,
-    pub next: Option<DeleteRecoveryCursor>,
-}
-
-#[derive(Debug, Clone, PartialEq, Serialize, Deserialize)]
-#[serde(deny_unknown_fields)]
-pub struct DeleteRecoveryJournal {
-    pub protocol_version: u32,
-    pub volume: String,
-    pub object: String,
-    pub transaction: Uuid,
-    pub bucket_incarnation: Option<Uuid>,
-    pub original_digest: Option<DeleteRecoveryDigest>,
-    pub target_digest: Option<DeleteRecoveryDigest>,
-    pub data_dirs: Vec<Uuid>,
-    pub deleted_versions: Vec<Uuid>,
-    pub requests: Vec<FileInfo>,
-    pub original_versions: Vec<DeleteRecoveryVersionIdentity>,
-    #[serde(default)]
-    pub predecessors: Vec<Uuid>,
-}
-
-impl DeleteRecoveryJournal {
-    /// Checks source generations only; the coordinator must supply a certified
-    /// decision and the receiver must retain its namespace/incarnation fences.
-    pub(crate) fn matches_adoption_source(&self, current: &[DeleteRecoveryVersionIdentity]) -> bool {
-        if self.requests.is_empty() {
-            return false;
-        }
-        let mut originals = std::collections::HashMap::with_capacity(self.original_versions.len());
-        for version in &self.original_versions {
-            originals.entry(version.version_id.unwrap_or_default()).or_insert(version);
-        }
-        let mut locals = std::collections::HashMap::with_capacity(current.len());
-        for version in current {
-            locals.entry(version.version_id.unwrap_or_default()).or_insert(version);
-        }
-        self.requests.iter().all(|request| {
-            if request.deleted && request.mark_deleted {
-                return current.first() == self.original_versions.first();
-            }
-            let id = request.version_id.unwrap_or_default();
-            let Some(original) = originals.get(&id) else {
-                return false;
-            };
-            match locals.get(&id) {
-                Some(local) => local == original,
-                None => !id.is_nil(),
-            }
-        })
-    }
-
-    /// Matches immutable local preparation intent against an independently
-    /// inspected same-transaction commit receipt. This match is not authority
-    /// by itself; the coordinator must supply the certified receipt.
-    pub(crate) fn matches_prepared_authority(&self, authority: &Self) -> bool {
-        if self.protocol_version != DELETE_RECOVERY_PROTOCOL_VERSION
-            || authority.protocol_version != DELETE_RECOVERY_PROTOCOL_VERSION
-            || self.transaction.is_nil()
-            || self.transaction != authority.transaction
-            || self.volume != authority.volume
-            || self.object != authority.object
-            || self.bucket_incarnation != authority.bucket_incarnation
-            || self.bucket_incarnation.is_some_and(|id| id.is_nil())
-            || self.requests.is_empty()
-            || self.requests != authority.requests
-            || self.deleted_versions.iter().collect::<std::collections::HashSet<_>>()
-                != authority.deleted_versions.iter().collect::<std::collections::HashSet<_>>()
-        {
-            return false;
-        }
-        let sources: std::collections::HashMap<_, _> = self
-            .original_versions
-            .iter()
-            .map(|version| (version.version_id.unwrap_or_default(), version))
-            .collect();
-        let certified: std::collections::HashMap<_, _> = authority
-            .original_versions
-            .iter()
-            .map(|version| (version.version_id.unwrap_or_default(), version))
-            .collect();
-        if sources.len() != self.original_versions.len() || certified.len() != authority.original_versions.len() {
-            return false;
-        }
-        self.requests.iter().all(|request| {
-            let (local, remote) = if request.deleted && request.mark_deleted {
-                (self.original_versions.first(), authority.original_versions.first())
-            } else {
-                let id = request.version_id.unwrap_or_default();
-                (sources.get(&id).copied(), certified.get(&id).copied())
-            };
-            match (local, remote) {
-                (Some(local), Some(remote)) => local == remote,
-                (None, None) => self.original_digest.is_none() && authority.original_digest.is_none(),
-                _ => false,
-            }
-        })
-    }
-}
-
-#[derive(Debug, Clone, PartialEq, Eq, Hash, Serialize, Deserialize)]
-#[serde(deny_unknown_fields)]
-pub struct DeleteRecoveryVersionIdentity {
-    pub version_id: Option<Uuid>,
-    pub mod_time: Option<OffsetDateTime>,
-    pub data_dir: Option<Uuid>,
-    pub deleted: bool,
-    pub metadata_digest: DeleteRecoveryDigest,
-}
-
-impl DeleteRecoveryVersionIdentity {
-    pub(crate) fn from_file_info(info: FileInfo) -> error::Result<Self> {
-        local::delete_recovery::generation(info)
-    }
-}
-
-#[derive(Debug, Clone, Copy, PartialEq, Eq, Serialize, Deserialize)]
-pub enum DeleteRecoveryState {
-    Absent,
-    NewPrepared,
-    QuorumCommitted,
-    CommitFinished,
-    LegacyAmbiguous,
-}
-
-#[derive(Debug, Clone, PartialEq, Serialize, Deserialize)]
-#[serde(deny_unknown_fields)]
-pub struct DeleteRecoveryInspection {
-    pub protocol_version: u32,
-    pub id: DeleteTransactionId,
-    pub state: DeleteRecoveryState,
-    pub journal: Option<DeleteRecoveryJournal>,
-    pub current_digest: Option<DeleteRecoveryDigest>,
-    pub current_versions: Vec<DeleteRecoveryVersionIdentity>,
-    pub backup_digest: Option<DeleteRecoveryDigest>,
-    pub rollback_data_complete: bool,
-    pub commit_data_safe: bool,
-    pub explicit_delete_safe: bool,
-    pub fingerprint: DeleteRecoveryDigest,
-}
-
-#[derive(Debug, Clone, PartialEq, Serialize, Deserialize)]
-pub enum DeleteRecoveryAction {
-    PublishCommit,
-    FinishCommit,
-    FinalizeCommit,
-    Rollback,
-    DiscardLegacy,
-    AdoptCommit { authority: Box<DeleteRecoveryJournal> },
-}
-
-#[derive(Debug, Clone, Serialize, Deserialize)]
-#[serde(deny_unknown_fields)]
-pub enum DeleteRecoveryRpcOperation {
-    Discover {
-        object: Option<String>,
-        cursor: Option<DeleteRecoveryCursor>,
-        limit: u32,
-    },
-    Inspect {
-        object: String,
-        transaction: Uuid,
-    },
-    Settle {
-        object: String,
-        transaction: Uuid,
-        expected_fingerprint: DeleteRecoveryDigest,
-        action: DeleteRecoveryAction,
-    },
-    ApplyVersion {
-        object: String,
-        file_info: Box<FileInfo>,
-        force_del_marker: bool,
-        opts: DeleteOptions,
-    },
-    ApplyVersions {
-        versions: Vec<FileInfoVersions>,
-        opts: DeleteOptions,
-    },
-}
-
-impl DeleteRecoveryRpcOperation {
-    /// Full metadata application and adoption share the existing mutation envelope.
-    pub fn max_wire_payload_bytes(&self) -> usize {
-        match self {
-            Self::ApplyVersion { .. }
-            | Self::ApplyVersions { .. }
-            | Self::Settle {
-                action: DeleteRecoveryAction::AdoptCommit { .. },
-                ..
-            } => rustfs_protos::DELETE_RECOVERY_MAX_PAYLOAD_BYTES,
-            _ => rustfs_protos::DELETE_RECOVERY_MAX_CONTROL_PAYLOAD_BYTES,
-        }
-    }
-}
-
-#[derive(Debug, Clone, Serialize, Deserialize)]
-#[serde(deny_unknown_fields)]
-pub enum DeleteRecoveryRpcReply {
-    Discovered(DeleteRecoveryPage),
-    Inspected(DeleteRecoveryInspection),
-    Settled(DeleteRecoveryInspection),
-    AppliedVersion,
-    AppliedVersions(Vec<Option<u32>>),
 }
 
 #[derive(Debug, Clone, Serialize, Deserialize)]
@@ -2352,8 +1872,6 @@ mod tests {
             undo_delete: false,
             old_data_dir: Some(Uuid::new_v4()),
             expected_delete_marker: None,
-            delete_recovery_bucket_incarnation: None,
-            recoverable_delete: false,
         };
 
         assert!(opts.recursive);
@@ -2526,8 +2044,6 @@ mod tests {
             cleanup_data_dir: Some(uuid),
             sign: Some(signature.clone()),
             old_current_size: Some(OldCurrentSize::Present(42)),
-            old_current_source_checked: false,
-            old_current_source: None,
         };
 
         assert_eq!(resp.old_data_dir, Some(uuid));
@@ -2548,8 +2064,6 @@ mod tests {
                 cleanup_data_dir: Some(Uuid::new_v4()),
                 sign: Some(vec![0x01, 0x02, 0x03]),
                 old_current_size,
-                old_current_source_checked: false,
-                old_current_source: None,
             };
 
             let encoded = rmp_serde::encode::to_vec_named(&resp).expect("named msgpack should encode");
@@ -2587,8 +2101,6 @@ mod tests {
         assert_eq!(decoded.cleanup_data_dir, None);
         assert_eq!(decoded.sign, legacy.sign);
         assert_eq!(decoded.old_current_size, None);
-        assert!(!decoded.old_current_source_checked);
-        assert!(decoded.old_current_source.is_none());
     }
 
     /// Test constants
@@ -2784,183 +2296,3 @@ mod tests {
         }
     }
 }
-
-#[cfg(test)]
-mod delete_recovery_timeout_tests {
-    use super::*;
-    #[test]
-    fn recovery_timeout_defaults_and_validates_range() {
-        assert_eq!(
-            parse_delete_recovery_timeout(None).expect("default timeout"),
-            std::time::Duration::from_secs(900)
-        );
-        for seconds in ["30", "900", "86400"] {
-            assert!(parse_delete_recovery_timeout(Some(seconds)).is_ok());
-        }
-        for value in ["", "invalid", "0", "2", "29", "86401", "-1", "18446744073709551616"] {
-            assert!(parse_delete_recovery_timeout(Some(value)).is_err());
-        }
-    }
-}
-
-#[cfg(test)]
-mod delete_recovery_wire_limit_tests {
-    use super::*;
-
-    #[test]
-    fn boxed_apply_version_preserves_existing_wire_payload() {
-        #[derive(Serialize)]
-        enum PriorOperation<'a> {
-            ApplyVersion {
-                object: &'a str,
-                file_info: &'a FileInfo,
-                force_del_marker: bool,
-                opts: &'a DeleteOptions,
-            },
-        }
-        let file_info = FileInfo {
-            name: "object".into(),
-            version_id: Some(Uuid::new_v4()),
-            metadata: std::collections::HashMap::from([("metadata".into(), "value".into())]),
-            ..Default::default()
-        };
-        let opts = DeleteOptions {
-            old_data_dir: Some(Uuid::new_v4()),
-            recoverable_delete: true,
-            delete_recovery_bucket_incarnation: Some(Uuid::new_v4()),
-            ..Default::default()
-        };
-        let prior = PriorOperation::ApplyVersion {
-            object: "object",
-            file_info: &file_info,
-            force_del_marker: true,
-            opts: &opts,
-        };
-        let encoded = rmp_serde::to_vec_named(&prior).expect("encode the previous unboxed request shape");
-        let operation = DeleteRecoveryRpcOperation::ApplyVersion {
-            object: "object".into(),
-            file_info: Box::new(file_info.clone()),
-            force_del_marker: true,
-            opts: opts.clone(),
-        };
-        assert_eq!(rmp_serde::to_vec_named(&operation).expect("encode boxed request"), encoded);
-        assert_eq!(
-            serde_json::to_value(&operation).expect("boxed request JSON"),
-            serde_json::to_value(prior).expect("previous request JSON")
-        );
-        let decoded: DeleteRecoveryRpcOperation =
-            rustfs_protos::decode_delete_recovery_payload(&encoded).expect("decode the previous payload");
-        assert_eq!(rmp_serde::to_vec_named(&decoded).expect("re-encode decoded prior request"), encoded);
-    }
-    #[test]
-    fn adoption_and_inspection_reply_support_large_version_journals() {
-        let transaction = Uuid::new_v4();
-        let request = FileInfo {
-            metadata: std::collections::HashMap::from([("metadata".into(), "x".repeat(2048))]),
-            ..Default::default()
-        };
-        let journal = DeleteRecoveryJournal {
-            protocol_version: DELETE_RECOVERY_PROTOCOL_VERSION,
-            volume: "bucket".into(),
-            object: "object".into(),
-            transaction,
-            bucket_incarnation: None,
-            original_digest: None,
-            target_digest: None,
-            data_dirs: Vec::new(),
-            deleted_versions: Vec::new(),
-            requests: vec![request; 1000],
-            original_versions: Vec::new(),
-            predecessors: Vec::new(),
-        };
-        let operation = DeleteRecoveryRpcOperation::Settle {
-            object: "object".into(),
-            transaction,
-            expected_fingerprint: [0; 32],
-            action: DeleteRecoveryAction::AdoptCommit {
-                authority: Box::new(journal.clone()),
-            },
-        };
-        let encoded = rmp_serde::to_vec_named(&operation).expect("encode large adoption");
-        assert!(encoded.len() > rustfs_protos::DELETE_RECOVERY_MAX_CONTROL_PAYLOAD_BYTES);
-        assert!(encoded.len() < operation.max_wire_payload_bytes());
-        let decoded: DeleteRecoveryRpcOperation =
-            rustfs_protos::decode_delete_recovery_payload(&encoded).expect("large adoption can decode");
-        assert_eq!(decoded.max_wire_payload_bytes(), rustfs_protos::DELETE_RECOVERY_MAX_PAYLOAD_BYTES);
-        let reply = DeleteRecoveryRpcReply::Inspected(DeleteRecoveryInspection {
-            protocol_version: DELETE_RECOVERY_PROTOCOL_VERSION,
-            id: DeleteTransactionId {
-                object: "object".into(),
-                transaction,
-            },
-            state: DeleteRecoveryState::QuorumCommitted,
-            journal: Some(journal),
-            current_digest: None,
-            current_versions: Vec::new(),
-            backup_digest: None,
-            rollback_data_complete: false,
-            commit_data_safe: true,
-            explicit_delete_safe: false,
-            fingerprint: [0; 32],
-        });
-        let encoded = rmp_serde::to_vec_named(&reply).expect("encode large inspection");
-        assert!(encoded.len() > rustfs_protos::DELETE_RECOVERY_MAX_CONTROL_PAYLOAD_BYTES);
-        assert!(rustfs_protos::decode_delete_recovery_payload::<DeleteRecoveryRpcReply>(&encoded).is_ok());
-        let inspect = DeleteRecoveryRpcOperation::Inspect {
-            object: "object".into(),
-            transaction,
-        };
-        assert_eq!(inspect.max_wire_payload_bytes(), rustfs_protos::DELETE_RECOVERY_MAX_CONTROL_PAYLOAD_BYTES);
-    }
-}
-
-#[cfg(test)]
-mod delete_recovery_adoption_tests {
-    use super::*;
-
-    #[test]
-    fn source_matching_preserves_non_null_absence_and_rejects_foreign_or_missing_null() {
-        let id = Uuid::new_v4();
-        let source = DeleteRecoveryVersionIdentity {
-            version_id: Some(id),
-            mod_time: None,
-            data_dir: None,
-            deleted: false,
-            metadata_digest: [1; 32],
-        };
-        let mut journal = DeleteRecoveryJournal {
-            protocol_version: DELETE_RECOVERY_PROTOCOL_VERSION,
-            volume: "bucket".into(),
-            object: "object".into(),
-            transaction: Uuid::new_v4(),
-            bucket_incarnation: None,
-            original_digest: None,
-            target_digest: None,
-            data_dirs: Vec::new(),
-            deleted_versions: vec![id],
-            requests: vec![FileInfo {
-                version_id: Some(id),
-                ..Default::default()
-            }],
-            original_versions: vec![source.clone()],
-            predecessors: Vec::new(),
-        };
-        assert!(journal.matches_adoption_source(&[]), "certified non-null removal can already be absent");
-        assert!(journal.matches_adoption_source(std::slice::from_ref(&source)));
-        let mut foreign = source.clone();
-        foreign.metadata_digest = [2; 32];
-        assert!(!journal.matches_adoption_source(std::slice::from_ref(&foreign)));
-        journal.original_versions.push(foreign.clone());
-        assert!(
-            journal.matches_adoption_source(&[source.clone(), foreign.clone()]),
-            "retain original first-match semantics"
-        );
-        assert!(!journal.matches_adoption_source(&[foreign, source]));
-        journal.requests[0].version_id = None;
-        journal.original_versions[0].version_id = None;
-        assert!(!journal.matches_adoption_source(&[]), "null absence does not establish its generation");
-        assert!(journal.matches_adoption_source(std::slice::from_ref(&journal.original_versions[0])));
-        journal.requests.clear();
-        assert!(!journal.matches_adoption_source(&journal.original_versions));
-    }
-}

crates/ecstore/src/error/mod.rs

@@ -17,8 +17,8 @@
 use crate::bucket::error::BucketMetadataError;
 use crate::disk::error::DiskError;
 use crate::storage_api_contracts::{error::StorageErrorCode, range::HTTPRangeError};
+use rustfs_s3_types::S3ErrorCode;
 use rustfs_utils::path::decode_dir_object;
-use s3s::S3ErrorCode;
 
 pub type Error = StorageError;
 pub type Result<T> = core::result::Result<T, Error>;

crates/ecstore/src/io_support/bitrot.rs

@@ -174,14 +174,10 @@ async fn open_reader_source(
     metrics_path: Option<&'static str>,
 ) -> disk::error::Result<Option<BoxedObjectReader>> {
     if let Some(data) = inline_data {
-        #[cfg(test)]
-        crate::set_disk::disk_call_counters::record_shard_open(path, true);
         let mut reader = Cursor::new(data);
         reader.set_position(u64::try_from(offset).map_err(|_| DiskError::FileCorrupt)?);
         Ok(Some(ShardReader::InMemory(reader)))
     } else if let Some(disk) = disk {
-        #[cfg(test)]
-        crate::set_disk::disk_call_counters::record_shard_open(path, false);
         open_disk_reader(disk, bucket, path, offset, length, use_mmap_read, metrics_path)
             .await
             .map(Some)

crates/ecstore/src/io_support/rio.rs

@@ -442,19 +442,12 @@ impl WritePlan {
         }
 
         if let Some(encryption) = self.encryption {
-            let etag_nonce = match encryption.mode {
-                WriteEncryptionMode::Singlepart { base_nonce } | WriteEncryptionMode::MultipartLegacy { base_nonce, .. } => {
-                    base_nonce
-                }
-                WriteEncryptionMode::SinglepartObjectKey | WriteEncryptionMode::MultipartObjectKey { .. } => [0u8; 12],
-            };
-            let protected_reader = rustfs_rio::EtagReader::new_encrypted(reader, encryption.key_bytes, etag_nonce)?;
             reader = match encryption.mode {
                 WriteEncryptionMode::SinglepartObjectKey => HashReader::from_reader(
                     #[cfg(feature = "rio-v2")]
-                    EncryptReader::new_with_object_key(protected_reader, encryption.key_bytes),
+                    EncryptReader::new_with_object_key(reader, encryption.key_bytes),
                     #[cfg(not(feature = "rio-v2"))]
-                    EncryptReader::new(protected_reader, encryption.key_bytes, [0u8; 12]),
+                    EncryptReader::new(reader, encryption.key_bytes, [0u8; 12]),
                     HashReader::SIZE_PRESERVE_LAYER,
                     actual_size,
                     None,
@@ -464,12 +457,12 @@ impl WritePlan {
                 WriteEncryptionMode::Singlepart { base_nonce } => {
                     #[cfg(not(feature = "rio-v2"))]
                     let encrypt_reader = if encryption_frame_v2_enabled() {
-                        EncryptReader::new_v2(protected_reader, encryption.key_bytes, base_nonce)
+                        EncryptReader::new_v2(reader, encryption.key_bytes, base_nonce)
                     } else {
-                        EncryptReader::new(protected_reader, encryption.key_bytes, base_nonce)
+                        EncryptReader::new(reader, encryption.key_bytes, base_nonce)
                     };
                     #[cfg(feature = "rio-v2")]
-                    let encrypt_reader = EncryptReader::new(protected_reader, encryption.key_bytes, base_nonce);
+                    let encrypt_reader = EncryptReader::new(reader, encryption.key_bytes, base_nonce);
                     HashReader::from_reader(encrypt_reader, HashReader::SIZE_PRESERVE_LAYER, actual_size, None, None, false)?
                 }
                 WriteEncryptionMode::MultipartLegacy {
@@ -478,25 +471,20 @@ impl WritePlan {
                 } => {
                     #[cfg(not(feature = "rio-v2"))]
                     let encrypt_reader = if encryption_frame_v2_enabled() {
-                        EncryptReader::new_multipart_v2(protected_reader, encryption.key_bytes, base_nonce, multipart_part_number)
+                        EncryptReader::new_multipart_v2(reader, encryption.key_bytes, base_nonce, multipart_part_number)
                     } else {
-                        EncryptReader::new_multipart(protected_reader, encryption.key_bytes, base_nonce, multipart_part_number)
+                        EncryptReader::new_multipart(reader, encryption.key_bytes, base_nonce, multipart_part_number)
                     };
                     #[cfg(feature = "rio-v2")]
                     let encrypt_reader =
-                        EncryptReader::new_multipart(protected_reader, encryption.key_bytes, base_nonce, multipart_part_number);
+                        EncryptReader::new_multipart(reader, encryption.key_bytes, base_nonce, multipart_part_number);
                     HashReader::from_reader(encrypt_reader, HashReader::SIZE_PRESERVE_LAYER, actual_size, None, None, false)?
                 }
                 WriteEncryptionMode::MultipartObjectKey { multipart_part_number } => HashReader::from_reader(
                     #[cfg(feature = "rio-v2")]
-                    EncryptReader::new_multipart_with_object_key(protected_reader, encryption.key_bytes, multipart_part_number),
+                    EncryptReader::new_multipart_with_object_key(reader, encryption.key_bytes, multipart_part_number),
                     #[cfg(not(feature = "rio-v2"))]
-                    EncryptReader::new_multipart(
-                        protected_reader,
-                        encryption.key_bytes,
-                        [0u8; 12],
-                        multipart_part_number as usize,
-                    ),
+                    EncryptReader::new_multipart(reader, encryption.key_bytes, [0u8; 12], multipart_part_number as usize),
                     HashReader::SIZE_PRESERVE_LAYER,
                     actual_size,
                     None,
@@ -509,7 +497,6 @@ impl WritePlan {
         // `ignore_value` deliberately avoids a second hasher over compressed or
         // encrypted bytes. The inner reader still validates the plaintext request
         // checksum while this outer reader exposes the request checksum context.
-        reader.encrypted_etag = encrypted;
         reader.add_non_trailing_checksum(checksum, true)?;
         reader.set_trailer(trailer);
 
@@ -526,36 +513,6 @@ mod tests {
     use std::io::Cursor;
     use tokio::io::AsyncReadExt;
 
-    #[tokio::test]
-    async fn write_plan_encryption_persists_protected_etags_for_every_mode() {
-        let data = b"Salary 2026: 85000 EUR";
-        let size = i64::try_from(data.len()).expect("test data size");
-        let mut control =
-            HashReader::from_stream(Cursor::new(data), size, size, None, None, false).expect("create control reader");
-        control.read_to_end(&mut Vec::new()).await.expect("read control plaintext");
-        let md5 = control.try_resolve_etag().expect("control MD5");
-        for encryption in [
-            WriteEncryption::singlepart([1; 32], [1; 12]),
-            WriteEncryption::singlepart_object_key([1; 32]),
-            WriteEncryption::multipart([1; 32], [1; 12], 1),
-            WriteEncryption::multipart_object_key([1; 32], 1),
-        ] {
-            let plaintext = HashReader::from_stream(Cursor::new(data), size, size, Some(md5.clone()), None, false)
-                .expect("create checked reader");
-            let mut reader = WritePlan::new()
-                .with_encryption(encryption)
-                .apply(plaintext, size)
-                .expect("apply encryption");
-            let mut ciphertext = Vec::new();
-            reader.read_to_end(&mut ciphertext).await.expect("stream ciphertext");
-            assert_ne!(ciphertext, data);
-            let tag = reader.try_resolve_etag().expect("resolve persisted encrypted ETag");
-            assert_ne!(tag, md5, "stored ETag must not reveal the plaintext MD5");
-            assert_eq!(tag.len(), 32, "multipart ETags require a 16-byte hexadecimal part tag");
-            assert_eq!(reader.try_resolve_etag().as_ref(), Some(&tag));
-        }
-    }
-
     async fn assert_non_trailing_checksum_survives(plan: WritePlan) {
         let plaintext = b"checksum-context-through-write-plan".repeat(256);
         let actual_size = plaintext.len() as i64;

crates/ecstore/src/object_api/mod.rs

@@ -102,14 +102,10 @@ mod body_cache_hook;
 mod encryption;
 mod hook_slot;
 mod object_mutation_hook;
-mod read_condition;
 mod readers;
 mod types;
 pub(crate) mod write_commit_context;
 
-pub use read_condition::{GetObjectReadCondition, GetObjectReadMetadata, with_get_object_read_condition};
-pub(crate) use read_condition::{get_object_read_condition_is_active, get_object_read_condition_is_terminal};
-
 #[cfg(test)]
 mod persisted_metadata_keys_tests;
 

crates/ecstore/src/object_api/object_api_utils.rs

@@ -29,7 +29,7 @@ use crate::object_api::{GetObjectReader, ObjectInfo, ObjectOptions};
 use crate::storage_api_contracts::range::HTTPRangeSpec;
 use rustfs_filemeta::ObjectPartInfo;
 use rustfs_rio::HashReader;
-use s3s::S3ErrorCode;
+use rustfs_s3_types::S3ErrorCode;
 
 //#[derive(Clone)]
 pub struct PutObjReader {
@@ -231,6 +231,45 @@ mod tests {
         assert_eq!(result.2, 1024 * 1024);
     }
 
+    #[test]
+    fn test_new_getobjectreader_rejects_a_range_past_the_object_as_invalid_range() {
+        let oi = multipart_object_info();
+        let past_the_end = Some(HTTPRangeSpec {
+            is_suffix_length: false,
+            start: oi.size,
+            end: oi.size + 10,
+        });
+
+        let err = new_getobjectreader(&past_the_end, &oi, &ObjectOptions::default(), &HeaderMap::new())
+            .err()
+            .expect("a range starting at the object size must be rejected");
+
+        assert_eq!(err.code, S3ErrorCode::InvalidRange);
+        assert_eq!(err.code.as_str(), "InvalidRange");
+        assert_eq!(err.key, None);
+        assert_eq!(err.bucket_name, None);
+    }
+
+    #[test]
+    fn test_new_getobjectreader_rejects_an_unknown_size_without_a_range_as_invalid_range() {
+        let oi = ObjectInfo {
+            bucket: "bucket".to_string(),
+            name: "object".to_string(),
+            size: -1,
+            ..Default::default()
+        };
+
+        let err = new_getobjectreader(&None, &oi, &ObjectOptions::default(), &HeaderMap::new())
+            .err()
+            .expect("an object of unknown size cannot be read without a range");
+
+        assert_eq!(err.code, S3ErrorCode::InvalidRange);
+        assert_eq!(err.code.as_str(), "InvalidRange");
+        assert_eq!(err.message, "Invalid range");
+        assert_eq!(err.key.as_deref(), Some("object"));
+        assert_eq!(err.bucket_name.as_deref(), Some("bucket"));
+    }
+
     #[test]
     fn test_to_s3s_etag() {
         // Test unquoted ETag - should become strong etag

crates/ecstore/src/object_api/read_condition.rs

@@ -1,113 +0,0 @@
-// Copyright 2024 RustFS Team
-//
-// Licensed under the Apache License, Version 2.0 (the "License");
-// you may not use this file except in compliance with the License.
-// You may obtain a copy of the License at
-//
-//     http://www.apache.org/licenses/LICENSE-2.0
-//
-// Unless required by applicable law or agreed to in writing, software
-// distributed under the License is distributed on an "AS IS" BASIS,
-// WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
-// See the License for the specific language governing permissions and
-// limitations under the License.
-
-use rustfs_filemeta::{FileInfo, ObjectPartInfo};
-use std::future::Future;
-use std::sync::Arc;
-use time::OffsetDateTime;
-
-/// Borrowed fields needed to decide whether an object read needs a body.
-pub struct GetObjectReadMetadata<'a> {
-    pub bucket: &'a str,
-    pub object: &'a str,
-    pub etag: Option<&'a str>,
-    pub mod_time: Option<OffsetDateTime>,
-    pub parts: &'a [ObjectPartInfo],
-    pub delete_marker: bool,
-}
-
-impl<'a> GetObjectReadMetadata<'a> {
-    pub(crate) fn from_file_info(bucket: &'a str, object: &'a str, info: &'a FileInfo) -> Self {
-        Self {
-            bucket,
-            object,
-            etag: info.metadata.get("etag").map(String::as_str),
-            mod_time: info.mod_time,
-            parts: &info.parts,
-            delete_marker: info.deleted,
-        }
-    }
-}
-
-/// A pure decision on quorum-validated metadata. Authorization belongs to the caller.
-pub trait GetObjectReadCondition: Send + Sync {
-    fn is_terminal(&self, metadata: GetObjectReadMetadata<'_>) -> bool;
-}
-
-tokio::task_local! {
-    static READ_CONDITION: Arc<dyn GetObjectReadCondition>;
-}
-
-/// Applies a condition to this reader call, without inheriting it into shared producers.
-pub async fn with_get_object_read_condition<F: Future>(condition: Arc<dyn GetObjectReadCondition>, read: F) -> F::Output {
-    READ_CONDITION.scope(condition, read).await
-}
-
-pub(crate) fn get_object_read_condition_is_terminal(metadata: GetObjectReadMetadata<'_>) -> bool {
-    READ_CONDITION
-        .try_with(|condition| condition.is_terminal(metadata))
-        .unwrap_or(false)
-}
-
-pub(crate) fn get_object_read_condition_is_active() -> bool {
-    READ_CONDITION.try_with(|_| ()).is_ok()
-}
-
-#[cfg(test)]
-mod tests {
-    use super::*;
-
-    struct Stop;
-    impl GetObjectReadCondition for Stop {
-        fn is_terminal(&self, _: GetObjectReadMetadata<'_>) -> bool {
-            true
-        }
-    }
-    fn metadata() -> GetObjectReadMetadata<'static> {
-        GetObjectReadMetadata {
-            bucket: "bucket",
-            object: "object",
-            etag: Some("etag"),
-            mod_time: None,
-            parts: &[],
-            delete_marker: false,
-        }
-    }
-
-    #[tokio::test]
-    async fn read_condition_scope_does_not_escape_or_enter_shared_producers() {
-        assert!(!get_object_read_condition_is_terminal(metadata()));
-        with_get_object_read_condition(Arc::new(Stop), async {
-            assert!(get_object_read_condition_is_terminal(metadata()));
-            assert!(
-                !tokio::spawn(async { get_object_read_condition_is_terminal(metadata()) })
-                    .await
-                    .expect("shared producer")
-            );
-        })
-        .await;
-        assert!(!get_object_read_condition_is_terminal(metadata()));
-    }
-
-    #[tokio::test]
-    async fn dropping_a_pending_read_removes_its_condition_scope() {
-        let mut read = Box::pin(with_get_object_read_condition(Arc::new(Stop), async {
-            assert!(get_object_read_condition_is_terminal(metadata()));
-            std::future::pending::<()>().await;
-        }));
-        assert!(futures::poll!(&mut read).is_pending());
-        drop(read);
-        assert!(!get_object_read_condition_is_terminal(metadata()));
-    }
-}

crates/ecstore/src/object_api/readers.rs

@@ -528,14 +528,6 @@ pub struct GetObjectReader {
 }
 
 impl GetObjectReader {
-    pub(crate) fn condition_stopped(object_info: ObjectInfo) -> Self {
-        Self {
-            stream: Box::new(ConditionStoppedBody),
-            object_info,
-            buffered_body: None,
-            body_source: GetObjectBodySource::Unprobed,
-        }
-    }
     /// Builds a fully materialized reader from a cache-coordinated body.
     pub fn from_cache_body(mut object_info: ObjectInfo, body: Bytes) -> Result<Self> {
         object_info.size = i64::try_from(body.len()).map_err(|_| Error::other("cached GET body length exceeds i64::MAX"))?;
@@ -560,17 +552,6 @@ impl GetObjectReader {
     }
 }
 
-struct ConditionStoppedBody;
-
-impl AsyncRead for ConditionStoppedBody {
-    fn poll_read(self: Pin<&mut Self>, _cx: &mut Context<'_>, buf: &mut ReadBuf<'_>) -> Poll<std::io::Result<()>> {
-        if buf.remaining() == 0 {
-            return Poll::Ready(Ok(()));
-        }
-        Poll::Ready(Err(std::io::Error::other("object read condition must be handled before body access")))
-    }
-}
-
 enum ReadTransform {
     // Written but never read by production code: the enclosing struct already
     // carries the same pair as `storage_offset`/`storage_length`. They survive

crates/ecstore/src/object_api/types.rs

@@ -938,10 +938,6 @@ pub struct ObjectOptions {
     /// A genuine target write guard; loss fences alone do not authorize a borrowed write.
     #[doc(hidden)]
     pub write_lock_context: Option<WriteLockContext>,
-    /// In-process delete ownership for late filesystem completion. Never
-    /// serialized or copied into object metadata.
-    #[doc(hidden)]
-    pub delete_mutation_owner: Option<Arc<dyn Send + Sync>>,
     /// Internal read-only inspection must not enqueue metadata or payload repairs.
     pub suppress_read_repair: bool,
     /// Control-plane writers that immediately read or CAS the same namespace
@@ -1006,9 +1002,6 @@ pub struct ObjectOptions {
     /// Proves an upper layer holds the bucket lifecycle sentinel. A separate
     /// fence avoids recursively acquiring the read lock behind a queued writer.
     pub bucket_lifecycle_lock_fence: Option<NamespaceLockFence>,
-    /// Retains the actual lifecycle guard for an owned multipart commit. A
-    /// lock-loss fence alone does not keep a cancelled caller's lock alive.
-    pub bucket_lifecycle_lock_owner: Option<Arc<rustfs_lock::NamespaceLockGuard>>,
     pub replication_request: bool,
     /// True when the inbound request carried the
     /// `{x-rustfs-,x-minio-}source-proxy-request` header family with the
@@ -1165,10 +1158,8 @@ impl std::fmt::Debug for ObjectOptions {
             .field("http_preconditions", &self.http_preconditions.is_some())
             .field("delete_replication", &self.delete_replication.is_some())
             .field("delete_replication_config_snapshot", &self.delete_replication_config_snapshot)
-            .field("delete_mutation_owner", &self.delete_mutation_owner.is_some())
             .field("namespace_lock_fence", &self.namespace_lock_fence.is_some())
             .field("bucket_lifecycle_lock_fence", &self.bucket_lifecycle_lock_fence.is_some())
-            .field("bucket_lifecycle_lock_owner", &self.bucket_lifecycle_lock_owner.is_some())
             .field("replication_request", &self.replication_request)
             .field("proxy_request", &self.proxy_request)
             .field("proxy_header_set", &self.proxy_header_set)

crates/ecstore/src/runtime/global.rs

@@ -307,13 +307,6 @@ pub fn shutdown_background_services() {
     }
 }
 
-/// Wait up to `timeout` for object mutations that outlived their callers
-/// (see `InstanceContext::wait_for_detached_mutations`). Returns `false` when
-/// some are still running at the deadline.
-pub async fn wait_for_detached_mutations(timeout: std::time::Duration) -> bool {
-    current_ctx().wait_for_detached_mutations(timeout).await
-}
-
 /// Set the global lock client (first LocalClient created)
 ///
 /// # Arguments

crates/ecstore/src/runtime/instance.rs

@@ -59,8 +59,6 @@ use std::sync::{
 use tokio::sync::{Mutex, Notify, OnceCell, OwnedRwLockReadGuard, RwLock};
 use tokio::time::Instant;
 use tokio_util::sync::CancellationToken;
-use tokio_util::task::TaskTracker;
-use tracing::Instrument as _;
 use uuid::Uuid;
 
 const SCANNER_PUBLICATION_STATE_UNKNOWN: u8 = 0;
@@ -245,10 +243,6 @@ pub struct InstanceContext {
     suppress_tier_delete_journal_recovery: bool,
     transition_transaction_recovery_stores: std::sync::Mutex<HashSet<Uuid>>,
     tier_delete_journal_recovery_wakeup: tokio::sync::Notify,
-    /// Object mutations that run on their own task so that dropping the
-    /// caller (for example, an S3 client disconnect) cannot stop them between
-    /// their on-disk steps. Shutdown waits for them after the HTTP drain.
-    detached_mutations: TaskTracker,
 }
 
 impl InstanceContext {
@@ -299,7 +293,6 @@ impl InstanceContext {
             suppress_tier_delete_journal_recovery: false,
             transition_transaction_recovery_stores: std::sync::Mutex::new(HashSet::new()),
             tier_delete_journal_recovery_wakeup: tokio::sync::Notify::new(),
-            detached_mutations: TaskTracker::new(),
         }
     }
 
@@ -313,31 +306,6 @@ impl InstanceContext {
         self.lock_manager.clone()
     }
 
-    /// Run a multi-step mutation on its own task and wait for its output.
-    ///
-    /// Dropping the returned future leaves the task running to completion, so
-    /// the mutation, together with the locks and guards it owns, never stops
-    /// between two of its on-disk steps because its caller went away.
-    pub(crate) async fn run_detached_mutation<F>(&self, mutation: F) -> Result<F::Output, tokio::task::JoinError>
-    where
-        F: std::future::Future + Send + 'static,
-        F::Output: Send + 'static,
-    {
-        self.detached_mutations.spawn(mutation.in_current_span()).await
-    }
-
-    /// Wait up to `timeout` for every detached mutation to finish. Returns
-    /// `false` when some are still running at the deadline.
-    pub async fn wait_for_detached_mutations(&self, timeout: std::time::Duration) -> bool {
-        self.detached_mutations.close();
-        tokio::time::timeout(timeout, self.detached_mutations.wait()).await.is_ok()
-    }
-
-    #[cfg(test)]
-    pub(crate) fn detached_mutation_count(&self) -> usize {
-        self.detached_mutations.len()
-    }
-
     pub(crate) fn data_movement_operation_gate(&self) -> Arc<RwLock<()>> {
         Arc::clone(&self.data_movement_operation_gate)
     }
@@ -952,33 +920,6 @@ mod tests {
         }
     }
 
-    // A detached mutation keeps running after its caller is dropped, and
-    // shutdown waits for it instead of tearing it down mid-way.
-    #[tokio::test]
-    async fn detached_mutation_outlives_dropped_caller_and_is_awaited() {
-        let ctx = InstanceContext::new();
-        let finished = Arc::new(AtomicBool::new(false));
-        let (release_tx, release_rx) = tokio::sync::oneshot::channel::<()>();
-        let mut caller = Box::pin(ctx.run_detached_mutation({
-            let finished = Arc::clone(&finished);
-            async move {
-                let _ = release_rx.await;
-                finished.store(true, Ordering::Release);
-            }
-        }));
-        assert!(futures::poll!(caller.as_mut()).is_pending());
-        drop(caller);
-        assert_eq!(ctx.detached_mutation_count(), 1, "dropping the caller must not cancel the mutation");
-
-        assert!(
-            !ctx.wait_for_detached_mutations(std::time::Duration::from_millis(50)).await,
-            "the wait must report a mutation that is still running"
-        );
-        release_tx.send(()).expect("the detached mutation should still be waiting");
-        assert!(ctx.wait_for_detached_mutations(std::time::Duration::from_secs(5)).await);
-        assert!(finished.load(Ordering::Acquire));
-    }
-
     // A fresh context (before any update) reflects the initial all-false state.
     #[tokio::test]
     async fn fresh_context_is_all_false() {

crates/ecstore/src/runtime/sources.rs

@@ -50,7 +50,6 @@ use s3s::dto::BucketLifecycleConfiguration;
 use s3s::region::Region;
 use tokio::sync::{OwnedRwLockReadGuard, RwLock};
 use tokio_util::sync::CancellationToken;
-#[cfg(test)]
 use tonic::transport::Channel;
 use uuid::Uuid;
 
@@ -252,6 +251,10 @@ pub(crate) async fn root_disk_threshold_for_erasure_disk() -> Option<u64> {
     }
 }
 
+pub(crate) async fn cached_node_channel(addr: &str) -> Option<Channel> {
+    rustfs_common::cached_connection(addr).await
+}
+
 #[cfg(test)]
 pub(crate) async fn cache_test_node_channel(addr: String, channel: Channel) {
     rustfs_common::cache_connection(addr, channel).await;

crates/ecstore/src/services/rebalance/control.rs

@@ -18,8 +18,8 @@ use super::{
     encode_rebalance_stop_propagation_record,
 };
 use crate::core::pools::{
-    PoolMeta, PoolRebalanceActivationFence, PoolRebalanceWorkerFence, acquire_pool_activation_fleet_proof,
-    acquire_pool_rebalance_activation_locks, acquire_pool_rebalance_worker_locks, pool_meta_has_active_decommission,
+    PoolMeta, PoolRebalanceActivationFence, acquire_pool_activation_fleet_proof, acquire_pool_rebalance_activation_locks,
+    pool_meta_has_active_decommission,
 };
 use crate::error::{Error, Result};
 use crate::object_api::ObjectOptions;
@@ -77,33 +77,10 @@ fn ensure_rebalance_activation_pool_meta_allowed(meta: &PoolMeta) -> Result<()>
 }
 
 pub(super) enum RebalanceWorkerActivationFence {
-    Ready(Box<PoolRebalanceWorkerFence>),
+    Ready(Box<PoolRebalanceActivationFence>),
     NotStartedTerminal,
 }
 
-async fn persisted_rebalance_requires_worker_activation<S>(pool: Arc<S>, expected_id: &str) -> Result<bool>
-where
-    S: EcstoreObjectIO,
-{
-    let mut persisted = RebalanceMeta::new();
-    persisted
-        .load_with_opts(
-            pool,
-            ObjectOptions {
-                no_lock: true,
-                ..Default::default()
-            },
-        )
-        .await?;
-    if persisted.id != expected_id {
-        return Err(Error::other(format!(
-            "rebalance metadata changed before worker activation: expected {expected_id}, found {}",
-            persisted.id
-        )));
-    }
-    Ok(crate::services::rebalance::rebalance_requires_worker_activation(&persisted))
-}
-
 pub(super) struct RebalanceRunGuard {
     _local_guard: tokio::sync::OwnedRwLockReadGuard<()>,
     persisted_guard: rustfs_lock::NamespaceLockGuard,
@@ -600,58 +577,48 @@ impl ECStore {
     where
         S: EcstoreObjectIO + StorageNamespaceLocking<Error = Error, NamespaceLock = rustfs_lock::NamespaceLockWrapper>,
     {
-        // Lock order: pool_meta_save_gate -> rebalance.bin -> pool.bin. These
-        // shared fences admit a peer without waiting for the run's own entries.
+        // Lock order: pool_meta_save_gate -> rebalance.bin -> pool.bin.
         let mut pool_meta_guard = self.pool_meta_save_gate.lock().await;
         pool_meta_guard.ensure_write_safe("rebalance worker activation")?;
-        let mut worker_fence = acquire_pool_rebalance_worker_locks(pool.clone()).await?;
+        // Classify the durable rebalance record while holding both namespace
+        // fences. A terminal record is a no-op and must not depend on the
+        // notification subsystem having published a fleet proof yet.
+        let mut activation_fence = acquire_pool_rebalance_activation_locks(pool.clone(), None).await?;
         let pool_meta = self
-            .load_runtime_pool_meta_under_worker_fence(&mut pool_meta_guard, &worker_fence, "rebalance worker activation")
+            .load_runtime_pool_meta_under_activation_fence(&mut pool_meta_guard, &activation_fence, "rebalance worker activation")
             .await?;
         ensure_rebalance_activation_pool_meta_allowed(&pool_meta)?;
-        let needs_activation = persisted_rebalance_requires_worker_activation(pool, expected_id).await?;
-        worker_fence.ensure_held()?;
-        if !needs_activation {
+
+        let mut persisted = RebalanceMeta::new();
+        persisted
+            .load_with_opts(
+                pool,
+                ObjectOptions {
+                    no_lock: true,
+                    ..Default::default()
+                },
+            )
+            .await?;
+        if persisted.id != expected_id {
+            return Err(Error::other(format!(
+                "rebalance metadata changed before worker activation: expected {expected_id}, found {}",
+                persisted.id
+            )));
+        }
+
+        activation_fence.ensure_held()?;
+        if !crate::services::rebalance::rebalance_requires_worker_activation(&persisted) {
             return Ok(RebalanceWorkerActivationFence::NotStartedTerminal);
         }
 
-        let fleet_proof = acquire_pool_activation_fleet_proof(&self.ctx).await?;
-        worker_fence.set_fleet_proof(fleet_proof);
-        worker_fence.ensure_held()?;
-        Ok(RebalanceWorkerActivationFence::Ready(Box::new(worker_fence)))
-    }
-
-    pub(super) async fn fence_rebalance_worker_metadata_update<S>(
-        &self,
-        pool: Arc<S>,
-        expected_id: &str,
-    ) -> Result<Option<Box<PoolRebalanceActivationFence>>>
-    where
-        S: EcstoreObjectIO + StorageNamespaceLocking<Error = Error, NamespaceLock = rustfs_lock::NamespaceLockWrapper>,
-    {
-        // The admission read fences must already be dropped. Reacquire in the
-        // original order and revalidate every durable boundary across that gap.
-        let mut pool_meta_guard = self.pool_meta_save_gate.lock().await;
-        pool_meta_guard.ensure_write_safe("rebalance worker metadata update")?;
-        let mut activation_fence = acquire_pool_rebalance_activation_locks(pool.clone(), None).await?;
-        let pool_meta = self
-            .load_runtime_pool_meta_under_activation_fence(
-                &mut pool_meta_guard,
-                &activation_fence,
-                "rebalance worker metadata update",
-            )
-            .await?;
-        ensure_rebalance_activation_pool_meta_allowed(&pool_meta)?;
-        let needs_activation = persisted_rebalance_requires_worker_activation(pool, expected_id).await?;
-        activation_fence.ensure_held()?;
-        if !needs_activation {
-            return Ok(None);
-        }
-
+        // Active worker admission still requires the fail-closed fleet proof.
+        // Attach it immediately before the final fence validation so expiry or
+        // topology changes are checked again at every later commit boundary.
         let fleet_proof = acquire_pool_activation_fleet_proof(&self.ctx).await?;
         activation_fence.set_fleet_proof(fleet_proof);
         activation_fence.ensure_held()?;
-        Ok(Some(Box::new(activation_fence)))
+
+        Ok(RebalanceWorkerActivationFence::Ready(Box::new(activation_fence)))
     }
 
     #[tracing::instrument(skip_all)]
@@ -1849,107 +1816,6 @@ mod tests {
         assert_activation_locks_released(&store).await;
     }
 
-    #[tokio::test]
-    #[serial_test::serial]
-    async fn rebalance_init_missing_fleet_proof_preserves_metadata_and_rechecks_competing_run() {
-        let previous_id = "completed-before-capability-refresh";
-        let terminal = RebalanceMeta {
-            id: previous_id.to_string(),
-            stopped_at: Some(OffsetDateTime::now_utc()),
-            pool_stats: vec![RebalanceStats::default(); 2],
-            ..Default::default()
-        };
-        let (_temp_dirs, store, competing_store) = crate::services::rebalance::test_two_pool_stores(Some(terminal.clone())).await;
-        let disk_stats = vec![
-            DiskStat {
-                total_space: 100,
-                available_space: 0,
-            },
-            DiskStat {
-                total_space: 100,
-                available_space: 100,
-            },
-        ];
-        let initial_epoch = store.ctx.data_movement_operation_epoch();
-        let proof_guard = crate::services::notification_sys::without_cross_pool_fence_fleet_proof_for_test();
-        set_rebalance_disk_stats_override_for_test(store.id, disk_stats.clone());
-        let err = store
-            .init_rebalance_start(vec!["waiting-request".to_string()])
-            .await
-            .expect_err("initialization must fail closed while the live proof is missing");
-        assert!(
-            err.to_string()
-                .contains("pool activation requires a live fleet capability proof")
-        );
-        assert!(
-            crate::core::pools::is_pool_activation_fleet_proof_error(&err),
-            "the admin handler retries init only on this error shape: {err}"
-        );
-        assert_eq!(store.ctx.data_movement_operation_epoch(), initial_epoch);
-        {
-            let local = store.rebalance_meta.read().await;
-            let local = local.as_ref().expect("failed initialization must preserve the previous run");
-            assert_eq!(local.id, previous_id);
-            assert_eq!(local.stopped_at, terminal.stopped_at);
-            assert!(local.cancel.is_none(), "failed initialization must not start a worker");
-        }
-        let mut persisted = RebalanceMeta::new();
-        persisted
-            .load(store.pools[0].clone())
-            .await
-            .expect("failed initialization must leave durable metadata readable");
-        assert_eq!(persisted.id, previous_id);
-        assert_eq!(persisted.stopped_at, terminal.stopped_at);
-
-        drop(proof_guard);
-        set_rebalance_disk_stats_override_for_test(competing_store.id, disk_stats.clone());
-        let winning_id = competing_store
-            .init_rebalance_start(vec!["winning-request".to_string()])
-            .await
-            .expect("a ready request must be able to initialize one run after the failed attempt");
-        assert_ne!(winning_id, previous_id);
-
-        // The waiting request still holds its old local snapshot. Its retry must
-        // recheck durable state under the activation locks before publishing.
-        set_rebalance_disk_stats_override_for_test(store.id, disk_stats);
-        let remote_conflict = store
-            .init_rebalance_start(vec!["waiting-request".to_string()])
-            .await
-            .expect_err("the waiting request must not replace a durable competing run");
-        assert!(
-            matches!(
-                crate::data_movement::data_movement_stage_source(&remote_conflict),
-                Some(Error::RebalanceAlreadyRunning)
-            ),
-            "the save-stage error must retain the typed competing-run conflict: {remote_conflict:?}"
-        );
-        let local_conflict = competing_store
-            .init_rebalance_start(vec!["another-request".to_string()])
-            .await
-            .expect_err("a local retry must not replace its active run");
-        assert!(
-            matches!(local_conflict, Error::RebalanceAlreadyRunning),
-            "the local admission check must report the competing-run conflict: {local_conflict:?}"
-        );
-        persisted
-            .load(store.pools[0].clone())
-            .await
-            .expect("the competing run must remain durable after both rejected retries");
-        assert_eq!(persisted.id, winning_id);
-        assert_eq!(persisted.pool_stats[0].buckets, vec!["winning-request".to_string()]);
-        assert_eq!(
-            store
-                .rebalance_meta
-                .read()
-                .await
-                .as_ref()
-                .expect("old local run must survive")
-                .id,
-            previous_id
-        );
-        assert_eq!(store.ctx.data_movement_operation_epoch(), initial_epoch + 1);
-    }
-
     #[tokio::test]
     #[serial_test::serial]
     async fn rebalance_worker_skips_terminal_metadata_without_fleet_proof() {
@@ -2008,128 +1874,6 @@ mod tests {
         );
     }
 
-    async fn assert_worker_write_upgrade_revalidates_durable_state(change: &str) {
-        let rebalance_id = "rebalance-worker-write-upgrade";
-        let active = RebalanceMeta {
-            id: rebalance_id.to_string(),
-            percent_free_goal: 0.5,
-            pool_stats: vec![
-                RebalanceStats {
-                    participating: true,
-                    init_capacity: 100,
-                    info: RebalanceInfo {
-                        status: RebalStatus::Started,
-                        ..Default::default()
-                    },
-                    ..Default::default()
-                },
-                RebalanceStats::default(),
-            ],
-            ..Default::default()
-        };
-        let (_temp_dirs, store, peer) =
-            crate::services::rebalance::test_two_pool_stores_with_isolated_node_contexts(Some(active.clone())).await;
-        let arrived = Arc::new(tokio::sync::Notify::new());
-        let release = Arc::new(tokio::sync::Notify::new());
-        let worker = Arc::clone(&store);
-        let mut tasks = tokio::task::JoinSet::new();
-        tasks.spawn(
-            super::super::runtime::REBALANCE_WORKER_WRITE_UPGRADE_BARRIER
-                .scope((Arc::clone(&arrived), Arc::clone(&release)), async move {
-                    worker.start_rebalance_for_id(rebalance_id).await
-                }),
-        );
-        tokio::time::timeout(std::time::Duration::from_secs(30), arrived.notified())
-            .await
-            .expect("empty-queue activation must release its read fences before requesting write authority");
-
-        let mut changed = active.clone();
-        match change {
-            "stopped" => {
-                changed.stopped_at = Some(OffsetDateTime::now_utc());
-                changed.pool_stats[0].info.status = RebalStatus::Stopped;
-            }
-            "replaced" => changed.id = "replacement-rebalance-run".to_string(),
-            "decommission" => {
-                let mut pool_meta = peer.pool_meta.read().await.clone();
-                pool_meta.pools[0].decommission = Some(crate::core::pools::PoolDecommissionInfo {
-                    start_time: Some(OffsetDateTime::now_utc()),
-                    ..Default::default()
-                });
-                pool_meta
-                    .save(peer.pools.clone())
-                    .await
-                    .expect("a competing pool metadata writer must acquire the released pool fence");
-            }
-            _ => panic!("unknown durable state transition"),
-        }
-        changed
-            .save_with_opts(
-                peer.pools[0].clone(),
-                ObjectOptions {
-                    write_completion: crate::object_api::WriteCompletion::TailDrained,
-                    ..Default::default()
-                },
-            )
-            .await
-            .expect("a competing run writer must acquire the released rebalance fence");
-        release.notify_one();
-        let result = tokio::time::timeout(std::time::Duration::from_secs(30), tasks.join_next())
-            .await
-            .expect("activation must finish after the durable transition")
-            .expect("activation task must be tracked")
-            .expect("activation must not panic");
-        match change {
-            "stopped" => result.expect("a durably stopped run must become an activation no-op"),
-            "replaced" => assert!(
-                result
-                    .expect_err("a replaced run must reject stale activation")
-                    .to_string()
-                    .contains("expected"),
-            ),
-            "decommission" => assert!(matches!(result, Err(Error::DecommissionAlreadyRunning))),
-            _ => unreachable!(),
-        }
-        let mut persisted = RebalanceMeta::new();
-        persisted
-            .load(peer.pools[0].clone())
-            .await
-            .expect("the competing durable state must remain readable");
-        assert_eq!(persisted.id, changed.id, "activation must not overwrite the run identity");
-        assert_eq!(persisted.stopped_at, changed.stopped_at, "activation must not overwrite the stop marker");
-        assert_eq!(persisted.pool_stats[0].info.status, changed.pool_stats[0].info.status);
-        assert!(
-            store
-                .rebalance_meta
-                .read()
-                .await
-                .as_ref()
-                .expect("local metadata must remain present")
-                .cancel
-                .is_none(),
-            "rejected activation must not install a worker token",
-        );
-        assert_activation_locks_released(&store).await;
-    }
-
-    #[tokio::test]
-    #[serial_test::serial]
-    async fn rebalance_worker_write_upgrade_rechecks_stop() {
-        assert_worker_write_upgrade_revalidates_durable_state("stopped").await;
-    }
-
-    #[tokio::test]
-    #[serial_test::serial]
-    async fn rebalance_worker_write_upgrade_rechecks_run_identity() {
-        assert_worker_write_upgrade_revalidates_durable_state("replaced").await;
-    }
-
-    #[tokio::test]
-    #[serial_test::serial]
-    async fn rebalance_worker_write_upgrade_rechecks_decommission() {
-        assert_worker_write_upgrade_revalidates_durable_state("decommission").await;
-    }
-
     #[tokio::test]
     #[serial_test::serial]
     async fn rebalance_activation_adopts_commit_after_post_save_fence_loss() {

crates/ecstore/src/services/rebalance/entry.rs

@@ -50,7 +50,7 @@ fn ensure_rebalance_entry_active(cancel: &CancellationToken) -> Result<()> {
     Ok(())
 }
 
-#[cfg(any(test, feature = "test-util"))]
+#[cfg(test)]
 tokio::task_local! {
     static REBALANCE_ENTRY_RUN_FENCE_BARRIER: (Arc<tokio::sync::Notify>, Arc<tokio::sync::Notify>);
 }
@@ -268,7 +268,7 @@ impl ECStore {
             self.rebalance_run_guard(rebalance_id.as_ref(), "rebalance entry")
         })
         .await?;
-        #[cfg(any(test, feature = "test-util"))]
+        #[cfg(test)]
         if let Ok((arrived, release)) = REBALANCE_ENTRY_RUN_FENCE_BARRIER.try_with(Clone::clone) {
             arrived.notify_one();
             release.notified().await;
@@ -836,8 +836,7 @@ pub mod test_util {
         _temp_dirs: Vec<tempfile::TempDir>,
         store: Arc<ECStore>,
         cancel: CancellationToken,
-        entry_arrived: Arc<tokio::sync::Notify>,
-        entry_release: Arc<tokio::sync::Notify>,
+        barrier: crate::data_movement::SourceCleanupDeleteBarrier,
         stop_probe: super::super::control::RebalanceStopWaitProbe,
         entry_task: Option<tokio::task::JoinHandle<Result<RebalanceEntryOutcome>>>,
     }
@@ -898,36 +897,31 @@ pub mod test_util {
                 cached: Some(file_meta),
                 reusable: false,
             };
-            let entry_arrived = Arc::new(tokio::sync::Notify::new());
-            let entry_release = Arc::new(tokio::sync::Notify::new());
+            let barrier = crate::data_movement::SourceCleanupDeleteBarrier::install(bucket, object);
             let entry_store = Arc::clone(&store);
             let entry_set = Arc::clone(&set);
             let entry_cancel = cancel.clone();
-            let entry_task = tokio::spawn(REBALANCE_ENTRY_RUN_FENCE_BARRIER.scope(
-                (Arc::clone(&entry_arrived), Arc::clone(&entry_release)),
-                async move {
-                    entry_store
-                        .rebalance_entry(
-                            RebalanceEntryTarget {
-                                bucket: bucket.to_string(),
-                                pool_index: 0,
-                            },
-                            entry,
-                            entry_set,
-                            Arc::new(RebalanceBucketConfigs::default()),
-                            Arc::from(rebalance_id),
-                            entry_cancel,
-                        )
-                        .await
-                },
-            ));
+            let entry_task = tokio::spawn(async move {
+                entry_store
+                    .rebalance_entry(
+                        RebalanceEntryTarget {
+                            bucket: bucket.to_string(),
+                            pool_index: 0,
+                        },
+                        entry,
+                        entry_set,
+                        Arc::new(RebalanceBucketConfigs::default()),
+                        Arc::from(rebalance_id),
+                        entry_cancel,
+                    )
+                    .await
+            });
 
             Self {
                 _temp_dirs,
                 store,
                 cancel,
-                entry_arrived,
-                entry_release,
+                barrier,
                 stop_probe: super::super::control::RebalanceStopWaitProbe::install(rebalance_id),
                 entry_task: Some(entry_task),
             }
@@ -938,9 +932,7 @@ pub mod test_util {
         }
 
         pub async fn wait_until_entry_paused(&self) {
-            tokio::time::timeout(std::time::Duration::from_secs(30), self.entry_arrived.notified())
-                .await
-                .expect("the entry should acquire its run fence before pausing");
+            self.barrier.wait_until_paused().await;
         }
 
         pub async fn wait_until_admission_cancelled(&self) {
@@ -954,7 +946,7 @@ pub mod test_util {
         }
 
         pub fn release_entry(&self) {
-            self.entry_release.notify_one();
+            self.barrier.release();
         }
 
         pub async fn assert_entry_cancelled(&mut self) {
@@ -972,7 +964,7 @@ pub mod test_util {
 
     impl Drop for PausedRebalanceEntryTestFixture {
         fn drop(&mut self) {
-            self.entry_release.notify_one();
+            self.barrier.release();
             if let Some(task) = self.entry_task.take() {
                 task.abort();
             }
@@ -1440,7 +1432,7 @@ mod tests {
 
     #[tokio::test]
     #[serial_test::serial]
-    async fn real_rebalance_peer_activates_while_entry_retains_run_fence() {
+    async fn real_rebalance_entry_progresses_while_peer_activation_waits_for_run_fence() {
         const REBALANCE_ID: &str = "rebalance-peer-activation-lock-order";
         let (_temp_dirs, store, peer) = crate::services::rebalance::test_two_pool_stores_with_isolated_node_contexts(Some(
             active_rebalance_meta(REBALANCE_ID),
@@ -1456,7 +1448,7 @@ mod tests {
         let bucket = crate::disk::RUSTFS_META_BUCKET;
         let object = "rebalance-peer-activation-object";
         let version_id = uuid::Uuid::new_v4();
-        let payload = b"peer activation must not wait for this entry to drain".repeat(1024);
+        let payload = b"entry must drain before peer activation takes the pool fence".repeat(1024);
         let source_set = store.pools[0].get_disks_by_key(object);
         let target_set = store.pools[1].get_disks_by_key(object);
         let opts = ObjectOptions {
@@ -1516,12 +1508,7 @@ mod tests {
         );
         tokio::time::timeout(StdDuration::from_secs(30), attempted.notified())
             .await
-            .expect("peer activation must attempt the persisted rebalance fence");
-        tokio::time::timeout(StdDuration::from_secs(30), activation_result)
-            .await
-            .expect("peer activation must finish while the entry remains paused")
-            .expect("peer activation result should be sent")
-            .expect("joining the same run must not require its active entry to release the run fence");
+            .expect("peer activation must attempt the persisted rebalance write fence");
         release.notify_one();
 
         tokio::time::timeout(StdDuration::from_secs(30), async {
@@ -1536,6 +1523,10 @@ mod tests {
         })
         .await
         .expect("entry and peer activation must both make progress");
+        activation_result
+            .await
+            .expect("peer activation result should be sent")
+            .expect("peer activation must not time out behind the entry it blocks");
 
         let mut reader = target_set
             .get_object_reader(bucket, object, None, HeaderMap::new(), &opts)