feat(diagnostics): attribute storage reads to request origins

Propagate S3 operation, caller, and entry context across metadata fanout,
list workers, and PUT commit owners. Preserve each coalescer item's context
and explicit absence so internal reads remain distinguishable from S3 work.
Add bounded attribution counters and focused regression coverage.

Co-Authored-By: heihutu <heihutu@gmail.com>
Co-Authored-By: zhi22915 <qiuzgang@gmail.com>

.agents/skills/security-advisory-lessons/references/advisory-patterns.md

@@ -45,13 +45,11 @@ Update this file only when an advisory adds or changes a reusable lesson, affect
 - `GHSA-6r96-hmgc-726c`: request headers collided with lowercase server-derived condition keys such as `userid`, `groups`, `versionid`, and JWT/LDAP claims. Lesson: never let caller-controlled headers append to or replace server-derived policy context; reserve trusted condition keys and keep intentional request-header keys separate.
 - `GHSA-v9cp-qfw9-9pfp`: quantified negated string conditions applied negation after aggregation, transposing `ForAllValues` and `ForAnyValue` semantics. Lesson: push negation into the per-value predicate for quantified operators and test partially overlapping multi-value sets.
 - `GHSA-5w8r-p896-6vq2`: OPA policy mode skipped `ExistingObjectTag/*` loading, so tagged objects looked untagged to external policies. Lesson: external authorization plugins need the same object-tag and request context as built-in policy evaluation before they decide.
-- `GHSA-9wgh-5r68-49p7`: substituted policy-variable values could recreate `${...}` and loop inside a single resolution pass, never reaching the outer iteration limit. Lesson: bound substitutions within each pass and test cycles formed across otherwise literal resolver values; confirm whether attacker-controlled claims can reach those values before assigning remote impact.
 
 ### S3 object actions, copy, multipart, and upload policy validation
 
 - `GHSA-g8w9-qw9q-fghr` and `GHSA-xm99-m3gq-83g8`: a valid presigned `PutObject` accepted extra `x-amz-*` headers omitted from `SignedHeaders`; `x-amz-copy-source` could turn that upload capability into a cross-bucket read performed as the signer. Lesson: a presigned URL is a bounded capability; inspect all received security-sensitive headers and reject unsigned ones before selecting the S3 operation or reaching storage.
 - `GHSA-3ppv-fx5m-m749`: explicit `versionId` reads and copy sources authorized `s3:GetObject` instead of `s3:GetObjectVersion`. Lesson: version-specific object access must select version-specific actions for direct reads, `CopyObject`, and `UploadPartCopy`, with tests proving the backend is not reached on denial.
-- `GHSA-cg33-pxfp-qcj3` and `GHSA-65wq-gjw8-xr9w`: `HeadObject` and object-tagging paths authorized current-object actions while operating on an explicit historical `versionId`. Lesson: select the corresponding version action for metadata and tag reads and writes too; audit every operation that accepts a version ID.
 - `GHSA-gjph-xcgw-4p7w`: `PutObject` and browser POST accepted a caller-supplied `versionId` and replaced an unlocked historical version with only `s3:PutObject`. Lesson: only replication-authorized writes may choose an existing version ID; ordinary uploads must not rewrite version history.
 - `GHSA-x298-9x87-fvjq`: anonymous `ListObjectVersions` fell back to `ListBucket` and returned before public-access-block gates. Lesson: compatibility fallbacks must converge on the same post-authorization checks as direct grants, especially `RestrictPublicBuckets` and anonymous data-plane denies.
 - `GHSA-mx42-j6wv-px98`: `UploadPartCopy` missed source authorization and allowed cross-bucket object exfiltration. Lesson: multipart copy must enforce the same source and destination contract as `CopyObject`.
@@ -112,7 +110,6 @@ Update this file only when an advisory adds or changes a reusable lesson, affect
 
 - `GHSA-j548-9grx-fh4f`: Object Lock enforcement treated unreadable, fabricated, or unparsable bucket metadata as absent configuration and allowed retained objects to be deleted or expired. Lesson: retention must fail closed unless Object Lock absence is authoritative, and every delete, lifecycle, scanner, force-delete, and default-retention path needs the same state distinction.
 - `GHSA-28w4-rp2p-636w` and `GHSA-c7hx-vr97-45rg`: an authorized `DeleteBucket` with `x-rustfs-force-delete: true` removed active COMPLIANCE-retained versions even though ordinary version deletion was denied. Lesson: bucket force-delete must enforce retention across every version, including against root/owner callers.
-- `GHSA-q23w-fp7p-472v`: `PutObjectRetention` policy conditions read `x-amz-object-lock-*` headers while the handler applied mode and date from the XML body. Lesson: authorize the parsed retention that will be stored, or reject mismatched header/body values; compare retention dates as instants so offsets cannot change the applied bound.
 
 ### Serde deserialization and input validation
 
@@ -146,13 +143,13 @@ rg -n "deny_unknown_fields|serde.default|as u32|as usize|as i32" rustfs crates
 - IAM fixes: include import/update/list service-account cases with attacker-controlled parent, claims, access key, secret key, and policy.
 - Copy/upload fixes: include cross-bucket, cross-user, source-denied, destination-denied, copy-source-condition, and multipart completion cases.
 - Presigned upload fixes: include a valid presign with extra unsigned tagging, redirect, storage-class, and cross-bucket copy-source headers; require rejection before source or destination storage access, and verify explicitly signed equivalents still work.
-- Version-action fixes: include historical UUID, explicit current version, `null`, range, partNumber, presigned, STS/session, service-account, anonymous bucket-policy, copy source, multipart-copy source, `HeadObject`, and tag read/write cases.
+- Version-action fixes: include historical UUID, explicit current version, `null`, range, partNumber, presigned, STS/session, service-account, anonymous bucket-policy, copy source, and multipart-copy source cases.
 - Version-write fixes: a `PutObject`-only principal must not replace an existing version through `PutObject` or browser POST `?versionId=`; verify replication-authorized writes and versioned, suspended, and unversioned buckets separately.
-- Policy-condition fixes: include reserved-key header collisions, missing keys, partially overlapping multi-value sets, cyclic variable substitutions, plugin mode, and built-in policy mode.
+- Policy-condition fixes: include reserved-key header collisions, missing keys, partially overlapping multi-value sets, plugin mode, and built-in policy mode.
 - Path fixes: include encoded traversal, absolute path, nested traversal, archive entries with `..`, valid object keys that resemble traversal text but should be rejected, and canonical bucket/prefix boundary checks; test per-key `DeleteObjects` errors and prefix-scoped denials on versioned buckets.
 - Logging fixes: assert redacted output for structs and response bodies that may contain credentials.
 - IAM export fixes: assert exported archives omit plaintext user and service-account secrets unless the format deliberately encrypts or seals them.
 - RPC auth fixes: include captured metadata replay across two concrete methods, stale timestamps, wrong path, wrong method surrogate, wrong secret, and valid same-method calls.
 - Browser/CORS fixes: assert no credentials on reflected/default origins, correct behavior for explicit allowlists, and no same-origin script execution for previewed object content.
 - SSE fixes: inspect stored bytes and verify API metadata, read-back behavior, and on-disk ciphertext together; for ETags, upload equal plaintext under different keys and assert list-only callers cannot derive or correlate the plaintext digest, including multipart objects.
-- Object Lock fixes: include unreadable metadata, fabricated metadata defaults, unparsable config, confirmed absent config, COMPLIANCE/GOVERNANCE retention, lifecycle expiry, scanner sweeps, and force-delete paths; an authorized force-delete must preserve active COMPLIANCE versions. For `PutObjectRetention`, test conflicting header/body values and offset dates against policy bounds and read-back retention.
+- Object Lock fixes: include unreadable metadata, fabricated metadata defaults, unparsable config, confirmed absent config, COMPLIANCE/GOVERNANCE retention, lifecycle expiry, scanner sweeps, and force-delete paths; an authorized force-delete must preserve active COMPLIANCE versions.

.config/e2e-full-selection.txt

@@ -1,2 +1,2 @@
-sha256-darwin=1520cd3027c3a6924afc6a204eddd5475bc30f0109d4c2847456a66177263244
-sha256-linux=26155b7f1c5bf6bf596ff24a4136e6767ea4ee6461c32db7fbe1400791da2c28
+sha256-darwin=98dff0961c6eb21d6128bfd2d4a6100a624ddf1fb7db05dba360a5d3d0a310c2
+sha256-linux=890f5925b87e1bf3710c356253f3b2276d901e156df9ccc04ec9315374de6d22

.config/e2e-nightly-selection.txt

@@ -1,2 +1,2 @@
-sha256-darwin=347a0d4e5e547d5388703062ce66a66167a845b6e57ec591ccb4c732a567c68d
-sha256-linux=4f9efa47194279185bb68249e82c788fa147abe944b6c764050e3d47f31e7926
+sha256-darwin=efe68675f4676a0d2901242c5815b1f7af5cdec8231f7cccd408f6dacf5c75db
+sha256-linux=003ed6480935a0a95680cc061be143722fbd37bd477b8ff98607b1a064f3aef1

.config/make/tests.mak

@@ -27,7 +27,6 @@ script-tests: ## Run shell script tests
 	trap 'status=$$?; printf "ERROR: script-tests failed (exit %s): %s\n" "$$status" "$$BASH_COMMAND" >&2; exit "$$status"' ERR
 	./scripts/test_build_rustfs_options.sh
 	./scripts/test_docker_runtime_timezone.sh
-	sh ./scripts/test_ca_bundle.sh
 	./scripts/test_entrypoint_credentials.sh
 	./scripts/test_internode_grpc_ab_bench.sh
 	./scripts/test_object_batch_bench_enhanced.sh
@@ -53,9 +52,6 @@ script-tests: ## Run shell script tests
 	./scripts/check_embedded_secrets.sh --self-test
 	$(RUSTFS_PYTHON_BIN) ./scripts/check_test_wiring.py --self-test
 	$(RUSTFS_PYTHON_BIN) ./scripts/test_e2e_binary.py
-	$(RUSTFS_PYTHON_BIN) ./scripts/test_release_build_workflow.py
-	$(RUSTFS_PYTHON_BIN) ./scripts/test_release_glibc.py
-	$(RUSTFS_PYTHON_BIN) ./scripts/test_upgrade_workflow.py
 	$(RUSTFS_PYTHON_BIN) ./scripts/test_migration_gate_evidence.py
 	$(RUSTFS_PYTHON_BIN) ./scripts/ci_gate.py --self-test
 	$(RUSTFS_PYTHON_BIN) ./scripts/check_security_coverage.py --self-test

.config/nextest.toml

@@ -88,11 +88,6 @@ threads-required = "num-test-threads"
 filter = 'package(rustfs-ecstore) & test(/^(bucket::lifecycle::bucket_lifecycle_ops::tests::manual_transition_worker_result_recovery_marks_unknown_for_corrupt_marker|services::rebalance::entry::tests::real_rebalance_run_fence_loss_blocks_multipart_publication|store::init::tests::(batch_transitioned_delete_uses_free_version_per_item|decommission_entry_(allows_free_version_consumed_before_source_lock|rejects_subquorum_free_version_conflict_and_retains_source|skips_cleanup_only_marker_when_free_version_is_present)|dispatched_tier_delete_recovery_(checks_later_pool_then_commits_after_source_removal|finds_directory_source_on_encoded_set|retains_journal_on_source_metadata_error)|force_tier_remove_blocks_on_physical_free_version_hidden_by_other_pool|legacy_unknown_transition_delete_falls_back_for_single_batch_and_blocks_prefix|multi_pool_(recursive_prefix_rejects_legacy_or_hidden_merge_loser_before_delete|same_remote_tuple_(batch|single)_delete_waits_for_all_sources|same_tuple_recursive_prefix_uses_one_journal_owner|transitioned_delete_persists_one_free_version_per_remote_tuple)|recursive_prefix_partial_(pool|set)_failure_keeps_prepared_cleanup_owners|restored_transitioned_delete_uses_free_version_as_cleanup_owner|stable_transitioned_recursive_prefix_delete_uses_journal_owners|suspended_null_transition_delete_uses_free_version_as_sole_owner|tier_mutation_peer_handler_applies_prepare_commit_and_abort_idempotently|transition_response_loss_persists_unknown_outcome_for_provider_recovery|transition_transaction_recovery_(drops_record_after_confirmed_local_commit|keeps_cleanup_pending_local_commit)|transitioned_delete_(free_version_replays_after_store_restart|local_quorum_failure_rolls_back_without_cleanup_owner|uses_free_version_as_cleanup_owner)|versioned_delete_marker_keeps_transitioned_source_and_remote_object|versioned_explicit_transition_delete_preserves_other_version_then_allows_bucket_delete))$/)'
 setup = 'ecstore-large-stack'
 
-# These cross-crate real-storage fixtures need the same bounded stack as ECStore tests.
-[[profile.default.scripts]]
-filter = '(package(rustfs) & test(/^(admin::handlers::site_replication::tests::durable_iam_state_apply_iam_item_rejects_a_stale_recreate_after_a_replicated_delete|admin::service::federated_identity::tests::(binding_keeps_explicit_iam_mapping_strict|markerless_legacy_oidc_session_crosses_auth_and_iam_deletion_boundaries)|storage::scanner_dirty_journal::tests::(clear_callback_keeps_unknown_coverage_until_all_pending_work_is_confirmed|owned_journals_keep_peer_records_after_interleaved_writes_and_clear))$/)) | (package(rustfs-heal) & ((binary(heal_b5_versioned_regression_test) & test(/^serial_tests::(test_heal_unversioned_bucket_e2e|test_historical_null_recursive_heal_restores_physical_versions|test_recursive_deep_heal_reports_stale_delete_marker_repair)$/)) | (binary(usage_observation_read_repair_test) & test(=usage_observation_read_repair_survives_snapshot_cleanup)))) | (package(rustfs-test-utils) & test(=data_usage_snapshot_tests::data_usage_snapshot_recovery_wires_primary_backup_and_negative_cache))'
-setup = 'ecstore-base-stack'
-
 [[profile.default.scripts]]
 filter = 'package(rustfs-ecstore) | package(rustfs-s3select-api) | package(rustfs-scanner) | (package(rustfs) & test(/^(app::multipart_usecase::tests::concurrent_completions_share_durable_bucket_quota_reservations|app::object::delete::tests::compressed_delete_requests_update_observed_usage_without_releasing_quota_floor|app::object::internal_put::tests::internal_multipart_roundtrip_completes_and_abort_leaves_nothing|app::object::restore::tests::execute_restore_object_maps_failures_to_typed_s3_errors|storage::access::tests::(delete_object_access_captures_authorized_bucket_incarnation|copy_operations_reject_recreated_source_bucket_after_authorization|request_slot_keeps_bucket_policy_bound_to_its_store))$/))'
 setup = 'ecstore-base-stack'
@@ -344,11 +339,6 @@ threads-required = "num-test-threads"
 filter = 'package(rustfs-ecstore) & test(/^(bucket::lifecycle::bucket_lifecycle_ops::tests::manual_transition_worker_result_recovery_marks_unknown_for_corrupt_marker|services::rebalance::entry::tests::real_rebalance_run_fence_loss_blocks_multipart_publication|store::init::tests::(batch_transitioned_delete_uses_free_version_per_item|decommission_entry_(allows_free_version_consumed_before_source_lock|rejects_subquorum_free_version_conflict_and_retains_source|skips_cleanup_only_marker_when_free_version_is_present)|dispatched_tier_delete_recovery_(checks_later_pool_then_commits_after_source_removal|finds_directory_source_on_encoded_set|retains_journal_on_source_metadata_error)|force_tier_remove_blocks_on_physical_free_version_hidden_by_other_pool|legacy_unknown_transition_delete_falls_back_for_single_batch_and_blocks_prefix|multi_pool_(recursive_prefix_rejects_legacy_or_hidden_merge_loser_before_delete|same_remote_tuple_(batch|single)_delete_waits_for_all_sources|same_tuple_recursive_prefix_uses_one_journal_owner|transitioned_delete_persists_one_free_version_per_remote_tuple)|recursive_prefix_partial_(pool|set)_failure_keeps_prepared_cleanup_owners|restored_transitioned_delete_uses_free_version_as_cleanup_owner|stable_transitioned_recursive_prefix_delete_uses_journal_owners|suspended_null_transition_delete_uses_free_version_as_sole_owner|tier_mutation_peer_handler_applies_prepare_commit_and_abort_idempotently|transition_response_loss_persists_unknown_outcome_for_provider_recovery|transition_transaction_recovery_(drops_record_after_confirmed_local_commit|keeps_cleanup_pending_local_commit)|transitioned_delete_(free_version_replays_after_store_restart|local_quorum_failure_rolls_back_without_cleanup_owner|uses_free_version_as_cleanup_owner)|versioned_delete_marker_keeps_transitioned_source_and_remote_object|versioned_explicit_transition_delete_preserves_other_version_then_allows_bucket_delete))$/)'
 setup = 'ecstore-large-stack'
 
-# These cross-crate real-storage fixtures need the same bounded stack as ECStore tests.
-[[profile.ci.scripts]]
-filter = '(package(rustfs) & test(/^(admin::handlers::site_replication::tests::durable_iam_state_apply_iam_item_rejects_a_stale_recreate_after_a_replicated_delete|admin::service::federated_identity::tests::(binding_keeps_explicit_iam_mapping_strict|markerless_legacy_oidc_session_crosses_auth_and_iam_deletion_boundaries)|storage::scanner_dirty_journal::tests::(clear_callback_keeps_unknown_coverage_until_all_pending_work_is_confirmed|owned_journals_keep_peer_records_after_interleaved_writes_and_clear))$/)) | (package(rustfs-heal) & ((binary(heal_b5_versioned_regression_test) & test(/^serial_tests::(test_heal_unversioned_bucket_e2e|test_historical_null_recursive_heal_restores_physical_versions|test_recursive_deep_heal_reports_stale_delete_marker_repair)$/)) | (binary(usage_observation_read_repair_test) & test(=usage_observation_read_repair_survives_snapshot_cleanup)))) | (package(rustfs-test-utils) & test(=data_usage_snapshot_tests::data_usage_snapshot_recovery_wires_primary_backup_and_negative_cache))'
-setup = 'ecstore-base-stack'
-
 [[profile.ci.scripts]]
 filter = 'package(rustfs-ecstore) | package(rustfs-s3select-api) | package(rustfs-scanner) | (package(rustfs) & test(/^(app::multipart_usecase::tests::concurrent_completions_share_durable_bucket_quota_reservations|app::object::delete::tests::compressed_delete_requests_update_observed_usage_without_releasing_quota_floor|app::object::internal_put::tests::internal_multipart_roundtrip_completes_and_abort_leaves_nothing|app::object::restore::tests::execute_restore_object_maps_failures_to_typed_s3_errors|storage::access::tests::(delete_object_access_captures_authorized_bucket_incarnation|copy_operations_reject_recreated_source_bucket_after_authorization|request_slot_keeps_bucket_policy_bound_to_its_store))$/))'
 setup = 'ecstore-base-stack'

.docker/observability/grafana/dashboards/rustfs.json

@@ -1748,7 +1748,7 @@
             "uid": "${datasource}"
           },
           "editorMode": "code",
-          "expr": "max by (rustfs_cluster_id, job, bucket) (rustfs_cluster_usage_buckets_objects_count{rustfs_cluster_id=\"$storage_cluster\", job=~\"$job\", bucket=~\"$bucket\"})",
+          "expr": "max by (job, bucket) (rustfs_cluster_usage_buckets_objects_count{job=~\"$job\", bucket=~\"$bucket\"})",
           "legendFormat": "{{bucket}}",
           "range": true,
           "refId": "A"
@@ -1848,7 +1848,7 @@
             "uid": "${datasource}"
           },
           "editorMode": "code",
-          "expr": "max by (rustfs_cluster_id, job, bucket) (rustfs_cluster_usage_buckets_total_bytes{rustfs_cluster_id=\"$storage_cluster\", job=~\"$job\", bucket=~\"$bucket\"})",
+          "expr": "max by (job, bucket) (rustfs_cluster_usage_buckets_total_bytes{job=~\"$job\", bucket=~\"$bucket\"})",
           "legendFormat": "{{bucket}}",
           "range": true,
           "refId": "A"

.docker/observability/tests/bucket-usage.test.yml

@@ -1,48 +0,0 @@
-evaluation_interval: 1m
-tests:
-- name: bucket usage keeps clusters separate and deduplicates reporting nodes
-  interval: 1m
-  input_series:
-  - series: rustfs_cluster_usage_buckets_total_bytes{rustfs_cluster_id="a",job="rustfs",bucket="shared",service_instance_id="a1"}
-    values: '4096 1024'
-  - series: rustfs_cluster_usage_buckets_total_bytes{rustfs_cluster_id="a",job="rustfs",bucket="shared",service_instance_id="a2"}
-    values: '4096 1024'
-  - series: rustfs_cluster_usage_buckets_total_bytes{rustfs_cluster_id="b",job="rustfs",bucket="shared",service_instance_id="b1"}
-    values: '65536 65536'
-  - series: rustfs_cluster_usage_buckets_total_bytes{rustfs_cluster_id="a",job="rustfs",bucket="empty",service_instance_id="a1"}
-    values: '0 0'
-  - series: rustfs_cluster_usage_buckets_objects_count{rustfs_cluster_id="a",job="rustfs",bucket="shared",service_instance_id="a1"}
-    values: '3 1'
-  - series: rustfs_cluster_usage_buckets_objects_count{rustfs_cluster_id="a",job="rustfs",bucket="shared",service_instance_id="a2"}
-    values: '3 1'
-  - series: rustfs_cluster_usage_buckets_objects_count{rustfs_cluster_id="b",job="rustfs",bucket="shared",service_instance_id="b1"}
-    values: '70 70'
-  - series: rustfs_cluster_usage_buckets_objects_count{rustfs_cluster_id="a",job="rustfs",bucket="empty",service_instance_id="a1"}
-    values: '0 0'
-  promql_expr_test:
-  - expr: max by (rustfs_cluster_id, job, bucket) (rustfs_cluster_usage_buckets_total_bytes{rustfs_cluster_id="a",job=~"rustfs",bucket=~".*"})
-    eval_time: 1m
-    exp_samples:
-    - labels: '{rustfs_cluster_id="a",job="rustfs",bucket="shared"}'
-      value: 1024
-    - labels: '{rustfs_cluster_id="a",job="rustfs",bucket="empty"}'
-      value: 0
-  - expr: max by (rustfs_cluster_id, job, bucket) (rustfs_cluster_usage_buckets_objects_count{rustfs_cluster_id="a",job=~"rustfs",bucket=~".*"})
-    eval_time: 1m
-    exp_samples:
-    - labels: '{rustfs_cluster_id="a",job="rustfs",bucket="shared"}'
-      value: 1
-    - labels: '{rustfs_cluster_id="a",job="rustfs",bucket="empty"}'
-      value: 0
-  - expr: max by (rustfs_cluster_id, job, bucket) (rustfs_cluster_usage_buckets_total_bytes)
-    eval_time: 1m
-    exp_samples:
-    - labels: '{rustfs_cluster_id="a",job="rustfs",bucket="shared"}'
-      value: 1024
-    - labels: '{rustfs_cluster_id="a",job="rustfs",bucket="empty"}'
-      value: 0
-    - labels: '{rustfs_cluster_id="b",job="rustfs",bucket="shared"}'
-      value: 65536
-  - expr: rustfs_cluster_usage_buckets_total_bytes{rustfs_cluster_id="a",bucket="unknown"}
-    eval_time: 1m
-    exp_samples: []

.github/actions/setup-docker-compose/action.yml

@@ -1,39 +0,0 @@
-# Copyright 2024 RustFS Team
-#
-# Licensed under the Apache License, Version 2.0 (the "License");
-# you may not use this file except in compliance with the License.
-# You may obtain a copy of the License at
-#
-#     http://www.apache.org/licenses/LICENSE-2.0
-#
-# Unless required by applicable law or agreed to in writing, software
-# distributed under the License is distributed on an "AS IS" BASIS,
-# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
-# See the License for the specific language governing permissions and
-# limitations under the License.
-
-name: Set up Docker Compose
-description: Install the verified Compose CLI used by native Linux x86_64 Connect acceptance
-
-runs:
-  using: composite
-  steps:
-    - name: Install verified Docker Compose CLI plugin
-      shell: bash
-      run: |
-        set -euo pipefail
-        [[ $(uname -s) == Linux && $(uname -m) == x86_64 ]]
-        plugin_dir="${DOCKER_CONFIG:-$HOME/.docker}/cli-plugins"
-        mkdir -p "$plugin_dir"
-        compose_download=$(mktemp "${RUNNER_TEMP:?}/connect-compose.XXXXXX")
-        compose_staged=$(mktemp "$plugin_dir/.docker-compose.XXXXXX")
-        trap 'rm -f "$compose_download" "$compose_staged"' EXIT
-        curl --fail --location --retry 3 \
-          https://github.com/docker/compose/releases/download/v2.40.3/docker-compose-linux-x86_64 \
-          --output "$compose_download"
-        printf '%s  %s\n' \
-          dba9d98e1ba5bfe11d88c99b9bd32fc4a0624a30fafe68eea34d61a3e42fd372 \
-          "$compose_download" | sha256sum --check --strict
-        install -m 0755 "$compose_download" "$compose_staged"
-        mv -f "$compose_staged" "$plugin_dir/docker-compose"
-        [[ $(docker compose version --short) == 2.40.3 ]]

.github/actions/setup/action.yml

@@ -36,16 +36,6 @@ inputs:
     description: "Install cross-compilation tools"
     required: false
     default: "false"
-  zig-version:
-    description: >-
-      Zig version for cargo-zigbuild builds. Pinned so the `<target>.<glibc>`
-      baseline semantics cannot drift with an untested zig release.
-    required: false
-    default: "0.17.0"
-  cargo-zigbuild-version:
-    description: "cargo-zigbuild version installed for cross builds."
-    required: false
-    default: "0.23.4"
   target:
     description: "Target architecture to add"
     required: false
@@ -109,14 +99,10 @@ runs:
     - name: Install Zig
       if: inputs.install-cross-tools == 'true'
       uses: mlugg/setup-zig@d1434d08867e3ee9daa34448df10607b98908d29 # v2
-      with:
-        version: ${{ inputs.zig-version }}
 
     - name: Install cargo-zigbuild
       if: inputs.install-cross-tools == 'true'
       uses: taiki-e/install-action@a21ae4029b089b9ddc45704028756f51ab8abe48 # cargo-zigbuild
-      with:
-        tool: cargo-zigbuild@${{ inputs.cargo-zigbuild-version }}
 
     - name: Install cargo-nextest
       if: inputs.install-test-tools == 'true'
@@ -140,5 +126,5 @@ runs:
         # normalises, so cargo fingerprints stay valid.
         cache-all-crates: false
         cache-on-failure: true
-        shared-key: ${{ inputs.cache-shared-key }}${{ inputs.install-cross-tools == 'true' && format('-zig-{0}-zigbuild-{1}', inputs.zig-version, inputs.cargo-zigbuild-version) || '' }}
+        shared-key: ${{ inputs.cache-shared-key }}
         save-if: ${{ inputs.cache-save-if }}

.github/workflows/build.yml

@@ -213,12 +213,12 @@ jobs:
           fi
 
           all='{"include":[
-            {"target_id":"linux-x86_64-musl","os":"sm-standard-4","target":"x86_64-unknown-linux-musl","cross":false,"platform":"linux","rustflags":"","features":"","glibc":""},
-            {"target_id":"linux-aarch64-musl","os":"sm-standard-4-arm","target":"aarch64-unknown-linux-musl","cross":false,"platform":"linux","rustflags":"","features":"","glibc":""},
-            {"target_id":"linux-x86_64-gnu","os":"sm-standard-4","target":"x86_64-unknown-linux-gnu","cross":true,"platform":"linux","rustflags":"","features":"pyroscope","glibc":"2.31"},
-            {"target_id":"linux-aarch64-gnu","os":"sm-standard-4-arm","target":"aarch64-unknown-linux-gnu","cross":true,"platform":"linux","rustflags":"","features":"pyroscope","glibc":"2.31"},
-            {"target_id":"macos-aarch64","os":"macos-latest","target":"aarch64-apple-darwin","cross":false,"platform":"macos","rustflags":"","features":"pyroscope","glibc":""},
-            {"target_id":"windows-x86_64","os":"windows-latest","target":"x86_64-pc-windows-msvc","cross":false,"platform":"windows","rustflags":"","features":"","glibc":""}
+            {"target_id":"linux-x86_64-musl","os":"sm-standard-4","target":"x86_64-unknown-linux-musl","cross":false,"platform":"linux","rustflags":"","features":""},
+            {"target_id":"linux-aarch64-musl","os":"sm-standard-4-arm","target":"aarch64-unknown-linux-musl","cross":false,"platform":"linux","rustflags":"","features":""},
+            {"target_id":"linux-x86_64-gnu","os":"sm-standard-4","target":"x86_64-unknown-linux-gnu","cross":false,"platform":"linux","rustflags":"","features":"pyroscope"},
+            {"target_id":"linux-aarch64-gnu","os":"sm-standard-4-arm","target":"aarch64-unknown-linux-gnu","cross":false,"platform":"linux","rustflags":"","features":"pyroscope"},
+            {"target_id":"macos-aarch64","os":"macos-latest","target":"aarch64-apple-darwin","cross":false,"platform":"macos","rustflags":"","features":"pyroscope"},
+            {"target_id":"windows-x86_64","os":"windows-latest","target":"x86_64-pc-windows-msvc","cross":false,"platform":"windows","rustflags":"","features":""}
           ]}'
 
           if [[ "${selected}" == "all" ]]; then
@@ -246,8 +246,7 @@ jobs:
           echo "Selected platforms: ${selected}"
 
   build-rustfs:
-    # Acceptance workflows select artifacts by this stable platform name.
-    name: Build RustFS (${{ matrix.target_id }})
+    name: Build RustFS
     needs: [ build-check, prepare-platform-matrix ]
     if: needs.build-check.outputs.should_build == 'true' && needs.prepare-platform-matrix.result == 'success'
     runs-on: ${{ matrix.os }}
@@ -274,7 +273,7 @@ jobs:
         with:
           rust-version: stable
           target: ${{ matrix.target }}
-          cache-shared-key: build-${{ matrix.target }}${{ matrix.glibc && format('-glibc-{0}', matrix.glibc) || '' }}
+          cache-shared-key: build-${{ matrix.target }}
           # main only. A cache saved on refs/tags/X is scoped to that tag: no
           # other tag, no main run and no PR can restore it, so every release
           # cycle wrote up to 12 entries of 1-2GB (preview tag plus final tag,
@@ -322,8 +321,7 @@ jobs:
           download_console_assets() {
             local curl_bin="curl"
             local python_bin="python3"
-            # Preserve the reviewed 1.0.3 Console in this security-only release branch.
-            local console_api="https://api.github.com/repos/rustfs/console/releases/tags/v0.1.37"
+            local console_api="https://api.github.com/repos/rustfs/console/releases/latest"
             local console_json="console-release.json"
             local console_url
             local console_sha256
@@ -360,15 +358,12 @@ jobs:
               release = json.load(handle)
 
           tag = release.get("tag_name", "") or "unknown"
-          if tag != "v0.1.37":
-              raise SystemExit("Console release does not match the security-release pin")
-          expected_digest = "sha256:e6ff2c7e7695de824ec4cda91d8611d1aba5d8462e6809592c92a50b23f5b9ab"
 
           for asset in release.get("assets", []):
               name = asset.get("name", "")
               digest = asset.get("digest", "")
               url = asset.get("browser_download_url", "")
-              if name == "rustfs-console-v0.1.37.zip" and digest == expected_digest and url:
+              if name.startswith("rustfs-console-") and name.endswith(".zip") and digest.startswith("sha256:") and url:
                   sha256 = digest.split(":", 1)[1]
                   if not re.fullmatch(r"[0-9a-fA-F]{64}", sha256):
                       raise SystemExit(f"console zip asset has invalid sha256 digest: {sha256}")
@@ -415,53 +410,35 @@ jobs:
         shell: bash
         env:
           BUILD_TARGET: ${{ matrix.target }}
-          BUILD_OS: ${{ matrix.platform }}-${{ startsWith(matrix.target, 'aarch64-') && 'aarch64' || 'x86_64' }}
         run: |
           # Force rebuild by touching build.rs
           touch rustfs/build.rs
 
+          BINARY_NAMES=(rustfs)
+          if [[ "${{ matrix.platform }}" == "linux" ]]; then
+            BINARY_NAMES+=(rustfs-cli)
+          fi
           FEATURE_ARGS=()
           if [[ -n "${{ matrix.features }}" ]]; then
             FEATURE_ARGS=(--features "${{ matrix.features }}")
           fi
 
-          # gnu targets pin an explicit glibc baseline (`<target>.<glibc>` for
-          # cargo-zigbuild) so release binaries link no symbols newer than it
-          # and run on older distros such as RHEL/Rocky 9 (glibc 2.34); #8062.
-          TARGET_ARG="$BUILD_TARGET"
-          if [[ -n "${{ matrix.glibc }}" ]]; then
-            TARGET_ARG="${TARGET_ARG}.${{ matrix.glibc }}"
-          fi
-
-          PROFILE_ARGS=()
-          if [[ "$BUILD_TARGET" == *-linux-gnu && "${{ matrix.features }}" == "pyroscope" ]]; then
-            # The in-process profiler resolves names from the final executable.
-            # Retain function symbols without enabling DWARF source information.
-            PROFILE_ARGS=(--config 'profile.release.strip="none"')
-          fi
-          if [[ "${{ matrix.cross }}" == "true" ]]; then
-            cargo zigbuild --release --locked --target "$TARGET_ARG" "${FEATURE_ARGS[@]}" "${PROFILE_ARGS[@]}" -p rustfs --bin rustfs
-          else
-            cargo build --release --locked --target "$TARGET_ARG" "${FEATURE_ARGS[@]}" "${PROFILE_ARGS[@]}" -p rustfs --bin rustfs
-          fi
-
-          # All release targets use same-architecture runners, including GNU
-          # builds that use Zig only to select the glibc ABI floor.
-          bin="target/${{ matrix.target }}/release/rustfs"
-          if [[ -f "${bin}.exe" ]]; then
-            bin="${bin}.exe"
-          fi
-          version_output=$("$bin" --version)
-          if ! grep -Fxq "build target : ${BUILD_TARGET}" <<< "$version_output" || \
-             ! grep -Fxq "build os     : ${BUILD_OS}" <<< "$version_output" || \
-             ! grep -Fxq "rust channel : stable-${BUILD_TARGET}" <<< "$version_output"; then
-            echo "release binary reports an unexpected compile target" >&2
-            exit 1
-          fi
-          if [[ -n "${{ matrix.glibc }}" ]]; then
-            python3 scripts/release/check_glibc.py "${{ matrix.glibc }}" \
-              "target/${{ matrix.target }}/release/rustfs"
-          fi
+          # Build one binary at a time so release LTO links cannot exhaust a
+          # self-hosted runner by running concurrently.
+          for binary in "${BINARY_NAMES[@]}"; do
+            PROFILE_ARGS=()
+            if [[ "$binary" == "rustfs" && "$BUILD_TARGET" == *-linux-gnu && "${{ matrix.features }}" == "pyroscope" ]]; then
+              # The in-process profiler resolves names from the final executable.
+              # Retain function symbols without enabling DWARF source information.
+              PROFILE_ARGS=(--config 'profile.release.strip="none"')
+            fi
+            if [[ "${{ matrix.cross }}" == "true" ]]; then
+              # All cross targets in the matrix are Linux; zigbuild handles them.
+              cargo zigbuild --release --target ${{ matrix.target }} "${FEATURE_ARGS[@]}" "${PROFILE_ARGS[@]}" -p rustfs --bin "$binary"
+            else
+              cargo build --release --target ${{ matrix.target }} "${FEATURE_ARGS[@]}" "${PROFILE_ARGS[@]}" -p rustfs --bin "$binary"
+            fi
+          done
 
       - name: Verify macOS binary has no dynamic liblzma dependency
         if: matrix.platform == 'macos'
@@ -577,6 +554,9 @@ jobs:
           fi
 
           BINARY_FILES=("$BINARY_NAME")
+          if [[ "${{ matrix.platform }}" == "linux" ]]; then
+            BINARY_FILES+=("rustfs-cli")
+          fi
 
           if [[ "$TARGET" == *-linux-gnu && "${{ matrix.features }}" == "pyroscope" ]]; then
             BINARY_FILES+=(rustfs.cpu-symbol-catalog.json rustfs.cpu-symbol-catalog.sha256)
@@ -691,7 +671,7 @@ jobs:
           echo "🔧 Build type: ${BUILD_TYPE}"
           echo "📊 Version: ${VERSION}"
 
-      - name: Verify packaged Linux binary
+      - name: Verify packaged Linux binaries
         if: matrix.platform == 'linux'
         shell: bash
         run: |
@@ -699,7 +679,7 @@ jobs:
 
           package_file="${{ steps.package.outputs.package_file }}"
           archive_members="$(unzip -Z1 "$package_file" | sort)"
-          expected_members='rustfs'
+          expected_members=$'rustfs\nrustfs-cli'
           target="${{ matrix.target }}"
           features="${{ matrix.features }}"
           if [[ "$target" == *-linux-gnu && "$features" == "pyroscope" ]]; then
@@ -711,14 +691,13 @@ jobs:
             exit 1
           fi
 
-          # musl targets build natively; the gnu legs are same-arch
-          # cargo-zigbuild builds pinned to the release glibc baseline. Either
-          # way the binaries run on their runner, so executing them here
+          # Every Linux target builds natively, so executing the binaries here
           # proves each artifact matches its runner architecture.
-          binary_dir="$(mktemp -d)"
-          trap 'rm -rf "$binary_dir"' EXIT
-          unzip -q "$package_file" rustfs -d "$binary_dir"
-          "$binary_dir/rustfs" --version
+          cli_dir="$(mktemp -d)"
+          trap 'rm -rf "$cli_dir"' EXIT
+          unzip -q "$package_file" rustfs rustfs-cli -d "$cli_dir"
+          "$cli_dir/rustfs" --version
+          "$cli_dir/rustfs-cli" --help >/dev/null
 
       - name: Verify packaged console
         if: matrix.target == 'x86_64-unknown-linux-gnu' || matrix.target == 'aarch64-unknown-linux-gnu'

.github/workflows/cache-warm.yml

@@ -99,7 +99,7 @@ env:
   RUST_BACKTRACE: 1
 
 jobs:
-  # Readers: test-and-lint (workspace and ILM), build-rustfs-debug-binary,
+  # Readers: test-and-lint, test-ilm-integration-serial, build-rustfs-debug-binary,
   # e2e-tests, e2e-full.
   warm-ci-dev:
     name: Warm ci-dev

.github/workflows/ci.yml

@@ -168,10 +168,13 @@ jobs:
             echo "started_at=$(date --utc --iso-8601=seconds)"
           } > artifacts/test-and-lint/run-metadata.txt
 
-      # Keep full Clippy validation on main, merge groups, and scheduled/manual
-      # runs. PRs retain the complete default test suite and compile-free checks.
+      # Clippy runs before the test pass: lint failures are the most common
+      # CI-only breakage and should surface in minutes, not after 20+ minutes
+      # of tests.
+      # Sampled too: clippy is the natural control arm for any CARGO_BUILD_JOBS
+      # experiment, since --all-targets is check-only for workspace members and
+      # never links the ~100 test binaries the limit exists to throttle.
       - name: Run clippy lints
-        if: github.event_name != 'pull_request'
         run: |
           ./scripts/ci/resource_sampler.sh start clippy
           trap './scripts/ci/resource_sampler.sh stop' EXIT
@@ -280,49 +283,6 @@ jobs:
       - name: Verify rebalance/decommission migration proofs
         run: ./scripts/check_migration_gate_count.sh evidence artifacts/test-and-lint/core-test-listing.json target/nextest/ci/junit.xml
 
-      # The workspace already compiled these ignored tests. Keep the same Cargo
-      # package selection to reuse their binaries, then run them serially in
-      # separate processes because they share fixed ports and global state.
-      # test_transition_and_restore_flows was re-enabled by rustfs/backlog#1303:
-      # its "missing xl.meta on disk2" was a test-util bug (open_disk hardcoded
-      # disk_index 0), not an EC metadata-distribution issue.
-      # restore_object_usecase_reports_ongoing_conflict_and_completion was
-      # re-enabled by backlog#1304 (restore accepts serialize on a short CAS
-      # guard; the copy-back no longer holds the #4877 whole-copy-back lock,
-      # so the mid-restore ongoing read and fast 409 rejection it asserts are
-      # the implemented contract).
-      - name: Run ignored ILM integration tests serially
-        if: needs.classify-changes.outputs.mode == 'full'
-        env:
-          # Match the preceding workspace build and reuse its feature graph.
-          CARGO_BUILD_JOBS: ${{ (github.event_name == 'push' || github.event_name == 'workflow_dispatch') && '3' || '2' }}
-        run: |
-          mkdir -p artifacts/ilm-integration
-          set +e
-          NEXTEST_HIDE_PROGRESS_BAR=1 timeout --verbose --signal=TERM --kill-after=30s 80m \
-            cargo nextest run --profile default -j1 --run-ignored ignored-only \
-            --all --exclude e2e_test \
-            -E 'binary(lifecycle_integration_test) or (package(rustfs) and test(lifecycle_transition_api_test))' \
-            --status-level all --final-status-level all \
-            2>&1 | tee artifacts/ilm-integration/nextest.log
-          status=${PIPESTATUS[0]}
-          {
-            echo "exit_status=${status}"
-            echo "finished_at=$(date --utc --iso-8601=seconds)"
-            echo
-            echo "Remaining test-related processes:"
-            pgrep -af 'cargo|nextest|target/.*/deps/' || true
-          } > artifacts/ilm-integration/diagnostics.txt
-          exit "${status}"
-
-      - name: Upload ILM test diagnostics
-        if: always() && needs.classify-changes.outputs.mode == 'full'
-        uses: actions/upload-artifact@043fb46d1a93c77aae656e7c1c64a875d1fc6a0a # v7.0.1
-        with:
-          name: ilm-integration-${{ github.run_number }}-${{ github.run_attempt }}
-          path: artifacts/ilm-integration
-          retention-days: 3
-
   # This gate builds into a fresh target directory to isolate the E2E root.
   # Give its cold build a separate budget from workspace tests and migration proofs.
   offline-enrollment-e2e:
@@ -354,11 +314,83 @@ jobs:
       - name: Check offline enrollment E2E root boundary
         run: ./scripts/check_offline_enrollment_e2e.sh
 
+  # Dedicated serial lane for the ILM / lifecycle integration tests. These tests
+  # drive the object layer through process-global singletons (the GLOBAL_ENV
+  # ECStore, the global tier-config manager, background-expiry workers) and bind
+  # fixed ports (9002 for the scanner suite, 9003 for the app suite), so they are
+  # marked #[ignore] to keep them out of the default parallel `cargo nextest run
+  # --all` pass. Run them here explicitly with `--run-ignored ignored-only` and
+  # `-j1` so no two of them share global state or race for a port at once.
+  # serial_test's #[serial] does NOT serialize across nextest's process-per-test
+  # boundary (see .config/nextest.toml); `-j1` is what actually serializes them.
+  # See rustfs/backlog#1148 (ilm-1) and #1155.
+  test-ilm-integration-serial:
+    name: ILM Integration (serial)
+    if: needs.classify-changes.outputs.mode == 'full' && (github.event_name != 'pull_request' || github.event.action != 'closed')
+    needs: [ quick-checks, classify-changes ]
+    runs-on: sm-standard-4
+    timeout-minutes: 90
+    env:
+      FORCE_JAVASCRIPT_ACTIONS_TO_NODE24: "true"
+    steps:
+      - name: Checkout repository
+        uses: actions/checkout@f548e57e544e1ff5a4c46bf1e1b8685f8e4a348a # v7
+        with:
+          persist-credentials: false
+
+      - name: Setup Rust environment
+        uses: ./.github/actions/setup
+        with:
+          rust-version: stable
+          cache-shared-key: ci-dev
+          cache-save-if: 'false'
+          install-build-packaging-tools: 'false'
+
+      # test_transition_and_restore_flows was re-enabled by rustfs/backlog#1303:
+      # its "missing xl.meta on disk2" was a test-util bug (open_disk hardcoded
+      # disk_index 0), not an EC metadata-distribution issue.
+      # restore_object_usecase_reports_ongoing_conflict_and_completion was
+      # re-enabled by backlog#1304 (restore accepts serialize on a short CAS
+      # guard; the copy-back no longer holds the #4877 whole-copy-back lock,
+      # so the mid-restore ongoing read and fast 409 rejection it asserts are
+      # the implemented contract).
+      - name: Run ignored ILM integration tests serially
+        env:
+          # Match the measured Test and Lint link budget. The default exposed
+          # all 14 pod CPUs and a cold cache spent the full 80m compiling
+          # without starting one ILM test (main run 32982910990).
+          CARGO_BUILD_JOBS: ${{ (github.event_name == 'push' || github.event_name == 'workflow_dispatch') && '3' || '2' }}
+        run: |
+          mkdir -p artifacts/ilm-integration
+          set +e
+          NEXTEST_HIDE_PROGRESS_BAR=1 timeout --verbose --signal=TERM --kill-after=30s 80m \
+            cargo nextest run -j1 --run-ignored ignored-only \
+            -p rustfs-scanner -p rustfs \
+            -E 'binary(lifecycle_integration_test) or (package(rustfs) and test(lifecycle_transition_api_test))' \
+            --status-level all --final-status-level all \
+            2>&1 | tee artifacts/ilm-integration/nextest.log
+          status=${PIPESTATUS[0]}
+          {
+            echo "exit_status=${status}"
+            echo "finished_at=$(date --utc --iso-8601=seconds)"
+            echo
+            echo "Remaining test-related processes:"
+            pgrep -af 'cargo|nextest|target/.*/deps/' || true
+          } > artifacts/ilm-integration/diagnostics.txt
+          exit "${status}"
+
+      - name: Upload ILM test diagnostics
+        if: always()
+        uses: actions/upload-artifact@043fb46d1a93c77aae656e7c1c64a875d1fc6a0a # v7.0.1
+        with:
+          name: ilm-integration-${{ github.run_number }}-${{ github.run_attempt }}
+          path: |
+            artifacts/ilm-integration
+            target/nextest/ci/junit.xml
+
   test-and-lint-rio-v2:
     name: Test and Lint (rio-v2)
-    # Alternate feature configurations run on main, merge groups, and scheduled
-    # or manual validation; the default workspace remains the PR gate.
-    if: needs.classify-changes.outputs.mode == 'full' && github.event_name != 'pull_request'
+    if: needs.classify-changes.outputs.mode == 'full' && (github.event_name != 'pull_request' || github.event.action != 'closed')
     needs: [ quick-checks, classify-changes ]
     runs-on: sm-standard-4
     timeout-minutes: 90
@@ -474,7 +506,7 @@ jobs:
 
   test-and-lint-protocols:
     name: "Test and Lint (${{ matrix.features.name }})"
-    if: needs.classify-changes.outputs.mode == 'full' && github.event_name != 'pull_request'
+    if: needs.classify-changes.outputs.mode == 'full' && (github.event_name != 'pull_request' || github.event.action != 'closed')
     needs: [ quick-checks, classify-changes ]
     runs-on: sm-standard-4
     timeout-minutes: 90
@@ -1232,6 +1264,7 @@ jobs:
       - quick-checks
       - test-and-lint
       - offline-enrollment-e2e
+      - test-ilm-integration-serial
       - test-and-lint-rio-v2
       - connect-short-credential-boundary
       - test-and-lint-protocols
@@ -1265,6 +1298,7 @@ jobs:
       - quick-checks
       - test-and-lint
       - offline-enrollment-e2e
+      - test-ilm-integration-serial
       - test-and-lint-rio-v2
       - test-and-lint-protocols
       - build-rustfs-debug-binary

.github/workflows/connect-diagnostic-schedule-acceptance.yml

@@ -57,14 +57,6 @@ jobs:
           ref: ${{ inputs.connect_sha }}
           token: ${{ secrets.PF_TESTING_GH_TOKEN }}
 
-      - name: Set up Docker Compose
-        uses: ./.github/actions/setup-docker-compose
-
-      - name: Set up Python
-        uses: actions/setup-python@5fda3b95a4ea91299a34e894583c3862153e4b97 # v7.0.0
-        with:
-          python-version: "3.14"
-
       - name: Set up Node.js
         uses: actions/setup-node@820762786026740c76f36085b0efc47a31fe5020 # v7.0.0
         with:
@@ -102,7 +94,7 @@ jobs:
           ' <<<"$run" >/dev/null
           jobs=$(gh api --paginate --slurp "repos/${GITHUB_REPOSITORY}/actions/runs/${BUILD_RUN_ID}/jobs?per_page=100")
           jq -e '
-            [.[].jobs[] | select(.name | test("^Build RustFS \\(linux-x86_64-gnu\\)$"))] as $matches
+            [.[].jobs[] | select(.name | test("^Build RustFS \\(linux-x86_64-gnu, [a-z0-9-]+, x86_64-unknown-linux-gnu, false, linux, pyroscope\\)$"))] as $matches
             | ($matches | length) == 1 and $matches[0].conclusion == "success"
           ' <<<"$jobs" >/dev/null
           artifact=$(gh api "repos/${GITHUB_REPOSITORY}/actions/artifacts/${ARTIFACT_ID}")
@@ -145,7 +137,7 @@ jobs:
               assert outer.testzip() is None
               with zipfile.ZipFile(io.BytesIO(outer.read(package))) as inner:
                   entries = inner.infolist()
-                  assert {item.filename for item in entries} == {'rustfs', 'rustfs.cpu-symbol-catalog.json', 'rustfs.cpu-symbol-catalog.sha256'} and len(entries) == 3
+                  assert {item.filename for item in entries} == {'rustfs', 'rustfs-cli'} and len(entries) == 2
                   assert all(regular(item) and 0 < item.file_size <= 1073741824 for item in entries)
                   assert inner.testzip() is None
                   pathlib.Path('binary').mkdir()
@@ -170,7 +162,7 @@ jobs:
           connect-harness/web/node_modules/.bin/playwright install --with-deps chromium
           make -C connect-harness e2e-connected-dispatch-check
           (cd connect-harness && ./tests/connectivity/diagnostic-schedule-evidence.test.sh)
-          TMPDIR="$GITHUB_WORKSPACE" RUSTFS_BINARY="$GITHUB_WORKSPACE/binary/rustfs" \
+          RUSTFS_BINARY="$GITHUB_WORKSPACE/binary/rustfs" \
             RUSTFS_WORKTREE="$GITHUB_WORKSPACE/rustfs-source" \
             CONNECT_E2E_DIAGNOSTIC_SCHEDULE_EVIDENCE="$GITHUB_WORKSPACE/diagnostic-schedule-evidence.json" \
             make -C connect-harness e2e-connected E2E_SCENARIO=diagnostic-schedule

.github/workflows/connect-health-service-acceptance.yml

@@ -57,14 +57,6 @@ jobs:
           ref: ${{ inputs.connect_sha }}
           token: ${{ secrets.PF_TESTING_GH_TOKEN }}
 
-      - name: Set up Docker Compose
-        uses: ./.github/actions/setup-docker-compose
-
-      - name: Set up Python
-        uses: actions/setup-python@5fda3b95a4ea91299a34e894583c3862153e4b97 # v7.0.0
-        with:
-          python-version: "3.14"
-
       - name: Set up Node.js
         uses: actions/setup-node@820762786026740c76f36085b0efc47a31fe5020 # v7.0.0
         with:
@@ -135,7 +127,7 @@ jobs:
           ' <<<"$run" >/dev/null
           jobs=$(gh api --paginate --slurp "repos/rustfs/rustfs/actions/runs/${BUILD_RUN_ID}/jobs?per_page=100")
           jq -e '
-            [.[].jobs[] | select(.name | test("^Build RustFS \\(linux-x86_64-gnu\\)$"))] as $matches
+            [.[].jobs[] | select(.name | test("^Build RustFS \\(linux-x86_64-gnu, [a-z0-9-]+, x86_64-unknown-linux-gnu, false, linux, pyroscope\\)$"))] as $matches
             | ($matches | length) == 1 and $matches[0].conclusion == "success"
           ' <<<"$jobs" >/dev/null
           artifact=$(gh api "repos/rustfs/rustfs/actions/artifacts/${ARTIFACT_ID}")
@@ -180,7 +172,7 @@ jobs:
               assert outer.testzip() is None
               with zipfile.ZipFile(io.BytesIO(outer.read(package))) as inner:
                   entries = inner.infolist()
-                  assert {entry.filename for entry in entries} == {'rustfs', 'rustfs.cpu-symbol-catalog.json', 'rustfs.cpu-symbol-catalog.sha256'} and len(entries) == 3
+                  assert {entry.filename for entry in entries} == {'rustfs', 'rustfs-cli', 'rustfs.cpu-symbol-catalog.json', 'rustfs.cpu-symbol-catalog.sha256'} and len(entries) == 4
                   assert all(regular(entry) and 0 < entry.file_size <= 1073741824 for entry in entries)
                   assert inner.testzip() is None
                   pathlib.Path('binary').mkdir()
@@ -207,7 +199,7 @@ jobs:
           connect-harness/web/node_modules/.bin/playwright install --with-deps chromium
           make -C connect-harness e2e-connected-dispatch-check
           (cd connect-harness && ./tests/connectivity/health-service-evidence.test.sh)
-          TMPDIR="$GITHUB_WORKSPACE" RUSTFS_BINARY="$GITHUB_WORKSPACE/binary/rustfs" \
+          RUSTFS_BINARY="$GITHUB_WORKSPACE/binary/rustfs" \
             RUSTFS_WORKTREE="$GITHUB_WORKSPACE/rustfs-source" \
             CONNECT_E2E_HEALTH_SERVICE_EVIDENCE="$GITHUB_WORKSPACE/health-service-evidence.json" \
             make -C connect-harness e2e-connected E2E_SCENARIO=health-service

.github/workflows/connect-performance-drive-acceptance.yml

@@ -73,14 +73,6 @@ jobs:
           ref: ${{ inputs.connect_sha }}
           token: ${{ secrets.PF_TESTING_GH_TOKEN }}
 
-      - name: Set up Docker Compose
-        uses: ./.github/actions/setup-docker-compose
-
-      - name: Set up Python
-        uses: actions/setup-python@5fda3b95a4ea91299a34e894583c3862153e4b97 # v7.0.0
-        with:
-          python-version: "3.14"
-
       - name: Set up Node.js
         uses: actions/setup-node@820762786026740c76f36085b0efc47a31fe5020 # v7.0.0
         with:
@@ -115,7 +107,7 @@ jobs:
           [[ $(jq -r '.head_repository.full_name' <<<"$run") == "$GITHUB_REPOSITORY" ]]
           [[ $(jq -r '.name' <<<"$run") == "Build and Release" ]]
 
-          expected_job='Build RustFS (linux-x86_64-gnu)'
+          expected_job='Build RustFS (linux-x86_64-gnu, sm-standard-4, x86_64-unknown-linux-gnu, false, linux, pyroscope)'
           jobs=$(gh api --paginate --slurp "repos/${GITHUB_REPOSITORY}/actions/runs/${BUILD_RUN_ID}/jobs?per_page=100")
           jq -e --arg name "$expected_job" '
             [.[].jobs[] | select(.name == $name)] as $matches
@@ -162,7 +154,7 @@ jobs:
           npm --prefix connect-harness/web ci
           connect-harness/web/node_modules/.bin/playwright install --with-deps chromium
           make -C connect-harness e2e-connected-dispatch-check
-          TMPDIR="$GITHUB_WORKSPACE" RUSTFS_BINARY="$GITHUB_WORKSPACE/binary/rustfs" \
+          RUSTFS_BINARY="$GITHUB_WORKSPACE/binary/rustfs" \
             RUSTFS_WORKTREE="$GITHUB_WORKSPACE/rustfs-source" \
             CONNECT_E2E_PERFORMANCE_DRIVE_EVIDENCE="$GITHUB_WORKSPACE/performance-drive-service-job-evidence.json" \
             make -C connect-harness e2e-connected E2E_SCENARIO=performance-drive

.github/workflows/connect-performance-network-acceptance.yml

@@ -73,14 +73,6 @@ jobs:
           ref: ${{ inputs.connect_sha }}
           token: ${{ secrets.PF_TESTING_GH_TOKEN }}
 
-      - name: Set up Docker Compose
-        uses: ./.github/actions/setup-docker-compose
-
-      - name: Set up Python
-        uses: actions/setup-python@5fda3b95a4ea91299a34e894583c3862153e4b97 # v7.0.0
-        with:
-          python-version: "3.14"
-
       - name: Set up Node.js
         uses: actions/setup-node@820762786026740c76f36085b0efc47a31fe5020 # v7.0.0
         with:
@@ -115,7 +107,7 @@ jobs:
           [[ $(jq -r '.head_repository.full_name' <<<"$run") == "$GITHUB_REPOSITORY" ]]
           [[ $(jq -r '.name' <<<"$run") == "Build and Release" ]]
 
-          expected_job='Build RustFS (linux-x86_64-gnu)'
+          expected_job='Build RustFS (linux-x86_64-gnu, sm-standard-4, x86_64-unknown-linux-gnu, false, linux, pyroscope)'
           jobs=$(gh api --paginate --slurp "repos/${GITHUB_REPOSITORY}/actions/runs/${BUILD_RUN_ID}/jobs?per_page=100")
           jq -e --arg name "$expected_job" '
             [.[].jobs[] | select(.name == $name)] as $matches
@@ -162,7 +154,7 @@ jobs:
           npm --prefix connect-harness/web ci
           connect-harness/web/node_modules/.bin/playwright install --with-deps chromium
           make -C connect-harness e2e-connected-dispatch-check
-          TMPDIR="$GITHUB_WORKSPACE" RUSTFS_BINARY="$GITHUB_WORKSPACE/binary/rustfs" \
+          RUSTFS_BINARY="$GITHUB_WORKSPACE/binary/rustfs" \
             RUSTFS_WORKTREE="$GITHUB_WORKSPACE/rustfs-source" \
             CONNECT_E2E_PERFORMANCE_NETWORK_EVIDENCE="$GITHUB_WORKSPACE/performance-network-service-job-evidence.json" \
             make -C connect-harness e2e-connected E2E_SCENARIO=performance-network

.github/workflows/connect-profile-cpu-acceptance.yml

@@ -77,14 +77,6 @@ jobs:
           ref: ${{ inputs.connect_sha }}
           token: ${{ secrets.PF_TESTING_GH_TOKEN }}
 
-      - name: Set up Docker Compose
-        uses: ./.github/actions/setup-docker-compose
-
-      - name: Set up Python
-        uses: actions/setup-python@5fda3b95a4ea91299a34e894583c3862153e4b97 # v7.0.0
-        with:
-          python-version: "3.14"
-
       - name: Set up Node.js
         uses: actions/setup-node@820762786026740c76f36085b0efc47a31fe5020 # v7.0.0
         with:
@@ -126,7 +118,7 @@ jobs:
           [[ $(jq -r '.head_repository.full_name' <<<"$run") == "$GITHUB_REPOSITORY" ]]
           [[ $(jq -r '.name' <<<"$run") == "Build and Release" ]]
 
-          expected_job='Build RustFS (linux-x86_64-gnu)'
+          expected_job='Build RustFS (linux-x86_64-gnu, sm-standard-4, x86_64-unknown-linux-gnu, false, linux, pyroscope)'
           jobs=$(github_api "repos/${GITHUB_REPOSITORY}/actions/runs/${BUILD_RUN_ID}/jobs?per_page=100")
           jq -e --arg name "$expected_job" '
             [.jobs[] | select(.name == $name)] as $matches
@@ -181,7 +173,7 @@ jobs:
           npm --prefix connect-harness/web ci
           connect-harness/web/node_modules/.bin/playwright install --with-deps chromium
           make -C connect-harness e2e-connected-dispatch-check
-          TMPDIR="$GITHUB_WORKSPACE" RUSTFS_BINARY="$GITHUB_WORKSPACE/binary/rustfs" \
+          RUSTFS_BINARY="$GITHUB_WORKSPACE/binary/rustfs" \
             RUSTFS_WORKTREE="$GITHUB_WORKSPACE/rustfs-source" \
             CONNECT_E2E_CPU_CATALOG="$GITHUB_WORKSPACE/binary/rustfs.cpu-symbol-catalog.json" \
             CONNECT_E2E_CPU_CATALOG_SHA256="$CATALOG_SHA256" \

.github/workflows/connect-profile-memory-acceptance.yml

@@ -57,14 +57,6 @@ jobs:
           ref: ${{ inputs.connect_sha }}
           token: ${{ secrets.PF_TESTING_GH_TOKEN }}
 
-      - name: Set up Docker Compose
-        uses: ./.github/actions/setup-docker-compose
-
-      - name: Set up Python
-        uses: actions/setup-python@5fda3b95a4ea91299a34e894583c3862153e4b97 # v7.0.0
-        with:
-          python-version: "3.14"
-
       - name: Set up Node.js
         uses: actions/setup-node@820762786026740c76f36085b0efc47a31fe5020 # v7.0.0
         with:
@@ -135,7 +127,7 @@ jobs:
           ' <<<"$run" >/dev/null
           jobs=$(gh api --paginate --slurp "repos/rustfs/rustfs/actions/runs/${BUILD_RUN_ID}/jobs?per_page=100")
           jq -e '
-            [.[].jobs[] | select(.name | test("^Build RustFS \\(linux-x86_64-gnu\\)$"))] as $matches
+            [.[].jobs[] | select(.name | test("^Build RustFS \\(linux-x86_64-gnu, [a-z0-9-]+, x86_64-unknown-linux-gnu, false, linux, pyroscope\\)$"))] as $matches
             | ($matches | length) == 1 and $matches[0].conclusion == "success"
           ' <<<"$jobs" >/dev/null
           artifact=$(gh api "repos/rustfs/rustfs/actions/artifacts/${ARTIFACT_ID}")
@@ -180,7 +172,7 @@ jobs:
               assert outer.testzip() is None
               with zipfile.ZipFile(io.BytesIO(outer.read(package))) as inner:
                   entries = inner.infolist()
-                  expected = {'rustfs', 'rustfs.cpu-symbol-catalog.json', 'rustfs.cpu-symbol-catalog.sha256'}
+                  expected = {'rustfs', 'rustfs-cli', 'rustfs.cpu-symbol-catalog.json', 'rustfs.cpu-symbol-catalog.sha256'}
                   assert {entry.filename for entry in entries} == expected and len(entries) == len(expected)
                   assert all(regular(entry) and 0 < entry.file_size <= 1073741824 for entry in entries)
                   assert inner.testzip() is None
@@ -206,7 +198,7 @@ jobs:
           connect-harness/web/node_modules/.bin/playwright install --with-deps chromium
           make -C connect-harness e2e-connected-dispatch-check
           (cd connect-harness && ./tests/connectivity/profile-memory-evidence.test.sh)
-          TMPDIR="$GITHUB_WORKSPACE" RUSTFS_BINARY="$GITHUB_WORKSPACE/binary/rustfs" \
+          RUSTFS_BINARY="$GITHUB_WORKSPACE/binary/rustfs" \
             RUSTFS_WORKTREE="$GITHUB_WORKSPACE/rustfs-source" \
             CONNECT_E2E_PROFILE_MEMORY_EVIDENCE="$GITHUB_WORKSPACE/profile-memory-evidence.json" \
             make -C connect-harness e2e-connected E2E_SCENARIO=profile-memory

.github/workflows/connect-profile-threads-acceptance.yml

@@ -81,14 +81,6 @@ jobs:
           ref: ${{ inputs.connect_sha }}
           token: ${{ secrets.PF_TESTING_GH_TOKEN }}
 
-      - name: Set up Docker Compose
-        uses: ./.github/actions/setup-docker-compose
-
-      - name: Set up Python
-        uses: actions/setup-python@5fda3b95a4ea91299a34e894583c3862153e4b97 # v7.0.0
-        with:
-          python-version: "3.14"
-
       - name: Set up Node.js
         uses: actions/setup-node@820762786026740c76f36085b0efc47a31fe5020 # v7.0.0
         with:
@@ -156,7 +148,7 @@ jobs:
           [[ $(jq -r '.head_repository.full_name' <<<"$run") == "$GITHUB_REPOSITORY" ]]
           [[ $(jq -r '.name' <<<"$run") == "Build and Release" ]]
 
-          expected_job='Build RustFS (linux-x86_64-gnu)'
+          expected_job='Build RustFS (linux-x86_64-gnu, sm-standard-4, x86_64-unknown-linux-gnu, false, linux, pyroscope)'
           jobs=$(gh api --paginate --slurp "repos/${GITHUB_REPOSITORY}/actions/runs/${BUILD_RUN_ID}/jobs?per_page=100")
           jq -e --arg name "$expected_job" '
             [.[].jobs[] | select(.name == $name)] as $matches
@@ -209,7 +201,7 @@ jobs:
           npm --prefix connect-harness/web ci
           connect-harness/web/node_modules/.bin/playwright install --with-deps chromium
           make -C connect-harness e2e-connected-dispatch-check
-          TMPDIR="$GITHUB_WORKSPACE" RUSTFS_BINARY="$GITHUB_WORKSPACE/binary/rustfs" \
+          RUSTFS_BINARY="$GITHUB_WORKSPACE/binary/rustfs" \
             RUSTFS_WORKTREE="$GITHUB_WORKSPACE/rustfs-source" \
             CONNECT_E2E_PROFILE_THREADS_EVIDENCE="$GITHUB_WORKSPACE/profile-threads-service-job-evidence.json" \
             make -C connect-harness e2e-connected E2E_SCENARIO=profile-threads

.github/workflows/connect-top-api-acceptance.yml

@@ -73,14 +73,6 @@ jobs:
           ref: ${{ inputs.connect_sha }}
           token: ${{ secrets.PF_TESTING_GH_TOKEN }}
 
-      - name: Set up Docker Compose
-        uses: ./.github/actions/setup-docker-compose
-
-      - name: Set up Python
-        uses: actions/setup-python@5fda3b95a4ea91299a34e894583c3862153e4b97 # v7.0.0
-        with:
-          python-version: "3.14"
-
       - name: Set up Node.js
         uses: actions/setup-node@820762786026740c76f36085b0efc47a31fe5020 # v7.0.0
         with:
@@ -115,7 +107,7 @@ jobs:
           [[ $(jq -r '.head_repository.full_name' <<<"$run") == "$GITHUB_REPOSITORY" ]]
           [[ $(jq -r '.name' <<<"$run") == "Build and Release" ]]
 
-          expected_job='Build RustFS (linux-x86_64-gnu)'
+          expected_job='Build RustFS (linux-x86_64-gnu, sm-standard-4, x86_64-unknown-linux-gnu, false, linux, pyroscope)'
           jobs=$(gh api --paginate --slurp "repos/${GITHUB_REPOSITORY}/actions/runs/${BUILD_RUN_ID}/jobs?per_page=100")
           jq -e --arg name "$expected_job" '
             [.[].jobs[] | select(.name == $name)] as $matches
@@ -162,7 +154,7 @@ jobs:
           npm --prefix connect-harness/web ci
           connect-harness/web/node_modules/.bin/playwright install --with-deps chromium
           make -C connect-harness e2e-connected-dispatch-check
-          TMPDIR="$GITHUB_WORKSPACE" RUSTFS_BINARY="$GITHUB_WORKSPACE/binary/rustfs" \
+          RUSTFS_BINARY="$GITHUB_WORKSPACE/binary/rustfs" \
             RUSTFS_WORKTREE="$GITHUB_WORKSPACE/rustfs-source" \
             CONNECT_E2E_TOP_API_EVIDENCE="$GITHUB_WORKSPACE/top-api-service-job-evidence.json" \
             make -C connect-harness e2e-connected E2E_SCENARIO=top-api

.github/workflows/connect-top-disk-acceptance.yml

@@ -68,14 +68,6 @@ jobs:
           ref: ${{ inputs.connect_sha }}
           token: ${{ secrets.PF_TESTING_GH_TOKEN }}
 
-      - name: Set up Docker Compose
-        uses: ./.github/actions/setup-docker-compose
-
-      - name: Set up Python
-        uses: actions/setup-python@5fda3b95a4ea91299a34e894583c3862153e4b97 # v7.0.0
-        with:
-          python-version: "3.14"
-
       - name: Set up Node.js
         uses: actions/setup-node@820762786026740c76f36085b0efc47a31fe5020 # v7.0.0
         with:
@@ -146,7 +138,7 @@ jobs:
           ' <<<"$run" >/dev/null
           jobs=$(gh api --paginate --slurp "repos/rustfs/rustfs/actions/runs/${BUILD_RUN_ID}/jobs?per_page=100")
           jq -e '
-            [.[].jobs[] | select(.name | test("^Build RustFS \\(linux-x86_64-gnu\\)$"))] as $matches
+            [.[].jobs[] | select(.name | test("^Build RustFS \\(linux-x86_64-gnu, [a-z0-9-]+, x86_64-unknown-linux-gnu, false, linux, pyroscope\\)$"))] as $matches
             | ($matches | length) == 1 and $matches[0].conclusion == "success"
           ' <<<"$jobs" >/dev/null
           artifact=$(gh api "repos/rustfs/rustfs/actions/artifacts/${ARTIFACT_ID}")
@@ -191,7 +183,7 @@ jobs:
               assert outer.testzip() is None
               with zipfile.ZipFile(io.BytesIO(outer.read(package))) as inner:
                   entries = inner.infolist()
-                  expected = {'rustfs', 'rustfs.cpu-symbol-catalog.json', 'rustfs.cpu-symbol-catalog.sha256'}
+                  expected = {'rustfs', 'rustfs-cli', 'rustfs.cpu-symbol-catalog.json', 'rustfs.cpu-symbol-catalog.sha256'}
                   assert {entry.filename for entry in entries} == expected and len(entries) == len(expected)
                   assert all(regular(entry) and 0 < entry.file_size <= 1073741824 for entry in entries)
                   assert inner.testzip() is None
@@ -221,7 +213,7 @@ jobs:
           npm --prefix connect-harness/web ci
           connect-harness/web/node_modules/.bin/playwright install --with-deps chromium
           make -C connect-harness e2e-connected-dispatch-check
-          TMPDIR="$GITHUB_WORKSPACE" RUSTFS_BINARY="$GITHUB_WORKSPACE/binary/rustfs" \
+          RUSTFS_BINARY="$GITHUB_WORKSPACE/binary/rustfs" \
             RUSTFS_WORKTREE="$GITHUB_WORKSPACE/rustfs-source" \
             CONNECT_E2E_TOP_DISK_EVIDENCE="$GITHUB_WORKSPACE/top-disk-service-job-evidence.json" \
             make -C connect-harness e2e-connected E2E_SCENARIO=top-disk

.github/workflows/connect-top-locks-acceptance.yml

@@ -73,14 +73,6 @@ jobs:
           ref: ${{ inputs.connect_sha }}
           token: ${{ secrets.PF_TESTING_GH_TOKEN }}
 
-      - name: Set up Docker Compose
-        uses: ./.github/actions/setup-docker-compose
-
-      - name: Set up Python
-        uses: actions/setup-python@5fda3b95a4ea91299a34e894583c3862153e4b97 # v7.0.0
-        with:
-          python-version: "3.14"
-
       - name: Set up Node.js
         uses: actions/setup-node@820762786026740c76f36085b0efc47a31fe5020 # v7.0.0
         with:
@@ -115,7 +107,7 @@ jobs:
           [[ $(jq -r '.head_repository.full_name' <<<"$run") == "$GITHUB_REPOSITORY" ]]
           [[ $(jq -r '.name' <<<"$run") == "Build and Release" ]]
 
-          expected_job='Build RustFS (linux-x86_64-gnu)'
+          expected_job='Build RustFS (linux-x86_64-gnu, sm-standard-4, x86_64-unknown-linux-gnu, false, linux, pyroscope)'
           jobs=$(gh api --paginate --slurp "repos/${GITHUB_REPOSITORY}/actions/runs/${BUILD_RUN_ID}/jobs?per_page=100")
           jq -e --arg name "$expected_job" '
             [.[].jobs[] | select(.name == $name)] as $matches
@@ -162,7 +154,7 @@ jobs:
           npm --prefix connect-harness/web ci
           connect-harness/web/node_modules/.bin/playwright install --with-deps chromium
           make -C connect-harness e2e-connected-dispatch-check
-          TMPDIR="$GITHUB_WORKSPACE" RUSTFS_BINARY="$GITHUB_WORKSPACE/binary/rustfs" \
+          RUSTFS_BINARY="$GITHUB_WORKSPACE/binary/rustfs" \
             RUSTFS_WORKTREE="$GITHUB_WORKSPACE/rustfs-source" \
             CONNECT_E2E_TOP_LOCKS_EVIDENCE="$GITHUB_WORKSPACE/top-locks-service-job-evidence.json" \
             make -C connect-harness e2e-connected E2E_SCENARIO=top-locks

.github/workflows/connect-top-rpc-acceptance.yml

@@ -73,14 +73,6 @@ jobs:
           ref: ${{ inputs.connect_sha }}
           token: ${{ secrets.PF_TESTING_GH_TOKEN }}
 
-      - name: Set up Docker Compose
-        uses: ./.github/actions/setup-docker-compose
-
-      - name: Set up Python
-        uses: actions/setup-python@5fda3b95a4ea91299a34e894583c3862153e4b97 # v7.0.0
-        with:
-          python-version: "3.14"
-
       - name: Set up Node.js
         uses: actions/setup-node@820762786026740c76f36085b0efc47a31fe5020 # v7.0.0
         with:
@@ -115,7 +107,7 @@ jobs:
           [[ $(jq -r '.head_repository.full_name' <<<"$run") == "$GITHUB_REPOSITORY" ]]
           [[ $(jq -r '.name' <<<"$run") == "Build and Release" ]]
 
-          expected_job='Build RustFS (linux-x86_64-gnu)'
+          expected_job='Build RustFS (linux-x86_64-gnu, sm-standard-4, x86_64-unknown-linux-gnu, false, linux, pyroscope)'
           jobs=$(gh api --paginate --slurp "repos/${GITHUB_REPOSITORY}/actions/runs/${BUILD_RUN_ID}/jobs?per_page=100")
           jq -e --arg name "$expected_job" '
             [.[].jobs[] | select(.name == $name)] as $matches
@@ -162,7 +154,7 @@ jobs:
           npm --prefix connect-harness/web ci
           connect-harness/web/node_modules/.bin/playwright install --with-deps chromium
           make -C connect-harness e2e-connected-dispatch-check
-          TMPDIR="$GITHUB_WORKSPACE" RUSTFS_BINARY="$GITHUB_WORKSPACE/binary/rustfs" \
+          RUSTFS_BINARY="$GITHUB_WORKSPACE/binary/rustfs" \
             RUSTFS_WORKTREE="$GITHUB_WORKSPACE/rustfs-source" \
             CONNECT_E2E_TOP_RPC_EVIDENCE="$GITHUB_WORKSPACE/top-rpc-service-job-evidence.json" \
             make -C connect-harness e2e-connected E2E_SCENARIO=top-rpc

.github/workflows/e2e-upgrade.yml

@@ -96,8 +96,38 @@ jobs:
           overwrite: true
 
   upgrade:
-    name: Upgrade compatibility
+    name: ${{ matrix.name }}
     needs: build
+    strategy:
+      fail-fast: false
+      matrix:
+        include:
+          # The two `_from_rc2_` tests keep their names: they assert
+          # release-independent object contracts and pass unchanged against the
+          # newer pinned source, so renaming them would only churn history and
+          # the CI required-check names. UPGRADE_SOURCE_VERSION above is the
+          # single source of truth for which release they actually run against.
+          - name: Direct upgrade from the previous release
+            test: direct_upgrade_from_rc2_preserves_object_contracts
+            artifact: direct-upgrade
+          - name: Mixed-version rolling upgrade from the previous release
+            test: rolling_upgrade_from_rc2_preserves_mixed_version_contracts
+            artifact: mixed-version-upgrade
+          - name: Bucket configuration survives the upgrade
+            test: direct_upgrade_from_previous_release_preserves_bucket_configuration
+            artifact: bucket-config-upgrade
+          - name: Rollback reads current bucket metadata
+            test: rollback_to_previous_release_reads_current_bucket_metadata
+            artifact: bucket-config-rollback
+          - name: ODM configuration recovery after rc.5 rollback
+            test: rc5_rollback_requires_restoring_odm_configuration
+            artifact: odm-config-rollback
+          - name: Multipart layouts survive the rc.5 upgrade
+            test: direct_upgrade_from_rc5_preserves_multipart_layouts
+            artifact: multipart-layout-upgrade
+          - name: rc.5 multipart replication baseline
+            test: rc5_baseline_replicates_multipart_layouts
+            artifact: multipart-layout-baseline
     runs-on: sm-standard-2
     timeout-minutes: 60
     env:
@@ -112,7 +142,8 @@ jobs:
         uses: ./.github/actions/setup
         with:
           cache-shared-key: e2e-upgrade-tests
-          cache-save-if: ${{ github.ref == 'refs/heads/main' }}
+          # One main-branch consumer warms the shared test-harness dependencies.
+          cache-save-if: ${{ github.ref == 'refs/heads/main' && matrix.artifact == 'direct-upgrade' }}
           install-build-packaging-tools: "false"
           install-test-tools: "false"
 
@@ -139,49 +170,22 @@ jobs:
           echo "RUSTFS_UPGRADE_SOURCE_BINARY=$SOURCE_DIR/rustfs" >> "$GITHUB_ENV"
           echo "RUSTFS_E2E_LOG_DIR=$RUNNER_TEMP/rustfs-upgrade-logs" >> "$GITHUB_ENV"
 
-      # Reuse one compiled harness, while keeping each case in its own process.
-      # Continue after a failure to retain the former matrix's complete verdict.
-      - name: Run upgrade compatibility tests
-        shell: bash
+      - name: Run upgrade compatibility test
         env:
+          RUSTFS_SCANNER_HEAL_G09_EVIDENCE_DIR: ${{ runner.temp }}/rustfs-upgrade-g09-evidence/${{ matrix.artifact }}
           NO_PROXY: 127.0.0.1,localhost,::1
         run: |
           export no_proxy="$NO_PROXY"
-          cases=(
-            direct-upgrade:direct_upgrade_from_rc2_preserves_object_contracts
-            mixed-version-upgrade:rolling_upgrade_from_rc2_preserves_mixed_version_contracts
-            bucket-config-upgrade:direct_upgrade_from_previous_release_preserves_bucket_configuration
-            bucket-config-rollback:rollback_to_previous_release_reads_current_bucket_metadata
-            odm-config-rollback:rc5_rollback_requires_restoring_odm_configuration
-            multipart-layout-upgrade:direct_upgrade_from_rc5_preserves_multipart_layouts
-            multipart-layout-baseline:rc5_baseline_replicates_multipart_layouts
-          )
-          status=0
-          for entry in "${cases[@]}"; do
-            artifact="${entry%%:*}"
-            test="${entry#*:}"
-            export RUSTFS_SCANNER_HEAL_G09_EVIDENCE_DIR="$RUNNER_TEMP/rustfs-upgrade-g09-evidence/$artifact"
-            export RUSTFS_E2E_LOG_DIR="$RUNNER_TEMP/rustfs-upgrade-logs/$artifact"
-            mkdir -p "$RUSTFS_E2E_LOG_DIR"
-            echo "::group::$test"
-            if python3 scripts/e2e_binary.py run -- cargo test --locked -p e2e_test \
-              "upgrade_compatibility_test::$test" -- --ignored --exact --nocapture \
-              2>&1 | tee "$RUSTFS_E2E_LOG_DIR/test.log"; then
-              echo "PASS: $test"
-            else
-              echo "::error::Upgrade compatibility failed: $test"
-              status=1
-            fi
-            echo "::endgroup::"
-          done
-          exit "$status"
+          python3 scripts/e2e_binary.py run -- cargo test --locked -p e2e_test \
+            "upgrade_compatibility_test::${{ matrix.test }}" \
+            -- --ignored --exact --nocapture
 
       - name: Upload scanner/heal G09 evidence
         if: always()
         uses: actions/upload-artifact@043fb46d1a93c77aae656e7c1c64a875d1fc6a0a # v7.0.1
         with:
-          name: upgrade-g09-evidence-${{ github.run_number }}
-          path: ${{ runner.temp }}/rustfs-upgrade-g09-evidence
+          name: ${{ matrix.artifact }}-g09-evidence-${{ github.run_number }}
+          path: ${{ runner.temp }}/rustfs-upgrade-g09-evidence/${{ matrix.artifact }}
           if-no-files-found: ignore
           retention-days: 14
 
@@ -189,7 +193,7 @@ jobs:
         if: always()
         uses: actions/upload-artifact@043fb46d1a93c77aae656e7c1c64a875d1fc6a0a # v7.0.1
         with:
-          name: upgrade-server-logs-${{ github.run_number }}
+          name: ${{ matrix.artifact }}-server-logs-${{ github.run_number }}
           path: ${{ runner.temp }}/rustfs-upgrade-logs
           if-no-files-found: warn
           retention-days: 14

.github/workflows/nightly-gnu.yml

@@ -80,24 +80,13 @@ jobs:
       - name: Setup Rust environment
         uses: ./.github/actions/setup
         with:
-          cache-shared-key: build-x86_64-unknown-linux-gnu-glibc-2.31
+          cache-shared-key: build-x86_64-unknown-linux-gnu
           cache-save-if: 'false'
-          install-cross-tools: 'true'
-          target: x86_64-unknown-linux-gnu
           install-build-packaging-tools: 'false'
           install-test-tools: 'false'
 
       - name: Build RustFS
-        # Pin the glibc baseline (`<target>.<glibc>` for cargo-zigbuild) so
-        # nightlies run on older distros such as RHEL/Rocky 9 (glibc 2.34); #8062.
-        run: cargo zigbuild --release --locked --target x86_64-unknown-linux-gnu.2.31 -p rustfs --bins
-
-      - name: Verify glibc symbol baseline
-        run: |
-          python3 scripts/release/check_glibc.py 2.31 \
-            target/x86_64-unknown-linux-gnu/release/rustfs \
-            target/x86_64-unknown-linux-gnu/release/rustfs-cli
-          target/x86_64-unknown-linux-gnu/release/rustfs --version | grep -Fx 'build target : x86_64-unknown-linux-gnu'
+        run: cargo build --release --locked --target x86_64-unknown-linux-gnu -p rustfs --bins
 
       - name: Build DEB package
         id: deb

.github/workflows/oidc-keycloak.yml

@@ -20,9 +20,8 @@ on:
       - ".github/workflows/oidc-keycloak.yml"
       - "crates/config/src/constants/oidc.rs"
       - "crates/iam/src/federation/**"
-      - "crates/iam/src/oidc/**"
+      - "crates/iam/src/oidc.rs"
       - "rustfs/src/admin/handlers/oidc.rs"
-      - "rustfs/src/admin/service/oidc_config.rs"
       - "rustfs/src/admin/handlers/sts.rs"
       - "scripts/test/oidc_keycloak_live.sh"
       - "scripts/test/fixtures/keycloak-rustfs-ci-realm.json"
@@ -32,9 +31,8 @@ on:
       - ".github/workflows/oidc-keycloak.yml"
       - "crates/config/src/constants/oidc.rs"
       - "crates/iam/src/federation/**"
-      - "crates/iam/src/oidc/**"
+      - "crates/iam/src/oidc.rs"
       - "rustfs/src/admin/handlers/oidc.rs"
-      - "rustfs/src/admin/service/oidc_config.rs"
       - "rustfs/src/admin/handlers/sts.rs"
       - "scripts/test/oidc_keycloak_live.sh"
       - "scripts/test/fixtures/keycloak-rustfs-ci-realm.json"

.vscode/launch.json

@@ -7,9 +7,7 @@
             "name": "Debug(only) executable 'rustfs'",
             "env": {
               "RUST_LOG": "rustfs=info,ecstore=info,s3s=info,iam=info",
-              "RUSTFS_SKIP_BACKGROUND_TASK": "on",
-              // Local test volumes share one physical disk.
-              "RUSTFS_UNSAFE_BYPASS_DISK_CHECK": "true"
+              "RUSTFS_SKIP_BACKGROUND_TASK": "on"
               //"RUSTFS_OBS_LOG_DIRECTORY": "./deploy/logs",
               // "RUSTFS_POLICY_PLUGIN_URL":"http://localhost:8181/v1/data/rustfs/authz/allow",
               // "RUSTFS_POLICY_PLUGIN_AUTH_TOKEN":"your-opa-token"
@@ -46,7 +44,6 @@
             "env": {
                 "RUST_LOG": "rustfs=debug,ecstore=info,s3s=debug,iam=debug",
                 "RUSTFS_SKIP_BACKGROUND_TASK": "on",
-                "RUSTFS_UNSAFE_BYPASS_DISK_CHECK": "true",
                 //"RUSTFS_OBS_LOG_DIRECTORY": "./deploy/logs",
                 // "RUSTFS_POLICY_PLUGIN_URL":"http://localhost:8181/v1/data/rustfs/authz/allow",
                 // "RUSTFS_POLICY_PLUGIN_AUTH_TOKEN":"your-opa-token" 
@@ -120,8 +117,7 @@
             "env": {
                 "RUSTFS_ACCESS_KEY": "rustfsadmin",
                 "RUSTFS_SECRET_KEY": "rustfsadmin",
-                "RUSTFS_VOLUMES": "./target/volume/test{1...4}",
-                "RUSTFS_UNSAFE_BYPASS_DISK_CHECK": "true",
+                //"RUSTFS_VOLUMES": "./target/volume/test{1...4}",
                 "RUSTFS_ADDRESS": ":9000",
                 "RUSTFS_CONSOLE_ENABLE": "true",
                 // "RUSTFS_OBS_TRACE_ENDPOINT": "http://127.0.0.1:4318/v1/traces", // jeager otlp http endpoint
@@ -139,31 +135,42 @@
         {
             "type": "lldb",
             "request": "launch",
-            "name": "Debug test_transition_and_restore_flows",
+            "name": "Debug test_lifecycle_transition_basic",
             "cargo": {
                 "args": [
                     "test",
-                    "--no-run",
                     "-p",
                     "rustfs-scanner",
                     "--test",
                     "lifecycle_integration_test",
+                    "serial_tests::test_lifecycle_transition_basic",
                     "-j",
                     "1"
-                ],
-                "filter": {
-                    "name": "lifecycle_integration_test",
-                    "kind": "test"
-                }
+                ]
             },
-            "args": [
-                "serial_tests::test_transition_and_restore_flows",
-                "--exact",
-                "--ignored",
-                "--test-threads=1"
-            ],
+            "args": [],
             "cwd": "${workspaceFolder}"
         },
+        {
+            "name": "Debug executable target/debug/test",
+            "type":  "lldb",
+            "request": "launch",
+            "program": "${workspaceFolder}/target/debug/deps/lifecycle_integration_test-5915cbfcab491b3b",
+            "args": [
+              "--skip",
+              "test_lifecycle_expiry_basic",
+              "--skip",
+              "test_lifecycle_expiry_deletemarker",
+              //"--skip",
+              //"test_lifecycle_transition_basic",
+            ],
+            "cwd": "${workspaceFolder}",
+            //"stopAtEntry": false,
+            //"preLaunchTask": "cargo build",
+            "sourceLanguages": [
+                "rust"
+            ],
+        },
         {
             "name": "Debug executable target/debug/rustfs with sse kms",
             "type":  "lldb",
@@ -278,7 +285,7 @@
                 "RUSTFS_COMPRESSION_ENABLED": "true",
                 "RUSTFS_KMS_ENABLE": "true",
                 "RUSTFS_KMS_BACKEND": "local",
-                "RUSTFS_KMS_KEY_DIR": "${workspaceFolder}/target/rio-v2/kms-key-dir",
+                "RUSTFS_KMS_KEY_DIR": "./target/rio-v2/kms-key-dir",
                 "RUSTFS_KMS_LOCAL_MASTER_KEY": "my-secret-key", // Some Password
                 "RUSTFS_KMS_DEFAULT_KEY_ID": "rustfs-master-key",
             },
@@ -300,7 +307,6 @@
                 "RUSTFS_ACCESS_KEY": "rustfsadmin",
                 "RUSTFS_SECRET_KEY": "rustfsadmin",
                 "RUSTFS_VOLUMES": "./target/e2e-test/test{1...4}",
-                "RUSTFS_UNSAFE_BYPASS_DISK_CHECK": "true",
                 "RUSTFS_REGION": "us-east-1",
                 "RUSTFS_ADDRESS": ":9000",
                 "RUSTFS_CONSOLE_ENABLE": "true",
@@ -309,7 +315,7 @@
 
                 "RUSTFS_KMS_ENABLE": "true",
                 "RUSTFS_KMS_BACKEND": "local",
-                "RUSTFS_KMS_KEY_DIR": "${workspaceFolder}/target/e2e-key-dir",
+                "RUSTFS_KMS_KEY_DIR": "./target/e2e-key-dir",
                 "RUSTFS_KMS_LOCAL_MASTER_KEY": "my-secret-key", // Some Password
                 "RUSTFS_KMS_DEFAULT_KEY_ID": "rustfs-master-key",
             },

Cargo.toml

@@ -74,7 +74,7 @@ edition = "2024"
 license = "Apache-2.0"
 repository = "https://github.com/rustfs/rustfs"
 rust-version = "1.99.0"
-version = "1.0.4"
+version = "1.0.1"
 homepage = "https://rustfs.com"
 description = "RustFS is a high-performance distributed object storage software built using Rust, one of the most popular languages worldwide. "
 keywords = ["RustFS", "Minio", "object-storage", "filesystem", "s3"]
@@ -91,62 +91,62 @@ redundant_clone = "warn"
 
 [workspace.dependencies]
 # RustFS Internal Crates
-rustfs = { path = "./rustfs", version = "1.0.4" }
-rustfs-heal = { path = "crates/heal", version = "1.0.4" }
-rustfs-heal-contracts = { path = "crates/heal-contracts", version = "1.0.4" }
-rustfs-scanner-metrics = { path = "crates/scanner-metrics", version = "1.0.4" }
-rustfs-audit = { path = "crates/audit", version = "1.0.4" }
-rustfs-checksums = { path = "crates/checksums", version = "1.0.4" }
-rustfs-common = { path = "crates/common", version = "1.0.4" }
-rustfs-data-usage = { path = "crates/data-usage", version = "1.0.4" }
-rustfs-config = { path = "./crates/config", version = "1.0.4" }
-rustfs-concurrency = { path = "./crates/concurrency", version = "1.0.4" }
-rustfs-credentials = { path = "crates/credentials", version = "1.0.4" }
-rustfs-crypto = { path = "crates/crypto", version = "1.0.4" }
-rustfs-ecstore = { path = "crates/ecstore", version = "1.0.4" }
-rustfs-filemeta = { path = "crates/filemeta", version = "1.0.4" }
-rustfs-iam = { path = "crates/iam", version = "1.0.4" }
-rustfs-keystone = { path = "crates/keystone", version = "1.0.4" }
-rustfs-license = { path = "crates/license", version = "1.0.4" }
-rustfs-lifecycle = { path = "crates/lifecycle", version = "1.0.4" }
-rustfs-kms = { path = "crates/kms", version = "1.0.4" }
-rustfs-lock = { path = "crates/lock", version = "1.0.4" }
-rustfs-madmin = { path = "crates/madmin", version = "1.0.4" }
-rustfs-notify = { path = "crates/notify", version = "1.0.4" }
-rustfs-io-metrics = { path = "crates/io-metrics", version = "1.0.4" }
-rustfs-io-core = { path = "crates/io-core", version = "1.0.4" }
-rustfs-object-capacity = { path = "crates/object-capacity", version = "1.0.4" }
-rustfs-object-data-cache = { path = "crates/object-data-cache", version = "1.0.4", default-features = false }
-rustfs-log-analyzer = { path = "crates/log-analyzer", version = "1.0.4" }
-rustfs-obs = { path = "crates/obs", version = "1.0.4" }
-rustfs-policy = { path = "crates/policy", version = "1.0.4" }
-rustfs-protos = { path = "crates/protos", version = "1.0.4" }
-rustfs-protocols = { path = "crates/protocols", version = "1.0.4" }
-rustfs-replication = { path = "crates/replication", version = "1.0.4" }
-rustfs-rio = { path = "crates/rio", version = "1.0.4" }
-rustfs-rio-v2 = { path = "crates/rio-v2", version = "1.0.4" }
-rustfs-s3-client = { path = "crates/s3-client", version = "1.0.4" }
-rustfs-s3-types = { path = "crates/s3-types", version = "1.0.4" }
-rustfs-s3-ops = { path = "crates/s3-ops", version = "1.0.4" }
-rustfs-s3select-api = { path = "crates/s3select-api", version = "1.0.4" }
-rustfs-s3select-query = { path = "crates/s3select-query", version = "1.0.4" }
-rustfs-scanner = { path = "crates/scanner", version = "1.0.4" }
-rustfs-security-governance = { path = "crates/security-governance", version = "1.0.4" }
-rustfs-extension-schema = { path = "crates/extension-schema", version = "1.0.4" }
-rustfs-signer = { path = "crates/signer", version = "1.0.4" }
-rustfs-storage-api = { path = "crates/storage-api", version = "1.0.4" }
-rustfs-trusted-proxies = { path = "crates/trusted-proxies", version = "1.0.4" }
-rustfs-targets = { path = "crates/targets", version = "1.0.4" }
-rustfs-test-utils = { path = "crates/test-utils", version = "1.0.4" }
-rustfs-tls-runtime = { path = "crates/tls-runtime", version = "1.0.4" }
-rustfs-utils = { path = "crates/utils", version = "1.0.4" }
-rustfs-zip = { path = "./crates/zip", version = "1.0.4" }
+rustfs = { path = "./rustfs", version = "1.0.1" }
+rustfs-heal = { path = "crates/heal", version = "1.0.1" }
+rustfs-heal-contracts = { path = "crates/heal-contracts", version = "1.0.1" }
+rustfs-scanner-metrics = { path = "crates/scanner-metrics", version = "1.0.1" }
+rustfs-audit = { path = "crates/audit", version = "1.0.1" }
+rustfs-checksums = { path = "crates/checksums", version = "1.0.1" }
+rustfs-common = { path = "crates/common", version = "1.0.1" }
+rustfs-data-usage = { path = "crates/data-usage", version = "1.0.1" }
+rustfs-config = { path = "./crates/config", version = "1.0.1" }
+rustfs-concurrency = { path = "./crates/concurrency", version = "1.0.1" }
+rustfs-credentials = { path = "crates/credentials", version = "1.0.1" }
+rustfs-crypto = { path = "crates/crypto", version = "1.0.1" }
+rustfs-ecstore = { path = "crates/ecstore", version = "1.0.1" }
+rustfs-filemeta = { path = "crates/filemeta", version = "1.0.1" }
+rustfs-iam = { path = "crates/iam", version = "1.0.1" }
+rustfs-keystone = { path = "crates/keystone", version = "1.0.1" }
+rustfs-license = { path = "crates/license", version = "1.0.1" }
+rustfs-lifecycle = { path = "crates/lifecycle", version = "1.0.1" }
+rustfs-kms = { path = "crates/kms", version = "1.0.1" }
+rustfs-lock = { path = "crates/lock", version = "1.0.1" }
+rustfs-madmin = { path = "crates/madmin", version = "1.0.1" }
+rustfs-notify = { path = "crates/notify", version = "1.0.1" }
+rustfs-io-metrics = { path = "crates/io-metrics", version = "1.0.1" }
+rustfs-io-core = { path = "crates/io-core", version = "1.0.1" }
+rustfs-object-capacity = { path = "crates/object-capacity", version = "1.0.1" }
+rustfs-object-data-cache = { path = "crates/object-data-cache", version = "1.0.1", default-features = false }
+rustfs-log-analyzer = { path = "crates/log-analyzer", version = "1.0.1" }
+rustfs-obs = { path = "crates/obs", version = "1.0.1" }
+rustfs-policy = { path = "crates/policy", version = "1.0.1" }
+rustfs-protos = { path = "crates/protos", version = "1.0.1" }
+rustfs-protocols = { path = "crates/protocols", version = "1.0.1" }
+rustfs-replication = { path = "crates/replication", version = "1.0.1" }
+rustfs-rio = { path = "crates/rio", version = "1.0.1" }
+rustfs-rio-v2 = { path = "crates/rio-v2", version = "1.0.1" }
+rustfs-s3-client = { path = "crates/s3-client", version = "1.0.1" }
+rustfs-s3-types = { path = "crates/s3-types", version = "1.0.1" }
+rustfs-s3-ops = { path = "crates/s3-ops", version = "1.0.1" }
+rustfs-s3select-api = { path = "crates/s3select-api", version = "1.0.1" }
+rustfs-s3select-query = { path = "crates/s3select-query", version = "1.0.1" }
+rustfs-scanner = { path = "crates/scanner", version = "1.0.1" }
+rustfs-security-governance = { path = "crates/security-governance", version = "1.0.1" }
+rustfs-extension-schema = { path = "crates/extension-schema", version = "1.0.1" }
+rustfs-signer = { path = "crates/signer", version = "1.0.1" }
+rustfs-storage-api = { path = "crates/storage-api", version = "1.0.1" }
+rustfs-trusted-proxies = { path = "crates/trusted-proxies", version = "1.0.1" }
+rustfs-targets = { path = "crates/targets", version = "1.0.1" }
+rustfs-test-utils = { path = "crates/test-utils", version = "1.0.1" }
+rustfs-tls-runtime = { path = "crates/tls-runtime", version = "1.0.1" }
+rustfs-utils = { path = "crates/utils", version = "1.0.1" }
+rustfs-zip = { path = "./crates/zip", version = "1.0.1" }
 
 # Async Runtime and Networking
 async-channel = "2.5.0"
 async_zip = { default-features = false, version = "0.0.19" }
 mysql_async = { default-features = false, version = "0.37.1" }
-async-compression = { version = "0.4.51" }
+async-compression = { version = "0.4.50" }
 async-trait = "0.1.92"
 async-nats = { version = "0.50.0", default-features = false }
 axum = "0.8.9"
@@ -155,10 +155,9 @@ futures-core = "0.3.34"
 futures-lite = "2.6.1"
 futures-util = "0.3.34"
 pollster = "1.0.1"
-pulsar = { default-features = false, version = "=6.9.0" }
-lapin = { default-features = false, version = "4.12.3" }
+pulsar = { default-features = false, version = "6.9.0" }
+lapin = { default-features = false, version = "4.12.0" }
 hyper = { version = "1.12.0" }
-h2 = { version = "0.4.20" }
 hyper-rustls = { default-features = false, version = "0.27.10" }
 hyper-util = { version = "0.1.21" }
 http = "1.5.0"
@@ -169,11 +168,11 @@ minlz = "1.2.3"
 reqwest = "0.13.5"
 rustfs-kafka-async = { version = "1.4.0" }
 socket2 = { version = "0.6.5" }
-tokio = { version = "1.54.1" }
+tokio = { version = "1.53.2" }
 tokio-rustls = { default-features = false, version = "0.26.6" }
 tokio-stream = { version = "0.1.19" }
 tokio-test = "0.4.6"
-tokio-util = { version = "0.7.20" }
+tokio-util = { version = "0.7.19" }
 tonic = { version = "0.14.6" }
 tonic-prost = { version = "0.14.6" }
 tonic-prost-build = { version = "0.14.6" }
@@ -201,7 +200,7 @@ rmp-serde = { version = "1.3.1" }
 serde = { version = "1.0.229" }
 serde_ignored = { version = "0.1" }
 serde_with = { version = "3", default-features = false, features = ["macros", "std"] }
-serde_json = { version = "1.0.154" }
+serde_json = { version = "1.0.151" }
 serde_urlencoded = "0.7.1"
 
 # Cryptography and Security
@@ -220,12 +219,12 @@ openidconnect = { default-features = false, version = "4.0" }
 pbkdf2 = "0.13.0"
 p256 = { version = "0.14.0", features = ["ecdsa", "pkcs8"] }
 rsa = { version = "=0.10.0-rc.18" }
-rustls = { default-features = false, version = "0.23.46" }
+rustls = { default-features = false, version = "0.23.45" }
 rustls-native-certs = "0.8"
 rustls-pki-types = "1.15.1"
 x509-parser = "0.18.1"
 sha1 = "0.11.0"
-sha2 = "0.11.1"
+sha2 = "0.11.0"
 subtle = "2.6"
 zeroize = { version = "1.9.1" }
 proptest = "1"
@@ -243,22 +242,22 @@ tokio-postgres-rustls = "0.14.0"
 
 # Utilities and Tools
 anyhow = "1.0.104"
-arc-swap = "1.9.3"
+arc-swap = "1.9.2"
 rustfs-tokio-tar = { version = "0.7.1" }
 # Candidate Snowball parser versions exercised by rustfs-zip compatibility fixtures.
 tar-codec = "0.0.14"
 tar-framing = "0.0.14"
-atoi = "3.1.1"
+atoi = "3.1.0"
 atomic_enum = "0.3.0"
-aws-config = { version = "1.13.0" }
-aws-credential-types = { version = "1.3.1" }
-aws-sdk-kms = { default-features = false, version = "1.126.0" }
-aws-sdk-s3 = { default-features = false, version = "1.155.0" }
-aws-sdk-sts = { default-features = false, version = "1.122.0" }
-aws-smithy-async = { version = "1.3.1" }
-aws-smithy-http-client = { default-features = false, version = "1.5.1" }
-aws-smithy-runtime-api = { version = "1.19.1" }
-aws-smithy-types = { version = "1.9.0" }
+aws-config = { version = "1.12.0" }
+aws-credential-types = { version = "1.3.0" }
+aws-sdk-kms = { default-features = false, version = "1.123.0" }
+aws-sdk-s3 = { default-features = false, version = "1.152.0" }
+aws-sdk-sts = { default-features = false, version = "1.119.0" }
+aws-smithy-async = { version = "1.3.0" }
+aws-smithy-http-client = { default-features = false, version = "1.5.0" }
+aws-smithy-runtime-api = { version = "1.19.0" }
+aws-smithy-types = { version = "1.8.1" }
 base64-simd = "0.8.0"
 brotli = "9.0.0"
 clap = { version = "4.6.7" }
@@ -272,12 +271,12 @@ crossbeam-utils = "0.8.23"
 datafusion = { default-features = false, version = "55.2.0" }
 derive_builder = "0.20.2"
 ed25519-dalek = "3.0.0"
-enumset = "1.1.15"
+enumset = "1.1.14"
 faster-hex = "1.0.0"
 flate2 = "1.1.10"
 glob = "0.3.4"
-google-cloud-storage = "1.21.0"
-google-cloud-auth = "1.18.0"
+google-cloud-storage = "1.20.0"
+google-cloud-auth = "1.17.0"
 hashbrown = { version = "0.17.1" }
 # Base32 for RFC 6238 TOTP shared secrets (RFC 4648 unpadded, the alphabet
 # every authenticator app expects). Already in the graph transitively.
@@ -287,11 +286,11 @@ highway = { version = "1.3.0" }
 hostname = "0.4.2"
 ipnetwork = { version = "0.21.1" }
 lazy_static = "1.5.1"
-libc = "0.2.191"
+libc = "0.2.190"
 libsystemd = "0.7.2"
 local-ip-address = "0.6.13"
 log = "0.4"
-memmap2 = "0.9.12"
+memmap2 = "0.9.11"
 lz4 = "1.28.1"
 matchit = "0.9.2"
 md-5 = "0.11.0"
@@ -317,7 +316,7 @@ rustfs-erasure-codec = { version = "9.0.0" }
 reed-solomon-simd = "3.1.0"
 regex = { version = "1.13.1" }
 rumqttc = { package = "rumqttc-next", version = "0.34.0" }
-redis = { version = "1.8.0" }
+redis = { version = "1.7.1" }
 rustify = { version = "0.7", default-features = false }
 rustix = { version = "1.1.5" }
 rust-embed = { version = "8.13.0" }
@@ -327,7 +326,7 @@ s3s-sigv4 = { git = "https://github.com/s3s-project/s3s", rev = "5761ddfe4c509fc
 serial_test = "4.0.1"
 shadow-rs = { default-features = false, version = "2.0.0" }
 siphasher = "1.0.4"
-smallvec = { version = "1.16.4" }
+smallvec = { version = "1.16.2" }
 compact_str = "0.10.0"
 snap = "1.1.2"
 starshard = { version = "2.4.0" }
@@ -346,7 +345,7 @@ tracing-subscriber = { version = "0.3.23" }
 transform-stream = "0.3.1"
 url = "2.5.8"
 urlencoding = "2.1.3"
-uuid = { version = "1.29.0" }
+uuid = { version = "1.27.0" }
 vaultrs = { version = "0.8.0" }
 tar = "0.4.46"
 walkdir = "2.5.0"
@@ -354,26 +353,26 @@ winapi-util = "0.1.11"
 windows = { version = "0.62.2" }
 windows-sys = "0.61.2"
 xxhash-rust = { version = "0.8.19" }
-zip = "9.1.0"
-zstd = "0.14.3"
+zip = "8.6.0"
+zstd = "0.14.0"
 
 # Observability and Metrics
 metrics = "0.24.6"
 metrics-util = "0.20"
-dial9 = { version = "0.5.5", default-features = false }
-opentelemetry = { version = "0.33.1" }
-opentelemetry-appender-tracing = { version = "0.33.1" }
-opentelemetry-otlp = { version = "0.33.1" }
-opentelemetry-proto = { version = "0.33.1", default-features = false, features = ["metrics", "gen-tonic-messages"] }
-opentelemetry_sdk = { version = "0.33.1" }
-opentelemetry-semantic-conventions = { version = "0.33.1" }
-opentelemetry-stdout = { version = "0.33.1" }
-pyroscope = { version = "2.1.5" }
+dial9 = { version = "0.5.3", default-features = false }
+opentelemetry = { version = "0.33.0" }
+opentelemetry-appender-tracing = { version = "0.33.0" }
+opentelemetry-otlp = { version = "0.33.0" }
+opentelemetry-proto = { version = "0.33.0", default-features = false, features = ["metrics", "gen-tonic-messages"] }
+opentelemetry_sdk = { version = "0.33.0" }
+opentelemetry-semantic-conventions = { version = "0.33.0" }
+opentelemetry-stdout = { version = "0.33.0" }
+pyroscope = { version = "2.1.1" }
 
 # FTP and SFTP
 libunftp = { version = "0.23.1" }
 unftp-core = "0.1.0"
-suppaftp = { version = "12.2.1" }
+suppaftp = { version = "12.1.1" }
 rcgen = { version = "0.14.10", default-features = false, features = ["aws_lc_rs", "crypto", "pem"] }
 russh = { version = "0.64.1" }
 russh-sftp = "3.0.1"
@@ -384,7 +383,7 @@ dav-server = "0.11.0"
 # Performance Analysis and Memory Profiling
 rustfs-mimalloc = { version = "0.5.6" }
 # Preserve Unicode focus filters until rustfs/backlog#2302 is resolved.
-hotpath = { version = "0.28.7", default-features = false }
+hotpath = { version = "0.28.5", default-features = false }
 # Snapshot testing for output format regression detection
 insta = { version = "1.49" }
 

Dockerfile

@@ -18,13 +18,6 @@ ARG TARGETARCH
 ARG RELEASE=latest
 
 RUN apk add --no-cache ca-certificates curl jq unzip
-# Keep the native store safe even if the distribution CA package lags distrust.
-COPY scripts/check_ca_bundle.sh /tmp/check_ca_bundle.sh
-RUN sed -i '/^mozilla\/emSign_ECC_Root_CA_-_G3.crt$/s/^/!/' /etc/ca-certificates.conf && \
-    update-ca-certificates --fresh && \
-    sh /tmp/check_ca_bundle.sh /etc/ssl/certs/ca-certificates.crt /etc/ssl/certs && \
-    rm /tmp/check_ca_bundle.sh
-
 WORKDIR /build
 
 RUN set -eux; \
@@ -105,13 +98,7 @@ RUN apk upgrade --no-cache && \
     tzdata \
     && test "$(TZ=Asia/Kolkata date +%z)" = "+0530"
 
-# Keep the native store safe even if the distribution CA package lags distrust.
-COPY scripts/check_ca_bundle.sh /tmp/check_ca_bundle.sh
-RUN sed -i '/^mozilla\/emSign_ECC_Root_CA_-_G3.crt$/s/^/!/' /etc/ca-certificates.conf && \
-    update-ca-certificates --fresh && \
-    sh /tmp/check_ca_bundle.sh /etc/ssl/certs/ca-certificates.crt /etc/ssl/certs && \
-    rm /tmp/check_ca_bundle.sh
-
+COPY --from=build /etc/ssl/certs/ca-certificates.crt /etc/ssl/certs/
 COPY --from=build /build/rustfs /usr/bin/rustfs
 COPY entrypoint.sh /entrypoint.sh
 

Dockerfile.glibc

@@ -24,13 +24,6 @@ RUN apt-get update && apt-get install -y --no-install-recommends \
     unzip \
     && rm -rf /var/lib/apt/lists/*
 
-# Keep the native store safe even if the distribution CA package lags distrust.
-COPY scripts/check_ca_bundle.sh /tmp/check_ca_bundle.sh
-RUN sed -i '/^mozilla\/emSign_ECC_Root_CA_-_G3.crt$/s/^/!/' /etc/ca-certificates.conf && \
-    update-ca-certificates --fresh && \
-    sh /tmp/check_ca_bundle.sh /etc/ssl/certs/ca-certificates.crt /etc/ssl/certs && \
-    rm /tmp/check_ca_bundle.sh
-
 WORKDIR /build
 
 RUN set -eux; \
@@ -110,13 +103,6 @@ RUN apt-get update && apt-get upgrade -y \
     && test "$(TZ=Asia/Kolkata date +%z)" = "+0530" \
     && rm -rf /var/lib/apt/lists/*
 
-# Keep the native store safe even if the distribution CA package lags distrust.
-COPY scripts/check_ca_bundle.sh /tmp/check_ca_bundle.sh
-RUN sed -i '/^mozilla\/emSign_ECC_Root_CA_-_G3.crt$/s/^/!/' /etc/ca-certificates.conf && \
-    update-ca-certificates --fresh && \
-    sh /tmp/check_ca_bundle.sh /etc/ssl/certs/ca-certificates.crt /etc/ssl/certs && \
-    rm /tmp/check_ca_bundle.sh
-
 COPY --from=build /build/rustfs /usr/bin/rustfs
 COPY entrypoint.sh /entrypoint.sh
 

crates/common/Cargo.toml

@@ -34,7 +34,6 @@ hotpath-alloc = ["hotpath", "hotpath/hotpath-alloc"]
 hotpath-cpu = ["hotpath", "hotpath/hotpath-cpu"]
 
 [dependencies]
-arc-swap.workspace = true
 hotpath.workspace = true
 tokio = { workspace = true, features = ["fs", "rt-multi-thread"] }
 tonic = { workspace = true, features = ["gzip", "deflate"] }

crates/common/src/globals.rs

@@ -12,10 +12,9 @@
 // See the License for the specific language governing permissions and
 // limitations under the License.
 
-use arc_swap::ArcSwap;
 use std::collections::HashMap;
+use std::sync::LazyLock;
 use std::sync::atomic::{AtomicU64, Ordering};
-use std::sync::{Arc, LazyLock};
 use tokio::sync::RwLock;
 use tonic::transport::Channel;
 
@@ -28,42 +27,6 @@ pub static GLOBAL_ROOT_CERT: LazyLock<RwLock<Option<Vec<u8>>>> = LazyLock::new(|
 pub static GLOBAL_MTLS_IDENTITY: LazyLock<RwLock<Option<MtlsIdentityPem>>> = LazyLock::new(|| RwLock::new(None));
 pub static GLOBAL_OUTBOUND_TLS_GENERATION: LazyLock<AtomicU64> = LazyLock::new(|| AtomicU64::new(0));
 
-#[derive(Clone, Default)]
-pub struct OutboundTlsSnapshot {
-    pub generation: u64,
-    pub root_ca_pem: Option<Vec<u8>>,
-    pub mtls_identity: Option<MtlsIdentityPem>,
-}
-
-static GLOBAL_OUTBOUND_TLS_STATE: LazyLock<ArcSwap<OutboundTlsSnapshot>> =
-    LazyLock::new(|| ArcSwap::from_pointee(OutboundTlsSnapshot::default()));
-
-pub fn get_global_outbound_tls_snapshot() -> Arc<OutboundTlsSnapshot> {
-    GLOBAL_OUTBOUND_TLS_STATE.load_full()
-}
-
-/// Publish identity, trust and generation together; the individual globals are legacy views.
-pub async fn publish_global_outbound_tls_snapshot(snapshot: OutboundTlsSnapshot) -> u64 {
-    GLOBAL_OUTBOUND_TLS_STATE.rcu(|current| {
-        if current.generation != 0
-            && current.root_ca_pem == snapshot.root_ca_pem
-            && current.mtls_identity == snapshot.mtls_identity
-        {
-            return Arc::clone(current);
-        }
-        Arc::new(OutboundTlsSnapshot {
-            generation: snapshot.generation.max(current.generation.saturating_add(1)),
-            root_ca_pem: snapshot.root_ca_pem.clone(),
-            mtls_identity: snapshot.mtls_identity.clone(),
-        })
-    });
-    let accepted = get_global_outbound_tls_snapshot();
-    *GLOBAL_ROOT_CERT.write().await = accepted.root_ca_pem.clone();
-    *GLOBAL_MTLS_IDENTITY.write().await = accepted.mtls_identity.clone();
-    GLOBAL_OUTBOUND_TLS_GENERATION.store(accepted.generation, Ordering::Relaxed);
-    accepted.generation
-}
-
 /// Log level to use when reporting cached gRPC connection eviction.
 #[derive(Clone, Copy, Debug, Eq, PartialEq)]
 pub enum ConnectionEvictionLogLevel {
@@ -122,27 +85,17 @@ pub async fn get_global_addr() -> String {
 /// # Arguments
 /// * `cert` - A vector of bytes representing the PEM-encoded root CA certificate.
 pub async fn set_global_root_cert(cert: Vec<u8>) {
-    GLOBAL_OUTBOUND_TLS_STATE.rcu(|current| {
-        let mut next = (**current).clone();
-        next.root_ca_pem = Some(cert.clone());
-        Arc::new(next)
-    });
     *GLOBAL_ROOT_CERT.write().await = Some(cert);
 }
 
 /// Clear the global root CA certificate for outbound gRPC clients.
 pub async fn clear_global_root_cert() {
-    GLOBAL_OUTBOUND_TLS_STATE.rcu(|current| {
-        let mut next = (**current).clone();
-        next.root_ca_pem = None;
-        Arc::new(next)
-    });
     *GLOBAL_ROOT_CERT.write().await = None;
 }
 
 /// Get the global root CA certificate for outbound gRPC clients.
 pub async fn get_global_root_cert() -> Option<Vec<u8>> {
-    get_global_outbound_tls_snapshot().root_ca_pem.clone()
+    GLOBAL_ROOT_CERT.read().await.clone()
 }
 
 /// Set the global mTLS identity (cert+key PEM) for outbound gRPC clients.
@@ -152,32 +105,22 @@ pub async fn get_global_root_cert() -> Option<Vec<u8>> {
 /// # Arguments
 /// * `identity` - An optional MtlsIdentityPem struct containing the cert and key PEM.
 pub async fn set_global_mtls_identity(identity: Option<MtlsIdentityPem>) {
-    GLOBAL_OUTBOUND_TLS_STATE.rcu(|current| {
-        let mut next = (**current).clone();
-        next.mtls_identity = identity.clone();
-        Arc::new(next)
-    });
     *GLOBAL_MTLS_IDENTITY.write().await = identity;
 }
 
 /// Get the global mTLS identity for outbound gRPC clients.
 pub async fn get_global_mtls_identity() -> Option<MtlsIdentityPem> {
-    get_global_outbound_tls_snapshot().mtls_identity.clone()
+    GLOBAL_MTLS_IDENTITY.read().await.clone()
 }
 
 /// Set the global outbound TLS generation.
 pub fn set_global_outbound_tls_generation(generation: u64) {
-    GLOBAL_OUTBOUND_TLS_STATE.rcu(|current| {
-        let mut next = (**current).clone();
-        next.generation = generation;
-        Arc::new(next)
-    });
     GLOBAL_OUTBOUND_TLS_GENERATION.store(generation, Ordering::Relaxed);
 }
 
 /// Get the global outbound TLS generation.
 pub fn get_global_outbound_tls_generation() -> u64 {
-    get_global_outbound_tls_snapshot().generation
+    GLOBAL_OUTBOUND_TLS_GENERATION.load(Ordering::Relaxed)
 }
 
 /// Evict a stale/dead connection from the global connection cache.
@@ -251,77 +194,8 @@ pub async fn clear_all_connections() {
 ///
 /// When present, gRPC clients will present this identity to servers requesting/requiring mTLS.
 /// When absent, clients proceed with standard server-authenticated TLS.
-#[derive(Clone, PartialEq, Eq)]
+#[derive(Clone, Debug)]
 pub struct MtlsIdentityPem {
     pub cert_pem: Vec<u8>,
     pub key_pem: Vec<u8>,
 }
-
-impl std::fmt::Debug for MtlsIdentityPem {
-    fn fmt(&self, f: &mut std::fmt::Formatter<'_>) -> std::fmt::Result {
-        f.debug_struct("MtlsIdentityPem")
-            .field("cert_bytes", &self.cert_pem.len())
-            .field("key_pem", &"[REDACTED]")
-            .finish()
-    }
-}
-
-#[cfg(test)]
-mod tls_snapshot_tests {
-    use super::*;
-
-    #[tokio::test]
-    async fn outbound_snapshot_publication_is_coherent_and_skips_unchanged_material() {
-        let start = get_global_outbound_tls_generation().saturating_add(1);
-        let snapshot = OutboundTlsSnapshot {
-            generation: start,
-            root_ca_pem: Some(vec![1]),
-            mtls_identity: Some(MtlsIdentityPem {
-                cert_pem: vec![1],
-                key_pem: vec![1],
-            }),
-        };
-        let first = publish_global_outbound_tls_snapshot(snapshot.clone()).await;
-        assert_eq!(
-            publish_global_outbound_tls_snapshot(OutboundTlsSnapshot {
-                generation: first.saturating_add(1),
-                ..snapshot
-            })
-            .await,
-            first
-        );
-        let writer = tokio::spawn(async move {
-            for tag in 2_u8..=255 {
-                publish_global_outbound_tls_snapshot(OutboundTlsSnapshot {
-                    generation: start.saturating_add(u64::from(tag)),
-                    root_ca_pem: Some(vec![tag]),
-                    mtls_identity: Some(MtlsIdentityPem {
-                        cert_pem: vec![tag],
-                        key_pem: vec![tag],
-                    }),
-                })
-                .await;
-                tokio::task::yield_now().await;
-            }
-        });
-        while !writer.is_finished() {
-            let state = get_global_outbound_tls_snapshot();
-            let identity = state.mtls_identity.as_ref().expect("identity in complete snapshot");
-            assert_eq!(state.root_ca_pem.as_ref().expect("roots in complete snapshot"), &identity.cert_pem);
-            assert_eq!(identity.cert_pem, identity.key_pem);
-            tokio::task::yield_now().await;
-        }
-        writer.await.expect("writer completes");
-    }
-
-    #[test]
-    fn mtls_identity_debug_redacts_private_key() {
-        let identity = MtlsIdentityPem {
-            cert_pem: b"public certificate".to_vec(),
-            key_pem: b"PRIVATE-KEY-MARKER".to_vec(),
-        };
-        let debug = format!("{identity:?}");
-        assert!(!debug.contains("PRIVATE-KEY-MARKER"));
-        assert!(debug.contains("[REDACTED]"));
-    }
-}

crates/e2e_test/src/common.rs

@@ -1542,15 +1542,6 @@ impl RustFSTestClusterEnvironment {
         self.start_node_from_binary(node_idx, &binary_path).await
     }
 
-    /// Start a process and wait only for TCP, allowing tests to bring up a
-    /// quorum one node at a time before the S3 service can become ready.
-    pub async fn start_node_process(&mut self, node_idx: usize) -> Result<(), Box<dyn std::error::Error + Send + Sync>> {
-        let binary_path = rustfs_binary_path();
-        let volumes_arg = self.build_volumes_arg();
-        self.spawn_node(node_idx, &binary_path, &volumes_arg)?;
-        self.wait_for_node_ready(&self.nodes[node_idx].address, node_idx).await
-    }
-
     /// Start one stopped cluster node with a specific RustFS binary while
     /// preserving the cluster's volume layout and that node's data directory.
     pub async fn start_node_from_binary(

crates/e2e_test/src/distributed/harness.rs

@@ -860,12 +860,10 @@ pub(crate) async fn start_rebalance(cluster: &RustFSTestClusterEnvironment) -> T
         if status.is_success() {
             break response;
         }
-        // The init stage answers 503 + Retry-After while the cluster capability
-        // proof is still missing; older binaries surfaced the same condition as
-        // a 500 carrying the proof marker.
-        let waiting_for_readiness = status == StatusCode::SERVICE_UNAVAILABLE
-            || (status == StatusCode::INTERNAL_SERVER_ERROR && response.contains("requires a live fleet capability proof"));
-        if waiting_for_readiness && Instant::now() < deadline {
+        if status == StatusCode::INTERNAL_SERVER_ERROR
+            && response.contains("requires a live fleet capability proof")
+            && Instant::now() < deadline
+        {
             sleep(Duration::from_secs(1)).await;
             continue;
         }

crates/e2e_test/src/heal_erasure_disk_rebuild_test.rs

@@ -1191,66 +1191,11 @@ mod tests {
         .into())
     }
 
-    fn scanner_cycle_completion_budget(
-        cycle_end: u64,
-        interval: Option<u64>,
-        now: u64,
-        active: bool,
-    ) -> Result<Duration, &'static str> {
-        if cycle_end > now {
-            return Err("scanner cycle timestamp is in the future");
-        }
-        if active {
-            return Ok(Duration::from_secs(60));
-        }
-        let interval = interval
-            .filter(|interval| *interval != 0)
-            .ok_or("scanner schedule has no positive interval")?;
-        // A new leader may already be sleeping before completing its first cycle.
-        let schedule_start = if cycle_end == 0 { now } else { cycle_end };
-        let next_cycle = schedule_start
-            .checked_add(interval)
-            .ok_or("scanner schedule timestamp overflows")?;
-        let budget = next_cycle
-            .saturating_sub(now)
-            .checked_add(60)
-            .ok_or("scanner completion budget overflows")?;
-        Ok(Duration::from_secs(budget))
-    }
-
-    #[test]
-    fn scanner_cycle_completion_budget_accounts_for_captured_retry_schedule() {
-        assert_eq!(
-            scanner_cycle_completion_budget(1_791_815_390, Some(256), 1_791_815_447, false),
-            Ok(Duration::from_secs(259)),
-        );
-        assert_eq!(scanner_cycle_completion_budget(100, Some(1), 110, false), Ok(Duration::from_secs(60)));
-        // A newly elected leader can advertise a sleep before its first completed cycle.
-        assert_eq!(
-            scanner_cycle_completion_budget(0, Some(289), 1_792_913_211, false),
-            Ok(Duration::from_secs(349)),
-        );
-    }
-
-    #[test]
-    fn scanner_cycle_completion_budget_does_not_grant_planned_sleep_to_active_cycle() {
-        assert_eq!(scanner_cycle_completion_budget(100, Some(256), 110, true), Ok(Duration::from_secs(60)));
-        assert_eq!(scanner_cycle_completion_budget(0, None, 110, true), Ok(Duration::from_secs(60)));
-    }
-
-    #[test]
-    fn scanner_cycle_completion_budget_rejects_invalid_schedule() {
-        for (end, interval, now) in [(101, 1, 100), (100, 0, 100), (u64::MAX, 1, u64::MAX), (1, u64::MAX - 1, 1)] {
-            assert!(scanner_cycle_completion_budget(end, Some(interval), now, false).is_err());
-        }
-    }
-
     async fn wait_for_scanner_cycle_after(
         cluster: &RustFSTestClusterEnvironment,
         previous_cycle_end: u64,
     ) -> Result<u64, Box<dyn Error + Send + Sync>> {
-        let mut deadline = Instant::now() + Duration::from_secs(60);
-        let mut accounted_schedules = HashSet::new();
+        let deadline = Instant::now() + Duration::from_secs(60);
         loop {
             let mut latest_cycle_end = 0;
             let mut versions_observed = false;
@@ -1292,48 +1237,14 @@ mod tests {
                 let versions_scanned = metrics["versions_scanned"]
                     .as_u64()
                     .ok_or("scanner status is missing its version-coverage counter")?;
-                let now = std::time::SystemTime::now().duration_since(std::time::UNIX_EPOCH)?.as_secs();
-                if cycle_end > now {
-                    return Err("scanner cycle timestamp is in the future".into());
-                }
-                let schedule = &status["cycle_schedule"];
-                if schedule["execution_role"].as_str() == Some("leader") {
-                    let active = metrics["current_cycle_active"]
-                        .as_bool()
-                        .ok_or("scanner status is missing its active-cycle flag")?;
-                    if active || schedule["effective_interval_available"].as_bool() == Some(true) {
-                        let interval = if active {
-                            None
-                        } else {
-                            Some(
-                                schedule["effective_interval_seconds"]
-                                    .as_u64()
-                                    .ok_or("scanner schedule is missing its effective interval")?,
-                            )
-                        };
-                        // Reconsider a changed leader or retry plan, but never renew
-                        // the deadline on repeated observations of the same sleep.
-                        if accounted_schedules.insert((node_index, cycle_end, interval)) {
-                            let budget = scanner_cycle_completion_budget(cycle_end, interval, now, active)?;
-                            let scheduled_deadline = Instant::now().checked_add(budget).ok_or("scanner deadline overflows")?;
-                            deadline = deadline.max(scheduled_deadline);
-                        }
-                    }
-                }
-                // Superseded walks complete all sets while withholding quota authority.
-                // Deferred and partial attempts do not certify completed scanning.
-                if matches!(metrics["last_cycle_result"].as_str(), Some("success" | "superseded")) {
-                    latest_cycle_end = latest_cycle_end.max(cycle_end);
-                }
+                latest_cycle_end = latest_cycle_end.max(cycle_end);
                 versions_observed |= versions_scanned > 0;
                 observations.push(format!(
-                    "node{node_index}: end={cycle_end}, versions={versions_scanned}, cycle={}, active={}, leader={}, result={}, partial_reason={}, schedule={}",
+                    "node{node_index}: end={cycle_end}, versions={versions_scanned}, cycle={}, active={}, leader={}, result={}",
                     metrics["current_cycle"],
                     metrics["current_cycle_active"],
                     metrics["leader_lock_state"],
                     metrics["last_cycle_result"],
-                    metrics["last_cycle_partial_reason"],
-                    schedule,
                 ));
             }
             // The coordinator records cycle completion, but remote workers
@@ -1644,8 +1555,7 @@ mod tests {
         }
 
         if background_enabled {
-            let baseline_cycle_floor = std::time::SystemTime::now().duration_since(std::time::UNIX_EPOCH)?.as_secs();
-            wait_for_scanner_cycle_after(&cluster, baseline_cycle_floor).await?;
+            wait_for_scanner_cycle_after(&cluster, 0).await?;
         }
 
         cluster.stop_node(1)?;
@@ -1988,16 +1898,11 @@ mod tests {
             let status_body = signed_admin_post(&status_url, None, &cluster.access_key, &cluster.secret_key).await?;
             let status: serde_json::Value = serde_json::from_str(&status_body)
                 .map_err(|err| format!("pre-interrupt background heal status is not JSON ({err}): {status_body}"))?;
-            let admin_active = background_rejoin_heal_evidence
-                || (status["state"].as_str() == Some("active")
-                    && status["healOperations"]["activeBySource"]["admin"].as_u64() == Some(1));
-            if admin_active && (background_enabled || only_admin_heal_is_active(&status)) {
+            if background_enabled || only_admin_heal_is_active(&status) {
                 break status;
             }
             if Instant::now() >= partial_deadline {
-                return Err(
-                    format!("heal tasks did not reach the required interruption state before the deadline: {status}").into(),
-                );
+                return Err(format!("non-admin heal did not drain before the interruption deadline: {status}").into());
             }
             sleep(Duration::from_millis(50)).await;
         };
@@ -2657,157 +2562,4 @@ mod tests {
         };
         assert!(!outage_candidate_targets_replacement(&insufficient_peer_shards, 12, replacement_set_slot));
     }
-    async fn two_node_heal_status(
-        cluster: &RustFSTestClusterEnvironment,
-        bucket: Option<&str>,
-        recreate: bool,
-        scan_mode: u8,
-    ) -> Result<serde_json::Value, Box<dyn Error + Send + Sync>> {
-        let url = format!("{}/rustfs/admin/v3/heal/{}", cluster.nodes[0].url, bucket.unwrap_or_default());
-        let options = serde_json::json!({
-            "recursive": true, "dryRun": false, "remove": false,
-            "recreate": recreate, "scanMode": scan_mode, "updateParity": false, "nolock": false,
-        });
-        let started =
-            start_root_heal_when_control_ready(&url, &options.to_string(), &cluster.access_key, &cluster.secret_key).await?;
-        let started: serde_json::Value = serde_json::from_str(&started)?;
-        let token = started["clientToken"]
-            .as_str()
-            .ok_or_else(|| format!("missing heal token: {started}"))?;
-        let status_url = format!("{url}?clientToken={token}");
-        let deadline = Instant::now() + Duration::from_secs(240);
-        loop {
-            let body = timeout(
-                Duration::from_secs(20),
-                signed_admin_post(&status_url, None, &cluster.access_key, &cluster.secret_key),
-            )
-            .await??;
-            let status: serde_json::Value = serde_json::from_str(&body)?;
-            match status["summary"].as_str() {
-                Some("finished" | "stopped") => return Ok(status),
-                Some("running") if Instant::now() < deadline => sleep(Duration::from_millis(250)).await,
-                _ => return Err(format!("heal did not reach a terminal state: {status}").into()),
-            }
-        }
-    }
-
-    async fn verify_two_node_directory_recovery(
-        remove_bucket: bool,
-        scan_mode: u8,
-        bucket_scoped: bool,
-        recreate: bool,
-    ) -> Result<(), Box<dyn Error + Send + Sync>> {
-        init_logging();
-        let mut cluster = RustFSTestClusterEnvironment::new(2).await?;
-        // Isolate explicit administrator recovery from scanner, MRF, and disk healing.
-        for key in [
-            "RUSTFS_HEAL_AUTO_HEAL_ENABLE",
-            "RUSTFS_HEAL_MRF_ENABLE",
-            "RUSTFS_SCANNER_ENABLED",
-            "RUSTFS_PUT_RENAME_EARLY_ACK_ENABLE",
-        ] {
-            cluster.set_env(key, "false");
-        }
-        cluster.start().await?;
-        let source = cluster.create_s3_client(0)?;
-        let target = cluster.create_s3_client(1)?;
-        let bucket = "two-node-directory-recovery";
-        source.create_bucket().bucket(bucket).send().await?;
-        let objects = [
-            ("empty", Vec::new()),
-            ("nested/inline", vec![0x3c; 1024]),
-            ("nested/external", (0_u8..=255).cycle().take(1024 * 1024).collect::<Vec<_>>()),
-        ];
-        for (key, bytes) in &objects {
-            source
-                .put_object()
-                .bucket(bucket)
-                .key(*key)
-                .body(ByteStream::from(bytes.clone()))
-                .send()
-                .await?;
-        }
-        let target_bucket = Path::new(&cluster.nodes[1].data_dir).join(bucket);
-        let original = objects
-            .iter()
-            .map(|(key, _)| census_object_version_on_disk(Path::new(&cluster.nodes[1].data_dir), bucket, key, None))
-            .collect::<Result<Vec<_>, _>>()?;
-        for census in &original {
-            assert!(census.has_xl_meta, "fixture must have a target metadata copy: {census:?}");
-        }
-        assert!(
-            !original[2].present_part_fingerprints.is_empty(),
-            "large fixture must exercise external parts"
-        );
-        let removed = Path::new(&cluster.temp_dir).join("removed");
-        tokio::fs::create_dir(&removed).await?;
-        if remove_bucket {
-            tokio::fs::rename(&target_bucket, removed.join(bucket)).await?;
-        } else {
-            for (index, (key, _)) in objects.iter().enumerate() {
-                tokio::fs::rename(target_bucket.join(key), removed.join(index.to_string())).await?;
-            }
-        }
-        let status = two_node_heal_status(&cluster, bucket_scoped.then_some(bucket), recreate, scan_mode).await?;
-        assert_eq!(status["summary"], "finished", "administrator recovery should complete: {status}");
-        assert_eq!(status["outcome"]["execution"]["state"], "completed");
-        assert_eq!(status["outcome"]["coverage"], "complete");
-        for ((key, _), before) in objects.iter().zip(&original) {
-            let after = census_object_version_on_disk(Path::new(&cluster.nodes[1].data_dir), bucket, key, None)?;
-            assert!(after.has_xl_meta, "metadata must be restored before GET: {key}");
-            assert_eq!(
-                after.present_part_fingerprints, before.present_part_fingerprints,
-                "external shard bytes must match: {key}"
-            );
-            assert_eq!(
-                after.inline_data_fingerprint, before.inline_data_fingerprint,
-                "inline shard bytes must match: {key}"
-            );
-        }
-        cluster.stop_node(0)?;
-        for (key, expected) in &objects {
-            let actual = timeout(Duration::from_secs(30), async {
-                target
-                    .get_object()
-                    .bucket(bucket)
-                    .key(*key)
-                    .send()
-                    .await?
-                    .body
-                    .collect()
-                    .await
-                    .map(|body| body.into_bytes())
-                    .map_err(Box::<dyn Error + Send + Sync>::from)
-            })
-            .await??;
-            assert_eq!(actual.len(), expected.len(), "target-only read length: {key}");
-            assert_eq!(sha256_hex(&actual), sha256_hex(expected), "target-only content: {key}");
-        }
-        Ok(())
-    }
-
-    #[tokio::test]
-    async fn test_two_node_bucket_discovery_and_explicit_recreate() -> Result<(), Box<dyn Error + Send + Sync>> {
-        verify_two_node_directory_recovery(true, 2, true, true).await
-    }
-
-    #[tokio::test]
-    async fn test_two_node_missing_bucket_default_cluster_recovery() -> Result<(), Box<dyn Error + Send + Sync>> {
-        verify_two_node_directory_recovery(true, 2, false, false).await
-    }
-
-    #[tokio::test]
-    async fn test_two_node_missing_bucket_default_scoped_recovery() -> Result<(), Box<dyn Error + Send + Sync>> {
-        verify_two_node_directory_recovery(true, 2, true, false).await
-    }
-
-    #[tokio::test]
-    async fn test_two_node_object_directory_recovery_normal() -> Result<(), Box<dyn Error + Send + Sync>> {
-        verify_two_node_directory_recovery(false, 1, false, false).await
-    }
-
-    #[tokio::test]
-    async fn test_two_node_object_directory_recovery_deep() -> Result<(), Box<dyn Error + Send + Sync>> {
-        verify_two_node_directory_recovery(false, 2, false, false).await
-    }
 }

crates/e2e_test/src/internode_rpc_signature_e2e_test.rs

@@ -264,7 +264,7 @@ async fn call_make_volume_response(
     request: MakeVolumeRequest,
     headers: HeaderMap,
 ) -> Result<Response<MakeVolumeResponse>, Status> {
-    let mut client = node_service_time_out_client_no_auth(url)
+    let mut client = node_service_time_out_client_no_auth(&url.to_string())
         .await
         .map_err(|err| Status::unavailable(format!("cannot reach the node service: {err}")))?;
     let mut rpc_request = Request::new(request);
@@ -273,7 +273,7 @@ async fn call_make_volume_response(
 }
 
 async fn call_ping_response(url: &str, headers: HeaderMap) -> Result<Response<PingResponse>, Status> {
-    let mut client = node_service_time_out_client_no_auth(url)
+    let mut client = node_service_time_out_client_no_auth(&url.to_string())
         .await
         .map_err(|err| Status::unavailable(format!("cannot reach the node service: {err}")))?;
     let mut rpc_request = Request::new(PingRequest {

crates/e2e_test/src/namespace_lock_quorum_test.rs

@@ -12,7 +12,7 @@
 // See the License for the specific language governing permissions and
 // limitations under the License.
 
-use crate::common::{ClusterTopology, RustFSTestClusterEnvironment};
+use crate::common::RustFSTestClusterEnvironment;
 use aws_sdk_s3::Client;
 use aws_sdk_s3::error::SdkError;
 use bytes::Bytes;
@@ -185,11 +185,7 @@ async fn wait_for_quorum_read_admission(clients: &[Client], bucket: &str) -> Tes
             return Ok(());
         }
         if tokio::time::Instant::now() >= deadline {
-            let diagnostics = clients[0].get_object().bucket(bucket).key("warm-small").send().await;
-            return Err(format!(
-                "read quorum did not become available after lease convergence for {bucket}: GET={diagnostics:?}"
-            )
-            .into());
+            return Err(format!("read quorum did not become available after lease convergence for {bucket}").into());
         }
         tokio::time::sleep(std::time::Duration::from_millis(100)).await;
     }
@@ -284,14 +280,11 @@ async fn test_cold_degraded_object_mutations_require_only_storage_quorum() -> Te
 async fn test_degraded_cluster_read_quorum_follows_erasure_layout() -> TestResult {
     crate::common::init_logging();
 
-    for (node_count, parity) in [(3, 1), (4, 2), (6, 3), (6, 2), (4, 0)] {
+    for (node_count, parity) in [(4, 2), (6, 3), (6, 2)] {
         let read_quorum = node_count - parity;
         let write_quorum = read_quorum + usize::from(read_quorum == parity);
         let mut cluster = RustFSTestClusterEnvironment::new(node_count).await?;
         cluster.set_env("RUSTFS_STORAGE_CLASS_STANDARD", format!("EC:{parity}"));
-        if parity == 0 {
-            cluster.set_env("RUSTFS_STORAGE_CLASS_RRS", "EC:0");
-        }
         // Wait for every seed fanout before removing any physical shard.
         cluster.set_env("RUSTFS_PUT_RENAME_EARLY_ACK_ENABLE", "false");
         cluster.set_env("RUSTFS_OBS_METRICS_EXPORT_ENABLED", "false");
@@ -430,277 +423,6 @@ async fn test_degraded_cluster_read_quorum_follows_erasure_layout() -> TestResul
     Ok(())
 }
 
-#[tokio::test]
-async fn test_cold_start_read_quorum_follows_erasure_layout() -> TestResult {
-    crate::common::init_logging();
-    for (node_count, parity) in [(3, 1), (4, 2), (6, 3), (6, 2), (4, 0)] {
-        let read_quorum = node_count - parity;
-        let write_quorum = read_quorum + usize::from(read_quorum == parity);
-        let mut cluster = RustFSTestClusterEnvironment::new(node_count).await?;
-        cluster.set_env("RUSTFS_STORAGE_CLASS_STANDARD", format!("EC:{parity}"));
-        if parity == 0 {
-            cluster.set_env("RUSTFS_STORAGE_CLASS_RRS", "EC:0");
-        }
-        cluster.set_env("RUSTFS_PUT_RENAME_EARLY_ACK_ENABLE", "false");
-        cluster.set_env("RUSTFS_OBS_METRICS_EXPORT_ENABLED", "false");
-        cluster.set_env("RUST_LOG", "warn");
-        cluster.start().await?;
-        let clients = cluster
-            .create_all_clients()?
-            .into_iter()
-            .map(|client| {
-                Client::from_conf(
-                    client
-                        .config()
-                        .to_builder()
-                        .retry_config(aws_sdk_s3::config::retry::RetryConfig::standard().with_max_attempts(1))
-                        .build(),
-                )
-            })
-            .collect::<Vec<_>>();
-        let bucket = format!("cold-read-quorum-{node_count}-{parity}");
-        clients[0].create_bucket().bucket(&bucket).send().await?;
-        let small = b"cold startup follows the stored EC layout".to_vec();
-        let large = vec![73; 256 * 1024];
-        for (key, body) in [
-            ("warm-small", &small),
-            ("warm-large", &large),
-            ("cold-large", &large),
-            ("below-quorum", &large),
-        ] {
-            clients[0]
-                .put_object()
-                .bucket(&bucket)
-                .key(key)
-                .body(Bytes::copy_from_slice(body).into())
-                .send()
-                .await?;
-            for node in &cluster.nodes {
-                let census =
-                    crate::chaos::census_object_version_on_disk(std::path::Path::new(&node.data_dir), &bucket, key, None)?;
-                assert!(census.is_complete(), "seed shard must exist before cold startup: {census:?}");
-                assert_eq!((census.data_blocks, census.parity_blocks), (Some(read_quorum), Some(parity)));
-            }
-        }
-        for node in 0..node_count {
-            cluster.stop_node(node)?;
-        }
-        // Rebuild every process's in-memory state below read quorum, then
-        // cross the EC boundary. A format bootstrap must not admit S3 reads
-        // merely because half the peers are online (e.g. six disks, EC:2).
-        for node in 0..read_quorum - 1 {
-            cluster.start_node_process(node).await?;
-        }
-        let error = clients[0]
-            .get_object()
-            .bucket(&bucket)
-            .key("below-quorum")
-            .send()
-            .await
-            .expect_err("cold startup below the configured EC read quorum must stay unavailable");
-        assert_eq!(error.raw_response().map(|response| response.status().as_u16()), Some(503));
-        cluster.start_node_process(read_quorum - 1).await?;
-        wait_for_quorum_read_admission(&clients[..read_quorum], &bucket).await?;
-        for client in clients.iter().take(read_quorum) {
-            assert_quorum_object_body(client, &bucket, "warm-small", &small).await?;
-            assert_quorum_object_body(client, &bucket, "cold-large", &large).await?;
-            let write = client
-                .put_object()
-                .bucket(&bucket)
-                .key("cold-quorum-write")
-                .body(Bytes::copy_from_slice(&small).into())
-                .send()
-                .await;
-            if read_quorum >= write_quorum {
-                write?;
-                assert_quorum_object_body(client, &bucket, "cold-quorum-write", &small).await?;
-            } else {
-                let error = write.expect_err("cold startup at read quorum must preserve the EC write quorum");
-                assert_eq!(error.raw_response().map(|response| response.status().as_u16()), Some(503));
-            }
-        }
-        for node in read_quorum..node_count {
-            cluster.start_node(node).await?;
-        }
-        for client in &clients {
-            assert_quorum_object_body(client, &bucket, "cold-large", &large).await?;
-        }
-    }
-    Ok(())
-}
-
-#[tokio::test]
-async fn test_four_by_six_cold_start_at_read_quorum() -> TestResult {
-    crate::common::init_logging();
-    let mut cluster = RustFSTestClusterEnvironment::with_topology(ClusterTopology::single_pool_multidrive(4, 6)).await?;
-    // Explicit URLs are consumed in order. Place one drive from each node in
-    // every set, matching a four-host, six-drive-per-host deployment.
-    let volumes = (0..6)
-        .flat_map(|drive| {
-            cluster
-                .nodes
-                .iter()
-                .map(move |node| format!("http://{}{}", node.address, node.data_dirs[drive]))
-        })
-        .collect::<Vec<_>>()
-        .join(" ");
-    cluster.set_env("RUSTFS_VOLUMES", volumes);
-    cluster.set_env("RUSTFS_ERASURE_SET_DRIVE_COUNT", "4");
-    cluster.set_env("RUSTFS_STORAGE_CLASS_STANDARD", "EC:2");
-    cluster.set_env("RUSTFS_PUT_RENAME_EARLY_ACK_ENABLE", "false");
-    cluster.set_env("RUSTFS_OBS_METRICS_EXPORT_ENABLED", "false");
-    cluster.set_env("RUST_LOG", "error");
-    cluster.start().await?;
-    let clients = cluster
-        .create_all_clients()?
-        .into_iter()
-        .map(|client| {
-            Client::from_conf(
-                client
-                    .config()
-                    .to_builder()
-                    .retry_config(aws_sdk_s3::config::retry::RetryConfig::standard().with_max_attempts(1))
-                    .build(),
-            )
-        })
-        .collect::<Vec<_>>();
-    let bucket = "cold-start-read-quorum";
-    clients[0].create_bucket().bucket(bucket).send().await?;
-    let mut fixtures = Vec::new();
-    let mut covered = [[false; 2]; 6];
-    for index in 0..128 {
-        let key = match index {
-            0 => "warm-small".to_string(),
-            1 => "warm-large".to_string(),
-            _ => format!("cold-{index}"),
-        };
-        let size_class = index % 2;
-        let size = if size_class == 0 { 71 } else { 256 * 1024 };
-        let body = (0..size)
-            .map(|offset| u8::try_from(offset % 251).expect("bounded fixture byte"))
-            .collect::<Vec<_>>();
-        clients[0]
-            .put_object()
-            .bucket(bucket)
-            .key(&key)
-            .body(Bytes::copy_from_slice(&body).into())
-            .send()
-            .await?;
-        for node in &cluster.nodes {
-            let mut complete_shards = 0;
-            for (drive, path) in node.data_dirs.iter().enumerate() {
-                let census = crate::chaos::census_object_version_on_disk(std::path::Path::new(path), bucket, &key, None)?;
-                if census.has_xl_meta {
-                    assert!(census.is_complete(), "seed shard must be complete before shutdown: {census:?}");
-                    assert_eq!((census.data_blocks, census.parity_blocks), (Some(2), Some(2)));
-                    covered[drive][size_class] = true;
-                    complete_shards += 1;
-                }
-            }
-            assert_eq!(complete_shards, 1, "each node must own one shard of every seed object");
-        }
-        fixtures.push((key, body));
-        if covered.iter().flatten().all(|covered| *covered) {
-            break;
-        }
-    }
-    assert!(
-        covered.iter().flatten().all(|covered| *covered),
-        "fixtures must cover all six sets with inline and large objects"
-    );
-
-    for first in 0..4 {
-        for second in first + 1..4 {
-            for node in 0..4 {
-                cluster.stop_node(node)?;
-            }
-            cluster.start_node_process(first).await?;
-            let error = clients[first]
-                .get_object()
-                .bucket(bucket)
-                .key("warm-large")
-                .send()
-                .await
-                .expect_err("one node cannot satisfy read quorum after a cold start");
-            assert_eq!(error.raw_response().map(|response| response.status().as_u16()), Some(503));
-            cluster.start_node_process(second).await?;
-            wait_for_quorum_read_admission(&[clients[first].clone(), clients[second].clone()], bucket).await?;
-            for node in [first, second] {
-                for (key, body) in &fixtures {
-                    assert_quorum_object_body(&clients[node], bucket, key, body).await?;
-                }
-                let range = clients[node]
-                    .get_object()
-                    .bucket(bucket)
-                    .key("warm-large")
-                    .range("bytes=17-1023")
-                    .send()
-                    .await?;
-                assert_eq!(range.body.collect().await?.into_bytes().as_ref(), &fixtures[1].1[17..1024]);
-                let listing = clients[node].list_objects_v2().bucket(bucket).send().await?;
-                for (key, _) in &fixtures {
-                    assert!(
-                        listing.contents().iter().any(|object| object.key() == Some(key.as_str())),
-                        "listing omitted {key}"
-                    );
-                }
-                let error = clients[node]
-                    .put_object()
-                    .bucket(bucket)
-                    .key("recovery-write")
-                    .body(Bytes::from_static(b"after recovery").into())
-                    .send()
-                    .await
-                    .expect_err("two nodes must not authorize a write");
-                assert_eq!(error.raw_response().map(|response| response.status().as_u16()), Some(503));
-            }
-            let error = clients[first]
-                .delete_object()
-                .bucket(bucket)
-                .key("warm-large")
-                .send()
-                .await
-                .expect_err("two nodes must not authorize a delete");
-            assert_eq!(error.raw_response().map(|response| response.status().as_u16()), Some(503));
-            assert_quorum_object_body(&clients[first], bucket, "warm-large", &fixtures[1].1).await?;
-
-            let remaining = (0..4).filter(|node| *node != first && *node != second).collect::<Vec<_>>();
-            cluster.start_node(remaining[0]).await?;
-            let deadline = tokio::time::Instant::now() + Duration::from_secs(90);
-            loop {
-                match clients[first]
-                    .put_object()
-                    .bucket(bucket)
-                    .key("recovery-write")
-                    .body(Bytes::from_static(b"after recovery").into())
-                    .send()
-                    .await
-                {
-                    Ok(_) => break,
-                    Err(error)
-                        if error.raw_response().is_some_and(|response| response.status().as_u16() == 503)
-                            && tokio::time::Instant::now() < deadline =>
-                    {
-                        tokio::time::sleep(Duration::from_millis(200)).await
-                    }
-                    Err(error) => return Err(error.into()),
-                }
-            }
-            for node in [first, second, remaining[0]] {
-                assert_quorum_object_body(&clients[node], bucket, "recovery-write", b"after recovery").await?;
-            }
-            cluster.start_node(remaining[1]).await?;
-            for client in &clients {
-                for (key, body) in &fixtures {
-                    assert_quorum_object_body(client, bucket, key, body).await?;
-                }
-                assert_quorum_object_body(client, bucket, "recovery-write", b"after recovery").await?;
-            }
-        }
-    }
-    Ok(())
-}
-
 async fn put_object(client: Client, payload: Vec<u8>, writer_id: usize) -> Result<(), String> {
     client
         .put_object()

crates/e2e_test/src/reliant/grpc_lock_server.rs

@@ -465,13 +465,6 @@ impl NodeService for MinimalLockNodeService {
         Err(Status::unimplemented("lock-only test server"))
     }
 
-    async fn recover_delete_transaction(
-        &self,
-        _request: Request<rustfs_protos::proto_gen::node_service::RecoverDeleteTransactionRequest>,
-    ) -> Result<Response<rustfs_protos::proto_gen::node_service::RecoverDeleteTransactionResponse>, Status> {
-        Err(Status::unimplemented("delete recovery is not supported by the lock fixture"))
-    }
-
     async fn settle_part_transaction(
         &self,
         _request: Request<rustfs_protos::proto_gen::node_service::SettlePartTransactionRequest>,

crates/e2e_test/src/replication_extension_test.rs

@@ -3945,15 +3945,7 @@ async fn test_bucket_replication_version_purge_of_non_inline_object_releases_sou
         (sse_key, sse_put.version_id().ok_or("SSE PUT omitted version ID")?.to_string()),
         (large_key, large_put.version_id().ok_or("large PUT omitted version ID")?.to_string()),
     ];
-    // Managed-SSE listings may omit ETags. Check the replicated version's body
-    // before deleting it rather than comparing keyless listing fingerprints.
-    for (key, version_id) in &purged {
-        let source_body = get_version_body(&source_client, &source_bucket, key, version_id).await?;
-        let expected_sha256: [u8; 32] = Sha256::digest(&source_body).into();
-        wait_for_replicated_sha256(&target_client, &target_bucket, key, expected_sha256).await?;
-        let target_body = get_version_body(&target_client, &target_bucket, key, version_id).await?;
-        assert_eq!(target_body, source_body, "replicated version body differs for {key}/{version_id}");
-    }
+    assert_replication_converged(&source_client, &source_bucket, &target_client, &target_bucket).await?;
 
     for (key, version_id) in &purged {
         source_client
@@ -3964,6 +3956,10 @@ async fn test_bucket_replication_version_purge_of_non_inline_object_releases_sou
             .send()
             .await?;
     }
+    assert_replication_converged(&source_client, &source_bucket, &target_client, &target_bucket).await?;
+    let target_state = list_replication_state(&target_client, &target_bucket).await?;
+    assert!(target_state.is_empty(), "target retained an explicitly purged version: {target_state:?}");
+
     // The purge state is applied on the source asynchronously after the target
     // acknowledges the delete; only then does the retained version go away and
     // the bucket become deletable. A listing that is empty while DeleteBucket
@@ -3988,9 +3984,6 @@ async fn test_bucket_replication_version_purge_of_non_inline_object_releases_sou
         }
     }
 
-    let target_state = list_replication_state(&target_client, &target_bucket).await?;
-    assert!(target_state.is_empty(), "target retained an explicitly purged version: {target_state:?}");
-
     Ok(())
 }
 

crates/e2e_test/src/sts_query_compat_test.rs

@@ -740,7 +740,7 @@ async fn test_bucket_tags_opa_contract() -> TestResult {
         .version_id(version)
         .send()
         .await?;
-    opa.expect_bucket_tags("s3:GetObjectVersion", source, Some("finance")).await?;
+    opa.expect_bucket_tags("s3:GetObject", source, Some("finance")).await?;
     client.list_objects_v2().bucket(source).send().await?;
     opa.expect_bucket_tags("s3:ListBucket", source, Some("finance")).await?;
     client.list_object_versions().bucket(source).send().await?;

crates/e2e_test/src/upgrade_compatibility_test.rs

@@ -2345,7 +2345,7 @@ fn layout_reported_parts(attributes: &aws_sdk_s3::operation::get_object_attribut
     attributes.object_parts().map(|parts| parts.parts().len())
 }
 
-async fn assert_layout_full_body(client: &Client, case: &LayoutCase, context: &str) -> TestResult {
+async fn assert_layout_readable(client: &Client, case: &LayoutCase, context: &str) -> TestResult {
     let label = case.label();
     let head = layout_head(client, case).await?;
     assert_eq!(
@@ -2362,12 +2362,7 @@ async fn assert_layout_full_body(client: &Client, case: &LayoutCase, context: &s
     let full = layout_get(client, case, None, None).await?.body.collect().await?.into_bytes();
     assert_eq!(full.len(), case.body.len(), "{context}: {label}: full GET length");
     assert!(full == case.body, "{context}: {label}: full GET body must equal the rc.5 upload");
-    Ok(())
-}
 
-async fn assert_layout_readable(client: &Client, case: &LayoutCase, context: &str) -> TestResult {
-    assert_layout_full_body(client, case, context).await?;
-    let label = case.label();
     if case.is_multipart_layout() {
         let first = case.part_sizes[0];
         let range = format!("bytes={}-{}", first - 32, first + 31);
@@ -2646,11 +2641,10 @@ async fn direct_upgrade_from_rc5_preserves_multipart_layouts() -> TestResult {
             "rc.5 wrote a legacy layout"
         );
     }
-    // Prove the rc.5 writer persisted every original byte before upgrading.
-    // Its compressed cross-part range reader has a legacy limitation; the
-    // upgraded reader must still satisfy every range and partNumber assertion.
+    // The rc.5 writer must itself still read what it wrote, so a later
+    // failure is attributable to the upgrade rather than to the fixture.
     for case in &cases {
-        assert_layout_full_body(&old_client, case, "rc.5").await?;
+        assert_layout_readable(&old_client, case, "rc.5").await?;
     }
 
     // Upgrade in place.

crates/ecstore/Cargo.toml

@@ -31,6 +31,7 @@ workspace = true
 
 [features]
 default = []
+diagnostic-read-version-caller = ["rustfs-io-metrics/diagnostic-read-version-caller"]
 gcs = ["dep:google-cloud-storage", "dep:google-cloud-auth"]
 # Compiles the controlled list-objects namespace-journal chaos injector into a
 # production binary (it is always available to tests). Off by default so the
@@ -183,7 +184,6 @@ sha2 = { workspace = true }
 hex-simd = { workspace = true }
 tempfile.workspace = true
 hyper = { workspace = true, features = ["http2", "http1", "server"] }
-h2 = { workspace = true }
 hyper-util = { workspace = true, features = ["tokio", "server-auto", "server-graceful", "tracing"] }
 hyper-rustls = { workspace = true, default-features = false, features = ["native-tokio", "http1", "tls12", "logging", "http2", "aws-lc-rs"] }
 hostname.workspace = true
@@ -249,6 +249,7 @@ windows-sys = { workspace = true, features = ["Win32_System_Ioctl"] }
 
 [dev-dependencies]
 aws-smithy-async.workspace = true
+rustfs-s3-ops.workspace = true
 tokio = { workspace = true, features = ["rt-multi-thread", "macros", "test-util", "fs"] }
 criterion = { workspace = true, features = ["html_reports"] }
 temp-env = { workspace = true, features = ["async_closure"] }

crates/ecstore/src/api/mod.rs

@@ -14,6 +14,15 @@
 
 //! Explicit ECStore public facades for outer crate compatibility boundaries.
 
+#[cfg(feature = "diagnostic-read-version-caller")]
+pub mod diagnostic {
+    pub use crate::cluster::rpc::runtime_sources::{
+        current_read_version_entry_source, observe_read_version_context_boundary, with_read_version_entry,
+        with_read_version_spawn_context,
+    };
+    pub use rustfs_io_metrics::internode_metrics::{ReadVersionContextBoundary, ReadVersionEntrySource};
+}
+
 pub mod admin {
     pub use crate::diagnostics::admin_server_info::{get_local_server_property, get_server_info};
 }
@@ -199,10 +208,10 @@ pub mod bucket {
             get_object_lock_config, get_object_lock_config_state, get_on_demand_migration_config,
             get_on_demand_migration_config_in, get_public_access_block_config, get_quota_config, get_replication_config,
             get_request_payment_config, get_sse_config, get_tagging_config, get_versioning_config, get_website_config,
-            init_bucket_metadata_sys, list_bucket_targets, reload_bucket_metadata, remove_bucket_metadata,
-            remove_bucket_metadata_if_absent, set_bucket_metadata, update, update_bucket_targets_under_transaction_lock,
-            update_config_with, update_if_incarnation, update_if_incarnation_at, update_quota_if_incarnation,
-            update_quota_if_incarnation_at, update_under_transaction_lock, update_under_transaction_lock_at,
+            init_bucket_metadata_sys, list_bucket_targets, reload_bucket_metadata, remove_bucket_metadata, set_bucket_metadata,
+            update, update_bucket_targets_under_transaction_lock, update_config_with, update_if_incarnation,
+            update_if_incarnation_at, update_quota_if_incarnation, update_quota_if_incarnation_at, update_under_transaction_lock,
+            update_under_transaction_lock_at,
         };
         #[cfg(feature = "test-util")]
         pub use crate::bucket::metadata_sys::{ConfigWriteLockProbe, test_support};
@@ -210,8 +219,7 @@ pub mod bucket {
 
     pub mod migration {
         pub use crate::bucket::migration::{
-            LegacyBlobDecryptFn, migration_startup_error, try_migrate_bucket_metadata, try_migrate_bucket_metadata_for_buckets,
-            try_migrate_iam_config,
+            LegacyBlobDecryptFn, migration_startup_error, try_migrate_bucket_metadata, try_migrate_iam_config,
         };
     }
 
@@ -423,9 +431,7 @@ pub mod disk {
     pub use crate::disk::os::{LocalPublicationPause, LocalPublicationStage};
     pub use crate::disk::{
         BATCH_READ_VERSION_MAX_ITEMS, BUCKET_META_PREFIX, BatchReadVersionItem, BatchReadVersionReq, BatchReadVersionResp,
-        CheckPartsResp, ConditionalFileUpdate, DELETE_RECOVERY_PROTOCOL_VERSION, DeleteOptions, DeleteRecoveryAction,
-        DeleteRecoveryCursor, DeleteRecoveryDigest, DeleteRecoveryInspection, DeleteRecoveryPage, DeleteRecoveryRpcOperation,
-        DeleteRecoveryRpcReply, DeleteRecoveryState, Disk, DiskAPI, DiskInfo, DiskInfoOptions, DiskLocation, DiskOption,
+        CheckPartsResp, ConditionalFileUpdate, DeleteOptions, Disk, DiskAPI, DiskInfo, DiskInfoOptions, DiskLocation, DiskOption,
         DiskStore, FileInfoVersions, FileReader, FileWriter, HEALING_MARKER_PATH, NsScannerOpenRequest, OldCurrentSize,
         PartTransactionAction, RUSTFS_META_BUCKET, ReadMultipleReq, ReadMultipleResp, ReadOptions, RenameDataResp,
         STORAGE_FORMAT_FILE, SnapshotLeaseToken, UpdateMetadataOpts, VolumeInfo, WalkDirOptions, new_disk,
@@ -522,14 +528,13 @@ pub mod integrity {
 pub mod object {
     pub use crate::object_api::{
         BLOCK_SIZE_V2, ERASURE_ALGORITHM, EncryptionResolutionError, EncryptionResolutionErrorKind, GetObjectBodyCacheHook,
-        GetObjectBodyCacheHookLookup, GetObjectBodySource, GetObjectReadCondition, GetObjectReadMetadata, GetObjectReader,
-        NamespaceLockFence, ObjectEncryptionResolver, ObjectInfo, ObjectLockConfigSnapshot, ObjectMutationHook, ObjectOptions,
-        PutObjReader, QuotaAdmission, RangedDecompressReader, ReadEncryptionMaterial, ReadEncryptionMode, ReadEncryptionRequest,
+        GetObjectBodyCacheHookLookup, GetObjectBodySource, GetObjectReader, NamespaceLockFence, ObjectEncryptionResolver,
+        ObjectInfo, ObjectLockConfigSnapshot, ObjectMutationHook, ObjectOptions, PutObjReader, QuotaAdmission,
+        RangedDecompressReader, ReadEncryptionMaterial, ReadEncryptionMode, ReadEncryptionRequest,
         SCANNER_PUBLICATION_LEASE_FENCE_METADATA_KEY, ScannerPublicationCommitScope, ScannerPublicationCommitStartError,
         ScannerPublicationCommitState, ShardIntegrityWriteMode, StreamConsumer, WriteCommitGuard, WriteCompletion,
         get_object_body_cache_plaintext_len, lookup_get_object_body_cache_hook, register_get_object_body_cache_hook,
         register_object_mutation_hook, unregister_get_object_body_cache_hook, unregister_object_mutation_hook,
-        with_get_object_read_condition,
     };
     pub use crate::store::{
         PrepareSelectObjectSnapshotError, PreparedGetObjectReader, SelectObjectSnapshot, SelectObjectSnapshotReadError,
@@ -594,7 +599,6 @@ pub mod set_disk {
     #[cfg(feature = "test-util")]
     pub mod test_util {
         pub use crate::bucket::quota::reservation::fail_next_quota_ledger_save_for_test;
-        pub use crate::set_disk::ConditionalReadBenchmarkMetadataGuard;
         pub use crate::set_disk::{MultipartCommitBarrier, MultipartCommitPause, PutObjectCommitBarrier, PutObjectCommitPause};
 
         /// Keep a namespace commit pending until the returned owner is dropped.

crates/ecstore/src/bucket/lifecycle/bucket_lifecycle_ops.rs

@@ -2417,16 +2417,6 @@ fn is_slow_down(err: &Error) -> bool {
     matches!(err, Error::SlowDown)
 }
 
-/// Lifecycle expiry is idempotent when the exact object or version has already
-/// disappeared between the scanner listing and the delete admission.
-///
-/// Keep this predicate narrower than bucket/volume "not found" helpers: an
-/// unavailable bucket or disk is an operational failure and must not be
-/// reported as a completed expiry.
-fn is_lifecycle_expiry_already_gone(err: &Error) -> bool {
-    is_err_object_not_found(err) || is_err_version_not_found(err)
-}
-
 /// Resolves the expiry worker count from the single documented knob,
 /// `RUSTFS_MAX_EXPIRY_WORKERS`: a set, parsable, non-zero value wins;
 /// anything else falls back to `min(cpus, 16)`. The historical
@@ -2461,7 +2451,13 @@ fn spawn_recovery_disposition_maintenance_once(api: Arc<ECStore>) -> Option<Join
         return None;
     }
     Some(tokio::spawn(async move {
-        run_recovery_disposition_maintenance_loop(api, cancel_token).await;
+        let maintenance = run_recovery_disposition_maintenance_loop(api, cancel_token);
+        #[cfg(feature = "diagnostic-read-version-caller")]
+        let maintenance = crate::cluster::rpc::runtime_sources::with_read_version_entry(
+            rustfs_io_metrics::internode_metrics::ReadVersionEntrySource::RecoveryDispositionMaintenance,
+            maintenance,
+        );
+        maintenance.await;
     }))
 }
 
@@ -3344,7 +3340,13 @@ fn spawn_tier_delete_journal_recovery_once(api: Arc<ECStore>) {
         return;
     }
     tokio::spawn(async move {
-        run_tier_delete_journal_recovery_loop(api, cancel_token).await;
+        let recovery = run_tier_delete_journal_recovery_loop(api, cancel_token);
+        #[cfg(feature = "diagnostic-read-version-caller")]
+        let recovery = crate::cluster::rpc::runtime_sources::with_read_version_entry(
+            rustfs_io_metrics::internode_metrics::ReadVersionEntrySource::LifecycleRecovery,
+            recovery,
+        );
+        recovery.await;
     });
 }
 
@@ -3363,7 +3365,13 @@ fn spawn_transition_transaction_recovery_once(api: Arc<ECStore>) {
         return;
     }
     tokio::spawn(async move {
-        run_transition_transaction_recovery_loop(api, cancel_token).await;
+        let recovery = run_transition_transaction_recovery_loop(api, cancel_token);
+        #[cfg(feature = "diagnostic-read-version-caller")]
+        let recovery = crate::cluster::rpc::runtime_sources::with_read_version_entry(
+            rustfs_io_metrics::internode_metrics::ReadVersionEntrySource::LifecycleRecovery,
+            recovery,
+        );
+        recovery.await;
     });
 }
 
@@ -4930,20 +4938,6 @@ async fn expire_transitioned_object_with_lock_lost_signal(
                 //audit_log_lifecycle(*oi, ILMExpiry, tags, traceFn);
                 Ok(dobj)
             }
-            Err(err) if is_lifecycle_expiry_already_gone(&err) => {
-                debug!(
-                    event = EVENT_LIFECYCLE_DELETE_COMPLETED,
-                    component = LOG_COMPONENT_ECSTORE,
-                    subsystem = LOG_SUBSYSTEM_LIFECYCLE,
-                    bucket = %oi.bucket,
-                    object = %oi.name,
-                    operation = "delete_restored_object",
-                    reason = "already_gone",
-                    "Lifecycle delete target was already absent"
-                );
-                crate::object_api::notify_object_mutation(&oi.bucket, &oi.name).await;
-                Ok(oi.clone())
-            }
             Err(err) => Err(std::io::Error::other(err)),
         };
     }
@@ -4954,8 +4948,6 @@ async fn expire_transitioned_object_with_lock_lost_signal(
     // pointing to a remote tier version that has already been removed. If this
     // only creates a delete marker, remote cleanup must be driven by persisted
     // free-version recovery rather than the visible delete result.
-    // Keep remote transitioned not-found errors visible: local metadata
-    // absence alone does not prove that the tier object was cleaned up.
     let dobj = match api.delete_object(&oi.bucket, &oi.name, opts).await {
         Ok(obj) => obj,
         Err(e) => {
@@ -5828,20 +5820,6 @@ async fn apply_expiry_on_non_transitioned_objects_with_lock_lost_signal(
         .await
     {
         Ok(dobj) => dobj,
-        Err(err) if is_lifecycle_expiry_already_gone(&err) => {
-            debug!(
-                event = EVENT_LIFECYCLE_DELETE_COMPLETED,
-                component = LOG_COMPONENT_ECSTORE,
-                subsystem = LOG_SUBSYSTEM_LIFECYCLE,
-                bucket = %oi.bucket,
-                object = %oi.name,
-                operation = "delete_object",
-                reason = "already_gone",
-                "Lifecycle delete target was already absent"
-            );
-            crate::object_api::notify_object_mutation(&oi.bucket, &oi.name).await;
-            return true;
-        }
         Err(e) => {
             error!(
                 event = EVENT_LIFECYCLE_DELETE_FAILED,
@@ -6081,19 +6059,18 @@ mod tests {
         cleanup_empty_multipart_sha_dirs_on_local_disks, cleanup_stale_multipart_uploads_once_at,
         enqueue_recovered_free_version_with_state, enqueue_transition_for_existing_objects_scoped,
         enqueue_transition_with_lifecycle, enqueue_transition_with_lifecycle_report, eval_action_from_lifecycle,
-        get_lock_acquire_timeout, is_lifecycle_expiry_already_gone, jitter_tier_free_version_recovery_delay,
-        lifecycle_action_blocked_by_replication, lifecycle_delete_all_versions_replication_scan, lifecycle_deleted_object,
-        lifecycle_replication_blocks_action, lifecycle_rule_has_date_expiration, lifecycle_version_delete_target,
-        manual_transition_duration_elapsed, manual_transition_has_more_after_limit, manual_transition_recovery_progress_sink,
-        manual_transition_version_marker, manual_transition_worker_failure_reason,
-        mark_delete_opts_skip_decommissioned_on_remote_success, merge_stale_multipart_candidate,
-        persist_manual_transition_job_progress_if_owned, persist_manual_transition_page_checkpoint,
-        recover_manual_transition_job, recover_manual_transition_jobs, resolve_tier_free_version_recovery_enabled,
-        resolve_transition_queue_capacity, resolve_transition_queue_send_timeout, resolve_transition_worker_count,
-        resolve_transition_workers_absolute_max, run_tier_free_version_recovery_coordinator, run_tier_free_version_recovery_loop,
-        select_restore_s3_location, set_lifecycle_observability_observer, set_recovered_free_version_enqueue_observer,
-        should_defer_date_expiry_for_recent_config_update, transitioned_cleanup_tuple, transitioned_object_delete_opts,
-        wait_for_tier_free_version_recovery,
+        get_lock_acquire_timeout, jitter_tier_free_version_recovery_delay, lifecycle_action_blocked_by_replication,
+        lifecycle_delete_all_versions_replication_scan, lifecycle_deleted_object, lifecycle_replication_blocks_action,
+        lifecycle_rule_has_date_expiration, lifecycle_version_delete_target, manual_transition_duration_elapsed,
+        manual_transition_has_more_after_limit, manual_transition_recovery_progress_sink, manual_transition_version_marker,
+        manual_transition_worker_failure_reason, mark_delete_opts_skip_decommissioned_on_remote_success,
+        merge_stale_multipart_candidate, persist_manual_transition_job_progress_if_owned,
+        persist_manual_transition_page_checkpoint, recover_manual_transition_job, recover_manual_transition_jobs,
+        resolve_tier_free_version_recovery_enabled, resolve_transition_queue_capacity, resolve_transition_queue_send_timeout,
+        resolve_transition_worker_count, resolve_transition_workers_absolute_max, run_tier_free_version_recovery_coordinator,
+        run_tier_free_version_recovery_loop, select_restore_s3_location, set_lifecycle_observability_observer,
+        set_recovered_free_version_enqueue_observer, should_defer_date_expiry_for_recent_config_update,
+        transitioned_cleanup_tuple, transitioned_object_delete_opts, wait_for_tier_free_version_recovery,
     };
     #[cfg(feature = "test-util")]
     use super::{delete_free_version_remote_object_then, encode_dir_object, get_transitioned_object_reader_with_tier_manager};
@@ -10552,26 +10529,6 @@ mod tests {
         );
     }
 
-    #[test]
-    fn lifecycle_expiry_already_gone_only_accepts_object_or_version_absence() {
-        assert!(is_lifecycle_expiry_already_gone(&Error::FileNotFound));
-        assert!(is_lifecycle_expiry_already_gone(&Error::ObjectNotFound(
-            "bucket".to_string(),
-            "object".to_string(),
-        )));
-        assert!(is_lifecycle_expiry_already_gone(&Error::FileVersionNotFound));
-        assert!(is_lifecycle_expiry_already_gone(&Error::VersionNotFound(
-            "bucket".to_string(),
-            "object".to_string(),
-            "version".to_string(),
-        )));
-
-        assert!(!is_lifecycle_expiry_already_gone(&Error::VolumeNotFound));
-        assert!(!is_lifecycle_expiry_already_gone(&Error::DiskNotFound));
-        assert!(!is_lifecycle_expiry_already_gone(&Error::ErasureReadQuorum));
-        assert!(!is_lifecycle_expiry_already_gone(&Error::MethodNotAllowed));
-    }
-
     #[tokio::test]
     async fn manual_transition_counts_already_transitioned_object() {
         let lc = latest_transition_lifecycle();
@@ -13446,9 +13403,7 @@ mod tests {
                 .bucket_incarnation_id_from_disk(&bucket)
                 .await
                 .expect("bucket incarnation");
-            // Replaying the same scanner snapshot after the delete is an
-            // already-gone success and must not be reported as a failure.
-            for (attempt_incarnation, expected) in [(Uuid::new_v4(), false), (incarnation, true), (incarnation, true)] {
+            for (attempt_incarnation, expected) in [(Uuid::new_v4(), false), (incarnation, true), (incarnation, false)] {
                 assert_eq!(
                     super::apply_expiry_on_non_transitioned_objects(
                         ecstore.clone(),

crates/ecstore/src/bucket/metadata_sys.rs

@@ -321,33 +321,6 @@ pub async fn remove_bucket_metadata(bucket: &str) -> Result<bool> {
     Ok(lock.remove(bucket).await)
 }
 
-/// Remove a peer's cached metadata only when the bucket is absent under the
-/// namespace read fence. A delayed delete notification must not withdraw the
-/// metadata of a same-name bucket that has already been recreated.
-pub async fn remove_bucket_metadata_if_absent(api: Arc<ECStore>, bucket: &str) -> Result<bool> {
-    if is_meta_bucketname(bucket) {
-        return Ok(false);
-    }
-    // Lock order: bucket namespace read -> metadata system read (released)
-    // -> metadata publish, matching peer reloads.
-    let namespace_lock = api.new_ns_lock(bucket, bucket).await?;
-    let namespace_guard = namespace_lock
-        .get_read_lock(crate::set_disk::get_lock_acquire_timeout())
-        .await?;
-    let sys = bucket_metadata_sys_of(&api.ctx)?;
-    let sys = sys.read().await.clone();
-    if sys
-        .bucket_exists(bucket, &namespace_guard, "peer bucket metadata removal existence check")
-        .await?
-    {
-        return Ok(false);
-    }
-    await_bucket_namespace_operation(Some(&namespace_guard), bucket, "peer bucket metadata removal", async {
-        Ok(sys.remove(bucket).await)
-    })
-    .await
-}
-
 fn start_refresh_buckets_metadata_loop(sys: Arc<RwLock<BucketMetadataSys>>) {
     let Some(cancel_token) = runtime_sources::background_services_cancel_token() else {
         warn!("bucket metadata refresh loop skipped because background cancellation token is not initialized");
@@ -525,36 +498,6 @@ pub(crate) async fn get_bucket_incarnation_id_in(ctx: &crate::runtime::instance:
     sys.get_bucket_incarnation_id_from_disk(bucket).await
 }
 
-/// Directory loss is recoverable only for a committed, non-retired generation.
-/// The caller retains the bucket lifecycle fence through the ensuing reconstruction.
-pub(crate) async fn validate_bucket_reconstruction_in(
-    ctx: &crate::runtime::instance::InstanceContext,
-    bucket: &str,
-    expected: Uuid,
-) -> Result<()> {
-    let api = bucket_metadata_sys_of(ctx)?.read().await.object_store();
-    let lock = api
-        .new_ns_lock(RUSTFS_META_BUCKET, &bucket_metadata_transaction_lock_key(bucket))
-        .await?;
-    let guard = lock.get_read_lock(crate::set_disk::get_lock_acquire_timeout()).await?;
-    let (metadata, persisted) = load_bucket_metadata_parse_with_presence(api.clone(), bucket, true).await?;
-    if !persisted
-        || !metadata.bucket_incarnation_sidecar
-        || expected.is_nil()
-        || metadata.bucket_incarnation_id != expected
-        || !metadata.bucket_creation_committed
-    {
-        return Err(Error::PreconditionFailed);
-    }
-    if crate::bucket::retirement::is_retired(api, bucket, expected).await? {
-        return Err(Error::BucketNotFound(bucket.to_owned()));
-    }
-    if guard.is_lock_lost() {
-        return Err(Error::PreconditionFailed);
-    }
-    Ok(())
-}
-
 /// Validate against disk while retaining an already-held Object Lock fence.
 pub(crate) async fn get_bucket_incarnation_id_for_options_in(
     ctx: &crate::runtime::instance::InstanceContext,
@@ -4990,117 +4933,6 @@ mod tests {
         .await;
     }
 
-    #[tokio::test]
-    #[serial]
-    async fn peer_reload_clears_negative_cache_after_remote_bucket_creation() {
-        let (dirs, node_a, node_b) = two_nodes_over_one_store().await;
-        let bucket = "cross-node-negative-cache-refresh";
-        let node_b_sys = node_b.read().await.clone();
-
-        assert_eq!(
-            node_b_sys.get_bucket_incarnation_id(bucket).await.unwrap_err(),
-            Error::BucketNotFound(bucket.to_string()),
-            "the probe must populate node B's missing-bucket cache"
-        );
-
-        for dir in &dirs {
-            std::fs::create_dir_all(dir.path().join(bucket)).expect("physical bucket should exist before creation metadata");
-        }
-        let created = BucketMetadata::new(bucket);
-        let created_incarnation = created.bucket_incarnation_id;
-        node_a
-            .read()
-            .await
-            .persist_new_and_set(created)
-            .await
-            .expect("node A should publish the created bucket metadata");
-
-        node_b_sys
-            .reload_from_store(bucket)
-            .await
-            .expect("peer reload should clear the stale negative cache");
-        assert_eq!(
-            node_b_sys.get_bucket_incarnation_id(bucket).await.unwrap(),
-            created_incarnation,
-            "node B should resolve the newly created bucket immediately"
-        );
-    }
-
-    #[tokio::test(flavor = "multi_thread", worker_threads = 4)]
-    #[serial]
-    async fn peer_delete_clears_metadata_for_absent_bucket() {
-        let (_dirs, store) = isolated_store_over_temp_disks().await;
-        init_bucket_metadata_sys(store.clone(), Vec::new()).await;
-        let sys = bucket_metadata_sys_of(&store.ctx).expect("metadata owner");
-        let bucket = "peer-delete-absent-bucket";
-
-        store
-            .make_bucket(bucket, &MakeBucketOptions::default())
-            .await
-            .expect("initial bucket creation");
-        let cached = sys.read().await.get(bucket).await.expect("initial cached metadata");
-        store
-            .delete_bucket(bucket, &DeleteBucketOptions::default())
-            .await
-            .expect("bucket deletion");
-
-        // Model a peer that still caches the metadata after the physical delete.
-        sys.read().await.set(bucket.to_string(), cached).await;
-        assert!(
-            remove_bucket_metadata_if_absent(store.clone(), bucket)
-                .await
-                .expect("conditional peer delete"),
-            "the peer must evict the deleted bucket's cached metadata"
-        );
-        let sys = sys.read().await;
-        assert!(matches!(sys.get(bucket).await, Err(Error::ConfigNotFound)));
-        assert_eq!(
-            sys.get_bucket_incarnation_id(bucket)
-                .await
-                .expect_err("deleted bucket lookup"),
-            Error::BucketNotFound(bucket.to_string()),
-        );
-    }
-
-    #[tokio::test(flavor = "multi_thread", worker_threads = 4)]
-    #[serial]
-    async fn delayed_peer_delete_preserves_recreated_bucket_metadata() {
-        let (_dirs, store) = isolated_store_over_temp_disks().await;
-        init_bucket_metadata_sys(store.clone(), Vec::new()).await;
-        let sys = bucket_metadata_sys_of(&store.ctx).expect("metadata owner");
-        let bucket = "delayed-peer-delete-recreated-bucket";
-
-        store
-            .make_bucket(bucket, &MakeBucketOptions::default())
-            .await
-            .expect("initial bucket creation");
-        let old_incarnation = store.bucket_incarnation_id_from_disk(bucket).await.expect("old incarnation");
-        store
-            .delete_bucket(bucket, &DeleteBucketOptions::default())
-            .await
-            .expect("bucket deletion");
-        store
-            .make_bucket(bucket, &MakeBucketOptions::default())
-            .await
-            .expect("bucket recreation");
-        let new_incarnation = store.bucket_incarnation_id_from_disk(bucket).await.expect("new incarnation");
-        assert_ne!(old_incarnation, new_incarnation);
-
-        let removed = remove_bucket_metadata_if_absent(store.clone(), bucket)
-            .await
-            .expect("conditional peer delete");
-        assert!(!removed, "a delayed delete must not remove metadata for a live recreated bucket");
-        assert_eq!(
-            sys.read()
-                .await
-                .get(bucket)
-                .await
-                .expect("recreated metadata should remain cached")
-                .bucket_incarnation_id,
-            new_incarnation,
-        );
-    }
-
     #[tokio::test(flavor = "multi_thread", worker_threads = 4)]
     #[serial]
     async fn stale_config_request_cannot_mutate_a_recreated_bucket() {

crates/ecstore/src/bucket/migration.rs

@@ -273,30 +273,6 @@ where
 
     let buckets: Vec<String> = buckets_list.into_iter().map(|b| b.name).collect();
 
-    try_migrate_bucket_metadata_for_buckets(store, &buckets).await
-}
-
-/// Migrate the already discovered startup candidates without repeating a
-/// majority-only namespace listing. Every migration write retains its quorum.
-pub async fn try_migrate_bucket_metadata_for_buckets<S>(store: Arc<S>, buckets: &[String]) -> Result<()>
-where
-    S: ObjectIO<
-            Error = Error,
-            RangeSpec = HTTPRangeSpec,
-            HeaderMap = HeaderMap,
-            ObjectOptions = ObjectOptions,
-            ObjectInfo = ObjectInfo,
-            GetObjectReader = GetObjectReader,
-            PutObjectReader = PutObjReader,
-        > + ObjectOperations<
-            Error = Error,
-            ObjectInfo = ObjectInfo,
-            ObjectOptions = ObjectOptions,
-            FileInfo = FileInfo,
-            ObjectToDelete = ObjectToDelete,
-            DeletedObject = DeletedObject,
-        >,
-{
     if buckets.is_empty() {
         debug!("No migrating bucket metadata found");
         return Ok(());

crates/ecstore/src/bucket/replication/replication_resyncer.rs

@@ -65,9 +65,9 @@ use super::replication_target_boundary::{
     SsecPassthroughGate, TargetClient, VersionIdentityCapability, is_replication_target_offline_error,
     replication_action_for_target_head, replication_complete_multipart_options, replication_delete_marker_purge_remove_options,
     replication_delete_remove_options, replication_force_delete_remove_options, replication_object_is_ssec_encrypted,
-    replication_put_object_header_size, replication_put_object_options, replication_source_etag_is_comparable,
-    replication_target_head_is_newer_null_version, resolve_read_api_version_id, ssec_passthrough_evidence_present,
-    ssec_passthrough_gate, version_identity_capability_from_put, version_identity_drifted,
+    replication_put_object_header_size, replication_put_object_options, replication_target_head_is_newer_null_version,
+    resolve_read_api_version_id, ssec_passthrough_evidence_present, ssec_passthrough_gate, version_identity_capability_from_put,
+    version_identity_drifted,
 };
 use super::replication_versioning_boundary::ReplicationVersioningStore;
 use super::runtime_boundary as runtime_sources;
@@ -349,22 +349,15 @@ fn replica_etag_verification_enabled() -> bool {
 /// target that stores a transformed payload (e.g. undecoded `aws-chunked`
 /// frames, #6853) returns the ETag of what it actually wrote. Reporting
 /// COMPLETED over such a replica is silent corruption, so a decidable
-/// mismatch fails the replication instead. Encrypted sources are exempt: an
-/// SSE-C ciphertext passthrough sends ciphertext while the source ETag is the
-/// plaintext MD5 (that path has its own HEAD-back audit), and a managed-SSE
-/// (SSE-S3 / SSE-KMS) source stores a keyed fingerprint, or a legacy plaintext
-/// MD5, that the ETag of the target's re-encrypted replica can neither confirm
-/// nor refute. The checksum header on the replication PUT stays the integrity
-/// check for those objects.
+/// mismatch fails the replication instead. An SSE-C ciphertext passthrough
+/// transfer is exempt: the wire bytes are ciphertext while the source ETag is
+/// the plaintext MD5, and that path has its own HEAD-back audit.
 fn verify_single_part_replica(
     object_info: &ObjectInfo,
     response: &RemotePutObjectResponse,
     ciphertext_passthrough: bool,
 ) -> std::result::Result<(), std::io::Error> {
-    if ciphertext_passthrough
-        || !replication_source_etag_is_comparable(&object_info.user_defined)
-        || !replica_etag_verification_enabled()
-    {
+    if ciphertext_passthrough || !replica_etag_verification_enabled() {
         return Ok(());
     }
     if single_part_replica_etag_mismatch(object_info.etag.as_deref(), response.etag.as_deref()) {
@@ -455,26 +448,6 @@ fn is_version_id_format_mismatch(err: &SdkError<HeadObjectError>) -> bool {
     is_version_id_mismatch(code, raw_status)
 }
 
-/// A version-addressed HEAD rejected because the target cannot address the
-/// source version id (AWS S3: bodiless 400). Narrower than
-/// [`is_version_id_format_mismatch`]: a bodiless 403 may be a genuine
-/// `s3:GetObjectVersion` denial, so it is not accepted where a mutation or a
-/// completion verdict follows instead of another non-destructive HEAD.
-fn rejects_source_version_id(err: &SdkError<HeadObjectError>) -> bool {
-    is_version_id_format_mismatch(err) && !has_raw_status(err, 403)
-}
-
-/// Whether the target's latest version of `object` is a delete marker. For
-/// targets that mint their own version ids, which cannot be asked about the
-/// source marker id: S3 answers a versionless HEAD on such a key with 404
-/// (405 on some targets) plus `x-amz-delete-marker: true`.
-async fn latest_target_version_is_delete_marker(tgt_client: &TargetClient, object: &str) -> bool {
-    match head_object_for_worker(tgt_client, &tgt_client.bucket, object, None).await {
-        Ok(head) => head.delete_marker == Some(true),
-        Err(e) => raw_header_value(e.as_ref(), "x-amz-delete-marker").as_deref() == Some("true"),
-    }
-}
-
 async fn mark_replication_target_offline_if_needed(target_client: &Arc<TargetClient>, err: &(impl Display + ?Sized)) {
     if is_replication_target_offline_error(err) {
         ReplicationTargetStore::mark_target_offline(target_client).await;
@@ -1873,17 +1846,7 @@ async fn verify_resync_head_result(
             // only absence proves completion. All other outcomes remain
             // failed (backlog#862 / #799 B13).
             let retryable = if roi.version_purge_status.is_empty() {
-                match classify_delete_marker_head_error(&err) {
-                    // A target that mints its own version ids never holds the
-                    // source marker id; verify the marker on the latest version.
-                    DeleteMarkerHeadOutcome::Ambiguous => {
-                        let mints_own = rejects_source_version_id(&err)
-                            || ReplicationTargetStore::version_identity_capability(&target_client.arn)
-                                .version_addressing_unreliable();
-                        !(mints_own && latest_target_version_is_delete_marker(target_client, &roi.name).await)
-                    }
-                    DeleteMarkerHeadOutcome::Exists | DeleteMarkerHeadOutcome::Missing => false,
-                }
+                matches!(classify_delete_marker_head_error(&err), DeleteMarkerHeadOutcome::Ambiguous)
             } else {
                 !matches!(classify_delete_marker_head_error(&err), DeleteMarkerHeadOutcome::Missing)
             };
@@ -3604,16 +3567,7 @@ async fn replicate_delete_to_target(
         }
     }
 
-    // A target that mints its own version ids never holds the source marker
-    // id, so a version-addressed pre-check cannot succeed there; the marker
-    // is created by the versionless DELETE below.
-    let version_addressable =
-        !ReplicationTargetStore::version_identity_capability(&tgt_client.arn).version_addressing_unreliable();
-    if version_addressable
-        && !is_version_purge
-        && dobj.delete_object.delete_marker
-        && dobj.delete_object.delete_marker_version_id.is_some()
-    {
+    if !is_version_purge && dobj.delete_object.delete_marker && dobj.delete_object.delete_marker_version_id.is_some() {
         match head_object_for_worker(
             tgt_client.as_ref(),
             &tgt_client.bucket,
@@ -3622,11 +3576,6 @@ async fn replicate_delete_to_target(
         )
         .await
         {
-            // The verdict is not known yet (e.g. no PUT since restart) and the
-            // target rejected the source id: learn it as the PUT path does.
-            Err(e) if rejects_source_version_id(e.as_ref()) => {
-                ReplicationTargetStore::record_version_identity_capability(&tgt_client.arn, VersionIdentityCapability::MintsOwn);
-            }
             Ok(head) if head.delete_marker == Some(true) => {
                 // Some targets expose an existing marker as a successful HEAD
                 // with the marker bit instead of the usual 405. Treat that as
@@ -5877,57 +5826,6 @@ mod tests {
     use super::super::replication_filemeta_boundary::ReplicateTargetDecision;
     use super::super::replication_object_decision_boundary::ReplicationMultipartPlanError;
 
-    #[test]
-    fn verify_single_part_replica_exempts_managed_sse_sources() {
-        // A new encrypted write stores a keyed fingerprint; a target that
-        // computes its own ETag answers with the MD5 of the plaintext it
-        // received. Both are 32 hex characters, so without the exemption the
-        // audit would fail every SSE-S3 / SSE-KMS replica (rustfs/backlog).
-        let fingerprint = "00112233445566778899aabbccddeeff";
-        let replica_md5 = "ffeeddccbbaa99887766554433221100";
-        let response = RemotePutObjectResponse {
-            version_id: None,
-            etag: Some(replica_md5.to_string()),
-        };
-        for sse in ["AES256", "aws:kms"] {
-            let object_info = ObjectInfo {
-                bucket: "source".to_string(),
-                name: "encrypted".to_string(),
-                etag: Some(fingerprint.to_string()),
-                user_defined: HashMap::from([("x-amz-server-side-encryption".to_string(), sse.to_string())]).into(),
-                ..Default::default()
-            };
-            verify_single_part_replica(&object_info, &response, false)
-                .unwrap_or_else(|err| panic!("{sse}: a managed-SSE source must not be audited against the replica MD5: {err}"));
-        }
-    }
-
-    #[test]
-    fn verify_single_part_replica_still_fails_plaintext_mismatch() {
-        if !replica_etag_verification_enabled() {
-            return;
-        }
-        let object_info = ObjectInfo {
-            bucket: "source".to_string(),
-            name: "plain".to_string(),
-            etag: Some("00112233445566778899aabbccddeeff".to_string()),
-            ..Default::default()
-        };
-        let mismatch = RemotePutObjectResponse {
-            version_id: None,
-            etag: Some("ffeeddccbbaa99887766554433221100".to_string()),
-        };
-        let err = verify_single_part_replica(&object_info, &mismatch, false)
-            .expect_err("a plaintext source whose replica MD5 differs must fail the replication");
-        assert_eq!(err.to_string(), REPLICA_ETAG_MISMATCH_ERROR);
-
-        let matching = RemotePutObjectResponse {
-            version_id: None,
-            etag: object_info.etag.clone(),
-        };
-        verify_single_part_replica(&object_info, &matching, false).expect("matching plaintext MD5s pass the audit");
-    }
-
     #[test]
     fn multipart_read_plan_preserves_legacy_plain_part_ranges() {
         const MIB: usize = 1024 * 1024;
@@ -6777,93 +6675,6 @@ mod tests {
         assert!(seen[0].starts_with("HEAD "), "{seen:?}");
     }
 
-    /// AWS S3 answers a HEAD for a version id it did not mint with a bodiless
-    /// 400. On a target with no verdict yet, that teaches `MintsOwn` and the
-    /// marker is still created with one versionless DELETE.
-    #[tokio::test]
-    async fn live_delete_marker_worker_learns_mints_own_from_bodiless_400() {
-        let (endpoint, server) = spawn_scripted_target_server(2, |line| {
-            if line.starts_with("HEAD ") {
-                empty_response("400 Bad Request")
-            } else if line.starts_with("DELETE ") {
-                empty_response("204 No Content")
-            } else {
-                empty_response("500 Unexpected")
-            }
-        });
-        let target = test_target_client(endpoint);
-        register_test_target(&target).await;
-        let dobj = marker_creation_dobj(&target.arn);
-
-        let rinfo = replicate_delete_to_target(&dobj, target.clone(), None).await;
-
-        assert_eq!(rinfo.replication_status, ReplicationStatusType::Completed, "{rinfo:?}");
-        assert_eq!(
-            ReplicationTargetStore::version_identity_capability(&target.arn),
-            VersionIdentityCapability::MintsOwn
-        );
-        let seen = server.join().expect("test HTTP server should finish");
-        assert_eq!(seen.len(), 2, "{seen:?}");
-        assert!(seen[0].starts_with("HEAD "), "{seen:?}");
-        assert!(seen[1].starts_with("DELETE "), "{seen:?}");
-        assert!(!seen[1].contains("versionId="), "marker creation must remain versionless: {seen:?}");
-    }
-
-    /// A bodiless 403 on the version-addressed HEAD may be a genuine
-    /// GetObjectVersion denial on an id-adopting target. It must fail closed:
-    /// no DELETE, and no `MintsOwn` verdict cached for the ARN.
-    #[tokio::test]
-    async fn live_delete_marker_worker_keeps_bodiless_403_failure_closed() {
-        let (endpoint, server) = spawn_scripted_target_server(1, |line| {
-            if line.starts_with("HEAD ") {
-                empty_response("403 Forbidden")
-            } else if line.starts_with("DELETE ") {
-                empty_response("204 No Content")
-            } else {
-                empty_response("500 Unexpected")
-            }
-        });
-        let target = test_target_client(endpoint);
-        register_test_target(&target).await;
-        let dobj = marker_creation_dobj(&target.arn);
-
-        let rinfo = replicate_delete_to_target(&dobj, target.clone(), None).await;
-
-        assert_eq!(rinfo.replication_status, ReplicationStatusType::Failed, "{rinfo:?}");
-        assert_ne!(
-            ReplicationTargetStore::version_identity_capability(&target.arn),
-            VersionIdentityCapability::MintsOwn
-        );
-        let seen = server.join().expect("test HTTP server should finish");
-        assert_eq!(seen.len(), 1, "a denied HEAD must not trigger DELETE: {seen:?}");
-        assert!(seen[0].starts_with("HEAD "), "{seen:?}");
-    }
-
-    /// A target already known to mint its own version ids is never asked for
-    /// the source marker id: the marker is created by the versionless DELETE.
-    #[tokio::test]
-    async fn live_delete_marker_worker_skips_version_head_on_mints_own_target() {
-        let (endpoint, server) = spawn_scripted_target_server(1, |line| {
-            if line.starts_with("DELETE ") {
-                empty_response("204 No Content")
-            } else {
-                empty_response("500 Unexpected")
-            }
-        });
-        let target = test_target_client(endpoint);
-        register_test_target(&target).await;
-        ReplicationTargetStore::record_version_identity_capability(&target.arn, VersionIdentityCapability::MintsOwn);
-        let dobj = marker_creation_dobj(&target.arn);
-
-        let rinfo = replicate_delete_to_target(&dobj, target.clone(), None).await;
-
-        assert_eq!(rinfo.replication_status, ReplicationStatusType::Completed, "{rinfo:?}");
-        let seen = server.join().expect("test HTTP server should finish");
-        assert_eq!(seen.len(), 1, "no version-addressed HEAD on a MintsOwn target: {seen:?}");
-        assert!(seen[0].starts_with("DELETE "), "{seen:?}");
-        assert!(!seen[0].contains("versionId="), "marker creation must remain versionless: {seen:?}");
-    }
-
     /// A target answers `HEAD ?versionId=<delete marker>` with a bodiless
     /// 405, so the SDK reports no error code. That is the marker having
     /// propagated, not a failure (rustfs/backlog#2479).
@@ -6954,76 +6765,6 @@ mod tests {
         server.join().expect("test HTTP server should finish");
     }
 
-    /// AWS answers `HEAD ?versionId=<source marker id>` with a bodiless 400.
-    /// The marker is verified through the latest version instead: a
-    /// versionless 404 with `x-amz-delete-marker: true` proves it propagated.
-    #[tokio::test]
-    async fn resync_verification_accepts_latest_delete_marker_when_target_rejects_source_id() {
-        let (endpoint, server) = spawn_scripted_target_server(2, |line| {
-            if line.contains("versionId=") {
-                empty_response("400 Bad Request")
-            } else {
-                "HTTP/1.1 404 Not Found\r\nX-Amz-Delete-Marker: true\r\nX-Amz-Version-Id: target-marker\r\nContent-Length: 0\r\nConnection: close\r\n\r\n"
-                    .to_string()
-            }
-        });
-        let target = test_target_client(endpoint);
-        let roi = delete_marker_roi();
-        let mut st = TargetReplicationResyncStatus::default();
-
-        let head_result =
-            head_object_for_worker(target.as_ref(), &target.bucket, &roi.name, roi.version_id.map(|v| v.to_string())).await;
-        let (size, err) = verify_resync_head_result(head_result, &roi, &mut st, &target).await;
-
-        assert!(err.is_none(), "{err:?}");
-        assert_eq!((size, st.replicated_count, st.failed_count), (0, 1, 0));
-        let seen = server.join().expect("test HTTP server should finish");
-        assert!(!seen[1].contains("versionId="), "the re-check must address the latest version: {seen:?}");
-    }
-
-    /// A live latest version on a target that rejects the source id means the
-    /// marker has not propagated: the verification stays failed.
-    #[tokio::test]
-    async fn resync_verification_fails_delete_marker_when_latest_target_version_is_live() {
-        let (endpoint, server) = spawn_scripted_target_server(2, |line| {
-            if line.contains("versionId=") {
-                empty_response("400 Bad Request")
-            } else {
-                empty_response("200 OK")
-            }
-        });
-        let target = test_target_client(endpoint);
-        let roi = delete_marker_roi();
-        let mut st = TargetReplicationResyncStatus::default();
-
-        let head_result =
-            head_object_for_worker(target.as_ref(), &target.bucket, &roi.name, roi.version_id.map(|v| v.to_string())).await;
-        let (size, err) = verify_resync_head_result(head_result, &roi, &mut st, &target).await;
-
-        assert!(err.is_some(), "a live latest version must not verify the marker");
-        assert_eq!((size, st.replicated_count, st.failed_count), (0, 0, 1));
-        server.join().expect("test HTTP server should finish");
-    }
-
-    /// A bodiless 403 may be a genuine permission denial: it is not evidence
-    /// that the target mints its own ids, so no latest-version re-check runs.
-    #[tokio::test]
-    async fn resync_verification_keeps_delete_marker_failed_on_bodiless_403() {
-        let (endpoint, server) = spawn_scripted_target_server(1, |_| empty_response("403 Forbidden"));
-        let target = test_target_client(endpoint);
-        let roi = delete_marker_roi();
-        let mut st = TargetReplicationResyncStatus::default();
-
-        let head_result =
-            head_object_for_worker(target.as_ref(), &target.bucket, &roi.name, roi.version_id.map(|v| v.to_string())).await;
-        let (size, err) = verify_resync_head_result(head_result, &roi, &mut st, &target).await;
-
-        assert!(err.is_some(), "a 403 must not count as a propagated delete marker");
-        assert_eq!((size, st.replicated_count, st.failed_count), (0, 0, 1));
-        let seen = server.join().expect("test HTTP server should finish");
-        assert_eq!(seen.len(), 1, "{seen:?}");
-    }
-
     /// Serves exactly `requests` connections, answering each from
     /// `respond(request_line)`, and returns the request lines it saw. Reads
     /// the whole request (headers plus `Content-Length` body, honoring

crates/ecstore/src/bucket/replication/replication_target_boundary.rs

@@ -157,19 +157,6 @@ pub(crate) fn replication_object_is_ssec_encrypted(user_defined: &HashMap<String
     rustfs_replication::is_ssec_encrypted(user_defined)
 }
 
-/// Whether the stored ETag of a replication source can be compared with the
-/// ETag a target returns for the bytes it received. Only an unencrypted source
-/// stores the MD5 of the bytes that go on the wire. A managed-SSE (SSE-S3 /
-/// SSE-KMS) or SSE-C source stores a keyed fingerprint (new writes) or a legacy
-/// plaintext MD5, and the target re-encrypts what it receives, so neither
-/// equality nor inequality of the two ETags proves anything about the replica.
-pub(crate) fn replication_source_etag_is_comparable(user_defined: &HashMap<String, String>) -> bool {
-    matches!(
-        classify_replication_source_encryption(user_defined),
-        ReplicationSourceEncryption::Plaintext
-    )
-}
-
 /// HeadObjectOutput adapter over the pure SSE-C passthrough evidence
 /// judgment owned by `rustfs-replication`: extract the echoed
 /// customer-algorithm header and let the crate-owned policy decide.
@@ -591,22 +578,6 @@ mod tests {
     use time::Duration;
     use uuid::Uuid;
 
-    #[test]
-    fn replication_source_etag_is_comparable_only_for_plaintext_sources() {
-        assert!(replication_source_etag_is_comparable(&HashMap::new()));
-        for (key, value) in [
-            (metadata_keys::SERVER_SIDE_ENCRYPTION, "AES256"),
-            (metadata_keys::SERVER_SIDE_ENCRYPTION, "aws:kms"),
-            (SSEC_ALGORITHM_HEADER, "AES256"),
-        ] {
-            let metadata = HashMap::from([(key.to_string(), value.to_string())]);
-            assert!(
-                !replication_source_etag_is_comparable(&metadata),
-                "{key}={value} must not expose its stored ETag to the replica audit"
-            );
-        }
-    }
-
     /// backlog#1735 A3b SSE key-case finding. Every RustFS writer persists the
     /// SSE intent as lowercase `x-amz-server-side-encryption`
     /// (`encryption_material_to_metadata`; header-derived keys come from a

crates/ecstore/src/cluster/rpc/client.rs

@@ -24,11 +24,10 @@ use crate::cluster::rpc::{gen_tonic_replay_scope_headers, gen_tonic_signature_he
 #[cfg(test)]
 use crate::cluster::rpc::{tonic_boot_epoch_challenge, tonic_boot_epoch_response_headers};
 use crate::disk::error::{DiskError, Error as DiskErrorType, RpcStatusError};
-#[cfg(test)]
 use crate::runtime::sources as runtime_sources;
 use http::{Request as HttpRequest, Response as HttpResponse, Uri};
 use rustfs_protos::{
-    ChannelClass, get_channel_for_class,
+    ChannelClass, create_new_channel, get_channel_for_class,
     proto_gen::node_service::{
         heal_control_service_client::HealControlServiceClient, node_service_client::NodeServiceClient,
         scanner_control_service_client::ScannerControlServiceClient,
@@ -54,7 +53,7 @@ use super::context_propagation::{inject_request_id_into_metadata, inject_trace_c
 /// 3. Subsequent calls will attempt fresh connections
 /// 4. If node is still down, connection will fail fast (3s timeout)
 pub async fn node_service_time_out_client(
-    addr: &str,
+    addr: &String,
     interceptor: TonicInterceptor,
 ) -> Result<NodeServiceClient<InterceptedService<AuthenticatedChannel, TonicInterceptor>>, Box<dyn Error>> {
     // Default to the latency-sensitive control channel; bulk `bytes` RPCs opt in via the
@@ -67,9 +66,12 @@ pub(crate) async fn scanner_control_time_out_client(
     interceptor: TonicInterceptor,
 ) -> crate::error::Result<ScannerControlServiceClient<InterceptedService<AuthenticatedChannel, TonicInterceptor>>> {
     let interceptor = interceptor.with_rpc_audience(addr)?;
-    let channel = get_channel_for_class(addr, ChannelClass::Control)
-        .await
-        .map_err(|err| crate::error::Error::other(err.to_string()))?;
+    let channel = match runtime_sources::cached_node_channel(addr).await {
+        Some(channel) => channel,
+        None => create_new_channel(addr)
+            .await
+            .map_err(|err| crate::error::Error::other(err.to_string()))?,
+    };
     let channel = ReplayScopeChannel::new(channel, interceptor.replay_scope_audience());
     let limit = rustfs_protos::scoped_dirty_usage::SCOPED_DIRTY_USAGE_MAX_REQUEST_BYTES as usize;
     Ok(ScannerControlServiceClient::with_interceptor(channel, interceptor)
@@ -82,7 +84,10 @@ pub async fn heal_control_time_out_client(
     interceptor: TonicInterceptor,
 ) -> Result<HealControlServiceClient<InterceptedService<AuthenticatedChannel, TonicInterceptor>>, Box<dyn Error>> {
     let interceptor = interceptor.with_rpc_audience(addr)?;
-    let channel = get_channel_for_class(addr, ChannelClass::Control).await?;
+    let channel = match runtime_sources::cached_node_channel(addr).await {
+        Some(channel) => channel,
+        None => create_new_channel(addr).await?,
+    };
     let max_message_size = rustfs_protos::HEAL_CONTROL_RPC_MAX_MESSAGE_SIZE;
     let channel = ReplayScopeChannel::new(channel, interceptor.replay_scope_audience());
     Ok(HealControlServiceClient::with_interceptor(channel, interceptor)
@@ -95,7 +100,10 @@ pub async fn tier_mutation_control_time_out_client(
     interceptor: TonicInterceptor,
 ) -> Result<TierMutationControlServiceClient<InterceptedService<AuthenticatedChannel, TonicInterceptor>>, Box<dyn Error>> {
     let interceptor = interceptor.with_rpc_audience(addr)?;
-    let channel = get_channel_for_class(addr, ChannelClass::Control).await?;
+    let channel = match runtime_sources::cached_node_channel(addr).await {
+        Some(channel) => channel,
+        None => create_new_channel(addr).await?,
+    };
     let max_message_size = rustfs_protos::TIER_MUTATION_RPC_MAX_MESSAGE_SIZE;
     let channel = ReplayScopeChannel::new(channel, interceptor.replay_scope_audience());
     Ok(TierMutationControlServiceClient::with_interceptor(channel, interceptor)
@@ -110,12 +118,22 @@ pub async fn tier_mutation_control_time_out_client(
 /// from lock/health RPCs; everything else uses [`ChannelClass::Control`]. When isolation is
 /// disabled the two classes resolve to the same cached channel, i.e. legacy behavior.
 pub async fn node_service_time_out_client_for_class(
-    addr: &str,
+    addr: &String,
     interceptor: TonicInterceptor,
     class: ChannelClass,
 ) -> Result<NodeServiceClient<InterceptedService<AuthenticatedChannel, TonicInterceptor>>, Box<dyn Error>> {
     let interceptor = interceptor.with_rpc_audience(addr)?;
-    let channel = get_channel_for_class(addr, class).await?;
+    let channel = match class {
+        ChannelClass::Control => match runtime_sources::cached_node_channel(addr).await {
+            Some(channel) => {
+                debug!("Using cached gRPC channel for: {}", addr);
+                channel
+            }
+            // No cached connection, create new one.
+            None => create_new_channel(addr).await?,
+        },
+        ChannelClass::Bulk => get_channel_for_class(addr, ChannelClass::Bulk).await?,
+    };
 
     let max_message_size = rustfs_protos::internode_rpc_max_message_size();
     let channel = ReplayScopeChannel::new(channel, interceptor.replay_scope_audience());
@@ -124,24 +142,8 @@ pub async fn node_service_time_out_client_for_class(
         .max_encoding_message_size(max_message_size))
 }
 
-pub(crate) async fn node_service_delete_recovery_client(
-    addr: &str,
-    interceptor: TonicInterceptor,
-    timeout: std::time::Duration,
-) -> Result<NodeServiceClient<InterceptedService<AuthenticatedChannel, TonicInterceptor>>, DiskError> {
-    let interceptor = interceptor.with_rpc_audience(addr).map_err(DiskError::from)?;
-    let channel = rustfs_protos::create_channel_with_timeout(addr, timeout)
-        .await
-        .map_err(|error| DiskError::RemoteClientUnavailable(error.to_string()))?;
-    let channel = ReplayScopeChannel::new(channel, interceptor.replay_scope_audience());
-    let max_message_size = rustfs_protos::internode_rpc_max_message_size();
-    Ok(NodeServiceClient::with_interceptor(channel, interceptor)
-        .max_decoding_message_size(max_message_size)
-        .max_encoding_message_size(max_message_size))
-}
-
 pub async fn node_service_time_out_client_no_auth(
-    addr: &str,
+    addr: &String,
 ) -> Result<NodeServiceClient<InterceptedService<AuthenticatedChannel, TonicInterceptor>>, Box<dyn Error>> {
     node_service_time_out_client(addr, TonicInterceptor::NoOp(NoOpInterceptor)).await
 }
@@ -580,126 +582,6 @@ mod tests {
     use tracing_opentelemetry::OpenTelemetrySpanExt;
     use tracing_subscriber::{Registry, layer::SubscriberExt};
 
-    #[tokio::test]
-    async fn delete_recovery_channel_uses_its_scoped_deadline() {
-        use bytes::Bytes;
-        use http_body_util::StreamBody;
-        use hyper::{body::Frame, service::service_fn};
-        use hyper_util::rt::{TokioExecutor, TokioIo};
-        use rustfs_protos::proto_gen::node_service::{RecoverDeleteTransactionRequest, RecoverDeleteTransactionResponse};
-        use tonic_prost::prost::Message;
-        let listener = tokio::net::TcpListener::bind("127.0.0.1:0")
-            .await
-            .expect("bind recovery peer");
-        let addr = format!("http://{}", listener.local_addr().expect("peer address"));
-        let server = tokio::spawn(async move {
-            loop {
-                let (stream, _) = listener.accept().await.expect("accept recovery connection");
-                tokio::spawn(async move {
-                    let service = service_fn(|_request: HttpRequest<hyper::body::Incoming>| async {
-                        tokio::time::sleep(std::time::Duration::from_millis(500)).await;
-                        let payload = RecoverDeleteTransactionResponse {
-                            success: true,
-                            protocol_version: 1,
-                            result: Bytes::new(),
-                            error: None,
-                        }
-                        .encode_to_vec();
-                        let mut data = vec![0];
-                        data.extend_from_slice(&u32::try_from(payload.len()).expect("small response").to_be_bytes());
-                        data.extend_from_slice(&payload);
-                        let mut trailers = http::HeaderMap::new();
-                        trailers.insert("grpc-status", "0".parse().expect("status"));
-                        let frames: Vec<Result<Frame<Bytes>, std::convert::Infallible>> =
-                            vec![Ok(Frame::data(data.into())), Ok(Frame::trailers(trailers))];
-                        let mut response = HttpResponse::new(StreamBody::new(futures::stream::iter(frames)));
-                        response
-                            .headers_mut()
-                            .insert("content-type", "application/grpc".parse().expect("content type"));
-                        Ok::<_, std::convert::Infallible>(response)
-                    });
-                    let _ = hyper::server::conn::http2::Builder::new(TokioExecutor::new())
-                        .serve_connection(TokioIo::new(stream), service)
-                        .await;
-                });
-            }
-        });
-        let request = || RecoverDeleteTransactionRequest {
-            disk: "disk".into(),
-            volume: "bucket".into(),
-            protocol_version: 1,
-            operation: Bytes::new(),
-        };
-        // Seed an existing short-lived control channel to reproduce the inherited deadline.
-        let control = tonic::transport::Endpoint::from_shared(addr.clone())
-            .expect("test endpoint")
-            .timeout(std::time::Duration::from_millis(50))
-            .connect()
-            .await
-            .expect("control connect");
-        rustfs_common::cache_connection(addr.clone(), control).await;
-        let mut foreground = node_service_time_out_client(&addr, TonicInterceptor::NoOp(NoOpInterceptor))
-            .await
-            .expect("control client");
-        let error = foreground
-            .recover_delete_transaction(request())
-            .await
-            .expect_err("short control deadline");
-        assert!(matches!(error.code(), tonic::Code::Cancelled | tonic::Code::DeadlineExceeded));
-        // The 2-second test override bypasses the production range parser, which rejects 2.
-        let mut recovery = node_service_delete_recovery_client(
-            &addr,
-            TonicInterceptor::NoOp(NoOpInterceptor),
-            std::time::Duration::from_secs(2),
-        )
-        .await
-        .expect("scoped recovery client");
-        assert!(
-            recovery
-                .recover_delete_transaction(request())
-                .await
-                .expect("scoped deadline must survive control deadline")
-                .into_inner()
-                .success
-        );
-        server.abort();
-        rustfs_protos::evict_failed_connection(&addr).await;
-    }
-
-    #[tokio::test]
-    async fn rpc_client_factories_do_not_reuse_https_channels_without_tls_generation() {
-        let listener = std::net::TcpListener::bind("127.0.0.1:0").expect("reserve local address");
-        let addr = format!("https://{}", listener.local_addr().expect("local address"));
-        drop(listener);
-        for factory in 0..4 {
-            let placeholder = tonic::transport::Endpoint::from_static("http://127.0.0.1:1").connect_lazy();
-            runtime_sources::cache_test_node_channel(addr.clone(), placeholder).await;
-            let interceptor = TonicInterceptor::NoOp(NoOpInterceptor);
-            let _result = match factory {
-                0 => scanner_control_time_out_client(&addr, interceptor)
-                    .await
-                    .map(|_| ())
-                    .map_err(|e| e.to_string()),
-                1 => heal_control_time_out_client(&addr, interceptor)
-                    .await
-                    .map(|_| ())
-                    .map_err(|e| e.to_string()),
-                2 => tier_mutation_control_time_out_client(&addr, interceptor)
-                    .await
-                    .map(|_| ())
-                    .map_err(|e| e.to_string()),
-                _ => node_service_time_out_client(&addr, interceptor)
-                    .await
-                    .map(|_| ())
-                    .map_err(|e| e.to_string()),
-            };
-            assert!(
-                !runtime_sources::test_node_channel_is_cached(&addr).await,
-                "factory {factory} must evict an unbound HTTPS cache entry before attempting a fresh dial"
-            );
-        }
-    }
-
     #[derive(Clone)]
     struct EpochProofService {
         audience: String,

crates/ecstore/src/cluster/rpc/network_probe.rs

@@ -289,9 +289,13 @@ async fn client(
     >,
     NetworkPeerProbeError,
 > {
-    node_service_time_out_client_for_class(address, TonicInterceptor::Signature(gen_tonic_signature_interceptor()), class)
-        .await
-        .map_err(|_| NetworkPeerProbeError::Unreachable)
+    node_service_time_out_client_for_class(
+        &address.to_owned(),
+        TonicInterceptor::Signature(gen_tonic_signature_interceptor()),
+        class,
+    )
+    .await
+    .map_err(|_| NetworkPeerProbeError::Unreachable)
 }
 
 fn ping_request(payload: &[u8]) -> PingRequest {
@@ -325,24 +329,6 @@ fn map_status(status: tonic::Status) -> NetworkPeerProbeError {
     match status.code() {
         Code::Unavailable | Code::Unknown => NetworkPeerProbeError::Unreachable,
         Code::DeadlineExceeded => NetworkPeerProbeError::TimedOut,
-        Code::Cancelled | Code::Internal => {
-            let mut source = std::error::Error::source(&status);
-            while let Some(error) = source {
-                if error
-                    .downcast_ref::<h2::Error>()
-                    .is_some_and(|error| matches!(error.reason(), Some(h2::Reason::CANCEL | h2::Reason::NO_ERROR)))
-                {
-                    return NetworkPeerProbeError::Unreachable;
-                }
-                source = if let Some(error) = error.downcast_ref::<std::io::Error>() {
-                    // io::Error::source skips its immediate custom error.
-                    error.get_ref().map(|inner| inner as &(dyn std::error::Error + 'static))
-                } else {
-                    error.source()
-                };
-            }
-            NetworkPeerProbeError::ProtocolFailure
-        }
         _ => NetworkPeerProbeError::ProtocolFailure,
     }
 }
@@ -353,35 +339,6 @@ mod tests {
     use crate::layout::endpoint::Endpoint;
     use crate::layout::endpoints::{Endpoints, PoolEndpoints};
 
-    #[test]
-    fn peer_transport_resets_are_unreachable_without_reclassifying_application_errors() {
-        for reason in [h2::Reason::CANCEL, h2::Reason::NO_ERROR] {
-            let status = tonic::Status::from_error(Box::new(h2::Error::from(reason)));
-            assert!(matches!(status.code(), Code::Cancelled | Code::Internal));
-            let code = status.code();
-            assert_eq!(map_status(status), NetworkPeerProbeError::Unreachable);
-            let mut wrapped = tonic::Status::new(code, "wrapped transport reset");
-            wrapped.set_source(Arc::new(std::io::Error::other(h2::Error::from(reason))));
-            assert_eq!(map_status(wrapped), NetworkPeerProbeError::Unreachable);
-        }
-        for reason in [h2::Reason::PROTOCOL_ERROR, h2::Reason::COMPRESSION_ERROR] {
-            let status = tonic::Status::from_error(Box::new(h2::Error::from(reason)));
-            assert_eq!(map_status(status), NetworkPeerProbeError::ProtocolFailure);
-        }
-        for status in [
-            tonic::Status::cancelled("peer cancelled the application request"),
-            tonic::Status::internal("peer rejected the probe"),
-            tonic::Status::unauthenticated("authentication rejected"),
-            tonic::Status::permission_denied("authorization rejected"),
-        ] {
-            assert_eq!(map_status(status), NetworkPeerProbeError::ProtocolFailure);
-        }
-        assert_eq!(
-            map_status(tonic::Status::deadline_exceeded("probe deadline elapsed")),
-            NetworkPeerProbeError::TimedOut
-        );
-    }
-
     fn pacing(duration: Duration) -> Arc<DiagnosticPacing> {
         NetworkPeerProbeClient::from_endpoint_pools(&topology())
             .with_diagnostic_pacing(Instant::now(), duration)

crates/ecstore/src/cluster/rpc/remote_disk.rs

@@ -14,8 +14,7 @@
 
 use crate::cluster::rpc::client::{
     AuthenticatedChannel, TonicInterceptor, gen_tonic_signature_interceptor, is_network_like_disk_error,
-    node_service_delete_recovery_client, node_service_time_out_client, node_service_time_out_client_for_class,
-    node_service_time_out_client_no_auth,
+    node_service_time_out_client, node_service_time_out_client_for_class, node_service_time_out_client_no_auth,
 };
 use crate::cluster::rpc::internode_data_transport::{
     InternodeDataTransport, NsScannerCapabilityRequest, NsScannerStreamRequest, ReadStreamRequest, WalkDirStreamRequest,
@@ -23,11 +22,10 @@ use crate::cluster::rpc::internode_data_transport::{
 };
 use crate::disk::error::{Error, Result};
 use crate::disk::{
-    BatchReadVersionReq, BatchReadVersionResp, CheckPartsResp, ConditionalFileUpdate, DELETE_RECOVERY_PROTOCOL_VERSION,
-    DeleteOptions, DeleteRecoveryAction, DeleteRecoveryCursor, DeleteRecoveryDigest, DeleteRecoveryInspection,
-    DeleteRecoveryPage, DeleteRecoveryRpcOperation, DeleteRecoveryRpcReply, DiskAPI, DiskInfo, DiskInfoOptions, DiskLocation,
-    DiskOption, FileInfoVersions, FileReader, FileWriter, PartTransactionAction, ReadMultipleReq, ReadMultipleResp, ReadOptions,
-    RenameDataResp, SnapshotLeaseToken, UpdateMetadataOpts, VolumeInfo, WalkDirOptions, batch_read_version_one_by_one,
+    BatchReadVersionReq, BatchReadVersionResp, CheckPartsResp, ConditionalFileUpdate, DeleteOptions, DiskAPI, DiskInfo,
+    DiskInfoOptions, DiskLocation, DiskOption, FileInfoVersions, FileReader, FileWriter, PartTransactionAction, ReadMultipleReq,
+    ReadMultipleResp, ReadOptions, RenameDataResp, SnapshotLeaseToken, UpdateMetadataOpts, VolumeInfo, WalkDirOptions,
+    batch_read_version_one_by_one,
     disk_store::{
         DEFAULT_RUSTFS_DRIVE_ACTIVE_MONITORING, ENV_RUSTFS_DRIVE_ACTIVE_MONITORING, SKIP_IF_SUCCESS_BEFORE,
         get_drive_active_check_interval, get_drive_active_check_timeout, get_drive_disk_info_timeout, get_drive_list_dir_timeout,
@@ -61,10 +59,10 @@ use rustfs_protos::proto_gen::node_service::{
     CompareAndUpdateFileRequest, DeletePathsRequest, DeleteRequest, DeleteVersionRequest, DeleteVersionsRequest,
     DeleteVersionsResponse, DeleteVolumeRequest, DiskInfoRequest, ListDirRequest, ListVolumesRequest, MakeVolumeRequest,
     MakeVolumesRequest, PreparePartTransactionRequest, ReadAllRequest, ReadMetadataRequest, ReadMultipleRequest,
-    ReadMultipleResponse, ReadPartsRequest, ReadVersionRequest, ReadXlRequest, RecoverDeleteTransactionRequest,
-    RenameDataRequest, RenameFileRequest, SettlePartTransactionRequest, SnapshotLeaseReleaseRequest, SnapshotLeaseRenewRequest,
-    SnapshotLeaseRequest, SnapshotLeaseResponse, StatVolumeRequest, UpdateMetadataRequest, VerifyFileRequest, WriteAllRequest,
-    WriteMetadataRequest, node_service_client::NodeServiceClient,
+    ReadMultipleResponse, ReadPartsRequest, ReadVersionRequest, ReadXlRequest, RenameDataRequest, RenameFileRequest,
+    SettlePartTransactionRequest, SnapshotLeaseReleaseRequest, SnapshotLeaseRenewRequest, SnapshotLeaseRequest,
+    SnapshotLeaseResponse, StatVolumeRequest, UpdateMetadataRequest, VerifyFileRequest, WriteAllRequest, WriteMetadataRequest,
+    node_service_client::NodeServiceClient,
 };
 use serde::{Serialize, de::DeserializeOwned};
 use std::{
@@ -144,45 +142,6 @@ fn decode_delete_versions_errors(response: DeleteVersionsResponse, expected_len:
         .collect()
 }
 
-fn decode_delete_recovery_batch_errors(codes: Vec<Option<u32>>, expected_len: usize) -> Result<Vec<Option<DiskError>>> {
-    if codes.len() != expected_len {
-        return Err(Error::other("malformed recoverable batch delete response length"));
-    }
-    codes
-        .into_iter()
-        .map(|code| match code {
-            None => Ok(None),
-            Some(code) => DiskError::from_u32(code)
-                .map(Some)
-                .ok_or_else(|| Error::other("invalid recoverable batch delete error code")),
-        })
-        .collect()
-}
-
-fn validate_delete_recovery_inspection(
-    volume: &str,
-    object: &str,
-    transaction: Uuid,
-    inspection: &DeleteRecoveryInspection,
-) -> Result<()> {
-    if inspection.protocol_version != DELETE_RECOVERY_PROTOCOL_VERSION
-        || inspection.id.object != object
-        || inspection.id.transaction != transaction
-        || transaction.is_nil()
-    {
-        return Err(Error::other("delete recovery inspection does not match request"));
-    }
-    if let Some(journal) = &inspection.journal
-        && (journal.protocol_version != DELETE_RECOVERY_PROTOCOL_VERSION
-            || journal.volume != volume
-            || journal.object != object
-            || journal.transaction != transaction)
-    {
-        return Err(Error::other("delete recovery journal does not match request"));
-    }
-    Ok(())
-}
-
 fn snapshot_lease_token_from_response(response: SnapshotLeaseResponse) -> Result<SnapshotLeaseToken> {
     if !response.success {
         return Err(response.error.unwrap_or_default().into());
@@ -899,75 +858,6 @@ fn spawn_control_channel_prewarm(addr: String) {
 }
 
 impl RemoteDisk {
-    async fn recover_delete_transaction(
-        &self,
-        volume: &str,
-        operation: DeleteRecoveryRpcOperation,
-    ) -> Result<DeleteRecoveryRpcReply> {
-        let payload = rmp_serde::to_vec_named(&operation)?;
-        if payload.len() > operation.max_wire_payload_bytes() {
-            return Err(Error::other("delete recovery operation exceeds payload limit"));
-        }
-        let settle_timeout = if matches!(&operation, DeleteRecoveryRpcOperation::Settle { .. }) {
-            Some(crate::disk::get_delete_recovery_timeout()?)
-        } else {
-            None
-        };
-        self.execute_with_timeout(
-            || async {
-                let mut client = if let Some(timeout) = settle_timeout {
-                    if let Some(error) = self.offline_bypass_error() {
-                        return Err(error);
-                    }
-                    node_service_delete_recovery_client(
-                        &self.addr,
-                        TonicInterceptor::Signature(gen_tonic_signature_interceptor()),
-                        timeout,
-                    )
-                    .await?
-                } else {
-                    self.get_client().await?
-                };
-                let mut request = Request::new(RecoverDeleteTransactionRequest {
-                    disk: self.endpoint.to_string(),
-                    volume: volume.to_owned(),
-                    protocol_version: DELETE_RECOVERY_PROTOCOL_VERSION,
-                    operation: payload.into(),
-                });
-                let canonical_body = rustfs_protos::canonical_recover_delete_transaction_request_body(request.get_ref())
-                    .map_err(|_| Error::other("recover_delete_transaction request length cannot be represented"))?;
-                crate::cluster::rpc::set_tonic_canonical_body_digest(&mut request, &canonical_body).map_err(Error::other)?;
-                // UNIMPLEMENTED is deliberately propagated: older peers cannot settle transactions.
-                let response = client
-                    .recover_delete_transaction(request)
-                    .await
-                    .map_err(|status| {
-                        if status.code() == Code::Unimplemented {
-                            DiskError::UnsupportedDisk
-                        } else {
-                            status.into()
-                        }
-                    })?
-                    .into_inner();
-                if response.protocol_version != DELETE_RECOVERY_PROTOCOL_VERSION {
-                    return Err(DiskError::UnsupportedDisk);
-                }
-                if !response.success {
-                    return Err(response
-                        .error
-                        .ok_or_else(|| Error::other("missing delete recovery error"))?
-                        .into());
-                }
-                if response.error.is_some() {
-                    return Err(Error::other("malformed delete recovery success"));
-                }
-                rustfs_protos::decode_delete_recovery_payload(&response.result).map_err(Into::into)
-            },
-            settle_timeout.unwrap_or_else(get_max_timeout_duration),
-        )
-        .await
-    }
-
     async fn rename_file_with_durability(
         &self,
         src_volume: &str,
@@ -2030,12 +1920,6 @@ fn encode_msgpack<T: Serialize>(value: &T) -> Result<Vec<u8>> {
     encode_msgpack_with_capacity(value, MSGPACK_ENCODE_CAPACITY_HINT)
 }
 
-// Optional fields are omitted by serde; named maps keep rollback flags and
-// transaction UUIDs independent of field order across peer versions.
-fn encode_delete_options_msgpack(value: &DeleteOptions) -> Result<Vec<u8>> {
-    Ok(rmp_serde::to_vec_named(value)?)
-}
-
 fn encode_file_info_msgpack(value: &FileInfo) -> Result<Vec<u8>> {
     encode_msgpack_with_capacity(value, FILE_INFO_MSGPACK_ENCODE_CAPACITY_HINT)
 }
@@ -2726,7 +2610,7 @@ impl DiskAPI for RemoteDisk {
                 // `_bin` support for DeleteVersion is new (grpc-optimization P2); always dual-write
                 // JSON + msgpack until its fallback counter has read zero across a release window.
                 let file_info_bin = encode_file_info_msgpack(&fi)?;
-                let opts_bin = encode_delete_options_msgpack(&opts)?;
+                let opts_bin = encode_msgpack(&opts)?;
                 let conditional_marker = opts.expected_delete_marker.is_some();
                 let file_info = serde_json::to_string(&fi)?;
                 let opts = serde_json::to_string(&opts)?;
@@ -2798,7 +2682,7 @@ impl DiskAPI for RemoteDisk {
 
         // `_bin` support for DeleteVersions is new (grpc-optimization P2); always dual-write JSON +
         // msgpack until its fallback counter has read zero across a release window.
-        let opts_bin = match encode_delete_options_msgpack(&opts) {
+        let opts_bin = match encode_msgpack(&opts) {
             Ok(opts_bin) => opts_bin,
             Err(err) => {
                 let mut errors = Vec::with_capacity(versions.len());
@@ -3193,7 +3077,7 @@ impl DiskAPI for RemoteDisk {
                     opts_bin: opts_bin.into(),
                 });
 
-                crate::cluster::rpc::runtime_sources::record_remote_disk_grpc_read_version_request();
+                crate::cluster::rpc::runtime_sources::record_remote_disk_grpc_read_version_request(opts);
                 if let Some(request_payload_bytes) = request_payload_bytes {
                     crate::cluster::rpc::runtime_sources::record_remote_disk_grpc_read_version_sent_bytes(request_payload_bytes);
                 }
@@ -3271,6 +3155,7 @@ impl DiskAPI for RemoteDisk {
         record_batch_read_version_stage(INTERNODE_STAGE_BATCH_READ_VERSION_REQUEST_ENCODE, encode_started);
         let request_payload_bytes = batch_read_version_attribution_enabled
             .then(|| batch_read_version_request_payload_len(&req, &batch_read_version_req, &batch_read_version_req_bin));
+        let request_opts = req.opts;
         let batch_result = self
             .execute_with_timeout_for_op(
                 "batch_read_version",
@@ -3284,7 +3169,7 @@ impl DiskAPI for RemoteDisk {
                         batch_read_version_req_bin: batch_read_version_req_bin.into(),
                     });
 
-                    crate::cluster::rpc::runtime_sources::record_remote_disk_grpc_batch_read_version_request();
+                    crate::cluster::rpc::runtime_sources::record_remote_disk_grpc_batch_read_version_request(&request_opts);
                     if let Some(request_payload_bytes) = request_payload_bytes {
                         crate::cluster::rpc::runtime_sources::record_remote_disk_grpc_batch_read_version_sent_bytes(
                             request_payload_bytes.saturating_add(disk_len),
@@ -3777,138 +3662,6 @@ impl DiskAPI for RemoteDisk {
     }
 
     #[tracing::instrument(level = "trace", skip_all)]
-    async fn delete_version_recoverable(
-        &self,
-        volume: &str,
-        object: &str,
-        file_info: FileInfo,
-        force_del_marker: bool,
-        opts: DeleteOptions,
-    ) -> Result<()> {
-        match self
-            .recover_delete_transaction(
-                volume,
-                DeleteRecoveryRpcOperation::ApplyVersion {
-                    object: object.to_owned(),
-                    file_info: Box::new(file_info),
-                    force_del_marker,
-                    opts,
-                },
-            )
-            .await?
-        {
-            DeleteRecoveryRpcReply::AppliedVersion => Ok(()),
-            _ => Err(Error::other("unexpected recoverable delete reply")),
-        }
-    }
-
-    async fn delete_versions_recoverable(
-        &self,
-        volume: &str,
-        versions: Vec<FileInfoVersions>,
-        opts: DeleteOptions,
-    ) -> Vec<Option<DiskError>> {
-        let expected_len = versions.len();
-        let result = match self
-            .recover_delete_transaction(volume, DeleteRecoveryRpcOperation::ApplyVersions { versions, opts })
-            .await
-        {
-            Ok(DeleteRecoveryRpcReply::AppliedVersions(codes)) => decode_delete_recovery_batch_errors(codes, expected_len),
-            Ok(_) => Err(Error::other("unexpected recoverable batch delete reply")),
-            Err(error) => Err(error),
-        };
-        match result {
-            Ok(errors) => errors,
-            Err(error) => vec![Some(error); expected_len],
-        }
-    }
-
-    async fn discover_delete_transactions(
-        &self,
-        volume: &str,
-        object: Option<&str>,
-        cursor: Option<DeleteRecoveryCursor>,
-        limit: u32,
-    ) -> Result<DeleteRecoveryPage> {
-        match self
-            .recover_delete_transaction(
-                volume,
-                DeleteRecoveryRpcOperation::Discover {
-                    object: object.map(str::to_owned),
-                    cursor,
-                    limit,
-                },
-            )
-            .await?
-        {
-            DeleteRecoveryRpcReply::Discovered(page) => {
-                if object.is_some_and(|object| page.transactions.iter().any(|id| id.object != object)) {
-                    return Err(Error::other("delete recovery discovery escaped requested object"));
-                }
-                if page.transactions.len() > usize::try_from(limit).map_err(|_| Error::other("invalid discovery limit"))?
-                    || page
-                        .transactions
-                        .iter()
-                        .any(|id| id.transaction.is_nil() || id.object.is_empty())
-                {
-                    return Err(Error::other("malformed delete discovery page"));
-                }
-                Ok(page)
-            }
-            _ => Err(Error::other("unexpected delete discovery reply")),
-        }
-    }
-    async fn inspect_delete_transaction(
-        &self,
-        volume: &str,
-        object: &str,
-        transaction: Uuid,
-    ) -> Result<DeleteRecoveryInspection> {
-        match self
-            .recover_delete_transaction(
-                volume,
-                DeleteRecoveryRpcOperation::Inspect {
-                    object: object.to_owned(),
-                    transaction,
-                },
-            )
-            .await?
-        {
-            DeleteRecoveryRpcReply::Inspected(inspection) => {
-                validate_delete_recovery_inspection(volume, object, transaction, &inspection)?;
-                Ok(inspection)
-            }
-            _ => Err(Error::other("unexpected delete inspection reply")),
-        }
-    }
-    async fn settle_delete_transaction(
-        &self,
-        volume: &str,
-        object: &str,
-        transaction: Uuid,
-        expected_fingerprint: DeleteRecoveryDigest,
-        action: DeleteRecoveryAction,
-    ) -> Result<DeleteRecoveryInspection> {
-        match self
-            .recover_delete_transaction(
-                volume,
-                DeleteRecoveryRpcOperation::Settle {
-                    object: object.to_owned(),
-                    transaction,
-                    expected_fingerprint,
-                    action,
-                },
-            )
-            .await?
-        {
-            DeleteRecoveryRpcReply::Settled(inspection) => {
-                validate_delete_recovery_inspection(volume, object, transaction, &inspection)?;
-                Ok(inspection)
-            }
-            _ => Err(Error::other("unexpected delete settlement reply")),
-        }
-    }
-
     async fn settle_part_transaction(&self, volume: &str, path: &str, action: PartTransactionAction) -> Result<()> {
         self.execute_with_timeout(
             || async {
@@ -4322,45 +4075,6 @@ mod tests {
 
     static INIT: Once = Once::new();
 
-    #[test]
-    fn recoverable_batch_delete_rejects_invalid_error_codes_and_length() {
-        let errors = decode_delete_recovery_batch_errors(vec![None, Some(DiskError::FileNotFound.to_u32())], 2)
-            .expect("valid per-item errors");
-        assert!(errors[0].is_none());
-        assert_eq!(errors[1], Some(DiskError::FileNotFound));
-        assert!(decode_delete_recovery_batch_errors(vec![None], 2).is_err());
-        assert!(decode_delete_recovery_batch_errors(vec![Some(0)], 1).is_err());
-        assert!(decode_delete_recovery_batch_errors(vec![Some(u32::MAX)], 1).is_err());
-    }
-
-    #[test]
-    fn delete_recovery_inspection_rejects_wrong_transaction_object_and_protocol() {
-        let transaction = Uuid::new_v4();
-        let inspection = DeleteRecoveryInspection {
-            protocol_version: DELETE_RECOVERY_PROTOCOL_VERSION,
-            id: crate::disk::DeleteTransactionId {
-                object: "object".into(),
-                transaction,
-            },
-            state: crate::disk::DeleteRecoveryState::Absent,
-            journal: None,
-            current_digest: None,
-            current_versions: Vec::new(),
-            backup_digest: None,
-            rollback_data_complete: false,
-            commit_data_safe: false,
-            explicit_delete_safe: false,
-            fingerprint: [0; 32],
-        };
-        validate_delete_recovery_inspection("bucket", "object", transaction, &inspection).expect("matching inspection");
-        assert!(validate_delete_recovery_inspection("bucket", "other", transaction, &inspection).is_err());
-        assert!(validate_delete_recovery_inspection("bucket", "object", Uuid::new_v4(), &inspection).is_err());
-        assert!(validate_delete_recovery_inspection("bucket", "object", Uuid::nil(), &inspection).is_err());
-        let mut incompatible = inspection;
-        incompatible.protocol_version += 1;
-        assert!(validate_delete_recovery_inspection("bucket", "object", transaction, &incompatible).is_err());
-    }
-
     #[test]
     fn compare_and_update_wire_outcome_fails_closed() {
         assert_eq!(
@@ -4394,19 +4108,32 @@ mod tests {
     }
 
     #[test]
-    fn delete_options_named_wire_preserves_rollback_and_legacy_marker_fields() {
+    fn retired_marker_options_preserve_legacy_positional_wire_shape() {
         #[derive(Debug, serde::Serialize, serde::Deserialize, PartialEq)]
         struct LegacyDeleteOptions {
             recursive: bool,
             immediate: bool,
             undo_write: bool,
-            #[serde(default)]
             undo_delete: bool,
             old_data_dir: Option<Uuid>,
-            #[serde(default, skip_serializing_if = "Option::is_none")]
-            expected_delete_marker: Option<rustfs_filemeta::MetaDeleteMarker>,
         }
-        let transaction = Uuid::new_v4();
+        let legacy = LegacyDeleteOptions {
+            recursive: false,
+            immediate: false,
+            undo_write: false,
+            undo_delete: false,
+            old_data_dir: None,
+        };
+        let original = encode_msgpack(&legacy).unwrap();
+        let current = encode_msgpack(&DeleteOptions::default()).unwrap();
+        assert_eq!(current, original, "ordinary deletes must retain the older peer's positional payload");
+        assert_eq!(rmp_serde::from_slice::<LegacyDeleteOptions>(&current).unwrap(), legacy);
+        assert!(
+            rmp_serde::from_slice::<DeleteOptions>(&original)
+                .unwrap()
+                .expected_delete_marker
+                .is_none()
+        );
         let mut marker = FileInfo {
             deleted: true,
             version_id: Some(Uuid::new_v4()),
@@ -4415,69 +4142,12 @@ mod tests {
         };
         marker.set_delete_marker_incarnation(Uuid::new_v4());
         let marker = rustfs_filemeta::MetaDeleteMarker::from(marker);
-        for expected_delete_marker in [None, Some(marker)] {
-            for (undo_write, undo_delete) in [(true, false), (false, true)] {
-                let legacy = LegacyDeleteOptions {
-                    recursive: false,
-                    immediate: false,
-                    undo_write,
-                    undo_delete,
-                    old_data_dir: Some(transaction),
-                    expected_delete_marker: expected_delete_marker.clone(),
-                };
-                // Older senders use positional arrays. Their unchanged field prefix and
-                // missing optional fields must also decode on the named-map sender's peer.
-                for (encoding, payload) in [
-                    ("positional", rmp_serde::to_vec(&legacy).expect("encode legacy positional options")),
-                    ("named", rmp_serde::to_vec_named(&legacy).expect("encode legacy named options")),
-                ] {
-                    let decoded: DeleteOptions = rmp_serde::from_slice(&payload).expect("decode legacy options on current peer");
-                    assert_eq!(decoded.undo_write, undo_write, "{encoding} undo-write flag");
-                    assert_eq!(decoded.undo_delete, undo_delete, "{encoding} undo-delete flag");
-                    assert_eq!(decoded.old_data_dir, Some(transaction), "{encoding} transaction UUID");
-                    assert_eq!(decoded.expected_delete_marker, expected_delete_marker, "{encoding} marker precondition");
-                    assert!(decoded.delete_recovery_bucket_incarnation.is_none(), "{encoding} omitted incarnation");
-                    assert!(!decoded.recoverable_delete, "{encoding} omitted recovery flag");
-                }
-                for incarnation in [None, Some(Uuid::new_v4())] {
-                    let current = DeleteOptions {
-                        undo_write,
-                        undo_delete,
-                        old_data_dir: Some(transaction),
-                        expected_delete_marker: expected_delete_marker.clone(),
-                        delete_recovery_bucket_incarnation: incarnation,
-                        recoverable_delete: true,
-                        ..Default::default()
-                    };
-                    let payload = encode_delete_options_msgpack(&current).expect("encode current named options");
-                    let decoded: DeleteOptions = rmp_serde::from_slice(&payload).expect("decode current options on current peer");
-                    assert_eq!(decoded.undo_write, undo_write);
-                    assert_eq!(decoded.undo_delete, undo_delete);
-                    assert_eq!(decoded.old_data_dir, Some(transaction));
-                    assert_eq!(decoded.expected_delete_marker, expected_delete_marker);
-                    assert_eq!(decoded.delete_recovery_bucket_incarnation, incarnation);
-                    assert!(decoded.recoverable_delete);
-                    let decoded: LegacyDeleteOptions =
-                        rmp_serde::from_slice(&payload).expect("decode current options on legacy peer");
-                    assert_eq!(decoded, legacy, "unknown recovery fields cannot shift rollback or marker values");
-                }
-            }
-        }
-    }
-
-    #[test]
-    fn malformed_nonempty_delete_options_do_not_fall_back_to_json() {
         let options = DeleteOptions {
-            undo_write: true,
-            old_data_dir: Some(Uuid::new_v4()),
+            expected_delete_marker: Some(marker.clone()),
             ..Default::default()
         };
-        let json = serde_json::to_string(&options).expect("valid compatibility JSON");
-        assert!(decode_msgpack_or_json::<DeleteOptions>(&[0xc1], &json, "DeleteOptions").is_err());
-        let decoded: DeleteOptions =
-            decode_msgpack_or_json(&[], &json, "DeleteOptions").expect("absent MessagePack permits JSON compatibility");
-        assert!(decoded.undo_write);
-        assert_eq!(decoded.old_data_dir, options.old_data_dir);
+        let decoded: DeleteOptions = rmp_serde::from_slice(&encode_msgpack(&options).unwrap()).unwrap();
+        assert_eq!(decoded.expected_delete_marker, Some(marker));
     }
 
     #[test]

crates/ecstore/src/cluster/rpc/runtime_sources.rs

@@ -12,6 +12,7 @@
 // See the License for the specific language governing permissions and
 // limitations under the License.
 
+use crate::disk::ReadOptions;
 use rustfs_io_metrics::internode_metrics::{
     INTERNODE_MSGPACK_CODEC_JSON, INTERNODE_MSGPACK_CODEC_MSGPACK, INTERNODE_MSGPACK_DIRECTION_RESPONSE,
     INTERNODE_OPERATION_GRPC_BATCH_READ_VERSION, INTERNODE_OPERATION_GRPC_COMPARE_AND_UPDATE_FILE,
@@ -19,8 +20,320 @@ use rustfs_io_metrics::internode_metrics::{
     INTERNODE_OPERATION_GRPC_WRITE_ALL, INTERNODE_OPERATION_PUT_FILE_STREAM, INTERNODE_OPERATION_READ_FILE_STREAM,
     INTERNODE_TRANSPORT_BACKEND_GRPC, INTERNODE_TRANSPORT_BACKEND_TCP_HTTP, global_internode_metrics,
 };
+#[cfg(feature = "diagnostic-read-version-caller")]
+use rustfs_io_metrics::internode_metrics::{
+    ReadVersionRequestCaller, ReadVersionRequestMode, record_read_version_request_by_caller,
+};
 use std::time::Duration;
 
+#[cfg(feature = "diagnostic-read-version-caller")]
+tokio::task_local! {
+    static CURRENT_READ_VERSION_REQUEST_CALLER: std::cell::Cell<Option<ReadVersionRequestCaller>>;
+}
+
+#[cfg(feature = "diagnostic-read-version-caller")]
+pub(crate) fn read_version_request_caller_from_location(file: &str) -> ReadVersionRequestCaller {
+    let path = file.replace('\\', "/");
+    if path.ends_with("/store/peer.rs") {
+        ReadVersionRequestCaller::Peer
+    } else if path.ends_with("/store/init.rs") {
+        ReadVersionRequestCaller::Startup
+    } else if path.ends_with("/store/heal.rs") || path.ends_with("/set_disk/ops/heal.rs") {
+        ReadVersionRequestCaller::Heal
+    } else if path.ends_with("/set_disk/ops/heal_walk.rs") {
+        ReadVersionRequestCaller::Heal
+    } else if path.ends_with("/store/list_objects.rs") {
+        ReadVersionRequestCaller::ListObjects
+    } else if path.ends_with("/store/object.rs") || path.ends_with("/set_disk/ops/object.rs") {
+        ReadVersionRequestCaller::Object
+    } else if path.ends_with("/set_disk/core/metadata_read.rs") {
+        ReadVersionRequestCaller::MetadataRead
+    } else if path.ends_with("/set_disk/ops/multipart.rs") {
+        ReadVersionRequestCaller::Multipart
+    } else if path.ends_with("/disk/disk_store.rs") {
+        ReadVersionRequestCaller::DiskStore
+    } else if path.ends_with("/disk/mod.rs") {
+        ReadVersionRequestCaller::DiskDispatch
+    } else if path.ends_with("/set_disk/read.rs") {
+        ReadVersionRequestCaller::SetDiskRead
+    } else if path.ends_with("/set_disk/core/io_primitives.rs") {
+        ReadVersionRequestCaller::SetDiskIo
+    } else if path.ends_with("/services/rebalance/worker.rs") {
+        ReadVersionRequestCaller::Rebalance
+    } else if path.ends_with("/core/pools.rs") {
+        ReadVersionRequestCaller::Decommission
+    } else if path.ends_with("/tier_free_version_recovery.rs") {
+        ReadVersionRequestCaller::TierRecovery
+    } else if path.contains("/bucket/lifecycle/") {
+        ReadVersionRequestCaller::Lifecycle
+    } else if path.contains("/cache_value/") {
+        ReadVersionRequestCaller::CacheValue
+    } else if path.contains("/ecstore/src/") {
+        ReadVersionRequestCaller::Other
+    } else {
+        ReadVersionRequestCaller::Unattributed
+    }
+}
+
+#[cfg(feature = "diagnostic-read-version-caller")]
+pub(crate) fn current_read_version_request_caller() -> Option<ReadVersionRequestCaller> {
+    CURRENT_READ_VERSION_REQUEST_CALLER
+        .try_with(std::cell::Cell::get)
+        .ok()
+        .flatten()
+}
+
+#[cfg(feature = "diagnostic-read-version-caller")]
+pub(crate) async fn with_read_version_request_caller<F>(caller: ReadVersionRequestCaller, future: F) -> F::Output
+where
+    F: std::future::Future,
+{
+    CURRENT_READ_VERSION_REQUEST_CALLER
+        .scope(std::cell::Cell::new(Some(caller)), future)
+        .await
+}
+
+#[cfg(feature = "diagnostic-read-version-caller")]
+pub(crate) async fn with_read_version_request_caller_if_absent<F>(caller: ReadVersionRequestCaller, future: F) -> F::Output
+where
+    F: std::future::Future,
+{
+    if current_read_version_request_caller().is_some() {
+        future.await
+    } else {
+        with_read_version_request_caller(caller, future).await
+    }
+}
+
+#[cfg(all(test, feature = "diagnostic-read-version-caller"))]
+pub(crate) async fn with_optional_read_version_request_caller<F>(caller: Option<ReadVersionRequestCaller>, future: F) -> F::Output
+where
+    F: std::future::Future,
+{
+    match caller {
+        Some(caller) => with_read_version_request_caller(caller, future).await,
+        None => future.await,
+    }
+}
+
+#[cfg(feature = "diagnostic-read-version-caller")]
+use rustfs_io_metrics::internode_metrics::{ReadVersionAttribution, ReadVersionContextBoundary, ReadVersionEntrySource};
+#[cfg(feature = "diagnostic-read-version-caller")]
+use rustfs_io_metrics::s3_http_metrics::{S3HttpOperationContext, current_s3_http_operation_context};
+
+#[cfg(feature = "diagnostic-read-version-caller")]
+#[derive(Clone, Copy, Debug, Eq, PartialEq)]
+struct ReadVersionEntryContext {
+    source: ReadVersionEntrySource,
+    api_operation: Option<S3HttpOperationContext>,
+    caller: Option<ReadVersionRequestCaller>,
+    // Entry provenance stays fixed; the parent snapshot follows each handoff.
+    parent_api_operation: Option<S3HttpOperationContext>,
+    parent_caller: Option<ReadVersionRequestCaller>,
+}
+
+#[cfg(feature = "diagnostic-read-version-caller")]
+impl ReadVersionEntryContext {
+    fn new(
+        source: ReadVersionEntrySource,
+        api_operation: Option<S3HttpOperationContext>,
+        caller: Option<ReadVersionRequestCaller>,
+    ) -> Self {
+        Self {
+            source,
+            api_operation,
+            caller,
+            parent_api_operation: api_operation,
+            parent_caller: caller,
+        }
+    }
+}
+
+/// Capture values before spawning or enqueueing; absence is part of the value.
+#[cfg(feature = "diagnostic-read-version-caller")]
+#[derive(Clone, Copy, Debug, Eq, PartialEq)]
+pub(crate) struct ReadVersionDiagnosticContext {
+    pub(crate) api_operation: Option<S3HttpOperationContext>,
+    pub(crate) caller: Option<ReadVersionRequestCaller>,
+    entry: Option<ReadVersionEntryContext>,
+}
+
+#[cfg(feature = "diagnostic-read-version-caller")]
+tokio::task_local! {
+    static CURRENT_READ_VERSION_ENTRY: std::cell::Cell<Option<ReadVersionEntryContext>>;
+    static CURRENT_READ_VERSION_BATCH_CONTEXTS: Vec<ReadVersionDiagnosticContext>;
+}
+
+#[cfg(feature = "diagnostic-read-version-caller")]
+impl ReadVersionDiagnosticContext {
+    pub(crate) fn capture() -> Self {
+        Self {
+            api_operation: current_s3_http_operation_context(),
+            caller: current_read_version_request_caller(),
+            entry: CURRENT_READ_VERSION_ENTRY.try_with(std::cell::Cell::get).ok().flatten(),
+        }
+    }
+
+    pub(crate) fn capture_entry(source: ReadVersionEntrySource) -> Self {
+        let mut context = Self::capture();
+        if context.entry.is_none() {
+            context.entry = Some(ReadVersionEntryContext::new(source, context.api_operation, context.caller));
+            context.observe(ReadVersionContextBoundary::Entry);
+        }
+        context
+    }
+
+    pub(crate) fn observe(self, boundary: ReadVersionContextBoundary) {
+        self.observe_with_mode(boundary, None);
+    }
+
+    fn observe_with_mode(self, boundary: ReadVersionContextBoundary, mode: Option<ReadVersionRequestMode>) {
+        rustfs_io_metrics::internode_metrics::record_read_version_context_transition(
+            self.entry.map_or(ReadVersionEntrySource::Unclassified, |entry| entry.source),
+            boundary,
+            self.api_operation,
+            current_s3_http_operation_context(),
+            self.caller,
+            current_read_version_request_caller(),
+            mode,
+        );
+    }
+}
+
+#[cfg(feature = "diagnostic-read-version-caller")]
+pub fn with_read_version_entry<F>(source: ReadVersionEntrySource, future: F) -> impl std::future::Future<Output = F::Output>
+where
+    F: std::future::Future,
+{
+    // Box before constructing the scope future so large storage futures do not
+    // inflate every caller frame. Capture task-local values only when polled.
+    let future = Box::pin(future);
+    async move {
+        let context = ReadVersionDiagnosticContext::capture_entry(source);
+        CURRENT_READ_VERSION_ENTRY
+            .scope(std::cell::Cell::new(context.entry), future)
+            .await
+    }
+}
+
+/// Capture on the caller before spawning, then restore when the child is polled.
+#[cfg(feature = "diagnostic-read-version-caller")]
+pub fn with_read_version_spawn_context<F>(
+    source: ReadVersionEntrySource,
+    boundary: ReadVersionContextBoundary,
+    future: F,
+) -> impl std::future::Future<Output = F::Output>
+where
+    F: std::future::Future,
+{
+    let context = ReadVersionDiagnosticContext::capture_entry(source);
+    let future = Box::pin(future);
+    with_read_version_diagnostic_context(context, boundary, future)
+}
+
+/// Observe a named entry even when its scope retains an earlier source.
+#[cfg(feature = "diagnostic-read-version-caller")]
+pub fn observe_read_version_context_boundary(boundary: ReadVersionContextBoundary) {
+    ReadVersionDiagnosticContext::capture().observe(boundary);
+}
+
+/// The earliest observed storage entry, for diagnostic consumers only.
+#[cfg(feature = "diagnostic-read-version-caller")]
+pub fn current_read_version_entry_source() -> Option<ReadVersionEntrySource> {
+    CURRENT_READ_VERSION_ENTRY
+        .try_with(std::cell::Cell::get)
+        .ok()
+        .flatten()
+        .map(|entry| entry.source)
+}
+
+/// Restore only this item's context and observe the child after restoration.
+#[cfg(feature = "diagnostic-read-version-caller")]
+pub(crate) async fn with_read_version_diagnostic_context<F>(
+    context: ReadVersionDiagnosticContext,
+    boundary: ReadVersionContextBoundary,
+    future: F,
+) -> F::Output
+where
+    F: std::future::Future,
+{
+    let observed = async {
+        context.observe(boundary);
+        future.await
+    };
+    let caller = CURRENT_READ_VERSION_REQUEST_CALLER.scope(std::cell::Cell::new(context.caller), observed);
+    let mut forwarded_entry = context.entry;
+    if let Some(entry) = &mut forwarded_entry {
+        entry.parent_api_operation = context.api_operation;
+        entry.parent_caller = context.caller;
+    }
+    let entry = CURRENT_READ_VERSION_ENTRY.scope(std::cell::Cell::new(forwarded_entry), caller);
+    rustfs_io_metrics::s3_http_metrics::with_isolated_s3_http_operation_context(context.api_operation, entry).await
+}
+
+/// A batch has item contexts, not one owner's S3 task-local operation.
+#[cfg(feature = "diagnostic-read-version-caller")]
+pub(crate) async fn with_read_version_batch_contexts<F>(contexts: Vec<ReadVersionDiagnosticContext>, future: F) -> F::Output
+where
+    F: std::future::Future,
+{
+    let batch = CURRENT_READ_VERSION_BATCH_CONTEXTS.scope(contexts, future);
+    let caller = CURRENT_READ_VERSION_REQUEST_CALLER.scope(std::cell::Cell::new(None), batch);
+    let entry = CURRENT_READ_VERSION_ENTRY.scope(std::cell::Cell::new(None), caller);
+    rustfs_io_metrics::s3_http_metrics::with_isolated_s3_http_operation_context(None, entry).await
+}
+
+#[cfg(feature = "diagnostic-read-version-caller")]
+pub(crate) async fn with_batch_read_version_item_context<F>(index: usize, item_count: usize, future: F) -> F::Output
+where
+    F: std::future::Future,
+{
+    let context = CURRENT_READ_VERSION_BATCH_CONTEXTS
+        .try_with(|contexts| (contexts.len() == item_count).then(|| contexts.get(index).copied()).flatten())
+        .ok()
+        .flatten();
+    match context {
+        Some(context) => with_read_version_diagnostic_context(context, ReadVersionContextBoundary::UnaryFallback, future).await,
+        None => future.await,
+    }
+}
+
+#[cfg(feature = "diagnostic-read-version-caller")]
+fn common_read_version_attribution<T: Copy + Eq>(values: impl Iterator<Item = Option<T>>) -> ReadVersionAttribution<T> {
+    values
+        .map(|value| value.map_or(ReadVersionAttribution::Absent, ReadVersionAttribution::Common))
+        .reduce(|common, value| {
+            if common == value {
+                common
+            } else {
+                ReadVersionAttribution::Mixed
+            }
+        })
+        .unwrap_or(ReadVersionAttribution::Absent)
+}
+
+#[cfg(feature = "diagnostic-read-version-caller")]
+fn record_read_version_entry_to_rpc_context(opts: &ReadOptions) {
+    let context = ReadVersionDiagnosticContext::capture();
+    let mode = Some(read_version_request_mode(opts));
+    if let Some(entry) = context.entry {
+        ReadVersionDiagnosticContext {
+            api_operation: entry.api_operation,
+            caller: entry.caller,
+            ..context
+        }
+        .observe_with_mode(ReadVersionContextBoundary::EntryToRpc, mode);
+        ReadVersionDiagnosticContext {
+            api_operation: entry.parent_api_operation,
+            caller: entry.parent_caller,
+            ..context
+        }
+        .observe_with_mode(ReadVersionContextBoundary::RemoteRpc, mode);
+    } else {
+        context.observe_with_mode(ReadVersionContextBoundary::RemoteRpc, mode);
+    }
+}
+
 #[cfg(test)]
 use rustfs_io_metrics::internode_metrics::InternodeMetricsSnapshot;
 
@@ -106,7 +419,18 @@ pub(crate) fn record_remote_disk_grpc_read_all_request() {
         .record_outgoing_request_for_operation_and_backend(INTERNODE_OPERATION_GRPC_READ_ALL, INTERNODE_TRANSPORT_BACKEND_GRPC);
 }
 
-pub(crate) fn record_remote_disk_grpc_read_version_request() {
+#[cfg(feature = "diagnostic-read-version-caller")]
+fn read_version_request_mode(opts: &ReadOptions) -> ReadVersionRequestMode {
+    match (opts.read_data, opts.healing, opts.incl_free_versions) {
+        (false, false, false) => ReadVersionRequestMode::MetadataOnly,
+        (true, false, false) => ReadVersionRequestMode::Data,
+        (false, true, false) => ReadVersionRequestMode::Healing,
+        (false, false, true) => ReadVersionRequestMode::IncludeFreeVersions,
+        _ => ReadVersionRequestMode::Combined,
+    }
+}
+
+pub(crate) fn record_remote_disk_grpc_read_version_request(_opts: &ReadOptions) {
     if !rustfs_io_metrics::get_stage_metrics_enabled() {
         return;
     }
@@ -114,9 +438,19 @@ pub(crate) fn record_remote_disk_grpc_read_version_request() {
         INTERNODE_OPERATION_GRPC_READ_VERSION,
         INTERNODE_TRANSPORT_BACKEND_GRPC,
     );
+    #[cfg(feature = "diagnostic-read-version-caller")]
+    let api_operation = rustfs_io_metrics::s3_http_metrics::current_s3_http_operation_context();
+    #[cfg(feature = "diagnostic-read-version-caller")]
+    record_read_version_request_by_caller(
+        api_operation,
+        current_read_version_request_caller().unwrap_or_default(),
+        read_version_request_mode(_opts),
+    );
+    #[cfg(feature = "diagnostic-read-version-caller")]
+    record_read_version_entry_to_rpc_context(_opts);
 }
 
-pub(crate) fn record_remote_disk_grpc_batch_read_version_request() {
+pub(crate) fn record_remote_disk_grpc_batch_read_version_request(_opts: &ReadOptions) {
     if !rustfs_io_metrics::get_stage_metrics_enabled() {
         return;
     }
@@ -124,6 +458,30 @@ pub(crate) fn record_remote_disk_grpc_batch_read_version_request() {
         INTERNODE_OPERATION_GRPC_BATCH_READ_VERSION,
         INTERNODE_TRANSPORT_BACKEND_GRPC,
     );
+    #[cfg(feature = "diagnostic-read-version-caller")]
+    {
+        let attribution = CURRENT_READ_VERSION_BATCH_CONTEXTS.try_with(|contexts| {
+            (
+                common_read_version_attribution(contexts.iter().map(|context| context.api_operation)),
+                common_read_version_attribution(contexts.iter().map(|context| context.caller)),
+                common_read_version_attribution(contexts.iter().map(|context| context.entry.map(|entry| entry.source))),
+            )
+        });
+        let (api, caller, source) = attribution.unwrap_or_else(|_| {
+            let context = ReadVersionDiagnosticContext::capture();
+            (
+                common_read_version_attribution(std::iter::once(context.api_operation)),
+                common_read_version_attribution(std::iter::once(context.caller)),
+                common_read_version_attribution(std::iter::once(context.entry.map(|entry| entry.source))),
+            )
+        });
+        rustfs_io_metrics::internode_metrics::record_batch_read_version_request_by_caller(
+            api,
+            caller,
+            source,
+            read_version_request_mode(_opts),
+        );
+    }
 }
 
 pub(crate) fn record_remote_disk_grpc_batch_read_version_stage(stage: &'static str, duration: Duration) {
@@ -303,3 +661,529 @@ pub(crate) fn internode_metrics_snapshot_for_test() -> InternodeMetricsSnapshot
 pub(crate) fn internode_msgpack_json_decode_error_total_for_test() -> u64 {
     global_internode_metrics().msgpack_json_decode_error_total_for_test()
 }
+
+#[cfg(test)]
+mod tests {
+    use super::*;
+
+    #[cfg(feature = "diagnostic-read-version-caller")]
+    fn diagnostic_operation(operation: rustfs_s3_ops::S3Operation) -> S3HttpOperationContext {
+        let mut request = rustfs_io_metrics::s3_http_metrics::S3HttpRequestGuard::new("GET");
+        request.in_scope(|| {
+            rustfs_io_metrics::record_s3_op(operation);
+            current_s3_http_operation_context().expect("operation should be captured")
+        })
+    }
+
+    #[cfg(feature = "diagnostic-read-version-caller")]
+    #[test]
+    fn diagnostic_list_retains_background_entry_through_shared_workers() {
+        let runtime = tokio::runtime::Builder::new_current_thread()
+            .enable_all()
+            .build()
+            .expect("test runtime");
+        runtime.block_on(async {
+            tokio::spawn(with_read_version_entry(ReadVersionEntrySource::RecoveryDispositionMaintenance, async {
+                let entry = ReadVersionDiagnosticContext::capture();
+                assert_eq!(entry.api_operation, None);
+                assert_eq!(entry.caller, None);
+                let expected_entry = entry.entry.expect("background entry is observed");
+                assert_eq!(expected_entry.source.as_str(), "recovery_disposition_maintenance");
+                with_read_version_entry(ReadVersionEntrySource::ListObjects, async {
+                    with_read_version_request_caller(ReadVersionRequestCaller::ListObjects, async {
+                        let parent = ReadVersionDiagnosticContext::capture_entry(ReadVersionEntrySource::ListObjects);
+                        assert_eq!(parent.entry, Some(expected_entry));
+                        for boundary in [
+                            ReadVersionContextBoundary::ListProducer,
+                            ReadVersionContextBoundary::ListGather,
+                        ] {
+                            tokio::spawn(with_read_version_diagnostic_context(parent, boundary, async move {
+                                let child = ReadVersionDiagnosticContext::capture();
+                                assert_eq!(child.api_operation, None);
+                                assert_eq!(child.caller, Some(ReadVersionRequestCaller::ListObjects));
+                                let child_entry = child.entry.expect("worker retains background provenance");
+                                assert_eq!(child_entry.source, expected_entry.source);
+                                assert_eq!(child_entry.api_operation, None);
+                                assert_eq!(child_entry.caller, None);
+                                assert_eq!(child_entry.parent_api_operation, None);
+                                assert_eq!(child_entry.parent_caller, Some(ReadVersionRequestCaller::ListObjects));
+                            }))
+                            .await
+                            .expect("list worker completes");
+                        }
+                    })
+                    .await;
+                })
+                .await;
+                assert_eq!(ReadVersionDiagnosticContext::capture(), entry);
+            }))
+            .await
+            .expect("background task completes");
+            assert_eq!(ReadVersionDiagnosticContext::capture().entry, None);
+        });
+    }
+
+    #[cfg(feature = "diagnostic-read-version-caller")]
+    #[test]
+    fn diagnostic_spawn_scope_captures_parent_and_isolates_absence() {
+        struct LargeFuture([u8; 64 * 1024]);
+        impl std::future::Future for LargeFuture {
+            type Output = ReadVersionDiagnosticContext;
+            fn poll(self: std::pin::Pin<&mut Self>, _cx: &mut std::task::Context<'_>) -> std::task::Poll<Self::Output> {
+                assert_eq!(self.0[0], 7);
+                std::task::Poll::Ready(ReadVersionDiagnosticContext::capture())
+            }
+        }
+        let runtime = tokio::runtime::Builder::new_current_thread()
+            .enable_all()
+            .build()
+            .expect("test runtime");
+        let put = diagnostic_operation(rustfs_s3_ops::S3Operation::PutObject);
+        let get = diagnostic_operation(rustfs_s3_ops::S3Operation::GetObject);
+        runtime.block_on(async {
+            let child = rustfs_io_metrics::s3_http_metrics::with_isolated_s3_http_operation_context(
+                Some(put),
+                with_read_version_request_caller(
+                    ReadVersionRequestCaller::Object,
+                    with_read_version_entry(ReadVersionEntrySource::ConfigWrite, async {
+                        let child = with_read_version_spawn_context(
+                            ReadVersionEntrySource::AppPutOwner,
+                            ReadVersionContextBoundary::AppPutOwner,
+                            LargeFuture([7; 64 * 1024]),
+                        );
+                        assert!(std::mem::size_of_val(&child) < 1024, "spawn scope must not embed the storage future");
+                        tokio::spawn(child).await.expect("child task")
+                    }),
+                ),
+            )
+            .await;
+            assert_eq!(child.api_operation, Some(put));
+            assert_eq!(child.caller, Some(ReadVersionRequestCaller::Object));
+            assert_eq!(child.entry.expect("earlier source retained").source, ReadVersionEntrySource::ConfigWrite);
+            let internal = with_read_version_spawn_context(
+                ReadVersionEntrySource::AppPutOwner,
+                ReadVersionContextBoundary::AppPutOwner,
+                LargeFuture([7; 64 * 1024]),
+            );
+            let observed = rustfs_io_metrics::s3_http_metrics::with_isolated_s3_http_operation_context(
+                Some(get),
+                with_read_version_request_caller(
+                    ReadVersionRequestCaller::Peer,
+                    with_read_version_entry(ReadVersionEntrySource::ConfigRead, async {
+                        let observed = internal.await;
+                        assert_eq!(current_s3_http_operation_context(), Some(get));
+                        assert_eq!(current_read_version_entry_source(), Some(ReadVersionEntrySource::ConfigRead));
+                        observed
+                    }),
+                ),
+            )
+            .await;
+            assert_eq!(observed.api_operation, None);
+            assert_eq!(observed.caller, None);
+            assert_eq!(
+                observed.entry.expect("internal source retained").source,
+                ReadVersionEntrySource::AppPutOwner
+            );
+            assert_eq!(current_s3_http_operation_context(), None);
+            assert_eq!(current_read_version_entry_source(), None);
+        });
+    }
+
+    #[cfg(feature = "diagnostic-read-version-caller")]
+    #[test]
+    fn diagnostic_entry_keeps_large_future_off_the_scope_frame_and_captures_when_polled() {
+        struct LargeFuture([u8; 64 * 1024]);
+        impl std::future::Future for LargeFuture {
+            type Output = ReadVersionDiagnosticContext;
+            fn poll(self: std::pin::Pin<&mut Self>, _cx: &mut std::task::Context<'_>) -> std::task::Poll<Self::Output> {
+                assert_eq!(self.0[0], 7);
+                std::task::Poll::Ready(ReadVersionDiagnosticContext::capture())
+            }
+        }
+        let runtime = tokio::runtime::Builder::new_current_thread()
+            .enable_all()
+            .build()
+            .expect("test runtime");
+        let get = diagnostic_operation(rustfs_s3_ops::S3Operation::GetObject);
+        let read = with_read_version_entry(ReadVersionEntrySource::ObjectInfoRead, LargeFuture([7; 64 * 1024]));
+        assert!(std::mem::size_of_val(&read) < 1024, "scope frame must not embed the storage future");
+        assert_eq!(current_read_version_entry_source(), None);
+        let observed = runtime.block_on(rustfs_io_metrics::s3_http_metrics::with_isolated_s3_http_operation_context(
+            Some(get),
+            with_read_version_entry(ReadVersionEntrySource::ConfigRead, read),
+        ));
+        assert_eq!(observed.entry.expect("entry captured at poll").source, ReadVersionEntrySource::ConfigRead);
+        assert_eq!(observed.api_operation, Some(get));
+        assert_eq!(current_read_version_entry_source(), None);
+        assert_eq!(current_s3_http_operation_context(), None);
+    }
+
+    #[cfg(feature = "diagnostic-read-version-caller")]
+    #[test]
+    fn diagnostic_named_entry_is_observed_when_earlier_source_is_preserved() {
+        use metrics_util::debugging::{DebugValue, DebuggingRecorder};
+        let runtime = tokio::runtime::Builder::new_current_thread()
+            .enable_all()
+            .build()
+            .expect("runtime");
+        let recorder = DebuggingRecorder::new();
+        let snapshots = recorder.snapshotter();
+        rustfs_io_metrics::set_get_stage_metrics_enabled(true);
+        let get = diagnostic_operation(rustfs_s3_ops::S3Operation::GetObject);
+        metrics::with_local_recorder(&recorder, || {
+            runtime.block_on(async {
+                for earlier in [None, Some(ReadVersionEntrySource::ConfigRead)] {
+                    let api = earlier.map(|_| get);
+                    let entry = with_read_version_entry(ReadVersionEntrySource::DataUsageCacheRead, async {
+                        observe_read_version_context_boundary(ReadVersionContextBoundary::ScannerCacheLoad);
+                        assert_eq!(current_s3_http_operation_context(), api);
+                    });
+                    rustfs_io_metrics::s3_http_metrics::with_isolated_s3_http_operation_context(api, async {
+                        match earlier {
+                            Some(source) => with_read_version_entry(source, entry).await,
+                            None => entry.await,
+                        }
+                    })
+                    .await;
+                    assert_eq!(current_read_version_entry_source(), None);
+                    assert_eq!(current_s3_http_operation_context(), None);
+                }
+            });
+        });
+        let rows = snapshots.snapshot().into_vec();
+        for (source, api, state) in [
+            ("data_usage_cache_read", "unattributed", "absent"),
+            ("config_read", "s3:GetObject", "preserved"),
+        ] {
+            assert!(
+                rows.iter().any(|(key, _, _, value)| {
+                    let labels = key
+                        .key()
+                        .labels()
+                        .map(|label| (label.key(), label.value()))
+                        .collect::<std::collections::HashMap<_, _>>();
+                    key.key().name() == "rustfs_storage_diagnostic_context_transitions_total"
+                        && labels.get("boundary") == Some(&"scanner_cache_load")
+                        && labels.get("entry_source") == Some(&source)
+                        && labels.get("parent_api_operation") == Some(&api)
+                        && labels.get("child_api_operation") == Some(&api)
+                        && labels.get("api_context") == Some(&state)
+                        && matches!(value, DebugValue::Counter(1))
+                }),
+                "a named cache entry remains observable with source {source}"
+            );
+        }
+    }
+
+    #[cfg(feature = "diagnostic-read-version-caller")]
+    #[test]
+    fn diagnostic_context_records_absent_internal_entry_and_real_child_loss() {
+        use metrics_util::debugging::{DebugValue, DebuggingRecorder};
+        use rustfs_io_metrics::s3_http_metrics::{with_isolated_s3_http_operation_context, with_s3_http_operation_context};
+        let runtime = tokio::runtime::Builder::new_current_thread()
+            .enable_all()
+            .build()
+            .expect("test runtime");
+        let recorder = DebuggingRecorder::new();
+        let snapshots = recorder.snapshotter();
+        rustfs_io_metrics::set_get_stage_metrics_enabled(true);
+        let get = diagnostic_operation(rustfs_s3_ops::S3Operation::GetObject);
+        metrics::with_local_recorder(&recorder, || {
+            runtime.block_on(async {
+                with_read_version_entry(ReadVersionEntrySource::ConfigRead, async {
+                    let context = ReadVersionDiagnosticContext::capture();
+                    tokio::spawn(with_read_version_diagnostic_context(
+                        context,
+                        ReadVersionContextBoundary::FullWait,
+                        async {
+                            assert_eq!(current_s3_http_operation_context(), None);
+                            record_read_version_entry_to_rpc_context(&ReadOptions::default());
+                        },
+                    ))
+                    .await
+                    .expect("internal child should complete");
+                })
+                .await;
+                with_s3_http_operation_context(
+                    Some(get),
+                    with_read_version_entry(
+                        ReadVersionEntrySource::ObjectRead,
+                        with_read_version_request_caller(ReadVersionRequestCaller::Object, async {
+                            let context = ReadVersionDiagnosticContext::capture();
+                            tokio::spawn(with_read_version_diagnostic_context(
+                                context,
+                                ReadVersionContextBoundary::EarlyStop,
+                                async move {
+                                    assert_eq!(current_s3_http_operation_context(), Some(get));
+                                    record_read_version_entry_to_rpc_context(&ReadOptions::default());
+                                    with_isolated_s3_http_operation_context(None, async {
+                                        record_read_version_entry_to_rpc_context(&ReadOptions::default());
+                                    })
+                                    .await;
+                                    CURRENT_READ_VERSION_REQUEST_CALLER
+                                        .scope(std::cell::Cell::new(None), async {
+                                            record_read_version_entry_to_rpc_context(&ReadOptions::default());
+                                        })
+                                        .await;
+                                },
+                            ))
+                            .await
+                            .expect("external child should complete");
+                        }),
+                    ),
+                )
+                .await;
+                assert_eq!(ReadVersionDiagnosticContext::capture().entry, None);
+                record_read_version_entry_to_rpc_context(&ReadOptions::default());
+            })
+        });
+        let rows = snapshots.snapshot().into_vec();
+        let observed = |source: &str, state: &str, parent: &str, child: &str| {
+            rows.iter().any(|(key, _, _, value)| {
+                let labels = key
+                    .key()
+                    .labels()
+                    .map(|label| (label.key(), label.value()))
+                    .collect::<std::collections::HashMap<_, _>>();
+                key.key().name() == "rustfs_storage_diagnostic_context_transitions_total"
+                    && labels.get("entry_source") == Some(&source)
+                    && labels.get("boundary") == Some(&"remote_rpc")
+                    && labels.get("api_context") == Some(&state)
+                    && labels.get("parent_api_operation") == Some(&parent)
+                    && labels.get("child_api_operation") == Some(&child)
+                    && labels.get("mode") == Some(&"metadata_only")
+                    && matches!(value, DebugValue::Counter(1))
+            })
+        };
+        assert!(observed("config_read", "absent", "unattributed", "unattributed"));
+        assert!(observed("object_read", "preserved", "s3:GetObject", "s3:GetObject"));
+        assert!(observed("object_read", "lost", "s3:GetObject", "unattributed"));
+        assert!(observed("unclassified", "unobserved", "unattributed", "unattributed"));
+        assert!(
+            rows.iter().any(|(key, _, _, value)| {
+                let labels = key
+                    .key()
+                    .labels()
+                    .map(|label| (label.key(), label.value()))
+                    .collect::<std::collections::HashMap<_, _>>();
+                key.key().name() == "rustfs_storage_diagnostic_context_transitions_total"
+                    && labels.get("boundary") == Some(&"remote_rpc")
+                    && labels.get("parent_caller") == Some(&"object")
+                    && labels.get("child_caller") == Some(&"unattributed")
+                    && labels.get("caller_context") == Some(&"lost")
+                    && matches!(value, DebugValue::Counter(1))
+            }),
+            "RPC caller must be compared with its direct parent's seed, not an earlier unseeded entry"
+        );
+    }
+
+    #[cfg(feature = "diagnostic-read-version-caller")]
+    #[tokio::test]
+    async fn diagnostic_context_restores_ambient_scope_after_pending_item_is_cancelled() {
+        use std::future::Future;
+        let get = diagnostic_operation(rustfs_s3_ops::S3Operation::GetObject);
+        let parent = ReadVersionDiagnosticContext {
+            api_operation: Some(get),
+            caller: Some(ReadVersionRequestCaller::Object),
+            entry: Some(ReadVersionEntryContext::new(
+                ReadVersionEntrySource::ObjectRead,
+                Some(get),
+                Some(ReadVersionRequestCaller::Object),
+            )),
+        };
+        let internal = ReadVersionDiagnosticContext {
+            api_operation: None,
+            caller: None,
+            entry: Some(ReadVersionEntryContext::new(ReadVersionEntrySource::ConfigRead, None, None)),
+        };
+        with_read_version_diagnostic_context(parent, ReadVersionContextBoundary::Entry, async {
+            let mut pending = Box::pin(with_read_version_diagnostic_context(
+                internal,
+                ReadVersionContextBoundary::UnaryFallback,
+                async {
+                    assert_eq!(ReadVersionDiagnosticContext::capture(), internal);
+                    std::future::pending::<()>().await;
+                },
+            ));
+            std::future::poll_fn(|cx| {
+                assert!(pending.as_mut().poll(cx).is_pending());
+                std::task::Poll::Ready(())
+            })
+            .await;
+            assert_eq!(
+                ReadVersionDiagnosticContext::capture(),
+                parent,
+                "a pending item must not leak into its poller's scope"
+            );
+            drop(pending);
+            assert_eq!(
+                ReadVersionDiagnosticContext::capture(),
+                parent,
+                "cancellation must retain the ambient request"
+            );
+        })
+        .await;
+        assert_eq!(ReadVersionDiagnosticContext::capture().entry, None);
+        assert_eq!(current_s3_http_operation_context(), None);
+        assert_eq!(current_read_version_request_caller(), None);
+    }
+
+    #[cfg(feature = "diagnostic-read-version-caller")]
+    #[test]
+    fn diagnostic_batch_attributes_each_dimension_and_counts_one_rpc() {
+        use metrics_util::debugging::{DebugValue, DebuggingRecorder};
+        let runtime = tokio::runtime::Builder::new_current_thread()
+            .enable_all()
+            .build()
+            .expect("test runtime");
+        let recorder = DebuggingRecorder::new();
+        let snapshots = recorder.snapshotter();
+        rustfs_io_metrics::set_get_stage_metrics_enabled(true);
+        let get = diagnostic_operation(rustfs_s3_ops::S3Operation::GetObject);
+        let list = diagnostic_operation(rustfs_s3_ops::S3Operation::ListObjectsV2);
+        let context = |api, caller, source| ReadVersionDiagnosticContext {
+            api_operation: api,
+            caller,
+            entry: Some(ReadVersionEntryContext::new(source, api, caller)),
+        };
+        let external = context(Some(get), Some(ReadVersionRequestCaller::Object), ReadVersionEntrySource::ObjectRead);
+        let internal = context(None, None, ReadVersionEntrySource::ConfigRead);
+        let internal_object = context(None, Some(ReadVersionRequestCaller::Object), ReadVersionEntrySource::ConfigRead);
+        let heal = context(Some(get), Some(ReadVersionRequestCaller::Heal), ReadVersionEntrySource::ObjectRead);
+        let other_api = context(Some(list), Some(ReadVersionRequestCaller::Object), ReadVersionEntrySource::ObjectRead);
+        metrics::with_local_recorder(&recorder, || {
+            runtime.block_on(async {
+                for contexts in [
+                    vec![external, external],
+                    vec![external, internal],
+                    vec![external, internal_object],
+                    vec![external, heal],
+                    vec![external, other_api],
+                    vec![internal, internal],
+                ] {
+                    with_read_version_batch_contexts(contexts, async {
+                        assert_eq!(current_s3_http_operation_context(), None);
+                        assert_eq!(current_read_version_request_caller(), None);
+                        record_remote_disk_grpc_batch_read_version_request(&ReadOptions {
+                            read_data: true,
+                            ..Default::default()
+                        });
+                    })
+                    .await;
+                }
+            })
+        });
+        let labels = snapshots
+            .snapshot()
+            .into_vec()
+            .into_iter()
+            .filter(|(key, _, _, _)| {
+                key.key().name() == "rustfs_system_network_internode_batch_read_version_requests_by_caller_total"
+            })
+            .map(|(key, _, _, value)| {
+                assert!(matches!(value, DebugValue::Counter(1)), "batch must count one RPC, not its items");
+                let labels = key
+                    .key()
+                    .labels()
+                    .map(|label| (label.key().to_owned(), label.value().to_owned()))
+                    .collect::<std::collections::HashMap<_, _>>();
+                assert_eq!(labels["mode"], "data");
+                (labels["api_operation"].clone(), labels["caller"].clone(), labels["entry_source"].clone())
+            })
+            .collect::<std::collections::HashSet<_>>();
+        assert_eq!(labels.len(), 6);
+        for (api, caller, source) in [
+            ("s3:GetObject", "object", "object_read"),
+            ("mixed", "mixed", "mixed"),
+            ("mixed", "object", "mixed"),
+            ("s3:GetObject", "mixed", "object_read"),
+            ("mixed", "object", "object_read"),
+            ("unattributed", "unattributed", "config_read"),
+        ] {
+            assert!(labels.contains(&(api.to_owned(), caller.to_owned(), source.to_owned())));
+        }
+    }
+
+    #[test]
+    #[cfg(feature = "diagnostic-read-version-caller")]
+    fn read_version_request_mode_uses_only_bounded_read_flags() {
+        assert_eq!(read_version_request_mode(&ReadOptions::default()), ReadVersionRequestMode::MetadataOnly);
+        assert_eq!(
+            read_version_request_mode(&ReadOptions {
+                read_data: true,
+                ..Default::default()
+            }),
+            ReadVersionRequestMode::Data
+        );
+        assert_eq!(
+            read_version_request_mode(&ReadOptions {
+                healing: true,
+                ..Default::default()
+            }),
+            ReadVersionRequestMode::Healing
+        );
+        assert_eq!(
+            read_version_request_mode(&ReadOptions {
+                incl_free_versions: true,
+                ..Default::default()
+            }),
+            ReadVersionRequestMode::IncludeFreeVersions
+        );
+        assert_eq!(
+            read_version_request_mode(&ReadOptions {
+                read_data: true,
+                healing: true,
+                ..Default::default()
+            }),
+            ReadVersionRequestMode::Combined
+        );
+    }
+
+    #[cfg(feature = "diagnostic-read-version-caller")]
+    #[test]
+    fn read_version_request_caller_classification_uses_bounded_module_names() {
+        assert_eq!(
+            read_version_request_caller_from_location("/repo/crates/ecstore/src/store/peer.rs"),
+            ReadVersionRequestCaller::Peer
+        );
+        assert_eq!(
+            read_version_request_caller_from_location("/repo/crates/ecstore/src/set_disk/ops/heal.rs"),
+            ReadVersionRequestCaller::Heal
+        );
+        assert_eq!(
+            read_version_request_caller_from_location("/repo/crates/ecstore/src/store/object.rs"),
+            ReadVersionRequestCaller::Object
+        );
+        assert_eq!(
+            read_version_request_caller_from_location("/repo/crates/ecstore/src/set_disk/core/metadata_read.rs"),
+            ReadVersionRequestCaller::MetadataRead
+        );
+        assert_eq!(
+            read_version_request_caller_from_location("/repo/crates/ecstore/src/set_disk/ops/multipart.rs"),
+            ReadVersionRequestCaller::Multipart
+        );
+        assert_eq!(
+            read_version_request_caller_from_location("/repo/crates/ecstore/src/disk/disk_store.rs"),
+            ReadVersionRequestCaller::DiskStore
+        );
+        assert_eq!(
+            read_version_request_caller_from_location("/repo/crates/ecstore/src/other/module.rs"),
+            ReadVersionRequestCaller::Other
+        );
+        assert_eq!(
+            read_version_request_caller_from_location("/repo/external/module.rs"),
+            ReadVersionRequestCaller::Unattributed
+        );
+    }
+
+    #[cfg(feature = "diagnostic-read-version-caller")]
+    #[tokio::test]
+    async fn read_version_request_caller_context_is_scoped_to_the_read_future() {
+        assert_eq!(current_read_version_request_caller(), None);
+        let caller =
+            with_read_version_request_caller(ReadVersionRequestCaller::Peer, async { current_read_version_request_caller() })
+                .await;
+        assert_eq!(caller, Some(ReadVersionRequestCaller::Peer));
+        assert_eq!(current_read_version_request_caller(), None);
+    }
+}

crates/ecstore/src/config/com.rs

@@ -574,17 +574,20 @@ where
         >,
 {
     let h = HeaderMap::new();
-    let mut rd = api
-        .get_object_reader(RUSTFS_META_BUCKET, file, None, h, opts)
-        .await
-        .map_err(|err| {
-            if err == Error::FileNotFound || matches!(err, Error::ObjectNotFound(_, _)) {
-                Error::ConfigNotFound
-            } else {
-                warn!("read_config_with_metadata: err: {:?}, file: {}", err, file);
-                err
-            }
-        })?;
+    let read = api.get_object_reader(RUSTFS_META_BUCKET, file, None, h, opts);
+    #[cfg(feature = "diagnostic-read-version-caller")]
+    let read = crate::cluster::rpc::runtime_sources::with_read_version_entry(
+        rustfs_io_metrics::internode_metrics::ReadVersionEntrySource::ConfigRead,
+        read,
+    );
+    let mut rd = read.await.map_err(|err| {
+        if err == Error::FileNotFound || matches!(err, Error::ObjectNotFound(_, _)) {
+            Error::ConfigNotFound
+        } else {
+            warn!("read_config_with_metadata: err: {:?}, file: {}", err, file);
+            err
+        }
+    })?;
 
     let data = if let Some(max_bytes) = max_bytes {
         let object_size = usize::try_from(rd.object_info.size).map_err(|_| Error::CorruptedFormat)?;
@@ -807,7 +810,18 @@ where
         >,
 {
     let mut put_data = PutObjReader::from_vec(data);
-    match api.put_object(RUSTFS_META_BUCKET, file, &mut put_data, opts).await {
+    let write = api.put_object(RUSTFS_META_BUCKET, file, &mut put_data, opts);
+    #[cfg(feature = "diagnostic-read-version-caller")]
+    let write = crate::cluster::rpc::runtime_sources::with_read_version_entry(
+        rustfs_io_metrics::internode_metrics::ReadVersionEntrySource::ConfigWrite,
+        async {
+            crate::cluster::rpc::runtime_sources::observe_read_version_context_boundary(
+                rustfs_io_metrics::internode_metrics::ReadVersionContextBoundary::ConfigWrite,
+            );
+            write.await
+        },
+    );
+    match write.await {
         Ok(object_info) => Ok(object_info),
         Err(err) => {
             if log_error {
@@ -2869,6 +2883,11 @@ mod tests {
     #[derive(Debug, Default)]
     struct MetadataWriteStore {
         error: Option<Error>,
+        #[cfg(feature = "diagnostic-read-version-caller")]
+        observations:
+            std::sync::Mutex<Vec<(Option<rustfs_io_metrics::internode_metrics::ReadVersionEntrySource>, Option<&'static str>)>>,
+        #[cfg(feature = "diagnostic-read-version-caller")]
+        pending: bool,
     }
 
     #[async_trait::async_trait]
@@ -2899,7 +2918,20 @@ mod tests {
             _data: &mut Self::PutObjectReader,
             _opts: &Self::ObjectOptions,
         ) -> core::result::Result<Self::ObjectInfo, Self::Error> {
-            Err(self.error.clone().expect("test store error should be configured"))
+            #[cfg(feature = "diagnostic-read-version-caller")]
+            {
+                self.observations.lock().expect("observations lock").push((
+                    crate::cluster::rpc::runtime_sources::current_read_version_entry_source(),
+                    rustfs_io_metrics::s3_http_metrics::current_s3_http_operation_name(),
+                ));
+                if self.pending {
+                    std::future::pending::<()>().await;
+                }
+            }
+            match self.error.clone() {
+                Some(error) => Err(error),
+                None => Ok(ObjectInfo::default()),
+            }
         }
     }
 
@@ -2907,6 +2939,7 @@ mod tests {
     async fn save_config_preserves_retryable_system_volume_errors() {
         let store = Arc::new(MetadataWriteStore {
             error: Some(Error::BucketNotFound(RUSTFS_META_BUCKET.to_string())),
+            ..Default::default()
         });
         let error =
             save_config_with_opts_inner(store, "buckets/example/.metadata.bin", Vec::new(), &ObjectOptions::default(), false)
@@ -2918,6 +2951,59 @@ mod tests {
             Error::InsufficientWriteQuorum(RUSTFS_META_BUCKET.to_string(), "buckets/example/.metadata.bin".to_string())
         );
     }
+
+    #[cfg(feature = "diagnostic-read-version-caller")]
+    #[tokio::test]
+    async fn diagnostic_config_write_retains_origin_and_api_for_results_and_cancellation() {
+        use crate::cluster::rpc::runtime_sources::{current_read_version_entry_source, with_read_version_entry};
+        use rustfs_io_metrics::internode_metrics::ReadVersionEntrySource;
+        use rustfs_io_metrics::s3_http_metrics::{S3HttpRequestGuard, current_s3_http_operation_context};
+
+        let mut guard = S3HttpRequestGuard::new("PUT");
+        let put = guard.in_scope(|| {
+            rustfs_io_metrics::record_s3_op(rustfs_s3_ops::S3Operation::PutObject);
+            current_s3_http_operation_context().expect("captured PutObject")
+        });
+        for earlier in [None, Some(ReadVersionEntrySource::TierConfigReload)] {
+            for api in [None, Some(put)] {
+                for case in ["success", "error", "cancelled"] {
+                    let store = Arc::new(MetadataWriteStore {
+                        error: (case == "error").then_some(Error::VolumeNotFound),
+                        pending: case == "cancelled",
+                        ..Default::default()
+                    });
+                    let opts = ObjectOptions::default();
+                    let write = async {
+                        let result = tokio::time::timeout(
+                            std::time::Duration::from_secs(1),
+                            save_config_with_opts_inner(store.clone(), "buckets/diagnostic/state", Vec::new(), &opts, false),
+                        )
+                        .await;
+                        match case {
+                            "success" => assert!(result.expect("write completes").is_ok()),
+                            "error" => {
+                                assert!(matches!(result.expect("error completes"), Err(Error::InsufficientWriteQuorum(_, _))))
+                            }
+                            _ => assert!(result.is_err(), "pending write is cancelled"),
+                        }
+                    };
+                    rustfs_io_metrics::s3_http_metrics::with_isolated_s3_http_operation_context(api, async {
+                        match earlier {
+                            Some(source) => with_read_version_entry(source, write).await,
+                            None => write.await,
+                        }
+                    })
+                    .await;
+                    assert_eq!(
+                        *store.observations.lock().expect("observations lock"),
+                        vec![(Some(earlier.unwrap_or(ReadVersionEntrySource::ConfigWrite)), api.map(|_| "s3:PutObject"))]
+                    );
+                    assert_eq!(current_read_version_entry_source(), None, "write scope restores: {case}");
+                    assert_eq!(current_s3_http_operation_context(), None, "API scope restores: {case}");
+                }
+            }
+        }
+    }
     use rustfs_lock::client::LockClient;
     use rustfs_lock::client::local::LocalClient;
     use rustfs_lock::{LockError, LockInfo, LockResponse, LockStats};

crates/ecstore/src/core/pools.rs

@@ -3691,55 +3691,6 @@ fn activation_pool_meta_lock_error(err: rustfs_lock::LockError) -> Error {
     }
 }
 
-fn rebalance_worker_read_lock_error(err: rustfs_lock::LockError, object: &str) -> Error {
-    match err {
-        rustfs_lock::LockError::QuorumNotReached { required, achieved } => Error::NamespaceLockQuorumUnavailable {
-            mode: "read",
-            bucket: RUSTFS_META_BUCKET.to_string(),
-            object: object.to_string(),
-            required,
-            achieved,
-        },
-        other => Error::Lock(other),
-    }
-}
-
-/// Read-only admission to an already durable rebalance run. Keeping this
-/// separate from the activation writer prevents admission from authorizing a
-/// pool or rebalance metadata write under shared namespace locks.
-pub(crate) struct PoolRebalanceWorkerFence {
-    pool_meta_guard: rustfs_lock::NamespaceLockGuard,
-    rebalance_meta_guard: rustfs_lock::NamespaceLockGuard,
-    fleet_proof: Option<crate::services::notification_sys::CrossPoolFenceFleetProofToken>,
-}
-
-impl PoolRebalanceWorkerFence {
-    pub(crate) fn set_fleet_proof(
-        &mut self,
-        fleet_proof: Option<crate::services::notification_sys::CrossPoolFenceFleetProofToken>,
-    ) {
-        self.fleet_proof = fleet_proof;
-    }
-
-    pub(crate) fn ensure_held(&self) -> Result<()> {
-        if self.pool_meta_guard.is_lock_lost()
-            || self.pool_meta_guard.is_released()
-            || self.rebalance_meta_guard.is_lock_lost()
-            || self.rebalance_meta_guard.is_released()
-        {
-            return Err(Error::other("worker admission lock lost before rebalance activation"));
-        }
-        if self
-            .fleet_proof
-            .as_ref()
-            .is_some_and(|proof| !crate::services::notification_sys::cross_pool_fence_fleet_proof_matches(proof))
-        {
-            return Err(Error::other(POOL_ACTIVATION_FLEET_PROOF_EXPIRED));
-        }
-        Ok(())
-    }
-}
-
 pub(crate) struct PoolRebalanceActivationFence {
     pool_meta_guard: Arc<rustfs_lock::NamespaceLockGuard>,
     rebalance_meta_guard: Arc<rustfs_lock::NamespaceLockGuard>,
@@ -3811,35 +3762,6 @@ tokio::task_local! {
     pub(crate) static REBALANCE_ACTIVATION_LOCK_ATTEMPT: Arc<tokio::sync::Notify>;
 }
 
-pub(crate) async fn acquire_pool_rebalance_worker_locks<S>(pool: Arc<S>) -> Result<PoolRebalanceWorkerFence>
-where
-    S: crate::storage_api_contracts::namespace::NamespaceLocking<
-            Error = Error,
-            NamespaceLock = rustfs_lock::NamespaceLockWrapper,
-        >,
-{
-    // Match activation and entry admission: rebalance.bin -> pool.bin. Joining
-    // the same durable run must coexist with entries retaining their run fence.
-    let rebalance_meta_lock = pool.new_ns_lock(RUSTFS_META_BUCKET, REBAL_META_NAME).await?;
-    #[cfg(test)]
-    let _ = REBALANCE_ACTIVATION_LOCK_ATTEMPT.try_with(|attempted| attempted.notify_one());
-    let rebalance_meta_guard = rebalance_meta_lock
-        .get_read_lock(get_lock_acquire_timeout())
-        .await
-        .map_err(|err| rebalance_worker_read_lock_error(err, REBAL_META_NAME))?;
-    let pool_meta_lock = pool.new_ns_lock(RUSTFS_META_BUCKET, POOL_META_NAME).await?;
-    let pool_meta_guard = pool_meta_lock
-        .get_read_lock(get_lock_acquire_timeout())
-        .await
-        .map_err(|err| rebalance_worker_read_lock_error(err, POOL_META_NAME))?;
-
-    Ok(PoolRebalanceWorkerFence {
-        pool_meta_guard,
-        rebalance_meta_guard,
-        fleet_proof: None,
-    })
-}
-
 pub(crate) async fn acquire_pool_rebalance_activation_locks<S>(
     pool: Arc<S>,
     fleet_proof: Option<crate::services::notification_sys::CrossPoolFenceFleetProofToken>,
@@ -12508,18 +12430,6 @@ impl ECStore {
         Ok(pool_meta)
     }
 
-    pub(crate) async fn load_runtime_pool_meta_under_worker_fence(
-        &self,
-        write_state: &mut PoolMetaWriteState,
-        worker_fence: &PoolRebalanceWorkerFence,
-        operation: &str,
-    ) -> Result<PoolMeta> {
-        worker_fence.ensure_held()?;
-        let pool_meta = self.load_runtime_pool_meta_observing(write_state, operation).await?;
-        worker_fence.ensure_held()?;
-        Ok(pool_meta)
-    }
-
     pub(crate) async fn load_runtime_pool_meta_under_activation_fence(
         &self,
         write_state: &mut PoolMetaWriteState,

crates/ecstore/src/data_movement/scanner_backlog.rs

@@ -110,19 +110,17 @@ async fn read_replica(set: Arc<SetDisks>) -> Result<ScannerPauseBacklogRetiremen
         set_index: set.set_index,
         data: None,
     };
-    let reader = match set
-        .get_object_reader(
-            RUSTFS_META_BUCKET,
-            SCANNER_PAUSE_BACKLOG_PATH,
-            None,
-            HeaderMap::new(),
-            &ObjectOptions {
-                no_lock: true,
-                ..Default::default()
-            },
-        )
-        .await
-    {
+    let opts = ObjectOptions {
+        no_lock: true,
+        ..Default::default()
+    };
+    let read = set.get_object_reader(RUSTFS_META_BUCKET, SCANNER_PAUSE_BACKLOG_PATH, None, HeaderMap::new(), &opts);
+    #[cfg(feature = "diagnostic-read-version-caller")]
+    let read = crate::cluster::rpc::runtime_sources::with_read_version_entry(
+        rustfs_io_metrics::internode_metrics::ReadVersionEntrySource::ScannerBacklogRead,
+        read,
+    );
+    let reader = match read.await {
         Ok(reader) => reader,
         Err(err) if is_err_object_not_found(&err) || is_err_version_not_found(&err) => {
             return Ok(ScannerPauseBacklogRetirementRead { replica, etag: None });

crates/ecstore/src/data_usage/mod.rs

@@ -2529,19 +2529,17 @@ where
     use crate::object_api::ObjectOptions;
     use http::HeaderMap;
 
-    match store
-        .get_object_reader(
-            RUSTFS_META_BUCKET,
-            key,
-            None,
-            HeaderMap::new(),
-            &ObjectOptions {
-                no_lock: true,
-                ..Default::default()
-            },
-        )
-        .await
-    {
+    let opts = ObjectOptions {
+        no_lock: true,
+        ..Default::default()
+    };
+    let read = store.get_object_reader(RUSTFS_META_BUCKET, key, None, HeaderMap::new(), &opts);
+    #[cfg(feature = "diagnostic-read-version-caller")]
+    let read = crate::cluster::rpc::runtime_sources::with_read_version_entry(
+        rustfs_io_metrics::internode_metrics::ReadVersionEntrySource::DataUsageCacheRead,
+        read,
+    );
+    match read.await {
         // A cache object that fails to decode is treated as absent rather than
         // as an error: a corrupt cache should not stall the caller, and the
         // next scanner pass rewrites it.
@@ -3064,6 +3062,8 @@ mod tests {
         transient_failures: Mutex<usize>,
         reads: Mutex<Vec<String>>,
         terminal_error: Mutex<Option<Error>>,
+        #[cfg(feature = "diagnostic-read-version-caller")]
+        record_context: bool,
     }
 
     impl UsageCacheReadStore {
@@ -3072,6 +3072,8 @@ mod tests {
                 transient_failures: Mutex::new(n),
                 reads: Mutex::new(Vec::new()),
                 terminal_error: Mutex::new(None),
+                #[cfg(feature = "diagnostic-read-version-caller")]
+                record_context: false,
             }
         }
 
@@ -3080,6 +3082,8 @@ mod tests {
                 transient_failures: Mutex::new(0),
                 reads: Mutex::new(Vec::new()),
                 terminal_error: Mutex::new(Some(error)),
+                #[cfg(feature = "diagnostic-read-version-caller")]
+                record_context: false,
             }
         }
 
@@ -3106,6 +3110,12 @@ mod tests {
             _h: Self::HeaderMap,
             _opts: &Self::ObjectOptions,
         ) -> Result<Self::GetObjectReader, Self::Error> {
+            #[cfg(feature = "diagnostic-read-version-caller")]
+            if self.record_context {
+                crate::cluster::rpc::runtime_sources::record_remote_disk_grpc_read_version_request(
+                    &crate::disk::ReadOptions::default(),
+                );
+            }
             self.reads.lock().await.push(object.to_string());
             if let Some(error) = self.terminal_error.lock().await.clone() {
                 return Err(error);
@@ -3205,6 +3215,89 @@ mod tests {
         );
     }
 
+    #[cfg(feature = "diagnostic-read-version-caller")]
+    #[test]
+    fn diagnostic_usage_cache_read_preserves_api_and_earlier_entry() {
+        use crate::cluster::rpc::runtime_sources::with_read_version_entry;
+        use metrics_util::debugging::{DebugValue, DebuggingRecorder};
+        use rustfs_io_metrics::internode_metrics::ReadVersionEntrySource;
+        use rustfs_io_metrics::s3_http_metrics::{
+            S3HttpRequestGuard, current_s3_http_operation_context, with_s3_http_operation_context,
+        };
+
+        let runtime = tokio::runtime::Builder::new_current_thread()
+            .enable_all()
+            .build()
+            .expect("diagnostic usage cache runtime");
+        let recorder = DebuggingRecorder::new();
+        let snapshots = recorder.snapshotter();
+        rustfs_io_metrics::set_get_stage_metrics_enabled(true);
+        let mut request = S3HttpRequestGuard::new("GET");
+        let get = request.in_scope(|| {
+            rustfs_io_metrics::record_s3_op(rustfs_s3_ops::S3Operation::GetObject);
+            current_s3_http_operation_context().expect("dispatched GET operation")
+        });
+        request.response(200);
+        metrics::with_local_recorder(&recorder, || {
+            runtime.block_on(async {
+                let ambient = crate::cluster::rpc::runtime_sources::ReadVersionDiagnosticContext::capture();
+                for (operation, source) in [
+                    (None, None),
+                    (Some(get), None),
+                    (None, Some(ReadVersionEntrySource::ServerInfoPools)),
+                    (Some(get), Some(ReadVersionEntrySource::StoreObjectReader)),
+                ] {
+                    let store = UsageCacheReadStore {
+                        record_context: true,
+                        ..Default::default()
+                    };
+                    with_s3_http_operation_context(operation, async {
+                        let load = load_data_usage_cache(&store, "usage-cache");
+                        let cache = match source {
+                            Some(source) => with_read_version_entry(source, load).await,
+                            None => load.await,
+                        }
+                        .expect("missing diagnostic cache is harmless");
+                        assert!(cache.cache.is_empty());
+                        assert_eq!(store.read_keys().await.len(), 2, "both prefixed and legacy cache reads are observed");
+                    })
+                    .await;
+                    assert_eq!(
+                        crate::cluster::rpc::runtime_sources::ReadVersionDiagnosticContext::capture(),
+                        ambient,
+                        "cache reads must restore the caller's complete diagnostic scope"
+                    );
+                }
+                assert_eq!(current_s3_http_operation_context(), None);
+            });
+        });
+        let rows = snapshots.snapshot().into_vec();
+        for (source, operation, state) in [
+            ("data_usage_cache_read", "unattributed", "absent"),
+            ("data_usage_cache_read", "s3:GetObject", "preserved"),
+            ("server_info_pools", "unattributed", "absent"),
+            ("store_object_reader", "s3:GetObject", "preserved"),
+        ] {
+            assert!(
+                rows.iter().any(|(key, _, _, value)| {
+                    let labels = key
+                        .key()
+                        .labels()
+                        .map(|label| (label.key(), label.value()))
+                        .collect::<std::collections::HashMap<_, _>>();
+                    key.key().name() == "rustfs_storage_diagnostic_context_transitions_total"
+                        && labels.get("entry_source") == Some(&source)
+                        && labels.get("boundary") == Some(&"remote_rpc")
+                        && labels.get("parent_api_operation") == Some(&operation)
+                        && labels.get("child_api_operation") == Some(&operation)
+                        && labels.get("api_context") == Some(&state)
+                        && matches!(value, DebugValue::Counter(2))
+                }),
+                "cache reads must retain API and the earliest observed entry: {source}/{operation}"
+            );
+        }
+    }
+
     #[tokio::test]
     async fn data_usage_removal_surfaces_missing_system_volume_as_read_quorum() {
         for cause in [Error::BucketNotFound(RUSTFS_META_BUCKET.to_string()), Error::VolumeNotFound] {

crates/ecstore/src/diagnostics/admin_server_info.rs

@@ -15,9 +15,7 @@
 use crate::cluster::rpc::{
     ScannerBucketListing, TonicInterceptor, gen_tonic_signature_interceptor, node_service_time_out_client,
 };
-use crate::data_usage::{
-    DATA_USAGE_CACHE_NAME, DATA_USAGE_ROOT, load_admin_data_usage_from_backend_cached as load_server_info_usage_snapshot,
-};
+use crate::data_usage::{DATA_USAGE_CACHE_NAME, DATA_USAGE_ROOT, load_data_usage_from_backend_cached};
 use crate::error::{Error, Result};
 use crate::{
     disk::endpoint::{Endpoint, EndpointType},
@@ -311,7 +309,7 @@ pub async fn get_server_info(get_pools: bool) -> InfoMessage {
     if let Some(store) = runtime_sources::object_store_handle() {
         mode = ITEM_ONLINE;
         apply_data_usage_result(
-            load_server_info_usage_snapshot(store.clone()).await,
+            load_data_usage_from_backend_cached(store.clone()).await,
             &mut buckets,
             &mut objects,
             &mut versions,
@@ -695,12 +693,16 @@ async fn get_pools_info(all_disks: &[Disk]) -> Result<HashMap<i32, HashMap<i32,
 
         if erasure_set.id == 0 {
             erasure_set.id = d.set_index;
-            match load_data_usage_cache(
+            let load = load_data_usage_cache(
                 store.pools[d.pool_index as usize].disk_set[d.set_index as usize].as_ref(),
                 DATA_USAGE_CACHE_NAME,
-            )
-            .await
-            {
+            );
+            #[cfg(feature = "diagnostic-read-version-caller")]
+            let load = crate::cluster::rpc::runtime_sources::with_read_version_entry(
+                rustfs_io_metrics::internode_metrics::ReadVersionEntrySource::ServerInfoPools,
+                load,
+            );
+            match load.await {
                 Ok(cache) => apply_erasure_set_usage(&cache, erasure_set),
                 Err(_) => erasure_set.usage_error = Some(DATA_USAGE_UNAVAILABLE_ERROR.to_string()),
             }
@@ -984,93 +986,6 @@ mod tests {
         assert_eq!(usage.size, 6);
     }
 
-    #[tokio::test]
-    #[serial]
-    async fn server_info_reads_complete_observation_without_promoting_authority() {
-        use crate::bucket::metadata_sys::test_support::isolated_store_over_temp_disks;
-        use crate::config::com::{read_config, save_config};
-        use crate::data_usage::{
-            DATA_USAGE_OBJ_NAME_PATH, DATA_USAGE_OBSERVED_OBJ_NAME_PATH, invalidate_data_usage_snapshot_cache,
-            load_data_usage_from_backend_cached,
-        };
-        use rustfs_data_usage::{BucketUsageInfo, DataUsageInfo};
-        use std::time::{Duration, SystemTime};
-
-        let (_dirs, store) = isolated_store_over_temp_disks().await;
-        let primary = DataUsageInfo {
-            last_update: Some(SystemTime::UNIX_EPOCH + Duration::from_secs(1)),
-            scanner_epoch: Some(1),
-            usage_snapshot_bootstrap_pending: true,
-            usage_snapshot_converged: Some(false),
-            ..Default::default()
-        };
-        let observed = DataUsageInfo {
-            last_update: Some(SystemTime::UNIX_EPOCH + Duration::from_secs(2)),
-            scanner_epoch: Some(1),
-            scanner_cycle: Some(1),
-            buckets_count: 1,
-            objects_total_count: 3,
-            versions_total_count: 4,
-            delete_markers_total_count: 1,
-            objects_total_size: 6,
-            buckets_usage: [(
-                "bucket".to_string(),
-                BucketUsageInfo {
-                    objects_count: 3,
-                    versions_count: 4,
-                    delete_markers_count: 1,
-                    size: 6,
-                    ..Default::default()
-                },
-            )]
-            .into_iter()
-            .collect(),
-            usage_snapshot_complete: true,
-            usage_snapshot_converged: Some(false),
-            usage_snapshot_authoritative_baseline: Some(primary.snapshot_identity()),
-            ..Default::default()
-        };
-        let primary_bytes = serde_json::to_vec(&primary).expect("bootstrap should encode");
-        save_config(store.clone(), DATA_USAGE_OBJ_NAME_PATH.as_str(), primary_bytes.clone())
-            .await
-            .expect("bootstrap should persist");
-        save_config(
-            store.clone(),
-            DATA_USAGE_OBSERVED_OBJ_NAME_PATH.as_str(),
-            serde_json::to_vec(&observed).expect("observation should encode"),
-        )
-        .await
-        .expect("observation should persist");
-        invalidate_data_usage_snapshot_cache().await;
-
-        let displayed = super::load_server_info_usage_snapshot(store.clone())
-            .await
-            .expect("server info should load the complete observation");
-        assert_eq!(displayed.usage_snapshot_converged, Some(false));
-        let mut buckets = rustfs_madmin::Buckets::default();
-        let mut objects = rustfs_madmin::Objects::default();
-        let mut versions = rustfs_madmin::Versions::default();
-        let mut delete_markers = rustfs_madmin::DeleteMarkers::default();
-        let mut usage = rustfs_madmin::Usage::default();
-        apply_data_usage_result(Ok(displayed), &mut buckets, &mut objects, &mut versions, &mut delete_markers, &mut usage);
-        assert!(objects.error.is_none());
-        assert_eq!(
-            (buckets.count, objects.count, versions.count, delete_markers.count, usage.size),
-            (1, 3, 4, 1, 6)
-        );
-        let quota = load_data_usage_from_backend_cached(store.clone())
-            .await
-            .expect("quota should retain its bootstrap state");
-        assert!(!quota.is_complete_bucket_usage_snapshot());
-        assert_eq!(
-            read_config(store, DATA_USAGE_OBJ_NAME_PATH.as_str())
-                .await
-                .expect("primary should remain readable"),
-            primary_bytes
-        );
-        invalidate_data_usage_snapshot_cache().await;
-    }
-
     #[test]
     fn incomplete_data_usage_is_unavailable_in_server_info() {
         let mut buckets = rustfs_madmin::Buckets::default();

crates/ecstore/src/diagnostics/get.rs

@@ -182,7 +182,6 @@ pub(crate) const GET_METADATA_CACHE_REASON_DELETE_MARKER: &str = "delete_marker"
 pub(crate) const GET_METADATA_CACHE_REASON_DIST_ERASURE: &str = "dist_erasure";
 pub(crate) const GET_METADATA_CACHE_REASON_INCL_FREE_VERSIONS: &str = "incl_free_versions";
 pub(crate) const GET_METADATA_CACHE_REASON_INSUFFICIENT_CACHED_QUORUM: &str = "insufficient_cached_quorum";
-pub(crate) const GET_METADATA_CACHE_REASON_BODY_LAYOUT_REQUIRED: &str = "body_layout_required";
 pub(crate) const GET_METADATA_CACHE_REASON_META_BUCKET: &str = "meta_bucket";
 pub(crate) const GET_METADATA_CACHE_REASON_NO_LOCK: &str = "no_lock";
 pub(crate) const GET_METADATA_CACHE_REASON_NOT_FOUND_OR_EXPIRED: &str = "not_found_or_expired";

crates/ecstore/src/disk/disk_store.rs

@@ -467,11 +467,12 @@ impl LocalDiskWrapper {
         self.track_disk_health_mutation(
             "delete_version",
             DiskMetricMutation::Delete,
-            || {
+            || async {
                 Box::pin(
                     self.disk
                         .delete_version_with_namespace_owner(volume, path, fi, force_del_marker, opts, namespace_owner),
                 )
+                .await
             },
             get_max_timeout_duration(),
         )
@@ -1839,8 +1840,6 @@ impl LocalDiskWrapper {
                     undo_delete: false,
                     old_data_dir: None,
                     expected_delete_marker: None,
-                    recoverable_delete: false,
-                    delete_recovery_bucket_incarnation: None,
                 },
             )
             .await?;
@@ -2026,86 +2025,10 @@ impl LocalDiskWrapper {
     }
 
     /// Get the current disk ID
-    pub(crate) async fn delete_versions_with_namespace_owner(
-        &self,
-        volume: &str,
-        versions: Vec<FileInfoVersions>,
-        opts: DeleteOptions,
-        namespace_owner: Option<Arc<dyn Send + Sync>>,
-    ) -> Vec<Option<Error>> {
-        let version_count = versions.len();
-        self.metrics.record_operation_call("delete_versions");
-        // Check if disk is faulty before proceeding
-        if self.health.is_faulty() {
-            self.metrics.record_availability_error();
-            return vec![Some(DiskError::FaultyDisk); versions.len()];
-        }
-
-        // Check if disk is stale
-        if let Err(e) = self.check_disk_stale().await {
-            self.metrics.record_availability_error();
-            return vec![Some(e); versions.len()];
-        }
-
-        let _io_permit = match self.acquire_disk_io_permit().await {
-            Ok(permit) => permit,
-            Err(error) => return vec![Some(error); version_count],
-        };
-
-        // Record operation start
-        self.health.last_started.store(current_unix_nanos(), Ordering::Relaxed);
-        let waiting_guard = self.health.waiting_guard();
-        let metric_waiting_guard = self.metrics.waiting_guard();
-        let started = Instant::now();
-
-        // Execute the operation
-        let result = self
-            .disk
-            .delete_versions_with_namespace_owner(volume, versions, opts, namespace_owner)
-            .await;
-        self.metrics.record_operation_latency("delete_versions", started.elapsed());
-        self.record_batch_delete_error_metrics(&result);
-
-        drop(waiting_guard);
-        drop(metric_waiting_guard);
-        let has_err = result.iter().any(|e| e.is_some());
-        if !has_err {
-            // Log success and decrement waiting counter
-            self.health.record_operation_success(&self.endpoint(), "operation_success");
-            self.metrics.record_mutation_success(DiskMetricMutation::Delete);
-        }
-
-        result
-    }
-
     pub async fn get_current_disk_id(&self) -> Option<Uuid> {
         *self.disk_id.read().await
     }
 
-    pub(crate) async fn settle_delete_transaction_with_namespace_owner(
-        &self,
-        volume: &str,
-        object: &str,
-        transaction: uuid::Uuid,
-        expected_fingerprint: crate::disk::DeleteRecoveryDigest,
-        action: crate::disk::DeleteRecoveryAction,
-        namespace_owner: Option<Arc<dyn Send + Sync>>,
-    ) -> Result<crate::disk::DeleteRecoveryInspection> {
-        self.track_disk_health_mutation(
-            "settle_delete_transaction",
-            DiskMetricMutation::Delete,
-            || {
-                Box::pin(async {
-                    self.disk
-                        .delete_recovery_settle(volume, object, transaction, expected_fingerprint, action, namespace_owner)
-                        .await
-                })
-            },
-            crate::disk::get_delete_recovery_timeout()?,
-        )
-        .await
-    }
-
     /// Track disk health for an operation.
     /// This method should wrap disk operations to ensure health checking.
     pub async fn track_disk_health<T, F, Fut>(&self, operation: F, timeout_duration: Duration) -> Result<T>
@@ -2321,87 +2244,6 @@ impl LocalDiskWrapper {
 
 #[async_trait::async_trait]
 impl DiskAPI for LocalDiskWrapper {
-    async fn delete_version_recoverable(
-        &self,
-        volume: &str,
-        object: &str,
-        file_info: FileInfo,
-        force_del_marker: bool,
-        mut opts: DeleteOptions,
-    ) -> Result<()> {
-        crate::disk::validate_delete_recovery_options(&opts)?;
-        if object.starts_with('/') {
-            return Err(DiskError::InvalidPath);
-        }
-        opts.recoverable_delete = true;
-        self.delete_version(volume, object, file_info, force_del_marker, opts).await
-    }
-    async fn delete_versions_recoverable(
-        &self,
-        volume: &str,
-        versions: Vec<FileInfoVersions>,
-        mut opts: DeleteOptions,
-    ) -> Vec<Option<DiskError>> {
-        if let Err(err) = crate::disk::validate_delete_recovery_options(&opts) {
-            return vec![Some(err); versions.len()];
-        }
-        if versions.iter().any(|version| version.name.starts_with('/')) {
-            return vec![Some(DiskError::InvalidPath); versions.len()];
-        }
-        opts.recoverable_delete = true;
-        self.delete_versions(volume, versions, opts).await
-    }
-    async fn discover_delete_transactions(
-        &self,
-        volume: &str,
-        object: Option<&str>,
-        cursor: Option<crate::disk::DeleteRecoveryCursor>,
-        limit: u32,
-    ) -> Result<crate::disk::DeleteRecoveryPage> {
-        self.track_disk_health_with_op(
-            "discover_delete_transactions",
-            || async {
-                self.disk
-                    .discover_delete_transactions(volume, object, cursor.clone(), limit)
-                    .await
-            },
-            get_max_timeout_duration(),
-        )
-        .await
-    }
-    async fn inspect_delete_transaction(
-        &self,
-        volume: &str,
-        object: &str,
-        transaction: uuid::Uuid,
-    ) -> Result<crate::disk::DeleteRecoveryInspection> {
-        self.track_disk_health_with_op(
-            "inspect_delete_transaction",
-            || async { self.disk.inspect_delete_transaction(volume, object, transaction).await },
-            get_max_timeout_duration(),
-        )
-        .await
-    }
-    async fn settle_delete_transaction(
-        &self,
-        volume: &str,
-        object: &str,
-        transaction: uuid::Uuid,
-        expected_fingerprint: crate::disk::DeleteRecoveryDigest,
-        action: crate::disk::DeleteRecoveryAction,
-    ) -> Result<crate::disk::DeleteRecoveryInspection> {
-        self.track_disk_health_mutation(
-            "settle_delete_transaction",
-            DiskMetricMutation::Delete,
-            || async {
-                self.disk
-                    .settle_delete_transaction(volume, object, transaction, expected_fingerprint, action)
-                    .await
-            },
-            crate::disk::get_delete_recovery_timeout()?,
-        )
-        .await
-    }
     fn has_replacement_mount_lease(&self) -> bool {
         self.disk.has_replacement_mount_lease()
     }
@@ -2615,7 +2457,46 @@ impl DiskAPI for LocalDiskWrapper {
     }
 
     async fn delete_versions(&self, volume: &str, versions: Vec<FileInfoVersions>, opts: DeleteOptions) -> Vec<Option<Error>> {
-        self.delete_versions_with_namespace_owner(volume, versions, opts, None).await
+        let version_count = versions.len();
+        self.metrics.record_operation_call("delete_versions");
+        // Check if disk is faulty before proceeding
+        if self.health.is_faulty() {
+            self.metrics.record_availability_error();
+            return vec![Some(DiskError::FaultyDisk); versions.len()];
+        }
+
+        // Check if disk is stale
+        if let Err(e) = self.check_disk_stale().await {
+            self.metrics.record_availability_error();
+            return vec![Some(e); versions.len()];
+        }
+
+        let _io_permit = match self.acquire_disk_io_permit().await {
+            Ok(permit) => permit,
+            Err(error) => return vec![Some(error); version_count],
+        };
+
+        // Record operation start
+        self.health.last_started.store(current_unix_nanos(), Ordering::Relaxed);
+        let waiting_guard = self.health.waiting_guard();
+        let metric_waiting_guard = self.metrics.waiting_guard();
+        let started = Instant::now();
+
+        // Execute the operation
+        let result = self.disk.delete_versions(volume, versions, opts).await;
+        self.metrics.record_operation_latency("delete_versions", started.elapsed());
+        self.record_batch_delete_error_metrics(&result);
+
+        drop(waiting_guard);
+        drop(metric_waiting_guard);
+        let has_err = result.iter().any(|e| e.is_some());
+        if !has_err {
+            // Log success and decrement waiting counter
+            self.health.record_operation_success(&self.endpoint(), "operation_success");
+            self.metrics.record_mutation_success(DiskMetricMutation::Delete);
+        }
+
+        result
     }
 
     async fn delete_paths(&self, volume: &str, paths: &[String]) -> Result<()> {
@@ -2743,6 +2624,7 @@ impl DiskAPI for LocalDiskWrapper {
         .await
     }
 
+    #[track_caller]
     async fn read_version(
         &self,
         org_volume: &str,
@@ -2751,7 +2633,7 @@ impl DiskAPI for LocalDiskWrapper {
         version_id: &str,
         opts: &ReadOptions,
     ) -> Result<FileInfo> {
-        self.track_disk_health_with_op_timeout_action_and_mutation_with_io_permit(
+        let operation = self.track_disk_health_with_op_timeout_action_and_mutation_with_io_permit(
             "read_version",
             |io_permit| async move {
                 with_disk_io_permit(io_permit, self.disk.read_version(org_volume, volume, path, version_id, opts)).await
@@ -2759,8 +2641,20 @@ impl DiskAPI for LocalDiskWrapper {
             get_max_timeout_duration(),
             TimeoutHealthAction::MarkFailure,
             DiskMetricMutation::None,
-        )
-        .await
+        );
+        #[cfg(feature = "diagnostic-read-version-caller")]
+        {
+            if crate::cluster::rpc::runtime_sources::current_read_version_request_caller().is_some() {
+                operation.await
+            } else {
+                let caller = crate::cluster::rpc::runtime_sources::read_version_request_caller_from_location(
+                    std::panic::Location::caller().file(),
+                );
+                crate::cluster::rpc::runtime_sources::with_read_version_request_caller(caller, operation).await
+            }
+        }
+        #[cfg(not(feature = "diagnostic-read-version-caller"))]
+        operation.await
     }
 
     async fn read_xl(&self, volume: &str, path: &str, read_data: bool) -> Result<RawFileInfo> {
@@ -3566,7 +3460,7 @@ mod tests {
             .track_cleanup_mutation(
                 "delete",
                 DiskMetricMutation::Delete,
-                std::future::pending::<Result<()>>,
+                || std::future::pending::<Result<()>>(),
                 Duration::from_millis(10),
             )
             .await;

crates/ecstore/src/disk/error_reduce.rs

@@ -131,15 +131,6 @@ pub fn build_write_quorum_failure_summary(
 
 fn dominant_error_label(errors: &[Option<Error>], ignored_errs: &[Error], dominant_error: Option<&Error>) -> &'static str {
     let Some(dominant_error) = dominant_error else {
-        // The reducer also returns None when every failure is ignored for
-        // quorum voting. That case contains no successful nil entry.
-        if !errors.is_empty() && errors.iter().all(Option::is_some) {
-            return if errors.iter().all(|error| error.as_ref() == Some(&Error::DiskNotFound)) {
-                "disk_not_found"
-            } else {
-                "all_ignored"
-            };
-        }
         return "nil_dominated";
     };
 
@@ -396,9 +387,9 @@ mod tests {
     }
 
     #[test]
-    fn test_build_write_quorum_failure_summary_all_missing_is_disk_not_found() {
-        // Ignoring unavailable disks for quorum reduction must not imply a
-        // successful nil entry in the diagnostic label.
+    fn test_build_write_quorum_failure_summary_all_ignored_is_nil_dominated() {
+        // All disks offline (all DiskNotFound). The dominant error must be None and
+        // the label must be "nil_dominated" rather than falling back to "other_error".
         let errors = vec![
             Some(Error::DiskNotFound),
             Some(Error::DiskNotFound),
@@ -406,21 +397,7 @@ mod tests {
         ];
         let summary = build_write_quorum_failure_summary(&errors, OBJECT_OP_IGNORED_ERRS, 2);
         assert_eq!(summary.dominant_error, None);
-        assert_eq!(summary.achieved, 0);
-        assert_eq!(summary.dominant_error_label, "disk_not_found");
-    }
-
-    #[test]
-    fn test_build_write_quorum_failure_summary_all_ignored_has_no_success() {
-        let errors = [Some(Error::FaultyDisk), Some(Error::DiskNotFound)];
-        let summary = build_write_quorum_failure_summary(&errors, OBJECT_OP_IGNORED_ERRS, 2);
-        assert_eq!(summary.dominant_error, None);
-        assert_eq!(summary.achieved, 0);
-        assert_eq!(summary.dominant_error_label, "all_ignored");
-        assert_eq!(
-            reduce_write_quorum_errs(&errors, OBJECT_OP_IGNORED_ERRS, 2),
-            Some(Error::ErasureWriteQuorum)
-        );
+        assert_eq!(summary.dominant_error_label, "nil_dominated");
     }
 
     #[test]

crates/ecstore/src/disk/mod.rs

@@ -231,105 +231,6 @@ pub enum Disk {
 }
 
 impl Disk {
-    pub(crate) async fn delete_versions_with_namespace_owner(
-        &self,
-        volume: &str,
-        versions: Vec<FileInfoVersions>,
-        opts: DeleteOptions,
-        namespace_owner: Option<Arc<dyn Send + Sync>>,
-    ) -> Vec<Option<Error>> {
-        match self {
-            Self::Local(disk) => {
-                disk.delete_versions_with_namespace_owner(volume, versions, opts, namespace_owner)
-                    .await
-            }
-            Self::Remote(disk) => {
-                let _owner = namespace_owner;
-                disk.delete_versions(volume, versions, opts).await
-            }
-        }
-    }
-
-    pub(crate) async fn delete_versions_recoverable_with_namespace_owner(
-        &self,
-        volume: &str,
-        versions: Vec<FileInfoVersions>,
-        mut opts: DeleteOptions,
-        namespace_owner: Option<Arc<dyn Send + Sync>>,
-    ) -> Vec<Option<Error>> {
-        if let Err(error) = validate_delete_recovery_options(&opts) {
-            return vec![Some(error); versions.len()];
-        }
-        if versions.iter().any(|version| version.name.starts_with('/')) {
-            return vec![Some(DiskError::InvalidPath); versions.len()];
-        }
-        opts.recoverable_delete = true;
-        match self {
-            Self::Local(disk) => {
-                disk.delete_versions_with_namespace_owner(volume, versions, opts, namespace_owner)
-                    .await
-            }
-            Self::Remote(disk) => {
-                let _owner = namespace_owner;
-                disk.delete_versions_recoverable(volume, versions, opts).await
-            }
-        }
-    }
-
-    pub(crate) async fn settle_delete_transaction_with_namespace_owner(
-        &self,
-        volume: &str,
-        object: &str,
-        transaction: Uuid,
-        expected_fingerprint: DeleteRecoveryDigest,
-        action: DeleteRecoveryAction,
-        namespace_owner: Option<Arc<dyn Send + Sync>>,
-    ) -> Result<DeleteRecoveryInspection> {
-        match self {
-            Self::Local(disk) => {
-                disk.settle_delete_transaction_with_namespace_owner(
-                    volume,
-                    object,
-                    transaction,
-                    expected_fingerprint,
-                    action,
-                    namespace_owner,
-                )
-                .await
-            }
-            Self::Remote(disk) => {
-                let _owner = namespace_owner;
-                disk.settle_delete_transaction(volume, object, transaction, expected_fingerprint, action)
-                    .await
-            }
-        }
-    }
-
-    pub(crate) async fn delete_version_recoverable_with_namespace_owner(
-        &self,
-        volume: &str,
-        path: &str,
-        fi: FileInfo,
-        force_del_marker: bool,
-        mut opts: DeleteOptions,
-        namespace_owner: Option<Arc<dyn Send + Sync>>,
-    ) -> Result<()> {
-        validate_delete_recovery_options(&opts)?;
-        if path.starts_with('/') {
-            return Err(DiskError::InvalidPath);
-        }
-        opts.recoverable_delete = true;
-        match self {
-            Self::Local(disk) => {
-                disk.delete_version_with_namespace_owner(volume, path, fi, force_del_marker, opts, namespace_owner)
-                    .await
-            }
-            Self::Remote(disk) => {
-                disk.delete_version_recoverable(volume, path, fi, force_del_marker, opts)
-                    .await
-            }
-        }
-    }
     pub(crate) async fn set_disk_id_state(&self, id: Option<Uuid>) -> Result<()> {
         match self {
             Disk::Local(local_disk) => {
@@ -357,70 +258,6 @@ impl Disk {
 
 #[async_trait::async_trait]
 impl DiskAPI for Disk {
-    async fn delete_version_recoverable(
-        &self,
-        volume: &str,
-        object: &str,
-        file_info: FileInfo,
-        force_del_marker: bool,
-        opts: DeleteOptions,
-    ) -> Result<()> {
-        self.delete_version_recoverable_with_namespace_owner(volume, object, file_info, force_del_marker, opts, None)
-            .await
-    }
-    async fn delete_versions_recoverable(
-        &self,
-        volume: &str,
-        versions: Vec<FileInfoVersions>,
-        opts: DeleteOptions,
-    ) -> Vec<Option<DiskError>> {
-        match self {
-            Self::Local(disk) => disk.delete_versions_recoverable(volume, versions, opts).await,
-            Self::Remote(disk) => disk.delete_versions_recoverable(volume, versions, opts).await,
-        }
-    }
-    async fn discover_delete_transactions(
-        &self,
-        volume: &str,
-        object: Option<&str>,
-        cursor: Option<DeleteRecoveryCursor>,
-        limit: u32,
-    ) -> Result<DeleteRecoveryPage> {
-        match self {
-            Self::Local(disk) => disk.discover_delete_transactions(volume, object, cursor, limit).await,
-            Self::Remote(disk) => disk.discover_delete_transactions(volume, object, cursor, limit).await,
-        }
-    }
-    async fn inspect_delete_transaction(
-        &self,
-        volume: &str,
-        object: &str,
-        transaction: Uuid,
-    ) -> Result<DeleteRecoveryInspection> {
-        match self {
-            Self::Local(disk) => disk.inspect_delete_transaction(volume, object, transaction).await,
-            Self::Remote(disk) => disk.inspect_delete_transaction(volume, object, transaction).await,
-        }
-    }
-    async fn settle_delete_transaction(
-        &self,
-        volume: &str,
-        object: &str,
-        transaction: Uuid,
-        expected_fingerprint: DeleteRecoveryDigest,
-        action: DeleteRecoveryAction,
-    ) -> Result<DeleteRecoveryInspection> {
-        match self {
-            Self::Local(disk) => {
-                disk.settle_delete_transaction(volume, object, transaction, expected_fingerprint, action)
-                    .await
-            }
-            Self::Remote(disk) => {
-                disk.settle_delete_transaction(volume, object, transaction, expected_fingerprint, action)
-                    .await
-            }
-        }
-    }
     fn to_string(&self) -> String {
         match self {
             Disk::Local(local_disk) => local_disk.to_string(),
@@ -622,6 +459,7 @@ impl DiskAPI for Disk {
     }
 
     #[tracing::instrument(level = "trace", skip_all)]
+    #[track_caller]
     async fn read_version(
         &self,
         _org_volume: &str,
@@ -630,10 +468,20 @@ impl DiskAPI for Disk {
         version_id: &str,
         opts: &ReadOptions,
     ) -> Result<FileInfo> {
-        match self {
-            Disk::Local(local_disk) => local_disk.read_version(_org_volume, volume, path, version_id, opts).await,
-            Disk::Remote(remote_disk) => remote_disk.read_version(_org_volume, volume, path, version_id, opts).await,
+        let operation = async {
+            match self {
+                Disk::Local(local_disk) => local_disk.read_version(_org_volume, volume, path, version_id, opts).await,
+                Disk::Remote(remote_disk) => remote_disk.read_version(_org_volume, volume, path, version_id, opts).await,
+            }
+        };
+        #[cfg(feature = "diagnostic-read-version-caller")]
+        if crate::cluster::rpc::runtime_sources::current_read_version_request_caller().is_none() {
+            let caller = crate::cluster::rpc::runtime_sources::read_version_request_caller_from_location(
+                std::panic::Location::caller().file(),
+            );
+            return crate::cluster::rpc::runtime_sources::with_read_version_request_caller(caller, operation).await;
         }
+        operation.await
     }
 
     #[tracing::instrument(level = "trace", skip_all)]
@@ -1325,6 +1173,7 @@ pub trait DiskAPI: Debug + Send + Sync + 'static {
     }
     async fn write_metadata(&self, org_volume: &str, volume: &str, path: &str, fi: FileInfo) -> Result<()>;
     async fn update_metadata(&self, volume: &str, path: &str, fi: FileInfo, opts: &UpdateMetadataOpts) -> Result<()>;
+    #[track_caller]
     async fn read_version(
         &self,
         org_volume: &str,
@@ -1410,51 +1259,6 @@ pub trait DiskAPI: Debug + Send + Sync + 'static {
         Err(DiskError::MethodNotAllowed)
     }
     async fn delete(&self, volume: &str, path: &str, opt: DeleteOptions) -> Result<()>;
-    async fn discover_delete_transactions(
-        &self,
-        _volume: &str,
-        _object: Option<&str>,
-        _cursor: Option<DeleteRecoveryCursor>,
-        _limit: u32,
-    ) -> Result<DeleteRecoveryPage> {
-        Err(DiskError::MethodNotAllowed)
-    }
-    async fn delete_version_recoverable(
-        &self,
-        _volume: &str,
-        _object: &str,
-        _file_info: FileInfo,
-        _force_del_marker: bool,
-        _opts: DeleteOptions,
-    ) -> Result<()> {
-        Err(DiskError::MethodNotAllowed)
-    }
-    async fn delete_versions_recoverable(
-        &self,
-        _volume: &str,
-        versions: Vec<FileInfoVersions>,
-        _opts: DeleteOptions,
-    ) -> Vec<Option<DiskError>> {
-        vec![Some(DiskError::MethodNotAllowed); versions.len()]
-    }
-    async fn inspect_delete_transaction(
-        &self,
-        _volume: &str,
-        _object: &str,
-        _transaction: Uuid,
-    ) -> Result<DeleteRecoveryInspection> {
-        Err(DiskError::MethodNotAllowed)
-    }
-    async fn settle_delete_transaction(
-        &self,
-        _volume: &str,
-        _object: &str,
-        _transaction: Uuid,
-        _expected_fingerprint: DeleteRecoveryDigest,
-        _action: DeleteRecoveryAction,
-    ) -> Result<DeleteRecoveryInspection> {
-        Err(DiskError::MethodNotAllowed)
-    }
     // VerifyFile
     async fn verify_file(&self, volume: &str, path: &str, fi: &FileInfo) -> Result<CheckPartsResp>;
     // CheckParts
@@ -1493,13 +1297,15 @@ where
 {
     validate_batch_read_version_item_count(req.items.len())?;
     let parallelism = batch_read_version_server_parallelism();
+    #[cfg(feature = "diagnostic-read-version-caller")]
+    let item_count = req.items.len();
 
     let mut responses = stream::iter(req.items.into_iter().enumerate())
         .map(|(index, item)| async move {
-            match disk
-                .read_version(&item.org_volume, &item.volume, &item.path, &item.version_id, &req.opts)
-                .await
-            {
+            let read = disk.read_version(&item.org_volume, &item.volume, &item.path, &item.version_id, &req.opts);
+            #[cfg(feature = "diagnostic-read-version-caller")]
+            let read = crate::cluster::rpc::runtime_sources::with_batch_read_version_item_context(index, item_count, read);
+            match read.await {
                 Ok(file_info) => BatchReadVersionResp {
                     index,
                     path: item.path,
@@ -1799,269 +1605,6 @@ pub struct DeleteOptions {
     /// Remote calls carrying it must use DeleteRetiredMarker, never DeleteVersion.
     #[serde(default, skip_serializing_if = "Option::is_none")]
     pub expected_delete_marker: Option<rustfs_filemeta::MetaDeleteMarker>,
-    #[serde(default, skip_serializing_if = "Option::is_none")]
-    pub delete_recovery_bucket_incarnation: Option<Uuid>,
-    #[serde(default)]
-    pub recoverable_delete: bool,
-}
-
-pub const DELETE_RECOVERY_PROTOCOL_VERSION: u32 = 1;
-pub(crate) fn get_delete_recovery_timeout() -> Result<std::time::Duration> {
-    let value = std::env::var("RUSTFS_DELETE_RECOVERY_TIMEOUT_SECS");
-    match value {
-        Ok(value) => parse_delete_recovery_timeout(Some(&value)),
-        Err(std::env::VarError::NotPresent) => parse_delete_recovery_timeout(None),
-        Err(std::env::VarError::NotUnicode(_)) => Err(DiskError::other("invalid delete recovery timeout configuration")),
-    }
-}
-
-fn parse_delete_recovery_timeout(value: Option<&str>) -> Result<std::time::Duration> {
-    let seconds = match value {
-        None => 900,
-        Some(value) => value
-            .parse::<u64>()
-            .ok()
-            .filter(|seconds| (30..=86400).contains(seconds))
-            .ok_or_else(|| DiskError::other("delete recovery timeout must be an integer between 30 and 86400 seconds"))?,
-    };
-    Ok(std::time::Duration::from_secs(seconds))
-}
-
-pub(crate) fn validate_delete_recovery_options(opts: &DeleteOptions) -> Result<()> {
-    if opts.old_data_dir.is_none_or(|id| id.is_nil())
-        || opts.undo_write
-        || opts.undo_delete
-        || opts.recursive
-        || opts.immediate
-        || opts.expected_delete_marker.is_some()
-        || opts.delete_recovery_bucket_incarnation.is_some_and(|id| id.is_nil())
-    {
-        return Err(DiskError::FileCorrupt);
-    }
-    Ok(())
-}
-pub type DeleteRecoveryDigest = [u8; 32];
-
-#[derive(Debug, Clone, PartialEq, Eq, Serialize, Deserialize)]
-#[serde(deny_unknown_fields)]
-pub struct DeleteTransactionId {
-    pub object: String,
-    pub transaction: Uuid,
-}
-
-#[derive(Debug, Clone, PartialEq, Eq, Serialize, Deserialize)]
-#[serde(deny_unknown_fields)]
-pub struct DeleteRecoveryCursor {
-    pub token: Uuid,
-}
-
-#[derive(Debug, Clone, PartialEq, Eq, Serialize, Deserialize)]
-#[serde(deny_unknown_fields)]
-pub struct DeleteRecoveryPage {
-    pub transactions: Vec<DeleteTransactionId>,
-    pub next: Option<DeleteRecoveryCursor>,
-}
-
-#[derive(Debug, Clone, PartialEq, Serialize, Deserialize)]
-#[serde(deny_unknown_fields)]
-pub struct DeleteRecoveryJournal {
-    pub protocol_version: u32,
-    pub volume: String,
-    pub object: String,
-    pub transaction: Uuid,
-    pub bucket_incarnation: Option<Uuid>,
-    pub original_digest: Option<DeleteRecoveryDigest>,
-    pub target_digest: Option<DeleteRecoveryDigest>,
-    pub data_dirs: Vec<Uuid>,
-    pub deleted_versions: Vec<Uuid>,
-    pub requests: Vec<FileInfo>,
-    pub original_versions: Vec<DeleteRecoveryVersionIdentity>,
-    #[serde(default)]
-    pub predecessors: Vec<Uuid>,
-}
-
-impl DeleteRecoveryJournal {
-    /// Checks source generations only; the coordinator must supply a certified
-    /// decision and the receiver must retain its namespace/incarnation fences.
-    pub(crate) fn matches_adoption_source(&self, current: &[DeleteRecoveryVersionIdentity]) -> bool {
-        if self.requests.is_empty() {
-            return false;
-        }
-        let mut originals = std::collections::HashMap::with_capacity(self.original_versions.len());
-        for version in &self.original_versions {
-            originals.entry(version.version_id.unwrap_or_default()).or_insert(version);
-        }
-        let mut locals = std::collections::HashMap::with_capacity(current.len());
-        for version in current {
-            locals.entry(version.version_id.unwrap_or_default()).or_insert(version);
-        }
-        self.requests.iter().all(|request| {
-            if request.deleted && request.mark_deleted {
-                return current.first() == self.original_versions.first();
-            }
-            let id = request.version_id.unwrap_or_default();
-            let Some(original) = originals.get(&id) else {
-                return false;
-            };
-            match locals.get(&id) {
-                Some(local) => local == original,
-                None => !id.is_nil(),
-            }
-        })
-    }
-
-    /// Matches immutable local preparation intent against an independently
-    /// inspected same-transaction commit receipt. This match is not authority
-    /// by itself; the coordinator must supply the certified receipt.
-    pub(crate) fn matches_prepared_authority(&self, authority: &Self) -> bool {
-        if self.protocol_version != DELETE_RECOVERY_PROTOCOL_VERSION
-            || authority.protocol_version != DELETE_RECOVERY_PROTOCOL_VERSION
-            || self.transaction.is_nil()
-            || self.transaction != authority.transaction
-            || self.volume != authority.volume
-            || self.object != authority.object
-            || self.bucket_incarnation != authority.bucket_incarnation
-            || self.bucket_incarnation.is_some_and(|id| id.is_nil())
-            || self.requests.is_empty()
-            || self.requests != authority.requests
-            || self.deleted_versions.iter().collect::<std::collections::HashSet<_>>()
-                != authority.deleted_versions.iter().collect::<std::collections::HashSet<_>>()
-        {
-            return false;
-        }
-        let sources: std::collections::HashMap<_, _> = self
-            .original_versions
-            .iter()
-            .map(|version| (version.version_id.unwrap_or_default(), version))
-            .collect();
-        let certified: std::collections::HashMap<_, _> = authority
-            .original_versions
-            .iter()
-            .map(|version| (version.version_id.unwrap_or_default(), version))
-            .collect();
-        if sources.len() != self.original_versions.len() || certified.len() != authority.original_versions.len() {
-            return false;
-        }
-        self.requests.iter().all(|request| {
-            let (local, remote) = if request.deleted && request.mark_deleted {
-                (self.original_versions.first(), authority.original_versions.first())
-            } else {
-                let id = request.version_id.unwrap_or_default();
-                (sources.get(&id).copied(), certified.get(&id).copied())
-            };
-            match (local, remote) {
-                (Some(local), Some(remote)) => local == remote,
-                (None, None) => self.original_digest.is_none() && authority.original_digest.is_none(),
-                _ => false,
-            }
-        })
-    }
-}
-
-#[derive(Debug, Clone, PartialEq, Eq, Hash, Serialize, Deserialize)]
-#[serde(deny_unknown_fields)]
-pub struct DeleteRecoveryVersionIdentity {
-    pub version_id: Option<Uuid>,
-    pub mod_time: Option<OffsetDateTime>,
-    pub data_dir: Option<Uuid>,
-    pub deleted: bool,
-    pub metadata_digest: DeleteRecoveryDigest,
-}
-
-impl DeleteRecoveryVersionIdentity {
-    pub(crate) fn from_file_info(info: FileInfo) -> error::Result<Self> {
-        local::delete_recovery::generation(info)
-    }
-}
-
-#[derive(Debug, Clone, Copy, PartialEq, Eq, Serialize, Deserialize)]
-pub enum DeleteRecoveryState {
-    Absent,
-    NewPrepared,
-    QuorumCommitted,
-    CommitFinished,
-    LegacyAmbiguous,
-}
-
-#[derive(Debug, Clone, PartialEq, Serialize, Deserialize)]
-#[serde(deny_unknown_fields)]
-pub struct DeleteRecoveryInspection {
-    pub protocol_version: u32,
-    pub id: DeleteTransactionId,
-    pub state: DeleteRecoveryState,
-    pub journal: Option<DeleteRecoveryJournal>,
-    pub current_digest: Option<DeleteRecoveryDigest>,
-    pub current_versions: Vec<DeleteRecoveryVersionIdentity>,
-    pub backup_digest: Option<DeleteRecoveryDigest>,
-    pub rollback_data_complete: bool,
-    pub commit_data_safe: bool,
-    pub explicit_delete_safe: bool,
-    pub fingerprint: DeleteRecoveryDigest,
-}
-
-#[derive(Debug, Clone, PartialEq, Serialize, Deserialize)]
-pub enum DeleteRecoveryAction {
-    PublishCommit,
-    FinishCommit,
-    FinalizeCommit,
-    Rollback,
-    DiscardLegacy,
-    AdoptCommit { authority: Box<DeleteRecoveryJournal> },
-}
-
-#[derive(Debug, Clone, Serialize, Deserialize)]
-#[serde(deny_unknown_fields)]
-pub enum DeleteRecoveryRpcOperation {
-    Discover {
-        object: Option<String>,
-        cursor: Option<DeleteRecoveryCursor>,
-        limit: u32,
-    },
-    Inspect {
-        object: String,
-        transaction: Uuid,
-    },
-    Settle {
-        object: String,
-        transaction: Uuid,
-        expected_fingerprint: DeleteRecoveryDigest,
-        action: DeleteRecoveryAction,
-    },
-    ApplyVersion {
-        object: String,
-        file_info: Box<FileInfo>,
-        force_del_marker: bool,
-        opts: DeleteOptions,
-    },
-    ApplyVersions {
-        versions: Vec<FileInfoVersions>,
-        opts: DeleteOptions,
-    },
-}
-
-impl DeleteRecoveryRpcOperation {
-    /// Full metadata application and adoption share the existing mutation envelope.
-    pub fn max_wire_payload_bytes(&self) -> usize {
-        match self {
-            Self::ApplyVersion { .. }
-            | Self::ApplyVersions { .. }
-            | Self::Settle {
-                action: DeleteRecoveryAction::AdoptCommit { .. },
-                ..
-            } => rustfs_protos::DELETE_RECOVERY_MAX_PAYLOAD_BYTES,
-            _ => rustfs_protos::DELETE_RECOVERY_MAX_CONTROL_PAYLOAD_BYTES,
-        }
-    }
-}
-
-#[derive(Debug, Clone, Serialize, Deserialize)]
-#[serde(deny_unknown_fields)]
-pub enum DeleteRecoveryRpcReply {
-    Discovered(DeleteRecoveryPage),
-    Inspected(DeleteRecoveryInspection),
-    Settled(DeleteRecoveryInspection),
-    AppliedVersion,
-    AppliedVersions(Vec<Option<u32>>),
 }
 
 #[derive(Debug, Clone, Serialize, Deserialize)]
@@ -2352,8 +1895,6 @@ mod tests {
             undo_delete: false,
             old_data_dir: Some(Uuid::new_v4()),
             expected_delete_marker: None,
-            delete_recovery_bucket_incarnation: None,
-            recoverable_delete: false,
         };
 
         assert!(opts.recursive);
@@ -2784,183 +2325,3 @@ mod tests {
         }
     }
 }
-
-#[cfg(test)]
-mod delete_recovery_timeout_tests {
-    use super::*;
-    #[test]
-    fn recovery_timeout_defaults_and_validates_range() {
-        assert_eq!(
-            parse_delete_recovery_timeout(None).expect("default timeout"),
-            std::time::Duration::from_secs(900)
-        );
-        for seconds in ["30", "900", "86400"] {
-            assert!(parse_delete_recovery_timeout(Some(seconds)).is_ok());
-        }
-        for value in ["", "invalid", "0", "2", "29", "86401", "-1", "18446744073709551616"] {
-            assert!(parse_delete_recovery_timeout(Some(value)).is_err());
-        }
-    }
-}
-
-#[cfg(test)]
-mod delete_recovery_wire_limit_tests {
-    use super::*;
-
-    #[test]
-    fn boxed_apply_version_preserves_existing_wire_payload() {
-        #[derive(Serialize)]
-        enum PriorOperation<'a> {
-            ApplyVersion {
-                object: &'a str,
-                file_info: &'a FileInfo,
-                force_del_marker: bool,
-                opts: &'a DeleteOptions,
-            },
-        }
-        let file_info = FileInfo {
-            name: "object".into(),
-            version_id: Some(Uuid::new_v4()),
-            metadata: std::collections::HashMap::from([("metadata".into(), "value".into())]),
-            ..Default::default()
-        };
-        let opts = DeleteOptions {
-            old_data_dir: Some(Uuid::new_v4()),
-            recoverable_delete: true,
-            delete_recovery_bucket_incarnation: Some(Uuid::new_v4()),
-            ..Default::default()
-        };
-        let prior = PriorOperation::ApplyVersion {
-            object: "object",
-            file_info: &file_info,
-            force_del_marker: true,
-            opts: &opts,
-        };
-        let encoded = rmp_serde::to_vec_named(&prior).expect("encode the previous unboxed request shape");
-        let operation = DeleteRecoveryRpcOperation::ApplyVersion {
-            object: "object".into(),
-            file_info: Box::new(file_info.clone()),
-            force_del_marker: true,
-            opts: opts.clone(),
-        };
-        assert_eq!(rmp_serde::to_vec_named(&operation).expect("encode boxed request"), encoded);
-        assert_eq!(
-            serde_json::to_value(&operation).expect("boxed request JSON"),
-            serde_json::to_value(prior).expect("previous request JSON")
-        );
-        let decoded: DeleteRecoveryRpcOperation =
-            rustfs_protos::decode_delete_recovery_payload(&encoded).expect("decode the previous payload");
-        assert_eq!(rmp_serde::to_vec_named(&decoded).expect("re-encode decoded prior request"), encoded);
-    }
-    #[test]
-    fn adoption_and_inspection_reply_support_large_version_journals() {
-        let transaction = Uuid::new_v4();
-        let request = FileInfo {
-            metadata: std::collections::HashMap::from([("metadata".into(), "x".repeat(2048))]),
-            ..Default::default()
-        };
-        let journal = DeleteRecoveryJournal {
-            protocol_version: DELETE_RECOVERY_PROTOCOL_VERSION,
-            volume: "bucket".into(),
-            object: "object".into(),
-            transaction,
-            bucket_incarnation: None,
-            original_digest: None,
-            target_digest: None,
-            data_dirs: Vec::new(),
-            deleted_versions: Vec::new(),
-            requests: vec![request; 1000],
-            original_versions: Vec::new(),
-            predecessors: Vec::new(),
-        };
-        let operation = DeleteRecoveryRpcOperation::Settle {
-            object: "object".into(),
-            transaction,
-            expected_fingerprint: [0; 32],
-            action: DeleteRecoveryAction::AdoptCommit {
-                authority: Box::new(journal.clone()),
-            },
-        };
-        let encoded = rmp_serde::to_vec_named(&operation).expect("encode large adoption");
-        assert!(encoded.len() > rustfs_protos::DELETE_RECOVERY_MAX_CONTROL_PAYLOAD_BYTES);
-        assert!(encoded.len() < operation.max_wire_payload_bytes());
-        let decoded: DeleteRecoveryRpcOperation =
-            rustfs_protos::decode_delete_recovery_payload(&encoded).expect("large adoption can decode");
-        assert_eq!(decoded.max_wire_payload_bytes(), rustfs_protos::DELETE_RECOVERY_MAX_PAYLOAD_BYTES);
-        let reply = DeleteRecoveryRpcReply::Inspected(DeleteRecoveryInspection {
-            protocol_version: DELETE_RECOVERY_PROTOCOL_VERSION,
-            id: DeleteTransactionId {
-                object: "object".into(),
-                transaction,
-            },
-            state: DeleteRecoveryState::QuorumCommitted,
-            journal: Some(journal),
-            current_digest: None,
-            current_versions: Vec::new(),
-            backup_digest: None,
-            rollback_data_complete: false,
-            commit_data_safe: true,
-            explicit_delete_safe: false,
-            fingerprint: [0; 32],
-        });
-        let encoded = rmp_serde::to_vec_named(&reply).expect("encode large inspection");
-        assert!(encoded.len() > rustfs_protos::DELETE_RECOVERY_MAX_CONTROL_PAYLOAD_BYTES);
-        assert!(rustfs_protos::decode_delete_recovery_payload::<DeleteRecoveryRpcReply>(&encoded).is_ok());
-        let inspect = DeleteRecoveryRpcOperation::Inspect {
-            object: "object".into(),
-            transaction,
-        };
-        assert_eq!(inspect.max_wire_payload_bytes(), rustfs_protos::DELETE_RECOVERY_MAX_CONTROL_PAYLOAD_BYTES);
-    }
-}
-
-#[cfg(test)]
-mod delete_recovery_adoption_tests {
-    use super::*;
-
-    #[test]
-    fn source_matching_preserves_non_null_absence_and_rejects_foreign_or_missing_null() {
-        let id = Uuid::new_v4();
-        let source = DeleteRecoveryVersionIdentity {
-            version_id: Some(id),
-            mod_time: None,
-            data_dir: None,
-            deleted: false,
-            metadata_digest: [1; 32],
-        };
-        let mut journal = DeleteRecoveryJournal {
-            protocol_version: DELETE_RECOVERY_PROTOCOL_VERSION,
-            volume: "bucket".into(),
-            object: "object".into(),
-            transaction: Uuid::new_v4(),
-            bucket_incarnation: None,
-            original_digest: None,
-            target_digest: None,
-            data_dirs: Vec::new(),
-            deleted_versions: vec![id],
-            requests: vec![FileInfo {
-                version_id: Some(id),
-                ..Default::default()
-            }],
-            original_versions: vec![source.clone()],
-            predecessors: Vec::new(),
-        };
-        assert!(journal.matches_adoption_source(&[]), "certified non-null removal can already be absent");
-        assert!(journal.matches_adoption_source(std::slice::from_ref(&source)));
-        let mut foreign = source.clone();
-        foreign.metadata_digest = [2; 32];
-        assert!(!journal.matches_adoption_source(std::slice::from_ref(&foreign)));
-        journal.original_versions.push(foreign.clone());
-        assert!(
-            journal.matches_adoption_source(&[source.clone(), foreign.clone()]),
-            "retain original first-match semantics"
-        );
-        assert!(!journal.matches_adoption_source(&[foreign, source]));
-        journal.requests[0].version_id = None;
-        journal.original_versions[0].version_id = None;
-        assert!(!journal.matches_adoption_source(&[]), "null absence does not establish its generation");
-        assert!(journal.matches_adoption_source(std::slice::from_ref(&journal.original_versions[0])));
-        journal.requests.clear();
-        assert!(!journal.matches_adoption_source(&journal.original_versions));
-    }
-}

crates/ecstore/src/disk/os.rs

@@ -491,11 +491,11 @@ pub fn fsync_dir_std(dir: impl AsRef<Path>) -> io::Result<()> {
 
 pub(crate) fn fsync_directory_handle_std(dir: &Path, opened: Option<&std::fs::File>) -> io::Result<()> {
     #[cfg(test)]
-    fsync_dir_recorder::record(dir);
+    fsync_dir_recorder::record(dir.as_ref());
     #[cfg(unix)]
     {
         #[cfg(test)]
-        if let Some(kind) = fsync_dir_recorder::take_failure(dir) {
+        if let Some(kind) = fsync_dir_recorder::take_failure(dir.as_ref()) {
             return Err(io::Error::from(kind));
         }
         match opened {
@@ -2426,7 +2426,7 @@ pub(crate) fn read_destination_file_with_commit_guard(
         let Some(parent) = directories.destination.get() else {
             return Ok(None);
         };
-        match open_linux_relative(&parent.file, Path::new(name), OFlags::RDONLY, Mode::empty()) {
+        return match open_linux_relative(&parent.file, Path::new(name), OFlags::RDONLY, Mode::empty()) {
             Ok(mut file) => {
                 let mut data = Vec::new();
                 std::io::Read::read_to_end(&mut file, &mut data)?;
@@ -2434,7 +2434,7 @@ pub(crate) fn read_destination_file_with_commit_guard(
             }
             Err(err) if err.kind() == io::ErrorKind::NotFound => Ok(None),
             Err(err) => Err(err),
-        }
+        };
     }
     #[cfg(windows)]
     {
@@ -2504,9 +2504,9 @@ pub(crate) async fn rename_all_with_prepared_source(
             {
                 #[cfg(any(test, feature = "test-util"))]
                 prepared_publication_test_hooks::run(prepared_publication_test_hooks::Stage::PreparedRename, &dst_file_path);
-                commit_guard
+                return commit_guard
                     .directories
-                    .rename(&src_file_path, &dst_file_path, Some(&prepared_source))
+                    .rename(&src_file_path, &dst_file_path, Some(&prepared_source));
             }
             #[cfg(not(target_os = "linux"))]
             {
@@ -2542,7 +2542,7 @@ pub(crate) async fn rename_all_with_commit_guard(
         let destination = dst_file_path.as_ref().to_path_buf();
         let guard = commit_guard.clone();
         let _ = (base_dir, publication_root);
-        run_blocking_namespace_operation(lease, move || {
+        return run_blocking_namespace_operation(lease, move || {
             guard.directories.prepare_destination()?;
             #[cfg(test)]
             prepared_publication_test_hooks::run_rename_destination(&source, &destination);
@@ -2555,7 +2555,7 @@ pub(crate) async fn rename_all_with_commit_guard(
         })
         .await
         .map_err(to_file_error)
-        .map_err(DiskError::from)
+        .map_err(DiskError::from);
     }
     #[cfg(not(target_os = "linux"))]
     {
@@ -3890,7 +3890,7 @@ pub(crate) fn prepare_rename_commit_guard(
             Err(err) if err.kind() == io::ErrorKind::NotFound => {}
             Err(err) => return Err(err),
         }
-        Ok(RenameCommitGuard {
+        return Ok(RenameCommitGuard {
             directories: Arc::new(LinuxCommitDirectories {
                 root: publication_root.clone(),
                 source_path: source_parent.to_path_buf(),
@@ -3899,7 +3899,7 @@ pub(crate) fn prepare_rename_commit_guard(
                 destination,
                 destination_base: destination_base.to_path_buf(),
             }),
-        })
+        });
     }
     #[cfg(windows)]
     {

crates/ecstore/src/io_support/bitrot.rs

@@ -174,14 +174,10 @@ async fn open_reader_source(
     metrics_path: Option<&'static str>,
 ) -> disk::error::Result<Option<BoxedObjectReader>> {
     if let Some(data) = inline_data {
-        #[cfg(test)]
-        crate::set_disk::disk_call_counters::record_shard_open(path, true);
         let mut reader = Cursor::new(data);
         reader.set_position(u64::try_from(offset).map_err(|_| DiskError::FileCorrupt)?);
         Ok(Some(ShardReader::InMemory(reader)))
     } else if let Some(disk) = disk {
-        #[cfg(test)]
-        crate::set_disk::disk_call_counters::record_shard_open(path, false);
         open_disk_reader(disk, bucket, path, offset, length, use_mmap_read, metrics_path)
             .await
             .map(Some)

crates/ecstore/src/object_api/mod.rs

@@ -102,14 +102,10 @@ mod body_cache_hook;
 mod encryption;
 mod hook_slot;
 mod object_mutation_hook;
-mod read_condition;
 mod readers;
 mod types;
 pub(crate) mod write_commit_context;
 
-pub use read_condition::{GetObjectReadCondition, GetObjectReadMetadata, with_get_object_read_condition};
-pub(crate) use read_condition::{get_object_read_condition_is_active, get_object_read_condition_is_terminal};
-
 #[cfg(test)]
 mod persisted_metadata_keys_tests;
 

crates/ecstore/src/object_api/read_condition.rs

@@ -1,113 +0,0 @@
-// Copyright 2024 RustFS Team
-//
-// Licensed under the Apache License, Version 2.0 (the "License");
-// you may not use this file except in compliance with the License.
-// You may obtain a copy of the License at
-//
-//     http://www.apache.org/licenses/LICENSE-2.0
-//
-// Unless required by applicable law or agreed to in writing, software
-// distributed under the License is distributed on an "AS IS" BASIS,
-// WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
-// See the License for the specific language governing permissions and
-// limitations under the License.
-
-use rustfs_filemeta::{FileInfo, ObjectPartInfo};
-use std::future::Future;
-use std::sync::Arc;
-use time::OffsetDateTime;
-
-/// Borrowed fields needed to decide whether an object read needs a body.
-pub struct GetObjectReadMetadata<'a> {
-    pub bucket: &'a str,
-    pub object: &'a str,
-    pub etag: Option<&'a str>,
-    pub mod_time: Option<OffsetDateTime>,
-    pub parts: &'a [ObjectPartInfo],
-    pub delete_marker: bool,
-}
-
-impl<'a> GetObjectReadMetadata<'a> {
-    pub(crate) fn from_file_info(bucket: &'a str, object: &'a str, info: &'a FileInfo) -> Self {
-        Self {
-            bucket,
-            object,
-            etag: info.metadata.get("etag").map(String::as_str),
-            mod_time: info.mod_time,
-            parts: &info.parts,
-            delete_marker: info.deleted,
-        }
-    }
-}
-
-/// A pure decision on quorum-validated metadata. Authorization belongs to the caller.
-pub trait GetObjectReadCondition: Send + Sync {
-    fn is_terminal(&self, metadata: GetObjectReadMetadata<'_>) -> bool;
-}
-
-tokio::task_local! {
-    static READ_CONDITION: Arc<dyn GetObjectReadCondition>;
-}
-
-/// Applies a condition to this reader call, without inheriting it into shared producers.
-pub async fn with_get_object_read_condition<F: Future>(condition: Arc<dyn GetObjectReadCondition>, read: F) -> F::Output {
-    READ_CONDITION.scope(condition, read).await
-}
-
-pub(crate) fn get_object_read_condition_is_terminal(metadata: GetObjectReadMetadata<'_>) -> bool {
-    READ_CONDITION
-        .try_with(|condition| condition.is_terminal(metadata))
-        .unwrap_or(false)
-}
-
-pub(crate) fn get_object_read_condition_is_active() -> bool {
-    READ_CONDITION.try_with(|_| ()).is_ok()
-}
-
-#[cfg(test)]
-mod tests {
-    use super::*;
-
-    struct Stop;
-    impl GetObjectReadCondition for Stop {
-        fn is_terminal(&self, _: GetObjectReadMetadata<'_>) -> bool {
-            true
-        }
-    }
-    fn metadata() -> GetObjectReadMetadata<'static> {
-        GetObjectReadMetadata {
-            bucket: "bucket",
-            object: "object",
-            etag: Some("etag"),
-            mod_time: None,
-            parts: &[],
-            delete_marker: false,
-        }
-    }
-
-    #[tokio::test]
-    async fn read_condition_scope_does_not_escape_or_enter_shared_producers() {
-        assert!(!get_object_read_condition_is_terminal(metadata()));
-        with_get_object_read_condition(Arc::new(Stop), async {
-            assert!(get_object_read_condition_is_terminal(metadata()));
-            assert!(
-                !tokio::spawn(async { get_object_read_condition_is_terminal(metadata()) })
-                    .await
-                    .expect("shared producer")
-            );
-        })
-        .await;
-        assert!(!get_object_read_condition_is_terminal(metadata()));
-    }
-
-    #[tokio::test]
-    async fn dropping_a_pending_read_removes_its_condition_scope() {
-        let mut read = Box::pin(with_get_object_read_condition(Arc::new(Stop), async {
-            assert!(get_object_read_condition_is_terminal(metadata()));
-            std::future::pending::<()>().await;
-        }));
-        assert!(futures::poll!(&mut read).is_pending());
-        drop(read);
-        assert!(!get_object_read_condition_is_terminal(metadata()));
-    }
-}

crates/ecstore/src/object_api/readers.rs

@@ -528,14 +528,6 @@ pub struct GetObjectReader {
 }
 
 impl GetObjectReader {
-    pub(crate) fn condition_stopped(object_info: ObjectInfo) -> Self {
-        Self {
-            stream: Box::new(ConditionStoppedBody),
-            object_info,
-            buffered_body: None,
-            body_source: GetObjectBodySource::Unprobed,
-        }
-    }
     /// Builds a fully materialized reader from a cache-coordinated body.
     pub fn from_cache_body(mut object_info: ObjectInfo, body: Bytes) -> Result<Self> {
         object_info.size = i64::try_from(body.len()).map_err(|_| Error::other("cached GET body length exceeds i64::MAX"))?;
@@ -560,17 +552,6 @@ impl GetObjectReader {
     }
 }
 
-struct ConditionStoppedBody;
-
-impl AsyncRead for ConditionStoppedBody {
-    fn poll_read(self: Pin<&mut Self>, _cx: &mut Context<'_>, buf: &mut ReadBuf<'_>) -> Poll<std::io::Result<()>> {
-        if buf.remaining() == 0 {
-            return Poll::Ready(Ok(()));
-        }
-        Poll::Ready(Err(std::io::Error::other("object read condition must be handled before body access")))
-    }
-}
-
 enum ReadTransform {
     // Written but never read by production code: the enclosing struct already
     // carries the same pair as `storage_offset`/`storage_length`. They survive

crates/ecstore/src/object_api/types.rs

@@ -938,10 +938,6 @@ pub struct ObjectOptions {
     /// A genuine target write guard; loss fences alone do not authorize a borrowed write.
     #[doc(hidden)]
     pub write_lock_context: Option<WriteLockContext>,
-    /// In-process delete ownership for late filesystem completion. Never
-    /// serialized or copied into object metadata.
-    #[doc(hidden)]
-    pub delete_mutation_owner: Option<Arc<dyn Send + Sync>>,
     /// Internal read-only inspection must not enqueue metadata or payload repairs.
     pub suppress_read_repair: bool,
     /// Control-plane writers that immediately read or CAS the same namespace
@@ -1006,9 +1002,6 @@ pub struct ObjectOptions {
     /// Proves an upper layer holds the bucket lifecycle sentinel. A separate
     /// fence avoids recursively acquiring the read lock behind a queued writer.
     pub bucket_lifecycle_lock_fence: Option<NamespaceLockFence>,
-    /// Retains the actual lifecycle guard for an owned multipart commit. A
-    /// lock-loss fence alone does not keep a cancelled caller's lock alive.
-    pub bucket_lifecycle_lock_owner: Option<Arc<rustfs_lock::NamespaceLockGuard>>,
     pub replication_request: bool,
     /// True when the inbound request carried the
     /// `{x-rustfs-,x-minio-}source-proxy-request` header family with the
@@ -1165,10 +1158,8 @@ impl std::fmt::Debug for ObjectOptions {
             .field("http_preconditions", &self.http_preconditions.is_some())
             .field("delete_replication", &self.delete_replication.is_some())
             .field("delete_replication_config_snapshot", &self.delete_replication_config_snapshot)
-            .field("delete_mutation_owner", &self.delete_mutation_owner.is_some())
             .field("namespace_lock_fence", &self.namespace_lock_fence.is_some())
             .field("bucket_lifecycle_lock_fence", &self.bucket_lifecycle_lock_fence.is_some())
-            .field("bucket_lifecycle_lock_owner", &self.bucket_lifecycle_lock_owner.is_some())
             .field("replication_request", &self.replication_request)
             .field("proxy_request", &self.proxy_request)
             .field("proxy_header_set", &self.proxy_header_set)

crates/ecstore/src/runtime/sources.rs

@@ -50,7 +50,6 @@ use s3s::dto::BucketLifecycleConfiguration;
 use s3s::region::Region;
 use tokio::sync::{OwnedRwLockReadGuard, RwLock};
 use tokio_util::sync::CancellationToken;
-#[cfg(test)]
 use tonic::transport::Channel;
 use uuid::Uuid;
 
@@ -252,6 +251,10 @@ pub(crate) async fn root_disk_threshold_for_erasure_disk() -> Option<u64> {
     }
 }
 
+pub(crate) async fn cached_node_channel(addr: &str) -> Option<Channel> {
+    rustfs_common::cached_connection(addr).await
+}
+
 #[cfg(test)]
 pub(crate) async fn cache_test_node_channel(addr: String, channel: Channel) {
     rustfs_common::cache_connection(addr, channel).await;

crates/ecstore/src/services/rebalance/control.rs

@@ -18,8 +18,8 @@ use super::{
     encode_rebalance_stop_propagation_record,
 };
 use crate::core::pools::{
-    PoolMeta, PoolRebalanceActivationFence, PoolRebalanceWorkerFence, acquire_pool_activation_fleet_proof,
-    acquire_pool_rebalance_activation_locks, acquire_pool_rebalance_worker_locks, pool_meta_has_active_decommission,
+    PoolMeta, PoolRebalanceActivationFence, acquire_pool_activation_fleet_proof, acquire_pool_rebalance_activation_locks,
+    pool_meta_has_active_decommission,
 };
 use crate::error::{Error, Result};
 use crate::object_api::ObjectOptions;
@@ -77,33 +77,10 @@ fn ensure_rebalance_activation_pool_meta_allowed(meta: &PoolMeta) -> Result<()>
 }
 
 pub(super) enum RebalanceWorkerActivationFence {
-    Ready(Box<PoolRebalanceWorkerFence>),
+    Ready(Box<PoolRebalanceActivationFence>),
     NotStartedTerminal,
 }
 
-async fn persisted_rebalance_requires_worker_activation<S>(pool: Arc<S>, expected_id: &str) -> Result<bool>
-where
-    S: EcstoreObjectIO,
-{
-    let mut persisted = RebalanceMeta::new();
-    persisted
-        .load_with_opts(
-            pool,
-            ObjectOptions {
-                no_lock: true,
-                ..Default::default()
-            },
-        )
-        .await?;
-    if persisted.id != expected_id {
-        return Err(Error::other(format!(
-            "rebalance metadata changed before worker activation: expected {expected_id}, found {}",
-            persisted.id
-        )));
-    }
-    Ok(crate::services::rebalance::rebalance_requires_worker_activation(&persisted))
-}
-
 pub(super) struct RebalanceRunGuard {
     _local_guard: tokio::sync::OwnedRwLockReadGuard<()>,
     persisted_guard: rustfs_lock::NamespaceLockGuard,
@@ -600,58 +577,48 @@ impl ECStore {
     where
         S: EcstoreObjectIO + StorageNamespaceLocking<Error = Error, NamespaceLock = rustfs_lock::NamespaceLockWrapper>,
     {
-        // Lock order: pool_meta_save_gate -> rebalance.bin -> pool.bin. These
-        // shared fences admit a peer without waiting for the run's own entries.
+        // Lock order: pool_meta_save_gate -> rebalance.bin -> pool.bin.
         let mut pool_meta_guard = self.pool_meta_save_gate.lock().await;
         pool_meta_guard.ensure_write_safe("rebalance worker activation")?;
-        let mut worker_fence = acquire_pool_rebalance_worker_locks(pool.clone()).await?;
+        // Classify the durable rebalance record while holding both namespace
+        // fences. A terminal record is a no-op and must not depend on the
+        // notification subsystem having published a fleet proof yet.
+        let mut activation_fence = acquire_pool_rebalance_activation_locks(pool.clone(), None).await?;
         let pool_meta = self
-            .load_runtime_pool_meta_under_worker_fence(&mut pool_meta_guard, &worker_fence, "rebalance worker activation")
+            .load_runtime_pool_meta_under_activation_fence(&mut pool_meta_guard, &activation_fence, "rebalance worker activation")
             .await?;
         ensure_rebalance_activation_pool_meta_allowed(&pool_meta)?;
-        let needs_activation = persisted_rebalance_requires_worker_activation(pool, expected_id).await?;
-        worker_fence.ensure_held()?;
-        if !needs_activation {
+
+        let mut persisted = RebalanceMeta::new();
+        persisted
+            .load_with_opts(
+                pool,
+                ObjectOptions {
+                    no_lock: true,
+                    ..Default::default()
+                },
+            )
+            .await?;
+        if persisted.id != expected_id {
+            return Err(Error::other(format!(
+                "rebalance metadata changed before worker activation: expected {expected_id}, found {}",
+                persisted.id
+            )));
+        }
+
+        activation_fence.ensure_held()?;
+        if !crate::services::rebalance::rebalance_requires_worker_activation(&persisted) {
             return Ok(RebalanceWorkerActivationFence::NotStartedTerminal);
         }
 
-        let fleet_proof = acquire_pool_activation_fleet_proof(&self.ctx).await?;
-        worker_fence.set_fleet_proof(fleet_proof);
-        worker_fence.ensure_held()?;
-        Ok(RebalanceWorkerActivationFence::Ready(Box::new(worker_fence)))
-    }
-
-    pub(super) async fn fence_rebalance_worker_metadata_update<S>(
-        &self,
-        pool: Arc<S>,
-        expected_id: &str,
-    ) -> Result<Option<Box<PoolRebalanceActivationFence>>>
-    where
-        S: EcstoreObjectIO + StorageNamespaceLocking<Error = Error, NamespaceLock = rustfs_lock::NamespaceLockWrapper>,
-    {
-        // The admission read fences must already be dropped. Reacquire in the
-        // original order and revalidate every durable boundary across that gap.
-        let mut pool_meta_guard = self.pool_meta_save_gate.lock().await;
-        pool_meta_guard.ensure_write_safe("rebalance worker metadata update")?;
-        let mut activation_fence = acquire_pool_rebalance_activation_locks(pool.clone(), None).await?;
-        let pool_meta = self
-            .load_runtime_pool_meta_under_activation_fence(
-                &mut pool_meta_guard,
-                &activation_fence,
-                "rebalance worker metadata update",
-            )
-            .await?;
-        ensure_rebalance_activation_pool_meta_allowed(&pool_meta)?;
-        let needs_activation = persisted_rebalance_requires_worker_activation(pool, expected_id).await?;
-        activation_fence.ensure_held()?;
-        if !needs_activation {
-            return Ok(None);
-        }
-
+        // Active worker admission still requires the fail-closed fleet proof.
+        // Attach it immediately before the final fence validation so expiry or
+        // topology changes are checked again at every later commit boundary.
         let fleet_proof = acquire_pool_activation_fleet_proof(&self.ctx).await?;
         activation_fence.set_fleet_proof(fleet_proof);
         activation_fence.ensure_held()?;
-        Ok(Some(Box::new(activation_fence)))
+
+        Ok(RebalanceWorkerActivationFence::Ready(Box::new(activation_fence)))
     }
 
     #[tracing::instrument(skip_all)]
@@ -1849,107 +1816,6 @@ mod tests {
         assert_activation_locks_released(&store).await;
     }
 
-    #[tokio::test]
-    #[serial_test::serial]
-    async fn rebalance_init_missing_fleet_proof_preserves_metadata_and_rechecks_competing_run() {
-        let previous_id = "completed-before-capability-refresh";
-        let terminal = RebalanceMeta {
-            id: previous_id.to_string(),
-            stopped_at: Some(OffsetDateTime::now_utc()),
-            pool_stats: vec![RebalanceStats::default(); 2],
-            ..Default::default()
-        };
-        let (_temp_dirs, store, competing_store) = crate::services::rebalance::test_two_pool_stores(Some(terminal.clone())).await;
-        let disk_stats = vec![
-            DiskStat {
-                total_space: 100,
-                available_space: 0,
-            },
-            DiskStat {
-                total_space: 100,
-                available_space: 100,
-            },
-        ];
-        let initial_epoch = store.ctx.data_movement_operation_epoch();
-        let proof_guard = crate::services::notification_sys::without_cross_pool_fence_fleet_proof_for_test();
-        set_rebalance_disk_stats_override_for_test(store.id, disk_stats.clone());
-        let err = store
-            .init_rebalance_start(vec!["waiting-request".to_string()])
-            .await
-            .expect_err("initialization must fail closed while the live proof is missing");
-        assert!(
-            err.to_string()
-                .contains("pool activation requires a live fleet capability proof")
-        );
-        assert!(
-            crate::core::pools::is_pool_activation_fleet_proof_error(&err),
-            "the admin handler retries init only on this error shape: {err}"
-        );
-        assert_eq!(store.ctx.data_movement_operation_epoch(), initial_epoch);
-        {
-            let local = store.rebalance_meta.read().await;
-            let local = local.as_ref().expect("failed initialization must preserve the previous run");
-            assert_eq!(local.id, previous_id);
-            assert_eq!(local.stopped_at, terminal.stopped_at);
-            assert!(local.cancel.is_none(), "failed initialization must not start a worker");
-        }
-        let mut persisted = RebalanceMeta::new();
-        persisted
-            .load(store.pools[0].clone())
-            .await
-            .expect("failed initialization must leave durable metadata readable");
-        assert_eq!(persisted.id, previous_id);
-        assert_eq!(persisted.stopped_at, terminal.stopped_at);
-
-        drop(proof_guard);
-        set_rebalance_disk_stats_override_for_test(competing_store.id, disk_stats.clone());
-        let winning_id = competing_store
-            .init_rebalance_start(vec!["winning-request".to_string()])
-            .await
-            .expect("a ready request must be able to initialize one run after the failed attempt");
-        assert_ne!(winning_id, previous_id);
-
-        // The waiting request still holds its old local snapshot. Its retry must
-        // recheck durable state under the activation locks before publishing.
-        set_rebalance_disk_stats_override_for_test(store.id, disk_stats);
-        let remote_conflict = store
-            .init_rebalance_start(vec!["waiting-request".to_string()])
-            .await
-            .expect_err("the waiting request must not replace a durable competing run");
-        assert!(
-            matches!(
-                crate::data_movement::data_movement_stage_source(&remote_conflict),
-                Some(Error::RebalanceAlreadyRunning)
-            ),
-            "the save-stage error must retain the typed competing-run conflict: {remote_conflict:?}"
-        );
-        let local_conflict = competing_store
-            .init_rebalance_start(vec!["another-request".to_string()])
-            .await
-            .expect_err("a local retry must not replace its active run");
-        assert!(
-            matches!(local_conflict, Error::RebalanceAlreadyRunning),
-            "the local admission check must report the competing-run conflict: {local_conflict:?}"
-        );
-        persisted
-            .load(store.pools[0].clone())
-            .await
-            .expect("the competing run must remain durable after both rejected retries");
-        assert_eq!(persisted.id, winning_id);
-        assert_eq!(persisted.pool_stats[0].buckets, vec!["winning-request".to_string()]);
-        assert_eq!(
-            store
-                .rebalance_meta
-                .read()
-                .await
-                .as_ref()
-                .expect("old local run must survive")
-                .id,
-            previous_id
-        );
-        assert_eq!(store.ctx.data_movement_operation_epoch(), initial_epoch + 1);
-    }
-
     #[tokio::test]
     #[serial_test::serial]
     async fn rebalance_worker_skips_terminal_metadata_without_fleet_proof() {
@@ -2008,128 +1874,6 @@ mod tests {
         );
     }
 
-    async fn assert_worker_write_upgrade_revalidates_durable_state(change: &str) {
-        let rebalance_id = "rebalance-worker-write-upgrade";
-        let active = RebalanceMeta {
-            id: rebalance_id.to_string(),
-            percent_free_goal: 0.5,
-            pool_stats: vec![
-                RebalanceStats {
-                    participating: true,
-                    init_capacity: 100,
-                    info: RebalanceInfo {
-                        status: RebalStatus::Started,
-                        ..Default::default()
-                    },
-                    ..Default::default()
-                },
-                RebalanceStats::default(),
-            ],
-            ..Default::default()
-        };
-        let (_temp_dirs, store, peer) =
-            crate::services::rebalance::test_two_pool_stores_with_isolated_node_contexts(Some(active.clone())).await;
-        let arrived = Arc::new(tokio::sync::Notify::new());
-        let release = Arc::new(tokio::sync::Notify::new());
-        let worker = Arc::clone(&store);
-        let mut tasks = tokio::task::JoinSet::new();
-        tasks.spawn(
-            super::super::runtime::REBALANCE_WORKER_WRITE_UPGRADE_BARRIER
-                .scope((Arc::clone(&arrived), Arc::clone(&release)), async move {
-                    worker.start_rebalance_for_id(rebalance_id).await
-                }),
-        );
-        tokio::time::timeout(std::time::Duration::from_secs(30), arrived.notified())
-            .await
-            .expect("empty-queue activation must release its read fences before requesting write authority");
-
-        let mut changed = active.clone();
-        match change {
-            "stopped" => {
-                changed.stopped_at = Some(OffsetDateTime::now_utc());
-                changed.pool_stats[0].info.status = RebalStatus::Stopped;
-            }
-            "replaced" => changed.id = "replacement-rebalance-run".to_string(),
-            "decommission" => {
-                let mut pool_meta = peer.pool_meta.read().await.clone();
-                pool_meta.pools[0].decommission = Some(crate::core::pools::PoolDecommissionInfo {
-                    start_time: Some(OffsetDateTime::now_utc()),
-                    ..Default::default()
-                });
-                pool_meta
-                    .save(peer.pools.clone())
-                    .await
-                    .expect("a competing pool metadata writer must acquire the released pool fence");
-            }
-            _ => panic!("unknown durable state transition"),
-        }
-        changed
-            .save_with_opts(
-                peer.pools[0].clone(),
-                ObjectOptions {
-                    write_completion: crate::object_api::WriteCompletion::TailDrained,
-                    ..Default::default()
-                },
-            )
-            .await
-            .expect("a competing run writer must acquire the released rebalance fence");
-        release.notify_one();
-        let result = tokio::time::timeout(std::time::Duration::from_secs(30), tasks.join_next())
-            .await
-            .expect("activation must finish after the durable transition")
-            .expect("activation task must be tracked")
-            .expect("activation must not panic");
-        match change {
-            "stopped" => result.expect("a durably stopped run must become an activation no-op"),
-            "replaced" => assert!(
-                result
-                    .expect_err("a replaced run must reject stale activation")
-                    .to_string()
-                    .contains("expected"),
-            ),
-            "decommission" => assert!(matches!(result, Err(Error::DecommissionAlreadyRunning))),
-            _ => unreachable!(),
-        }
-        let mut persisted = RebalanceMeta::new();
-        persisted
-            .load(peer.pools[0].clone())
-            .await
-            .expect("the competing durable state must remain readable");
-        assert_eq!(persisted.id, changed.id, "activation must not overwrite the run identity");
-        assert_eq!(persisted.stopped_at, changed.stopped_at, "activation must not overwrite the stop marker");
-        assert_eq!(persisted.pool_stats[0].info.status, changed.pool_stats[0].info.status);
-        assert!(
-            store
-                .rebalance_meta
-                .read()
-                .await
-                .as_ref()
-                .expect("local metadata must remain present")
-                .cancel
-                .is_none(),
-            "rejected activation must not install a worker token",
-        );
-        assert_activation_locks_released(&store).await;
-    }
-
-    #[tokio::test]
-    #[serial_test::serial]
-    async fn rebalance_worker_write_upgrade_rechecks_stop() {
-        assert_worker_write_upgrade_revalidates_durable_state("stopped").await;
-    }
-
-    #[tokio::test]
-    #[serial_test::serial]
-    async fn rebalance_worker_write_upgrade_rechecks_run_identity() {
-        assert_worker_write_upgrade_revalidates_durable_state("replaced").await;
-    }
-
-    #[tokio::test]
-    #[serial_test::serial]
-    async fn rebalance_worker_write_upgrade_rechecks_decommission() {
-        assert_worker_write_upgrade_revalidates_durable_state("decommission").await;
-    }
-
     #[tokio::test]
     #[serial_test::serial]
     async fn rebalance_activation_adopts_commit_after_post_save_fence_loss() {

crates/ecstore/src/services/rebalance/entry.rs

@@ -50,7 +50,7 @@ fn ensure_rebalance_entry_active(cancel: &CancellationToken) -> Result<()> {
     Ok(())
 }
 
-#[cfg(any(test, feature = "test-util"))]
+#[cfg(test)]
 tokio::task_local! {
     static REBALANCE_ENTRY_RUN_FENCE_BARRIER: (Arc<tokio::sync::Notify>, Arc<tokio::sync::Notify>);
 }
@@ -268,7 +268,7 @@ impl ECStore {
             self.rebalance_run_guard(rebalance_id.as_ref(), "rebalance entry")
         })
         .await?;
-        #[cfg(any(test, feature = "test-util"))]
+        #[cfg(test)]
         if let Ok((arrived, release)) = REBALANCE_ENTRY_RUN_FENCE_BARRIER.try_with(Clone::clone) {
             arrived.notify_one();
             release.notified().await;
@@ -836,8 +836,7 @@ pub mod test_util {
         _temp_dirs: Vec<tempfile::TempDir>,
         store: Arc<ECStore>,
         cancel: CancellationToken,
-        entry_arrived: Arc<tokio::sync::Notify>,
-        entry_release: Arc<tokio::sync::Notify>,
+        barrier: crate::data_movement::SourceCleanupDeleteBarrier,
         stop_probe: super::super::control::RebalanceStopWaitProbe,
         entry_task: Option<tokio::task::JoinHandle<Result<RebalanceEntryOutcome>>>,
     }
@@ -898,36 +897,31 @@ pub mod test_util {
                 cached: Some(file_meta),
                 reusable: false,
             };
-            let entry_arrived = Arc::new(tokio::sync::Notify::new());
-            let entry_release = Arc::new(tokio::sync::Notify::new());
+            let barrier = crate::data_movement::SourceCleanupDeleteBarrier::install(bucket, object);
             let entry_store = Arc::clone(&store);
             let entry_set = Arc::clone(&set);
             let entry_cancel = cancel.clone();
-            let entry_task = tokio::spawn(REBALANCE_ENTRY_RUN_FENCE_BARRIER.scope(
-                (Arc::clone(&entry_arrived), Arc::clone(&entry_release)),
-                async move {
-                    entry_store
-                        .rebalance_entry(
-                            RebalanceEntryTarget {
-                                bucket: bucket.to_string(),
-                                pool_index: 0,
-                            },
-                            entry,
-                            entry_set,
-                            Arc::new(RebalanceBucketConfigs::default()),
-                            Arc::from(rebalance_id),
-                            entry_cancel,
-                        )
-                        .await
-                },
-            ));
+            let entry_task = tokio::spawn(async move {
+                entry_store
+                    .rebalance_entry(
+                        RebalanceEntryTarget {
+                            bucket: bucket.to_string(),
+                            pool_index: 0,
+                        },
+                        entry,
+                        entry_set,
+                        Arc::new(RebalanceBucketConfigs::default()),
+                        Arc::from(rebalance_id),
+                        entry_cancel,
+                    )
+                    .await
+            });
 
             Self {
                 _temp_dirs,
                 store,
                 cancel,
-                entry_arrived,
-                entry_release,
+                barrier,
                 stop_probe: super::super::control::RebalanceStopWaitProbe::install(rebalance_id),
                 entry_task: Some(entry_task),
             }
@@ -938,9 +932,7 @@ pub mod test_util {
         }
 
         pub async fn wait_until_entry_paused(&self) {
-            tokio::time::timeout(std::time::Duration::from_secs(30), self.entry_arrived.notified())
-                .await
-                .expect("the entry should acquire its run fence before pausing");
+            self.barrier.wait_until_paused().await;
         }
 
         pub async fn wait_until_admission_cancelled(&self) {
@@ -954,7 +946,7 @@ pub mod test_util {
         }
 
         pub fn release_entry(&self) {
-            self.entry_release.notify_one();
+            self.barrier.release();
         }
 
         pub async fn assert_entry_cancelled(&mut self) {
@@ -972,7 +964,7 @@ pub mod test_util {
 
     impl Drop for PausedRebalanceEntryTestFixture {
         fn drop(&mut self) {
-            self.entry_release.notify_one();
+            self.barrier.release();
             if let Some(task) = self.entry_task.take() {
                 task.abort();
             }
@@ -1440,7 +1432,7 @@ mod tests {
 
     #[tokio::test]
     #[serial_test::serial]
-    async fn real_rebalance_peer_activates_while_entry_retains_run_fence() {
+    async fn real_rebalance_entry_progresses_while_peer_activation_waits_for_run_fence() {
         const REBALANCE_ID: &str = "rebalance-peer-activation-lock-order";
         let (_temp_dirs, store, peer) = crate::services::rebalance::test_two_pool_stores_with_isolated_node_contexts(Some(
             active_rebalance_meta(REBALANCE_ID),
@@ -1456,7 +1448,7 @@ mod tests {
         let bucket = crate::disk::RUSTFS_META_BUCKET;
         let object = "rebalance-peer-activation-object";
         let version_id = uuid::Uuid::new_v4();
-        let payload = b"peer activation must not wait for this entry to drain".repeat(1024);
+        let payload = b"entry must drain before peer activation takes the pool fence".repeat(1024);
         let source_set = store.pools[0].get_disks_by_key(object);
         let target_set = store.pools[1].get_disks_by_key(object);
         let opts = ObjectOptions {
@@ -1516,12 +1508,7 @@ mod tests {
         );
         tokio::time::timeout(StdDuration::from_secs(30), attempted.notified())
             .await
-            .expect("peer activation must attempt the persisted rebalance fence");
-        tokio::time::timeout(StdDuration::from_secs(30), activation_result)
-            .await
-            .expect("peer activation must finish while the entry remains paused")
-            .expect("peer activation result should be sent")
-            .expect("joining the same run must not require its active entry to release the run fence");
+            .expect("peer activation must attempt the persisted rebalance write fence");
         release.notify_one();
 
         tokio::time::timeout(StdDuration::from_secs(30), async {
@@ -1536,6 +1523,10 @@ mod tests {
         })
         .await
         .expect("entry and peer activation must both make progress");
+        activation_result
+            .await
+            .expect("peer activation result should be sent")
+            .expect("peer activation must not time out behind the entry it blocks");
 
         let mut reader = target_set
             .get_object_reader(bucket, object, None, HeaderMap::new(), &opts)

crates/ecstore/src/services/rebalance/runtime.rs

@@ -27,11 +27,6 @@ use tokio::time::{Duration, Instant};
 use tokio_util::sync::CancellationToken;
 use tracing::{debug, error, info, warn};
 
-#[cfg(test)]
-tokio::task_local! {
-    pub(super) static REBALANCE_WORKER_WRITE_UPGRADE_BARRIER: (Arc<tokio::sync::Notify>, Arc<tokio::sync::Notify>);
-}
-
 pub(super) fn should_fail_repeated_rebalance_bucket_defer(
     deferred_buckets: &mut HashSet<String>,
     bucket: &str,
@@ -218,22 +213,6 @@ impl ECStore {
         }
 
         if must_persist {
-            // Shared admission grants no write authority. Release it before
-            // taking the exclusive metadata fences, then recheck stop/run/pool
-            // state before publishing the locally staged terminal candidate.
-            drop(activation_fence);
-            #[cfg(test)]
-            if let Ok((arrived, release)) = REBALANCE_WORKER_WRITE_UPGRADE_BARRIER.try_with(Clone::clone) {
-                arrived.notify_one();
-                release.notified().await;
-            }
-            let Some(activation_fence) = self
-                .fence_rebalance_worker_metadata_update(pool.clone(), expected_id.as_ref())
-                .await?
-            else {
-                cancel_tx.cancel();
-                return Ok(false);
-            };
             let save_result = resolve_rebalance_meta_save_result(
                 self.save_rebalance_meta_under_activation_fence(
                     pool,
@@ -263,14 +242,13 @@ impl ECStore {
                 cancel_tx.cancel();
                 return Err(err);
             }
-        } else {
-            if let Err(err) = activation_fence.ensure_held() {
-                let mut rebalance_meta = self.rebalance_meta.write().await;
-                rollback_local_rebalance_worker_activation(rebalance_meta.as_mut(), expected_id.as_ref(), &rx);
-                return Err(err);
-            }
-            drop(activation_fence);
         }
+        if !must_persist && let Err(err) = activation_fence.ensure_held() {
+            let mut rebalance_meta = self.rebalance_meta.write().await;
+            rollback_local_rebalance_worker_activation(rebalance_meta.as_mut(), expected_id.as_ref(), &rx);
+            return Err(err);
+        }
+        drop(activation_fence);
 
         if activation_outcome != RebalanceLocalActivationOutcome::Started {
             return Ok(must_persist);

crates/ecstore/src/services/tier/tier.rs

@@ -4715,6 +4715,23 @@ impl TierConfigMgr {
     }
 
     async fn reload_handle_with<S>(handle: &Arc<RwLock<Self>>, api: Arc<S>) -> io::Result<()>
+    where
+        S: EcstoreObjectIO
+            + EcstoreObjectOperations
+            + NamespaceLocking<Error = Error, NamespaceLock = rustfs_lock::NamespaceLockWrapper>
+            + ListOperations<Error = Error, ListObjectsV2Info = StorageListObjectsV2Info<ObjectInfo>>,
+    {
+        // Periodic and committed-mutation refreshes call this shared entry directly.
+        let reload = Self::reload_handle_inner(handle, api);
+        #[cfg(feature = "diagnostic-read-version-caller")]
+        let reload = crate::cluster::rpc::runtime_sources::with_read_version_entry(
+            rustfs_io_metrics::internode_metrics::ReadVersionEntrySource::TierConfigReload,
+            reload,
+        );
+        reload.await
+    }
+
+    async fn reload_handle_inner<S>(handle: &Arc<RwLock<Self>>, api: Arc<S>) -> io::Result<()>
     where
         S: EcstoreObjectIO
             + EcstoreObjectOperations
@@ -12533,6 +12550,51 @@ mod tests {
         assert!(reloaded.iter().any(|intent| intent.mutation_id == prepared_id));
     }
 
+    #[cfg(feature = "diagnostic-read-version-caller")]
+    #[tokio::test]
+    async fn diagnostic_reload_shared_entry_preserves_source_and_api() {
+        use crate::api::diagnostic::{ReadVersionEntrySource, current_read_version_entry_source, with_read_version_entry};
+        use rustfs_io_metrics::s3_http_metrics::{
+            S3HttpRequestGuard, current_s3_http_operation_context, with_isolated_s3_http_operation_context,
+        };
+
+        let store = Arc::new(CasConfigStore::default());
+        empty_mgr()
+            .save_tiering_config_if_current(store.clone(), None)
+            .await
+            .expect("empty tier config fixture should persist");
+        let mut request = S3HttpRequestGuard::new("GET");
+        let get = request.in_scope(|| {
+            rustfs_io_metrics::record_s3_op(rustfs_s3_ops::S3Operation::GetObject);
+            current_s3_http_operation_context().expect("GetObject context should be captured")
+        });
+
+        for (earlier, api) in [(None, None), (Some(ReadVersionEntrySource::ConfigRead), Some(get))] {
+            store.reload_contexts.lock().expect("context recorder").clear();
+            let handle = TierConfigMgr::new();
+            with_isolated_s3_http_operation_context(api, async {
+                let reload = TierConfigMgr::reload_handle_with(&handle, store.clone());
+                if let Some(source) = earlier {
+                    with_read_version_entry(source, reload).await
+                } else {
+                    reload.await
+                }
+            })
+            .await
+            .expect("shared reload entry should load config and scan intents");
+
+            let observed = store.reload_contexts.lock().expect("context recorder");
+            assert!(observed.iter().any(|(boundary, _, _)| *boundary == "reader"));
+            assert!(observed.iter().any(|(boundary, _, _)| *boundary == "list"));
+            let expected = earlier.unwrap_or(ReadVersionEntrySource::TierConfigReload);
+            assert!(observed.iter().all(|(_, source, operation)| {
+                *source == Some(expected) && *operation == api.map(|context| context.operation_name())
+            }));
+            assert_eq!(current_read_version_entry_source(), None);
+            assert_eq!(current_s3_http_operation_context(), None);
+        }
+    }
+
     #[tokio::test]
     async fn reload_handle_keeps_committed_block_until_publish_swaps_generation() {
         let store = Arc::new(CasConfigStore::default());
@@ -14725,8 +14787,17 @@ mod tests {
         release: tokio::sync::Notify,
     }
 
+    #[cfg(feature = "diagnostic-read-version-caller")]
+    type ReloadDiagnosticObservation = (
+        &'static str,
+        Option<rustfs_io_metrics::internode_metrics::ReadVersionEntrySource>,
+        Option<&'static str>,
+    );
+
     #[derive(Debug)]
     struct CasConfigStore {
+        #[cfg(feature = "diagnostic-read-version-caller")]
+        reload_contexts: Mutex<Vec<ReloadDiagnosticObservation>>,
         objects: tokio::sync::Mutex<HashMap<String, (Vec<u8>, String)>>,
         legacy_state: tokio::sync::Mutex<Option<Vec<u8>>>,
         if_match_race_rewrite: tokio::sync::Mutex<std::collections::VecDeque<(String, Vec<u8>)>>,
@@ -14750,6 +14821,8 @@ mod tests {
     impl Default for CasConfigStore {
         fn default() -> Self {
             Self {
+                #[cfg(feature = "diagnostic-read-version-caller")]
+                reload_contexts: Mutex::new(Vec::new()),
                 objects: tokio::sync::Mutex::new(HashMap::new()),
                 legacy_state: tokio::sync::Mutex::new(None),
                 if_match_race_rewrite: tokio::sync::Mutex::new(std::collections::VecDeque::new()),
@@ -14904,6 +14977,12 @@ mod tests {
             _headers: Self::HeaderMap,
             _opts: &Self::ObjectOptions,
         ) -> Result<Self::GetObjectReader> {
+            #[cfg(feature = "diagnostic-read-version-caller")]
+            self.reload_contexts.lock().expect("context recorder").push((
+                "reader",
+                crate::api::diagnostic::current_read_version_entry_source(),
+                rustfs_io_metrics::s3_http_metrics::current_s3_http_operation_name(),
+            ));
             let legacy_config_path = tier_config_path(TIER_CONFIG_LEGACY_FILE);
             let (data, etag) = if object == legacy_config_path {
                 let state = self.legacy_state.lock().await;
@@ -15259,6 +15338,12 @@ mod tests {
             _start_after: Option<String>,
             _incl_deleted: bool,
         ) -> Result<Self::ListObjectsV2Info> {
+            #[cfg(feature = "diagnostic-read-version-caller")]
+            self.reload_contexts.lock().expect("context recorder").push((
+                "list",
+                crate::api::diagnostic::current_read_version_entry_source(),
+                rustfs_io_metrics::s3_http_metrics::current_s3_http_operation_name(),
+            ));
             if matches!(
                 prefix,
                 crate::services::tier::tier_mutation_intent::TIER_MUTATION_INTENT_RECORD_PREFIX

crates/ecstore/src/set_disk/conditional_read_bench.rs

@@ -1,63 +0,0 @@
-// Copyright 2024 RustFS Team
-//
-// Licensed under the Apache License, Version 2.0 (the "License");
-// you may not use this file except in compliance with the License.
-// You may obtain a copy of the License at
-//
-//     http://www.apache.org/licenses/LICENSE-2.0
-//
-// Unless required by applicable law or agreed to in writing, software
-// distributed under the License is distributed on an "AS IS" BASIS,
-// WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
-// See the License for the specific language governing permissions and
-// limitations under the License.
-
-//! Test-util-only controls for the paired read benchmark's dedicated bucket.
-
-use std::sync::atomic::{AtomicBool, Ordering};
-
-static ACTIVE: AtomicBool = AtomicBool::new(false);
-
-/// Keeps metadata off the cache in the benchmark's isolated bucket so GET
-/// metadata reads exercise the configured slow disk. Body-cache policy
-/// remains intact.
-#[must_use]
-pub struct ConditionalReadBenchmarkMetadataGuard {
-    _private: (),
-}
-
-impl ConditionalReadBenchmarkMetadataGuard {
-    pub fn acquire() -> Self {
-        assert!(
-            ACTIVE
-                .compare_exchange(false, true, Ordering::AcqRel, Ordering::Acquire)
-                .is_ok()
-        );
-        Self { _private: () }
-    }
-}
-
-impl Drop for ConditionalReadBenchmarkMetadataGuard {
-    fn drop(&mut self) {
-        ACTIVE.store(false, Ordering::Release);
-    }
-}
-
-pub(super) fn cache_bypass_applies(bucket: &str) -> bool {
-    bucket == "conditional-bench" && ACTIVE.load(Ordering::Acquire)
-}
-
-#[cfg(test)]
-mod tests {
-    use super::*;
-
-    #[test]
-    fn benchmark_metadata_profile_is_scoped_to_its_bucket_and_owner() {
-        assert!(!cache_bypass_applies("conditional-bench"));
-        let owner = ConditionalReadBenchmarkMetadataGuard::acquire();
-        assert!(cache_bypass_applies("conditional-bench"));
-        assert!(!cache_bypass_applies("other-bucket"));
-        drop(owner);
-        assert!(!cache_bypass_applies("conditional-bench"));
-    }
-}

crates/ecstore/src/set_disk/core/io_primitives.rs

@@ -1260,10 +1260,6 @@ async fn try_create_bitrot_readers_via_batch_pread(
     }
 
     let requests: Vec<_> = batch_items.iter().map(|(_, p, off, len)| (p.clone(), *off, *len)).collect();
-    #[cfg(test)]
-    for (disk_index, _, _, _) in &batch_items {
-        disk_call_counters::record(object, disk_call_counters::KIND_DISK_SHARD_OPEN, *disk_index);
-    }
     let batch_results = batch_shard_pread(requests).await;
 
     let mut setup = BitrotReaderSetup::new(disks.len());
@@ -2451,17 +2447,6 @@ impl RenameConvergence {
     }
 }
 
-struct RenameDataObservations<'a> {
-    disks: &'a [Option<DiskStore>],
-    file_infos: &'a [FileInfo],
-    disk_versions: &'a [Option<Vec<u8>>],
-    errs: &'a [Option<DiskError>],
-    cleanup_data_dirs: &'a [Option<Uuid>],
-    old_current_sizes: &'a [Option<OldCurrentSize>],
-    old_current_source_checked: &'a [bool],
-    old_current_sources: &'a [Option<FileInfo>],
-}
-
 pub(in crate::set_disk) struct RenameDataCommit {
     pub(in crate::set_disk) online_disks: Vec<Option<DiskStore>>,
     /// Disks whose capacity must be marked for this commit. Under early ACK,
@@ -3618,21 +3603,17 @@ impl SetDisks {
         }
     }
 
-    #[allow(clippy::too_many_arguments)]
     fn rename_data_commit_from_observations(
-        observations: RenameDataObservations<'_>,
+        disks: &[Option<DiskStore>],
+        file_infos: &[FileInfo],
+        disk_versions: &[Option<Vec<u8>>],
+        errs: &[Option<DiskError>],
+        cleanup_data_dirs: &[Option<Uuid>],
+        old_current_sizes: &[Option<OldCurrentSize>],
+        old_current_source_checked: &[bool],
+        old_current_sources: &[Option<FileInfo>],
         write_quorum: usize,
     ) -> disk::error::Result<RenameDataCommit> {
-        let RenameDataObservations {
-            disks,
-            file_infos,
-            disk_versions,
-            errs,
-            cleanup_data_dirs,
-            old_current_sizes,
-            old_current_source_checked,
-            old_current_sources,
-        } = observations;
         let data_dir = Self::reduce_common_data_dir(cleanup_data_dirs, write_quorum);
         let convergence = Self::classify_rename_convergence(disk_versions, errs);
         let old_current_size = Self::reduce_common_old_current_size(old_current_sizes, write_quorum);
@@ -3959,16 +3940,14 @@ impl SetDisks {
 
                     if !sent_commit && success_count >= write_quorum {
                         let snapshot_commit = Self::rename_data_commit_from_observations(
-                            RenameDataObservations {
-                                disks: &coordinator_disks,
-                                file_infos: &file_infos,
-                                disk_versions: &disk_versions,
-                                errs: &errs,
-                                cleanup_data_dirs: &cleanup_data_dirs,
-                                old_current_sizes: &old_current_sizes,
-                                old_current_source_checked: &old_current_source_checked,
-                                old_current_sources: &old_current_sources,
-                            },
+                            &coordinator_disks,
+                            &file_infos,
+                            &disk_versions,
+                            &errs,
+                            &cleanup_data_dirs,
+                            &old_current_sizes,
+                            &old_current_source_checked,
+                            &old_current_sources,
                             write_quorum,
                         );
                         if let Ok(commit) = snapshot_commit.as_ref() {
@@ -6413,7 +6392,18 @@ impl SetDisks {
         // - If no_lock is false, we must have obtained the lock out side of this function
         // - If no_lock is true, we should not obtain locks
         lookup_opts.no_lock = true;
-        let oi = self.get_object_info(bucket, object, &lookup_opts).await;
+        let lookup = self.get_object_info(bucket, object, &lookup_opts);
+        #[cfg(feature = "diagnostic-read-version-caller")]
+        let lookup = crate::cluster::rpc::runtime_sources::with_read_version_entry(
+            rustfs_io_metrics::internode_metrics::ReadVersionEntrySource::WritePreconditionRead,
+            async {
+                crate::cluster::rpc::runtime_sources::observe_read_version_context_boundary(
+                    rustfs_io_metrics::internode_metrics::ReadVersionContextBoundary::WritePreconditionLookup,
+                );
+                lookup.await
+            },
+        );
+        let oi = lookup.await;
 
         match oi {
             Ok(oi) => {
@@ -6768,14 +6758,16 @@ pub(crate) mod disk_call_counters {
     pub const KIND_READ_VERSION: &str = "read_version";
     pub const KIND_BATCH_READ_VERSION: &str = "batch_read_version";
     pub const KIND_METADATA_SLOWTAIL_FAULT: &str = "metadata_slowtail_fault";
-    pub const KIND_INLINE_SHARD_OPEN: &str = "inline_shard_open";
-    pub const KIND_DISK_SHARD_OPEN: &str = "disk_shard_open";
-    /// Body verification, direct shard assembly, and erasure decoder entry points.
-    pub const KIND_BODY_DECODE: &str = "body_decode";
 
     /// Registry key: (object, kind, disk_index).
     type CountKey = (String, String, usize);
 
+    #[cfg(feature = "diagnostic-read-version-caller")]
+    type ReadVersionContext = (
+        Option<&'static str>,
+        Option<rustfs_io_metrics::internode_metrics::ReadVersionRequestCaller>,
+    );
+
     #[derive(Default)]
     struct Registry {
         /// Object names with an active observing scope. Only these accumulate,
@@ -6783,6 +6775,8 @@ pub(crate) mod disk_call_counters {
         observed: HashSet<String>,
         /// (object, kind, disk_index) -> call count.
         counts: HashMap<CountKey, u64>,
+        #[cfg(feature = "diagnostic-read-version-caller")]
+        read_version_contexts: HashMap<(String, usize), Vec<ReadVersionContext>>,
     }
 
     fn registry() -> &'static Mutex<Registry> {
@@ -6800,27 +6794,16 @@ pub(crate) mod disk_call_counters {
         *reg.counts
             .entry((object.to_string(), kind.to_string(), disk_index))
             .or_insert(0) += 1;
-    }
-
-    /// Matches shard paths to the observed object's namespace path. This also
-    /// records deferred opens running in spawned decoder tasks.
-    pub(crate) fn record_shard_open(path: &str, inline: bool) {
-        let mut reg = registry().lock().expect("disk call-counter registry poisoned");
-        let objects: Vec<_> = reg
-            .observed
-            .iter()
-            .filter(|object| {
-                path == object.as_str() || path.strip_prefix(object.as_str()).is_some_and(|rest| rest.starts_with('/'))
-            })
-            .cloned()
-            .collect();
-        let kind = if inline {
-            KIND_INLINE_SHARD_OPEN
-        } else {
-            KIND_DISK_SHARD_OPEN
-        };
-        for object in objects {
-            *reg.counts.entry((object, kind.to_owned(), 0)).or_insert(0) += 1;
+        #[cfg(feature = "diagnostic-read-version-caller")]
+        if kind == KIND_READ_VERSION {
+            let context = (
+                rustfs_io_metrics::s3_http_metrics::current_s3_http_operation_name(),
+                crate::cluster::rpc::runtime_sources::current_read_version_request_caller(),
+            );
+            reg.read_version_contexts
+                .entry((object.to_string(), disk_index))
+                .or_default()
+                .push(context);
         }
     }
 
@@ -6842,6 +6825,15 @@ pub(crate) mod disk_call_counters {
     }
 
     impl CallCounterScope {
+        #[cfg(feature = "diagnostic-read-version-caller")]
+        pub fn read_version_contexts_for_disk(&self, disk_index: usize) -> Vec<ReadVersionContext> {
+            let reg = registry().lock().expect("disk call-counter registry poisoned");
+            reg.read_version_contexts
+                .get(&(self.object.clone(), disk_index))
+                .cloned()
+                .unwrap_or_default()
+        }
+
         /// Total number of `kind` calls across all disks for the observed object.
         pub fn total(&self, kind: &str) -> u64 {
             let reg = registry().lock().expect("disk call-counter registry poisoned");
@@ -6867,6 +6859,8 @@ pub(crate) mod disk_call_counters {
             let mut reg = registry().lock().expect("disk call-counter registry poisoned");
             reg.observed.remove(&self.object);
             reg.counts.retain(|(obj, _, _), _| obj != &self.object);
+            #[cfg(feature = "diagnostic-read-version-caller")]
+            reg.read_version_contexts.retain(|(obj, _), _| obj != &self.object);
         }
     }
 }
@@ -8008,6 +8002,290 @@ mod tests {
         drop(dirs);
     }
 
+    #[cfg(feature = "diagnostic-read-version-caller")]
+    #[test]
+    fn diagnostic_coalescer_restores_each_item_after_delayed_and_threshold_flush() {
+        use crate::cluster::rpc::runtime_sources::{
+            ReadVersionDiagnosticContext, with_optional_read_version_request_caller, with_read_version_diagnostic_context,
+            with_read_version_entry,
+        };
+        use crate::set_disk::core::metadata_read::enqueue_coalesced_read_version;
+        use metrics_util::debugging::{DebugValue, DebuggingRecorder};
+        use rustfs_io_metrics::internode_metrics::{
+            ReadVersionContextBoundary, ReadVersionEntrySource, ReadVersionRequestCaller,
+        };
+        use rustfs_io_metrics::s3_http_metrics::{
+            S3HttpRequestGuard, current_s3_http_operation_context, with_isolated_s3_http_operation_context,
+        };
+        let operation = |op| {
+            let mut guard = S3HttpRequestGuard::new("GET");
+            guard.in_scope(|| {
+                rustfs_io_metrics::record_s3_op(op);
+                current_s3_http_operation_context().expect("operation should be captured")
+            })
+        };
+        let get = operation(rustfs_s3_ops::S3Operation::GetObject);
+        let list = operation(rustfs_s3_ops::S3Operation::ListObjectsV2);
+        let foreign = operation(rustfs_s3_ops::S3Operation::DeleteObject);
+        let runtime = tokio::runtime::Builder::new_current_thread()
+            .enable_all()
+            .build()
+            .expect("test runtime");
+        rustfs_io_metrics::set_get_stage_metrics_enabled(true);
+        for (items, delay) in [(4, "2000"), (crate::disk::BATCH_READ_VERSION_MAX_ITEMS, "10000")] {
+            let recorder = DebuggingRecorder::new();
+            let snapshots = recorder.snapshotter();
+            metrics::with_local_recorder(&recorder, || {
+                runtime.block_on(async {
+                    let bucket = "diagnostic-coalescer-context";
+                    let (_dirs, disks) = call_counter_local_disks(bucket, 1).await;
+                    let disk = disks[0].clone().expect("test disk should exist");
+                    for case in 0..4 {
+                        install_metadata_fanout_fileinfo(&disks, bucket, &format!("context-{case}"), None).await;
+                    }
+                    let mut contexts = Vec::new();
+                    for (api, caller, source) in [
+                        (Some(get), Some(ReadVersionRequestCaller::Object), ReadVersionEntrySource::ObjectRead),
+                        (None, None, ReadVersionEntrySource::ConfigRead),
+                        (
+                            Some(list),
+                            Some(ReadVersionRequestCaller::ListObjects),
+                            ReadVersionEntrySource::ListObjects,
+                        ),
+                        (None, Some(ReadVersionRequestCaller::Heal), ReadVersionEntrySource::LifecycleRecovery),
+                    ] {
+                        contexts.push(
+                            with_isolated_s3_http_operation_context(
+                                api,
+                                with_optional_read_version_request_caller(
+                                    caller,
+                                    with_read_version_entry(source, async { ReadVersionDiagnosticContext::capture() }),
+                                ),
+                            )
+                            .await,
+                        );
+                    }
+                    let opts = ReadOptions::default();
+                    let foreign_context = with_isolated_s3_http_operation_context(
+                        Some(foreign),
+                        with_optional_read_version_request_caller(
+                            Some(ReadVersionRequestCaller::Multipart),
+                            with_read_version_entry(ReadVersionEntrySource::ObjectRead, async {
+                                ReadVersionDiagnosticContext::capture()
+                            }),
+                        ),
+                    )
+                    .await;
+                    temp_env::async_with_vars([("RUSTFS_GET_METADATA_READ_VERSION_COALESCE_DELAY_MICROS", Some(delay))], async {
+                        with_read_version_diagnostic_context(foreign_context, ReadVersionContextBoundary::Entry, async {
+                            let reads = (0..items).map(|index| {
+                                let case = index % contexts.len();
+                                let item = BatchReadVersionItem {
+                                    org_volume: String::new(),
+                                    volume: bucket.to_owned(),
+                                    path: format!("context-{case}"),
+                                    version_id: String::new(),
+                                };
+                                with_read_version_diagnostic_context(
+                                    contexts[case],
+                                    ReadVersionContextBoundary::EarlyStop,
+                                    enqueue_coalesced_read_version(disk.clone(), item, &opts),
+                                )
+                            });
+                            let results = futures::future::join_all(reads).await;
+                            for (index, result) in results.into_iter().enumerate() {
+                                assert_eq!(result.expect("fallback read should succeed").name, format!("context-{}", index % 4));
+                            }
+                            assert_eq!(
+                                current_s3_http_operation_context(),
+                                Some(foreign),
+                                "flush must restore its caller's ambient scope"
+                            );
+                        })
+                        .await;
+                    })
+                    .await;
+                    assert_eq!(current_s3_http_operation_context(), None);
+                })
+            });
+            let rows = snapshots.snapshot().into_vec();
+            let item_label = items.to_string();
+            assert!(
+                rows.iter().any(|(key, _, _, value)| {
+                    let labels = key
+                        .key()
+                        .labels()
+                        .map(|label| (label.key(), label.value()))
+                        .collect::<std::collections::HashMap<_, _>>();
+                    key.key().name() == "rustfs_get_metadata_read_version_coalescer_total"
+                        && labels.get("event") == Some(&"attempted_batch")
+                        && labels.get("item_count") == Some(&item_label.as_str())
+                        && matches!(value, DebugValue::Counter(1))
+                }),
+                "all requests should flush in exactly one batch"
+            );
+            for (source, api, caller) in [
+                ("object_read", "s3:GetObject", "object"),
+                ("config_read", "unattributed", "unattributed"),
+                ("list_objects", "s3:ListObjectsV2", "list_objects"),
+                ("lifecycle_recovery", "unattributed", "heal"),
+            ] {
+                assert!(rows.iter().any(|(key, _, _, value)| {
+                    let labels = key.key().labels().map(|label| (label.key(), label.value())).collect::<std::collections::HashMap<_, _>>();
+                    key.key().name() == "rustfs_storage_diagnostic_context_transitions_total"
+                        && labels.get("boundary") == Some(&"unary_fallback")
+                        && labels.get("entry_source") == Some(&source)
+                        && labels.get("parent_api_operation") == Some(&api)
+                        && labels.get("child_api_operation") == Some(&api)
+                        && labels.get("parent_caller") == Some(&caller)
+                        && labels.get("child_caller") == Some(&caller)
+                        && matches!(value, DebugValue::Counter(count) if *count == u64::try_from(items / 4).expect("small item count"))
+                }), "each item must retain its own API, caller, and entry source");
+            }
+        }
+    }
+
+    #[cfg(feature = "diagnostic-read-version-caller")]
+    #[tokio::test(flavor = "multi_thread", worker_threads = 2)]
+    async fn metadata_full_wait_preserves_external_request_diagnostic_context() {
+        use crate::cluster::rpc::runtime_sources::with_read_version_request_caller;
+        use rustfs_io_metrics::internode_metrics::ReadVersionRequestCaller;
+        use rustfs_io_metrics::s3_http_metrics::{
+            S3HttpRequestGuard, current_s3_http_operation_context, with_s3_http_operation_context,
+        };
+
+        const DISKS: usize = 4;
+        let bucket = "full-wait-external-context-bucket";
+        let object = "full-wait-external-context-object";
+        let (_dirs, disks) = call_counter_local_disks(bucket, DISKS).await;
+        install_metadata_fanout_fileinfo(&disks, bucket, object, None).await;
+        let calls = disk_call_counters::observe(object);
+        let mut request = S3HttpRequestGuard::new("GET");
+        let operation = request
+            .in_scope(|| {
+                rustfs_io_metrics::record_s3_op(rustfs_s3_ops::S3Operation::GetObject);
+                current_s3_http_operation_context()
+            })
+            .expect("external request operation should be captured");
+        let (metadata, errors, _) = with_s3_http_operation_context(
+            Some(operation),
+            with_read_version_request_caller(
+                ReadVersionRequestCaller::Object,
+                SetDisks::read_all_fileinfo_observed(&disks, "", bucket, object, "", false, false, false, false, 2),
+            ),
+        )
+        .await
+        .expect("full-wait metadata read should complete");
+
+        assert_eq!(metadata.iter().filter(|info| info.name == object).count(), DISKS);
+        assert!(errors.iter().all(Option::is_none));
+        for disk_index in 0..DISKS {
+            assert_eq!(
+                calls.read_version_contexts_for_disk(disk_index),
+                vec![(Some("s3:GetObject"), Some(ReadVersionRequestCaller::Object))],
+                "every full-wait child must retain the external request context"
+            );
+        }
+        assert_eq!(current_s3_http_operation_context(), None);
+    }
+
+    #[cfg(feature = "diagnostic-read-version-caller")]
+    #[tokio::test(flavor = "multi_thread", worker_threads = 2)]
+    async fn metadata_full_wait_keeps_internal_reads_without_s3_context() {
+        use crate::cluster::rpc::runtime_sources::with_optional_read_version_request_caller;
+        use rustfs_io_metrics::internode_metrics::ReadVersionRequestCaller;
+
+        const DISKS: usize = 4;
+        let bucket = "full-wait-internal-context-bucket";
+        let (_dirs, disks) = call_counter_local_disks(bucket, DISKS).await;
+        for (case, caller) in [None, Some(ReadVersionRequestCaller::Heal)].into_iter().enumerate() {
+            let object = format!("full-wait-internal-context-object-{case}");
+            install_metadata_fanout_fileinfo(&disks, bucket, &object, None).await;
+            let calls = disk_call_counters::observe(&object);
+            let (metadata, errors, _) = with_optional_read_version_request_caller(
+                caller,
+                SetDisks::read_all_fileinfo_observed(&disks, "", bucket, &object, "", false, false, false, false, 2),
+            )
+            .await
+            .expect("internal full-wait metadata read should complete");
+
+            assert_eq!(metadata.iter().filter(|info| info.name == object).count(), DISKS);
+            assert!(errors.iter().all(Option::is_none));
+            for disk_index in 0..DISKS {
+                assert_eq!(
+                    calls.read_version_contexts_for_disk(disk_index),
+                    vec![(None, caller)],
+                    "internal reads must retain their caller without acquiring an external request context"
+                );
+            }
+        }
+    }
+
+    #[cfg(feature = "diagnostic-read-version-caller")]
+    #[tokio::test(flavor = "multi_thread", worker_threads = 2)]
+    async fn metadata_early_stop_preserves_external_and_internal_diagnostic_context() {
+        use crate::cluster::rpc::runtime_sources::with_optional_read_version_request_caller;
+        use rustfs_io_metrics::internode_metrics::ReadVersionRequestCaller;
+        use rustfs_io_metrics::s3_http_metrics::{
+            S3HttpRequestGuard, current_s3_http_operation_context, with_s3_http_operation_context,
+        };
+
+        const DISKS: usize = 4;
+        let bucket = "early-stop-diagnostic-context-bucket";
+        let (_dirs, disks) = call_counter_local_disks(bucket, DISKS).await;
+        let mut request = S3HttpRequestGuard::new("GET");
+        let operation = request
+            .in_scope(|| {
+                rustfs_io_metrics::record_s3_op(rustfs_s3_ops::S3Operation::GetObject);
+                current_s3_http_operation_context()
+            })
+            .expect("external request operation should be captured");
+
+        temp_env::async_with_vars(
+            [
+                ("RUSTFS_GET_METADATA_EARLY_STOP_ENABLE", Some("true")),
+                ("RUSTFS_GET_METADATA_DATA_READ_EARLY_STOP_ENABLE", Some("true")),
+                ("RUSTFS_GET_METADATA_EARLY_STOP_BOUNDED_FANOUT", Some("true")),
+            ],
+            async {
+                for (case, api, caller) in [
+                    (0, Some(operation), Some(ReadVersionRequestCaller::Object)),
+                    (1, None, None),
+                    (2, None, Some(ReadVersionRequestCaller::Heal)),
+                ] {
+                    let object = object_with_initial_data_shards(bucket, &format!("early-stop-context-object-{case}"), 2, 3);
+                    install_inline_metadata_fanout_fileinfo(&disks, bucket, &object, b"context probe", |_| {}).await;
+                    let calls = disk_call_counters::observe(&object);
+                    let (metadata, errors, _) = with_s3_http_operation_context(
+                        api,
+                        with_optional_read_version_request_caller(
+                            caller,
+                            SetDisks::read_all_fileinfo_observed(&disks, "", bucket, &object, "", true, false, false, true, 2),
+                        ),
+                    )
+                    .await
+                    .expect("early-stop metadata read should complete");
+
+                    assert!(metadata.iter().filter(|info| info.name == object).count() >= 2);
+                    assert!(errors.iter().all(Option::is_none));
+                    assert!(calls.total(disk_call_counters::KIND_READ_VERSION) >= 2);
+                    let expected_api = api.map(|context| context.operation_name());
+                    for disk_index in 0..DISKS {
+                        for context in calls.read_version_contexts_for_disk(disk_index) {
+                            assert_eq!(
+                                context,
+                                (expected_api, caller),
+                                "every polled early-stop child must retain its parent's diagnostic context"
+                            );
+                        }
+                    }
+                }
+            },
+        )
+        .await;
+        assert_eq!(current_s3_http_operation_context(), None);
+    }
+
     #[test]
     fn batch_read_version_response_mapping_preserves_index_and_errors() {
         let expected_items = vec![

crates/ecstore/src/set_disk/core/metadata_read.rs

@@ -66,7 +66,6 @@ use tokio::task::JoinSet;
 pub(in crate::set_disk) struct MetadataReadResult {
     pub(in crate::set_disk) slots: Vec<MetadataDiskObservation>,
     pub(in crate::set_disk) diagnostics: MetadataFanoutDiagnostics,
-    pub(in crate::set_disk) condition_stopped: bool,
 }
 
 impl MetadataReadResult {
@@ -171,6 +170,23 @@ fn read_version_coalescing_delay() -> Duration {
 struct CoalescedReadVersionRequest {
     item: BatchReadVersionItem,
     tx: oneshot::Sender<disk::error::Result<FileInfo>>,
+    #[cfg(feature = "diagnostic-read-version-caller")]
+    context: crate::cluster::rpc::runtime_sources::ReadVersionDiagnosticContext,
+}
+
+impl CoalescedReadVersionRequest {
+    fn new(item: BatchReadVersionItem, tx: oneshot::Sender<disk::error::Result<FileInfo>>) -> Self {
+        #[cfg(feature = "diagnostic-read-version-caller")]
+        let context = crate::cluster::rpc::runtime_sources::ReadVersionDiagnosticContext::capture();
+        #[cfg(feature = "diagnostic-read-version-caller")]
+        context.observe(rustfs_io_metrics::internode_metrics::ReadVersionContextBoundary::CoalescerEnqueue);
+        Self {
+            item,
+            tx,
+            #[cfg(feature = "diagnostic-read-version-caller")]
+            context,
+        }
+    }
 }
 
 #[derive(Clone, Debug, Eq, PartialEq)]
@@ -246,26 +262,49 @@ async fn read_version_via_coalescer(
     allow_coalescing: bool,
 ) -> disk::error::Result<FileInfo> {
     if !allow_coalescing || !read_version_coalescing_enabled() {
+        #[cfg(feature = "diagnostic-read-version-caller")]
+        return crate::cluster::rpc::runtime_sources::with_read_version_request_caller_if_absent(
+            rustfs_io_metrics::internode_metrics::ReadVersionRequestCaller::MetadataRead,
+            disk.read_version(org_bucket, bucket, object, version_id, opts),
+        )
+        .await;
+        #[cfg(not(feature = "diagnostic-read-version-caller"))]
         return disk.read_version(org_bucket, bucket, object, version_id, opts).await;
     }
     if !matches!(disk.as_ref(), Disk::Remote(_)) {
         record_read_version_coalescer_event("bypass_non_remote", 1);
+        #[cfg(feature = "diagnostic-read-version-caller")]
+        return crate::cluster::rpc::runtime_sources::with_read_version_request_caller_if_absent(
+            rustfs_io_metrics::internode_metrics::ReadVersionRequestCaller::MetadataRead,
+            disk.read_version(org_bucket, bucket, object, version_id, opts),
+        )
+        .await;
+        #[cfg(not(feature = "diagnostic-read-version-caller"))]
         return disk.read_version(org_bucket, bucket, object, version_id, opts).await;
     }
 
-    let (tx, rx) = oneshot::channel();
     let item = BatchReadVersionItem {
         org_volume: org_bucket.to_string(),
         volume: bucket.to_string(),
         path: object.to_string(),
         version_id: version_id.to_string(),
     };
+    enqueue_coalesced_read_version(disk, item, opts).await
+}
+
+pub(super) async fn enqueue_coalesced_read_version(
+    disk: DiskStore,
+    item: BatchReadVersionItem,
+    opts: &ReadOptions,
+) -> disk::error::Result<FileInfo> {
+    let (tx, rx) = oneshot::channel();
     let lane_key = ReadVersionCoalescerKey::new(&disk, opts);
+    let request = CoalescedReadVersionRequest::new(item, tx);
     let pending = {
         let mut coalescer = read_version_coalescer().lock().await;
         let lane = coalescer.lanes.entry(lane_key).or_default();
         let schedule_delayed_flush = lane.is_empty();
-        lane.push(CoalescedReadVersionRequest { item, tx });
+        lane.push(request);
         if lane.len() >= BATCH_READ_VERSION_MAX_ITEMS {
             coalescer.lanes.remove(&lane_key)
         } else if schedule_delayed_flush {
@@ -326,20 +365,24 @@ async fn flush_read_version_coalescer_pending(
 
     let mut senders = Vec::with_capacity(pending.len());
     let mut items = Vec::with_capacity(pending.len());
+    #[cfg(feature = "diagnostic-read-version-caller")]
+    let mut contexts = Vec::with_capacity(pending.len());
     for request in pending {
         senders.push(request.tx);
         items.push(request.item);
+        #[cfg(feature = "diagnostic-read-version-caller")]
+        contexts.push(request.context);
     }
 
     let expected_items = items.iter().map(ExpectedBatchReadVersionItem::from).collect::<Vec<_>>();
     record_read_version_coalescer_event("attempted_batch", items.len());
-    let result =
-        match tokio::time::timeout(get_drive_metadata_timeout(), disk.batch_read_version(BatchReadVersionReq { items, opts }))
-            .await
-        {
-            Ok(result) => result,
-            Err(_) => Err(DiskError::Timeout),
-        };
+    let batch = disk.batch_read_version(BatchReadVersionReq { items, opts });
+    #[cfg(feature = "diagnostic-read-version-caller")]
+    let batch = crate::cluster::rpc::runtime_sources::with_read_version_batch_contexts(contexts, batch);
+    let result = match tokio::time::timeout(get_drive_metadata_timeout(), batch).await {
+        Ok(result) => result,
+        Err(_) => Err(DiskError::Timeout),
+    };
     match result {
         Ok(responses) => {
             let map_started = batch_read_version_stage_timer();
@@ -657,8 +700,6 @@ async fn inline_body_miss_reason_from_observations<'a>(
         return Some(GET_METADATA_EARLY_STOP_REASON_DATA_READ_INLINE_BODY_VERIFY);
     };
 
-    #[cfg(test)]
-    crate::set_disk::disk_call_counters::record(object, crate::set_disk::disk_call_counters::KIND_BODY_DECODE, 0);
     match try_read_inline_data_shards_direct(&mut readers, erasure.data_shards, read_length, object_size).await {
         Some(body) if body.len() == object_size => None,
         _ => Some(GET_METADATA_EARLY_STOP_REASON_DATA_READ_INLINE_BODY_VERIFY),
@@ -1016,6 +1057,10 @@ impl SetDisks {
         let object: Arc<str> = Arc::from(object);
         let version_id: Arc<str> = Arc::from(version_id);
         let slowtail_fault = get_metadata_slowtail_fault_request(bucket.as_ref(), object.as_ref(), read_data);
+        #[cfg(feature = "diagnostic-read-version-caller")]
+        let diagnostic_context = crate::cluster::rpc::runtime_sources::ReadVersionDiagnosticContext::capture_entry(
+            rustfs_io_metrics::internode_metrics::ReadVersionEntrySource::MetadataRead,
+        );
         let futures = disks.iter().enumerate().map(|(disk_index, disk)| {
             let disk = disk.clone();
             let task_opts = opts;
@@ -1024,21 +1069,34 @@ impl SetDisks {
             let object = object.clone();
             let version_id = version_id.clone();
             let slowtail_fault = slowtail_fault.clone();
+            #[cfg(feature = "diagnostic-read-version-caller")]
+            let diagnostic_context = diagnostic_context;
             AbortOnDropJoinHandle(tokio::spawn(async move {
-                let response_start = observe.then(Instant::now);
-                let result = if let Some(disk) = disk {
-                    Self::record_read_version_call(&object, disk_index);
-                    if let Some(delay) = slowtail_fault.as_ref().and_then(|fault| fault.delay_for_disk(disk_index)) {
-                        Self::record_metadata_slowtail_fault(&object, disk_index);
-                        tokio::time::sleep(delay).await;
-                    }
-                    read_version_via_coalescer(disk, &org_bucket, &bucket, &object, &version_id, &task_opts, allow_coalescing)
-                        .await
-                } else {
-                    Err(DiskError::DiskNotFound)
+                let read = async move {
+                    let response_start = observe.then(Instant::now);
+                    let result = if let Some(disk) = disk {
+                        Self::record_read_version_call(&object, disk_index);
+                        if let Some(delay) = slowtail_fault.as_ref().and_then(|fault| fault.delay_for_disk(disk_index)) {
+                            Self::record_metadata_slowtail_fault(&object, disk_index);
+                            tokio::time::sleep(delay).await;
+                        }
+                        read_version_via_coalescer(disk, &org_bucket, &bucket, &object, &version_id, &task_opts, allow_coalescing)
+                            .await
+                    } else {
+                        Err(DiskError::DiskNotFound)
+                    };
+                    let elapsed = response_start.map(|start| start.elapsed());
+                    (result, elapsed)
                 };
-                let elapsed = response_start.map(|start| start.elapsed());
-                (result, elapsed)
+                #[cfg(feature = "diagnostic-read-version-caller")]
+                return crate::cluster::rpc::runtime_sources::with_read_version_diagnostic_context(
+                    diagnostic_context,
+                    rustfs_io_metrics::internode_metrics::ReadVersionContextBoundary::FullWait,
+                    read,
+                )
+                .await;
+                #[cfg(not(feature = "diagnostic-read-version-caller"))]
+                read.await
             }))
         });
 
@@ -1082,11 +1140,7 @@ impl SetDisks {
                 0,
             );
         }
-        Ok(MetadataReadResult {
-            slots,
-            diagnostics,
-            condition_stopped: false,
-        })
+        Ok(MetadataReadResult { slots, diagnostics })
     }
 
     #[allow(clippy::too_many_arguments)]
@@ -1129,11 +1183,14 @@ impl SetDisks {
         let mut next_fanout_index = 0usize;
         let mut scheduled_count = 0usize;
         let mut force_full_wait = false;
-        let read_condition_active = read_data && crate::object_api::get_object_read_condition_is_active();
         let mut final_miss_reason_override = None;
         let mut non_inline_candidate_eligible = None;
         let mut single_pending_hedge_deadline = None;
         let slowtail_fault = get_metadata_slowtail_fault_request(bucket.as_ref(), object.as_ref(), read_data);
+        #[cfg(feature = "diagnostic-read-version-caller")]
+        let diagnostic_context = crate::cluster::rpc::runtime_sources::ReadVersionDiagnosticContext::capture_entry(
+            rustfs_io_metrics::internode_metrics::ReadVersionEntrySource::MetadataRead,
+        );
         let spawn_read_version = |join_set: &mut JoinSet<(usize, disk::error::Result<FileInfo>, Duration)>,
                                   task_ids: &mut [Option<tokio::task::Id>],
                                   index: usize,
@@ -1144,27 +1201,44 @@ impl SetDisks {
             let object = object.clone();
             let version_id = version_id.clone();
             let slowtail_fault = slowtail_fault.clone();
+            #[cfg(feature = "diagnostic-read-version-caller")]
+            let diagnostic_context = diagnostic_context;
             let task = join_set.spawn(async move {
-                let response_start = Instant::now();
-                let result = if let Some(disk) = disk {
-                    #[cfg(test)]
-                    let _fanout_task_guard = rename_fanout_barrier::task_guard(&object);
-                    Self::record_read_version_call(&object, index);
-                    #[cfg(test)]
-                    Self::read_version_fanout_barrier(&object, index).await;
-                    if let Some(delay) = slowtail_fault.as_ref().and_then(|fault| fault.delay_for_disk(index)) {
-                        Self::record_metadata_slowtail_fault(&object, index);
+                let read = async move {
+                    let response_start = Instant::now();
+                    let result = if let Some(disk) = disk {
                         #[cfg(test)]
-                        rename_fanout_barrier::checkpoint(&object, index, rename_fanout_barrier::PHASE_METADATA_SLOWTAIL_FAULT)
+                        let _fanout_task_guard = rename_fanout_barrier::task_guard(&object);
+                        Self::record_read_version_call(&object, index);
+                        #[cfg(test)]
+                        Self::read_version_fanout_barrier(&object, index).await;
+                        if let Some(delay) = slowtail_fault.as_ref().and_then(|fault| fault.delay_for_disk(index)) {
+                            Self::record_metadata_slowtail_fault(&object, index);
+                            #[cfg(test)]
+                            rename_fanout_barrier::checkpoint(
+                                &object,
+                                index,
+                                rename_fanout_barrier::PHASE_METADATA_SLOWTAIL_FAULT,
+                            )
                             .await;
-                        tokio::time::sleep(delay).await;
-                    }
-                    read_version_via_coalescer(disk, &org_bucket, &bucket, &object, &version_id, &task_opts, allow_coalescing)
-                        .await
-                } else {
-                    Err(DiskError::DiskNotFound)
+                            tokio::time::sleep(delay).await;
+                        }
+                        read_version_via_coalescer(disk, &org_bucket, &bucket, &object, &version_id, &task_opts, allow_coalescing)
+                            .await
+                    } else {
+                        Err(DiskError::DiskNotFound)
+                    };
+                    (index, result, response_start.elapsed())
                 };
-                (index, result, response_start.elapsed())
+                #[cfg(feature = "diagnostic-read-version-caller")]
+                return crate::cluster::rpc::runtime_sources::with_read_version_diagnostic_context(
+                    diagnostic_context,
+                    rustfs_io_metrics::internode_metrics::ReadVersionContextBoundary::EarlyStop,
+                    read,
+                )
+                .await;
+                #[cfg(not(feature = "diagnostic-read-version-caller"))]
+                read.await
             });
             task_ids[index] = Some(task.id());
         };
@@ -1267,28 +1341,12 @@ impl SetDisks {
                 }
             }
 
-            if (!force_full_wait || read_condition_active)
+            if !force_full_wait
                 && let Some(decision) = accumulator
                     .early_stop_decision()
                     .or_else(|| accumulator.version_early_stop_decision())
             {
-                let condition_stopped = read_condition_active
-                    && decision.reason == crate::diagnostics::get::GET_METADATA_EARLY_STOP_REASON_VALID_QUORUM
-                    && accumulator.candidate.as_ref().is_some_and(|candidate| {
-                        !candidate.deleted
-                            && crate::object_api::get_object_read_condition_is_terminal(
-                                crate::object_api::GetObjectReadMetadata::from_file_info(
-                                    bucket.as_ref(),
-                                    object.as_ref(),
-                                    candidate,
-                                ),
-                            )
-                    });
-                let should_return_early = if condition_stopped {
-                    true
-                } else if force_full_wait {
-                    false
-                } else if read_data {
+                let should_return_early = if read_data {
                     match accumulator.candidate.as_ref() {
                         Some(_candidate) if non_inline_candidate_eligible == Some(true) => {
                             accumulator.candidate_has_read_reserve()
@@ -1357,11 +1415,7 @@ impl SetDisks {
                         cancelled_count,
                     );
                     let diagnostics = MetadataFanoutDiagnostics::new(fanout_start.elapsed(), observations);
-                    return Ok(MetadataReadResult {
-                        slots,
-                        diagnostics,
-                        condition_stopped,
-                    });
+                    return Ok(MetadataReadResult { slots, diagnostics });
                 }
             }
 
@@ -1439,11 +1493,7 @@ impl SetDisks {
         rustfs_io_metrics::record_get_object_metadata_early_stop_saved_responses(metrics_path, 0);
         rustfs_io_metrics::record_get_object_metadata_fanout_lifecycle(metrics_path, scheduled_count, scheduled_count, 0);
         let diagnostics = MetadataFanoutDiagnostics::new(fanout_start.elapsed(), observations);
-        Ok(MetadataReadResult {
-            slots,
-            diagnostics,
-            condition_stopped: false,
-        })
+        Ok(MetadataReadResult { slots, diagnostics })
     }
 
     /// Test-only seam that records one per-disk `read_version` metadata RPC for

crates/ecstore/src/set_disk/mod.rs

@@ -799,11 +799,6 @@ const DEFAULT_RUSTFS_GET_METADATA_TWO_PHASE_READ_PLAN_ENABLE: bool = false;
 const ENV_RUSTFS_GET_METADATA_EARLY_STOP_BOUNDED_FANOUT: &str = "RUSTFS_GET_METADATA_EARLY_STOP_BOUNDED_FANOUT";
 const DEFAULT_RUSTFS_GET_METADATA_EARLY_STOP_BOUNDED_FANOUT: bool = true;
 
-#[cfg(feature = "test-util")]
-mod conditional_read_bench;
-#[cfg(feature = "test-util")]
-pub use conditional_read_bench::ConditionalReadBenchmarkMetadataGuard;
-
 const ENV_RUSTFS_GET_METADATA_SLOWTAIL_FAULT_DELAY_MS: &str = "RUSTFS_GET_METADATA_SLOWTAIL_FAULT_DELAY_MS";
 const ENV_RUSTFS_GET_METADATA_SLOWTAIL_FAULT_DISKS: &str = "RUSTFS_GET_METADATA_SLOWTAIL_FAULT_DISKS";
 const ENV_RUSTFS_GET_METADATA_SLOWTAIL_FAULT_BUCKET: &str = "RUSTFS_GET_METADATA_SLOWTAIL_FAULT_BUCKET";
@@ -885,7 +880,6 @@ mod ctx;
 mod metadata;
 mod ops;
 pub(crate) use ops::bucket::BucketInfoQuorum;
-pub(crate) use ops::delete_recovery::{DeleteRecoveryContext, DeleteRecoverySummary};
 pub(crate) use ops::heal::AbsenceProofRequest;
 #[cfg(test)]
 pub(crate) use ops::heal::DanglingDeleteFailure;
@@ -930,22 +924,16 @@ pub use ops::heal_walk::HealWalkVersion;
 pub(in crate::set_disk) struct GetObjectFileInfo {
     owned: Option<OwnedGetObjectFileInfo>,
     shared: Option<Arc<GetObjectMetadataCacheEntry>>,
-    condition_stopped: bool,
-    late_metadata_fanout: Option<LateMetadataFanout>,
 }
 
 struct OwnedGetObjectFileInfo {
     fi: FileInfo,
     parts_metadata: Vec<FileInfo>,
     online_disks: Vec<Option<DiskStore>>,
+    late_metadata_fanout_disks: Option<Vec<Option<DiskStore>>>,
 }
 
-enum LateMetadataFanout {
-    CapturedDisks(Vec<Option<DiskStore>>),
-    RefreshCurrentSet,
-}
-
-type OwnedGetObjectFileInfoParts = (FileInfo, Vec<FileInfo>, Vec<Option<DiskStore>>, Option<LateMetadataFanout>);
+type OwnedGetObjectFileInfoParts = (FileInfo, Vec<FileInfo>, Vec<Option<DiskStore>>, Option<Vec<Option<DiskStore>>>);
 
 impl GetObjectFileInfo {
     fn owned(fi: FileInfo, parts_metadata: Vec<FileInfo>, online_disks: Vec<Option<DiskStore>>) -> Self {
@@ -954,10 +942,9 @@ impl GetObjectFileInfo {
                 fi,
                 parts_metadata,
                 online_disks,
+                late_metadata_fanout_disks: None,
             }),
             shared: None,
-            condition_stopped: false,
-            late_metadata_fanout: None,
         }
     }
 
@@ -967,25 +954,22 @@ impl GetObjectFileInfo {
         online_disks: Vec<Option<DiskStore>>,
         late_metadata_fanout_disks: Vec<Option<DiskStore>>,
     ) -> Self {
-        let mut snapshot = Self::owned(fi, parts_metadata, online_disks);
-        snapshot.late_metadata_fanout = Some(LateMetadataFanout::CapturedDisks(late_metadata_fanout_disks));
-        snapshot
-    }
-
-    fn shared(entry: Arc<GetObjectMetadataCacheEntry>) -> Self {
-        let condition_stopped = entry.capability == GetObjectMetadataCacheCapability::ConditionOnly;
         Self {
-            owned: None,
-            shared: Some(entry),
-            condition_stopped,
-            late_metadata_fanout: None,
+            owned: Some(OwnedGetObjectFileInfo {
+                fi,
+                parts_metadata,
+                online_disks,
+                late_metadata_fanout_disks: Some(late_metadata_fanout_disks),
+            }),
+            shared: None,
         }
     }
 
-    fn shared_with_late_metadata_refresh(entry: Arc<GetObjectMetadataCacheEntry>) -> Self {
-        let mut snapshot = Self::shared(entry);
-        snapshot.late_metadata_fanout = Some(LateMetadataFanout::RefreshCurrentSet);
-        snapshot
+    fn shared(entry: Arc<GetObjectMetadataCacheEntry>) -> Self {
+        Self {
+            owned: None,
+            shared: Some(entry),
+        }
     }
 
     fn fi(&self) -> &FileInfo {
@@ -1013,7 +997,9 @@ impl GetObjectFileInfo {
     }
 
     fn has_late_metadata_fanout(&self) -> bool {
-        self.late_metadata_fanout.is_some()
+        self.owned
+            .as_ref()
+            .is_some_and(|snapshot| snapshot.late_metadata_fanout_disks.is_some())
     }
 
     fn into_owned(self) -> (FileInfo, Vec<FileInfo>, Vec<Option<DiskStore>>) {
@@ -1022,22 +1008,16 @@ impl GetObjectFileInfo {
     }
 
     fn into_owned_with_late_metadata_fanout(self) -> OwnedGetObjectFileInfoParts {
-        let Self {
-            owned,
-            shared,
-            late_metadata_fanout,
-            ..
-        } = self;
-        match (owned, shared) {
-            (Some(snapshot), None) => (snapshot.fi, snapshot.parts_metadata, snapshot.online_disks, late_metadata_fanout),
+        match (self.owned, self.shared) {
+            (Some(snapshot), None) => (
+                snapshot.fi,
+                snapshot.parts_metadata,
+                snapshot.online_disks,
+                snapshot.late_metadata_fanout_disks,
+            ),
             (None, Some(entry)) => match Arc::try_unwrap(entry) {
-                Ok(entry) => (entry.fi, entry.parts_metadata, entry.online_disks, late_metadata_fanout),
-                Err(entry) => (
-                    entry.fi.clone(),
-                    entry.parts_metadata.clone(),
-                    entry.online_disks.clone(),
-                    late_metadata_fanout,
-                ),
+                Ok(entry) => (entry.fi, entry.parts_metadata, entry.online_disks, None),
+                Err(entry) => (entry.fi.clone(), entry.parts_metadata.clone(), entry.online_disks.clone(), None),
             },
             _ => unreachable!("GET metadata snapshot representation must be exclusive"),
         }
@@ -1169,248 +1149,6 @@ mod prepared_get_object_metadata_tests {
         assert!(take_prepared_get_object_metadata().is_none());
     }
 
-    struct StopReadForEtag(String);
-    impl crate::object_api::GetObjectReadCondition for StopReadForEtag {
-        fn is_terminal(&self, metadata: crate::object_api::GetObjectReadMetadata<'_>) -> bool {
-            !metadata.delete_marker && metadata.etag == Some(self.0.as_str())
-        }
-    }
-
-    #[tokio::test]
-    #[serial_test::serial(body_cache_hook)]
-    async fn read_condition_stops_on_quorum_before_slow_disk_and_body_verification() {
-        use crate::object_api::with_get_object_read_condition;
-        for size in [4096, 1_300_000] {
-            let (dirs, set) = make_local_set_disks(4, 2).await;
-            let bucket = "read-condition-quorum";
-            let object = object_with_initial_data_shards(bucket, &format!("read-condition-{size}"));
-            let opts = ObjectOptions {
-                no_lock: true,
-                ..Default::default()
-            };
-            set.make_bucket(bucket, &MakeBucketOptions::default())
-                .await
-                .expect("create bucket");
-            let mut put = PutObjReader::from_vec(vec![b'v'; size]);
-            let info = set.put_object(bucket, &object, &mut put, &opts).await.expect("write object");
-            for dir in &dirs {
-                let object_path = dir.path().join(bucket).join(&object);
-                if size == 4096 {
-                    let path = object_path.join("xl.meta");
-                    let mut bytes = tokio::fs::read(&path).await.expect("read inline metadata");
-                    *bytes.last_mut().expect("inline payload") ^= 1;
-                    tokio::fs::write(path, bytes).await.expect("damage inline data");
-                } else {
-                    tokio::fs::remove_dir_all(object_path.join(info.data_dir.expect("data directory").to_string()))
-                        .await
-                        .expect("remove shards");
-                }
-            }
-            let slow = bounded_initial_parity_disk_index(bucket, &object);
-            let barrier = rename_fanout_barrier::arm(&object, slow, rename_fanout_barrier::PHASE_READ_VERSION);
-            let calls = disk_call_counters::observe(&object);
-            let read_opts = ObjectOptions {
-                metadata_cache_safe: true,
-                ..opts
-            };
-            let followup_opts = read_opts.clone();
-            let condition = Arc::new(StopReadForEtag(info.etag.clone().expect("committed ETag")));
-            let set_for_read = Arc::clone(&set);
-            let object_for_read = object.clone();
-            let mut read = tokio::spawn(async move {
-                with_get_object_read_condition(
-                    condition,
-                    set_for_read.get_object_reader(bucket, &object_for_read, None, HeaderMap::new(), &read_opts),
-                )
-                .await
-            });
-            tokio::time::timeout(READ_VERSION_BARRIER_GUARD, barrier.wait_until_paused())
-                .await
-                .expect("slow metadata read is pending");
-            let mut reader = tokio::time::timeout(READ_VERSION_BARRIER_GUARD, &mut read)
-                .await
-                .expect("condition must not wait for the slow disk or decode damaged data")
-                .expect("reader task")
-                .expect("metadata-only result");
-            assert_eq!(reader.object_info.etag, info.etag);
-            assert_eq!(
-                calls.total(disk_call_counters::KIND_INLINE_SHARD_OPEN),
-                0,
-                "damaged inline data was never opened"
-            );
-            assert_eq!(
-                calls.total(disk_call_counters::KIND_DISK_SHARD_OPEN),
-                0,
-                "missing shards were never opened"
-            );
-            assert_eq!(
-                calls.total(disk_call_counters::KIND_BODY_DECODE),
-                0,
-                "the body verifier and decoder were never entered"
-            );
-            let mut body = Vec::new();
-            assert!(
-                reader.stream.read_to_end(&mut body).await.is_err(),
-                "terminal metadata cannot be served as a successful body"
-            );
-            assert!(body.is_empty());
-            let first_calls = calls.total(disk_call_counters::KIND_READ_VERSION);
-            drop(reader);
-            barrier.release();
-            let snapshot = set
-                .get_object_fileinfo_for_get_object_reader(bucket, &object, &followup_opts, true, true)
-                .await
-                .expect("fresh metadata after terminal scope");
-            assert!(!snapshot.condition_stopped);
-            assert!(
-                calls.total(disk_call_counters::KIND_READ_VERSION) > first_calls,
-                "an incomplete terminal snapshot must not satisfy the next metadata read from cache"
-            );
-        }
-    }
-
-    #[tokio::test]
-    #[serial_test::serial(body_cache_hook)]
-    async fn read_condition_skips_shard_open_and_decode_with_live_positive_controls() {
-        for size in [4096, 1_300_000] {
-            let (_dirs, set) = make_local_set_disks(4, 2).await;
-            let bucket = "read-condition-body-counters";
-            let object = object_with_initial_data_shards(bucket, &format!("body-counters-{size}"));
-            let opts = ObjectOptions {
-                no_lock: true,
-                ..Default::default()
-            };
-            let payload: Vec<_> = (0..size)
-                .map(|index| u8::try_from(index % 251).expect("pattern byte"))
-                .collect();
-            set.make_bucket(bucket, &MakeBucketOptions::default())
-                .await
-                .expect("create bucket");
-            let mut put = PutObjReader::from_vec(payload.clone());
-            let info = set.put_object(bucket, &object, &mut put, &opts).await.expect("write object");
-            let calls = disk_call_counters::observe(&object);
-            let reader = crate::object_api::with_get_object_read_condition(
-                Arc::new(StopReadForEtag(info.etag.expect("committed ETag"))),
-                set.get_object_reader(bucket, &object, None, HeaderMap::new(), &opts),
-            )
-            .await
-            .expect("terminal metadata reader");
-            drop(reader);
-            assert!(calls.total(disk_call_counters::KIND_READ_VERSION) > 0, "metadata really was read");
-            assert_eq!(
-                calls.total(disk_call_counters::KIND_INLINE_SHARD_OPEN),
-                0,
-                "no inline shard source opened"
-            );
-            assert_eq!(calls.total(disk_call_counters::KIND_DISK_SHARD_OPEN), 0, "no disk shard source opened");
-            assert_eq!(
-                calls.total(disk_call_counters::KIND_BODY_DECODE),
-                0,
-                "no body verifier or decoder entered"
-            );
-
-            let mut reader = crate::object_api::with_get_object_read_condition(
-                Arc::new(StopReadForEtag("different-etag".to_owned())),
-                set.get_object_reader(bucket, &object, None, HeaderMap::new(), &opts),
-            )
-            .await
-            .expect("unmatched condition opens the intact object");
-            let mut body = Vec::new();
-            reader.stream.read_to_end(&mut body).await.expect("positive-control body");
-            assert_eq!(body, payload);
-            let source_kind = if size == 4096 {
-                disk_call_counters::KIND_INLINE_SHARD_OPEN
-            } else {
-                disk_call_counters::KIND_DISK_SHARD_OPEN
-            };
-            assert!(
-                calls.total(source_kind) > 0,
-                "the same source counter observes the positive control: size={size}, inline={}, disk={}, decode={}",
-                calls.total(disk_call_counters::KIND_INLINE_SHARD_OPEN),
-                calls.total(disk_call_counters::KIND_DISK_SHARD_OPEN),
-                calls.total(disk_call_counters::KIND_BODY_DECODE),
-            );
-            assert!(
-                calls.total(disk_call_counters::KIND_BODY_DECODE) > 0,
-                "the same decode counter observes the positive control"
-            );
-        }
-    }
-
-    #[tokio::test]
-    #[serial_test::serial(body_cache_hook)]
-    async fn read_condition_rechecks_selected_metadata_before_body_access() {
-        struct Changing(std::sync::atomic::AtomicUsize);
-        impl crate::object_api::GetObjectReadCondition for Changing {
-            fn is_terminal(&self, _: crate::object_api::GetObjectReadMetadata<'_>) -> bool {
-                self.0.fetch_add(1, std::sync::atomic::Ordering::Relaxed) == 0
-            }
-        }
-        let (_dirs, set) = make_local_set_disks(4, 2).await;
-        let bucket = "read-condition-consistency";
-        let object = object_with_initial_data_shards(bucket, "read-condition-consistency");
-        let opts = ObjectOptions {
-            no_lock: true,
-            ..Default::default()
-        };
-        set.make_bucket(bucket, &MakeBucketOptions::default())
-            .await
-            .expect("create bucket");
-        let mut put = PutObjReader::from_vec(vec![b'x'; 4096]);
-        set.put_object(bucket, &object, &mut put, &opts).await.expect("write object");
-        let result = crate::object_api::with_get_object_read_condition(
-            Arc::new(Changing(std::sync::atomic::AtomicUsize::new(0))),
-            set.get_object_reader(bucket, &object, None, HeaderMap::new(), &opts),
-        )
-        .await;
-        assert!(result.is_err(), "an unverified terminal snapshot must never become a 200 reader");
-    }
-
-    #[tokio::test]
-    #[serial_test::serial(body_cache_hook)]
-    async fn read_condition_miss_preserves_inline_early_stop_and_exact_body() {
-        let (_dirs, set) = make_local_set_disks(4, 2).await;
-        let bucket = "read-condition-miss";
-        let object = object_with_initial_data_shards(bucket, "read-condition-miss");
-        let opts = ObjectOptions {
-            no_lock: true,
-            ..Default::default()
-        };
-        let payload: Vec<u8> = (0..8192)
-            .map(|index| u8::try_from(index % 251).expect("pattern byte"))
-            .collect();
-        set.make_bucket(bucket, &MakeBucketOptions::default())
-            .await
-            .expect("create bucket");
-        let mut put = PutObjReader::from_vec(payload.clone());
-        set.put_object(bucket, &object, &mut put, &opts)
-            .await
-            .expect("write inline object");
-        let slow = bounded_initial_parity_disk_index(bucket, &object);
-        let barrier = rename_fanout_barrier::arm(&object, slow, rename_fanout_barrier::PHASE_READ_VERSION);
-        let set_for_read = Arc::clone(&set);
-        let object_for_read = object.clone();
-        let mut read = tokio::spawn(async move {
-            crate::object_api::with_get_object_read_condition(
-                Arc::new(StopReadForEtag("other".to_owned())),
-                set_for_read.get_object_reader(bucket, &object_for_read, None, HeaderMap::new(), &opts),
-            )
-            .await
-        });
-        tokio::time::timeout(READ_VERSION_BARRIER_GUARD, barrier.wait_until_paused())
-            .await
-            .expect("slow metadata read is pending");
-        let mut reader = tokio::time::timeout(READ_VERSION_BARRIER_GUARD, &mut read)
-            .await
-            .expect("condition miss preserves the normal early stop")
-            .expect("reader task")
-            .expect("reader");
-        let mut bytes = Vec::new();
-        reader.stream.read_to_end(&mut bytes).await.expect("normal body");
-        assert_eq!(bytes, payload);
-        barrier.release();
-    }
-
     #[test]
     fn cache_hit_consumers_release_snapshot_at_legacy_ownership() {
         let fi = FileInfo {
@@ -1419,7 +1157,6 @@ mod prepared_get_object_metadata_tests {
         };
         let cached = Arc::new(GetObjectMetadataCacheEntry {
             created_at: Instant::now(),
-            capability: GetObjectMetadataCacheCapability::Full,
             parts_metadata: vec![fi.clone()],
             fi,
             online_disks: vec![None],
@@ -1428,11 +1165,11 @@ mod prepared_get_object_metadata_tests {
         let snapshot = GetObjectFileInfo::shared(Arc::clone(&cached));
 
         assert!(snapshot.has_valid_representation());
-        // Keep the original payload budget after moving the captured disks out
-        // of OwnedGetObjectFileInfo.
-        let owned_payload_size = std::mem::size_of::<(OwnedGetObjectFileInfo, Option<Vec<Option<DiskStore>>>)>();
-        assert!(std::mem::size_of::<GetObjectFileInfo>() >= owned_payload_size);
-        assert!(std::mem::size_of::<GetObjectFileInfo>() <= owned_payload_size + 2 * std::mem::size_of::<usize>());
+        assert!(std::mem::size_of::<GetObjectFileInfo>() >= std::mem::size_of::<OwnedGetObjectFileInfo>());
+        assert!(
+            std::mem::size_of::<GetObjectFileInfo>()
+                <= std::mem::size_of::<OwnedGetObjectFileInfo>() + 2 * std::mem::size_of::<usize>()
+        );
         assert_eq!(Arc::strong_count(&cached), 2, "a cache hit must add one snapshot reference");
         assert_eq!(snapshot.fi().name, "object");
         assert_eq!(snapshot.parts_metadata().len(), 1);
@@ -4740,15 +4477,6 @@ impl Hash for GetObjectMetadataCacheKey {
     }
 }
 
-#[derive(Clone, Copy, Debug, PartialEq, Eq)]
-enum GetObjectMetadataCacheCapability {
-    Full,
-    // A quorum-selected layout can serve body reads, but is not a mutation layout.
-    QuorumSelected,
-    // Quorum-validated metadata used only to evaluate a terminal read condition.
-    ConditionOnly,
-}
-
 #[derive(Debug)]
 struct GetObjectMetadataCacheEntry {
     #[allow(dead_code)] // Kept for debugging; moka handles TTL internally
@@ -4757,7 +4485,6 @@ struct GetObjectMetadataCacheEntry {
     parts_metadata: Vec<FileInfo>,
     online_disks: Vec<Option<DiskStore>>,
     read_quorum: usize,
-    capability: GetObjectMetadataCacheCapability,
 }
 
 #[derive(Clone, Debug)]

crates/ecstore/src/set_disk/ops/bucket.rs

@@ -31,7 +31,6 @@ use crate::disk::DiskAPI;
 pub(crate) enum BucketInfoQuorum {
     Read,
     Write,
-    HealCandidates,
 }
 
 impl SetDisks {
@@ -105,9 +104,7 @@ impl SetDisks {
             // class. A namespace read must intersect that majority; object
             // readers still enforce the persisted layout's data-shard quorum.
             BucketInfoQuorum::Read => reduce_read_quorum_errs(&errs, BUCKET_OP_IGNORED_ERRS, disk_count.div_ceil(2).max(1)),
-            BucketInfoQuorum::Write | BucketInfoQuorum::HealCandidates => {
-                reduce_write_quorum_errs(&errs, BUCKET_OP_IGNORED_ERRS, disk_count / 2 + 1)
-            }
+            BucketInfoQuorum::Write => reduce_write_quorum_errs(&errs, BUCKET_OP_IGNORED_ERRS, disk_count / 2 + 1),
         };
         if let Some(err) = error {
             return Err(err.into());
@@ -126,42 +123,8 @@ impl SetDisks {
     }
 
     pub(crate) async fn list_bucket_for_scanner(&self, _opts: &BucketOptions) -> Result<(Vec<BucketInfo>, bool)> {
-        self.list_bucket_with_quorum(BucketInfoQuorum::Write).await
-    }
-
-    pub(crate) async fn bucket_needs_reconstruction(&self, bucket: &str) -> Result<bool> {
-        let results = join_all(self.disk_inventory().await.into_iter().map(|disk| async move {
-            match disk {
-                Some(disk) => disk.stat_volume(bucket).await,
-                None => Err(DiskError::DiskNotFound),
-            }
-        }))
-        .await;
-        let mut present = false;
-        let mut missing = false;
-        for result in results {
-            match result {
-                Ok(_) => present = true,
-                Err(DiskError::VolumeNotFound) => missing = true,
-                Err(error) => return Err(error.into()),
-            }
-        }
-        if !present {
-            return Err(Error::VolumeNotFound);
-        }
-        Ok(missing)
-    }
-
-    pub(crate) async fn list_bucket_with_quorum(&self, quorum: BucketInfoQuorum) -> Result<(Vec<BucketInfo>, bool)> {
         let disks = self.disk_inventory().await;
         let write_quorum = (disks.len() / 2) + 1;
-        // Discovery uses the same namespace read boundary as stat_bucket.
-        // It grants no authority to mutate buckets or accept object reads.
-        let required = match quorum {
-            BucketInfoQuorum::Read => disks.len().div_ceil(2).max(1),
-            BucketInfoQuorum::Write => write_quorum,
-            BucketInfoQuorum::HealCandidates => disks.len().max(1),
-        };
 
         let mut futures = Vec::with_capacity(disks.len());
         for disk in disks {
@@ -190,13 +153,7 @@ impl SetDisks {
             }
         }
 
-        let error = match quorum {
-            BucketInfoQuorum::Read => reduce_read_quorum_errs(&errs, BUCKET_OP_IGNORED_ERRS, required),
-            BucketInfoQuorum::Write | BucketInfoQuorum::HealCandidates => {
-                reduce_write_quorum_errs(&errs, BUCKET_OP_IGNORED_ERRS, required)
-            }
-        };
-        if let Some(err) = error {
+        if let Some(err) = reduce_write_quorum_errs(&errs, BUCKET_OP_IGNORED_ERRS, write_quorum) {
             return Err(err.into());
         }
 
@@ -219,17 +176,10 @@ impl SetDisks {
             }
         }
 
-        let minimum_presence = if matches!(quorum, BucketInfoQuorum::HealCandidates) {
-            // Candidate presence is not mutation authority. Every disk must answer,
-            // and the heal owner validates the committed generation before recreation.
-            1
-        } else {
-            topology_complete &= counts.values().all(|(count, _)| *count >= write_quorum);
-            required
-        };
+        topology_complete &= counts.values().all(|(count, _)| *count >= write_quorum);
         let mut buckets = counts
             .into_values()
-            .filter_map(|(count, bucket)| (count >= minimum_presence).then_some(bucket))
+            .filter_map(|(count, bucket)| (count >= write_quorum).then_some(bucket))
             .collect::<Vec<_>>();
         buckets.sort_by(|left, right| left.name.cmp(&right.name));
         Ok((buckets, topology_complete))

crates/ecstore/src/set_disk/ops/mod.rs

@@ -19,7 +19,6 @@
 
 pub(crate) mod bitrot_self_verify;
 pub(crate) mod bucket;
-pub(crate) mod delete_recovery;
 pub(crate) mod heal;
 pub(crate) mod heal_walk;
 pub(crate) mod list;

crates/ecstore/src/set_disk/ops/multipart.rs

@@ -63,6 +63,7 @@ use crate::crash_inject::{self, CrashPoint};
 use crate::disk::DiskAPI;
 #[cfg(test)]
 use crate::disk::DiskOption;
+#[cfg(test)]
 use crate::disk::STORAGE_FORMAT_FILE;
 #[cfg(test)]
 use crate::disk::new_disk;
@@ -230,60 +231,6 @@ pub(crate) struct StaleMultipartCleanupGuard {
     lock_guard: ObjectLockDiagGuard,
 }
 
-/// Exact-key recovery is admitted only after every disk proves that the upload
-/// metadata is absent. Keep the checked disk generation and upload lock through
-/// deletion; a quorum miss is not proof that an upload is an orphan.
-/// Final object data, completion replay identity, and old-data cleanup receipts
-/// live under the destination bucket/object, never this staging path. This also
-/// permits removing an exact orphan from a previous bucket incarnation without
-/// claiming that its UploadID belongs to the current incarnation.
-pub(crate) struct OrphanMultipartCleanupGuard {
-    set: Arc<SetDisks>,
-    disks: Vec<Option<DiskStore>>,
-    upload_path: String,
-    lock_guard: Arc<ObjectLockDiagGuard>,
-    exists: bool,
-}
-
-impl OrphanMultipartCleanupGuard {
-    pub(crate) async fn delete(&self, bucket: &str, object: &str, opts: &ObjectOptions) -> Result<bool> {
-        fence_commit_on_lock_loss(Some(&self.lock_guard), "orphan_multipart_cleanup", &self.upload_path)?;
-        ensure_multipart_bucket_lifecycle_lock_held(bucket, object, opts)?;
-        if !self.exists {
-            return Ok(false);
-        }
-
-        let results = join_all(self.disks.iter().map(|disk| async {
-            let Some(disk) = disk else {
-                return Err(DiskError::DiskNotFound);
-            };
-            match disk
-                .delete(
-                    RUSTFS_META_MULTIPART_BUCKET,
-                    &self.upload_path,
-                    crate::disk::DeleteOptions {
-                        recursive: true,
-                        ..Default::default()
-                    },
-                )
-                .await
-            {
-                Err(DiskError::FileNotFound | DiskError::PathNotFound | DiskError::VolumeNotFound) => Ok(()),
-                result => result,
-            }
-        }))
-        .await;
-        self.set.record_capacity_scope_if_needed(None, &self.disks);
-        let errors = results.into_iter().map(|result| result.err()).collect::<Vec<_>>();
-        if let Some(err) = reduce_write_quorum_errs(&errors, OBJECT_OP_IGNORED_ERRS, self.disks.len()) {
-            return Err(to_object_err(err.into(), vec![bucket, object]));
-        }
-        fence_commit_on_lock_loss(Some(&self.lock_guard), "orphan_multipart_cleanup", &self.upload_path)?;
-        ensure_multipart_bucket_lifecycle_lock_held(bucket, object, opts)?;
-        Ok(true)
-    }
-}
-
 pub(crate) struct DataMovementMultipartAbortGuard {
     upload_path: String,
     write_quorum: Option<usize>,
@@ -610,34 +557,6 @@ fn multipart_bucket_incarnation_matches(metadata: &HashMap<String, String>, expe
     matches!(multipart_bucket_incarnation_id(metadata), Ok(Some(actual)) if actual == expected)
 }
 
-fn orphan_multipart_upload_id_is_canonical(upload_id: &str) -> bool {
-    let Ok(decoded) = base64_simd::URL_SAFE_NO_PAD.decode_to_vec(upload_id.as_bytes()) else {
-        return false;
-    };
-    let Ok(decoded) = std::str::from_utf8(&decoded) else {
-        return false;
-    };
-    let Some((_, raw)) = decoded.split_once('.') else {
-        return false;
-    };
-    let Some((uuid, timestamp)) = raw.split_once('x') else {
-        return false;
-    };
-    let Ok(id) = Uuid::parse_str(uuid) else {
-        return false;
-    };
-    if id.is_nil() || id.to_string() != uuid {
-        return false;
-    }
-    let Ok(timestamp_ns) = timestamp.parse::<i128>() else {
-        return false;
-    };
-    if timestamp_ns.to_string() != timestamp {
-        return false;
-    }
-    OffsetDateTime::from_unix_timestamp_nanos(timestamp_ns).is_ok()
-}
-
 fn validate_multipart_bucket_incarnation(
     metadata: &HashMap<String, String>,
     bucket: &str,
@@ -1327,114 +1246,6 @@ impl SetDisks {
         })
     }
 
-    pub(crate) async fn lock_orphan_multipart_cleanup(
-        self: &Arc<Self>,
-        bucket: &str,
-        object: &str,
-        upload_id: &str,
-        opts: &ObjectOptions,
-        held_guards: &[OrphanMultipartCleanupGuard],
-    ) -> Result<OrphanMultipartCleanupGuard> {
-        let invalid_upload = || StorageError::InvalidUploadID(bucket.to_owned(), object.to_owned(), upload_id.to_owned());
-        if opts.data_movement
-            || opts.no_lock
-            || self.local_lock_manager.is_disabled()
-            || !orphan_multipart_upload_id_is_canonical(upload_id)
-        {
-            return Err(invalid_upload());
-        }
-        // This path is derived from the caller's exact bucket/key and a canonical
-        // UUID/timestamp upload identity, never from the shared MPU directory list.
-        let upload_path = Self::get_upload_id_dir(bucket, object, upload_id);
-        // Pool labels do not partition the raw ObjectKey sent to lock clients.
-        // Reuse the actual owner only for this exact path and proven backend;
-        // independent distributed domains still need their own guards.
-        let distributed = self.ctx.is_dist_erasure().await;
-        let mut lock_guard = None;
-        for held in held_guards {
-            if held.upload_path != upload_path {
-                continue;
-            }
-            let shared_domain = match (distributed, held.set.ctx.is_dist_erasure().await) {
-                (false, false) => self.shares_namespace_lock_domain(&held.set).await,
-                (true, true) => {
-                    let same = super::super::same_distributed_lock_domain(&self.lockers, &held.set.lockers);
-                    if !same
-                        && self
-                            .lockers
-                            .iter()
-                            .any(|client| held.set.lockers.iter().any(|held_client| Arc::ptr_eq(client, held_client)))
-                    {
-                        // A guard for only part of another quorum is not proof
-                        // of ownership there, and acquiring it again may wait
-                        // on ourselves. Preserve all staging in this topology.
-                        return Err(Error::other("orphan multipart cleanup cannot cover partially overlapping lock domains"));
-                    }
-                    same
-                }
-                _ => false,
-            };
-            if shared_domain {
-                lock_guard = Some(Arc::clone(&held.lock_guard));
-                break;
-            }
-        }
-        let lock_guard = match lock_guard {
-            Some(guard) => guard,
-            None => Arc::new(
-                self.acquire_write_lock_diag("orphan_multipart_cleanup", RUSTFS_META_MULTIPART_BUCKET, &upload_path)
-                    .await?,
-            ),
-        };
-        fence_commit_on_lock_loss(Some(&lock_guard), "orphan_multipart_cleanup", &upload_path)?;
-        ensure_multipart_bucket_lifecycle_lock_held(bucket, object, opts)?;
-        let disks = self.disks.read().await.clone();
-        if disks.len() != self.set_drive_count || disks.is_empty() {
-            return Err(Error::ErasureReadQuorum);
-        }
-        // Raw full fan-out cannot quorum-short-circuit or interpret an empty
-        // version list as absence of the physical metadata file.
-        let (_, errors) = Self::read_all_raw_file_info(&disks, RUSTFS_META_MULTIPART_BUCKET, &upload_path, false).await;
-        if errors.len() != disks.len() {
-            return Err(Error::ErasureReadQuorum);
-        }
-        for error in errors {
-            match error {
-                Some(DiskError::FileNotFound | DiskError::VolumeNotFound) => {}
-                // Even one valid metadata copy may still be recoverable. Never
-                // convert a partial, corrupt, or unreachable upload into debris.
-                None => return Err(invalid_upload()),
-                Some(error) => return Err(to_object_err(error.into(), vec![bucket, object, upload_id])),
-            }
-        }
-        let mut exists = false;
-        for disk in &disks {
-            let Some(disk) = disk else {
-                return Err(Error::ErasureReadQuorum);
-            };
-            match disk.list_dir(bucket, RUSTFS_META_MULTIPART_BUCKET, &upload_path, -1).await {
-                Ok(entries) => {
-                    // Raw metadata reads report a directory named xl.meta as
-                    // FileNotFound. Its physical presence is unknown/corrupt
-                    // metadata, not proof that this staging can be deleted.
-                    if entries.iter().any(|entry| entry.trim_end_matches('/') == STORAGE_FORMAT_FILE) {
-                        return Err(to_object_err(DiskError::FileCorrupt.into(), vec![bucket, object, upload_id]));
-                    }
-                    exists = true;
-                }
-                Err(DiskError::FileNotFound | DiskError::VolumeNotFound) => {}
-                Err(error) => return Err(to_object_err(error.into(), vec![bucket, object, upload_id])),
-            }
-        }
-        Ok(OrphanMultipartCleanupGuard {
-            set: self.clone(),
-            disks,
-            upload_path,
-            lock_guard,
-            exists,
-        })
-    }
-
     #[allow(clippy::too_many_arguments)]
     pub(crate) async fn list_multipart_uploads_for_incarnation(
         &self,
@@ -1867,7 +1678,6 @@ impl crate::storage_api_contracts::multipart::MultipartOperations for SetDisks {
             drop(admission_guard);
         }
 
-        let mut commit_owns_cleanup = false;
         let result: Result<PartInfo> = async {
             let erasure =
                 Arc::new(coding::Erasure::try_new(fi.erasure.data_blocks, fi.erasure.parity_blocks, fi.erasure.block_size)
@@ -2071,171 +1881,120 @@ impl crate::storage_api_contracts::multipart::MultipartOperations for SetDisks {
                     .await?;
                 (Some(upload_guard), Some(part_guard))
             };
-            // Only guards owned by this operation may outlive its caller. Internal
-            // data-movement and borrowed-lock callers retain their synchronous
-            // lifetime; a fence by itself cannot transfer the outer lock lease.
-            let detach_commit_owner = _upload_commit_guard.is_some()
-                && opts.namespace_lock_fence.is_none()
-                && (opts.bucket_lifecycle_lock_fence.is_none() || opts.bucket_lifecycle_lock_owner.is_some());
-            let commit_set = self.clone();
-            let commit_bucket = bucket.to_owned();
-            let commit_object = object.to_owned();
-            let commit_upload_id = upload_id.to_owned();
-            let commit_upload_id_path = upload_id_path.clone();
-            let commit_tmp_part = tmp_part.clone();
-            let commit_opts = opts.clone();
-            commit_owns_cleanup = true;
-            // The body is already staged. Keep every commit guard through the
-            // prepare, publish, settlement/rollback and temporary cleanup I/O,
-            // even when the HTTP request stops awaiting this task.
-            let commit = async move {
-                let _upload_commit_guard = _upload_commit_guard;
-                let _part_commit_guard = _part_commit_guard;
-                let _capped_staging_guard = _capped_staging_guard;
-                let bucket = commit_bucket.as_str();
-                let object = commit_object.as_str();
-                let upload_id = commit_upload_id.as_str();
-                let upload_id_path = commit_upload_id_path;
-                let opts = &commit_opts;
-                let result: Result<PartInfo> = async {
-                    let (commit_fi, _) = commit_set
-                        .check_upload_id_exists_with_opts(bucket, object, upload_id, multipart_size_limit.is_some(), opts)
-                        .await?;
-                    let commit_size_limit = multipart_size_limit_from_metadata(&commit_fi.metadata)?;
-                    if commit_size_limit != multipart_size_limit {
-                        return Err(Error::InvalidArgument(
-                            "multipart upload".to_string(),
-                            SUFFIX_MAX_TOTAL_OBJECT_SIZE.to_string(),
-                            "size limit metadata changed or is missing".to_string(),
-                        ));
-                    }
-                    ensure_data_movement_upload_access(&commit_fi, bucket, object, upload_id, opts)?;
-                    ensure_multipart_bucket_incarnation(
-                        &commit_set.ctx,
-                        &commit_fi,
+            let (commit_fi, _) = self
+                .check_upload_id_exists_with_opts(bucket, object, upload_id, multipart_size_limit.is_some(), opts)
+                .await?;
+            let commit_size_limit = multipart_size_limit_from_metadata(&commit_fi.metadata)?;
+            if commit_size_limit != multipart_size_limit {
+                return Err(Error::InvalidArgument(
+                    "multipart upload".to_string(),
+                    SUFFIX_MAX_TOTAL_OBJECT_SIZE.to_string(),
+                    "size limit metadata changed or is missing".to_string(),
+                ));
+            }
+            ensure_data_movement_upload_access(&commit_fi, bucket, object, upload_id, opts)?;
+            ensure_multipart_bucket_incarnation(
+                &self.ctx,
+                &commit_fi,
+                bucket,
+                object,
+                upload_id,
+                opts.expected_bucket_incarnation_id,
+            )
+            .await?;
+            #[cfg(test)]
+            pause_multipart_commit(bucket, object, MultipartCommitPause::PutPartBeforeLockLost).await;
+            fence_commit_on_lock_loss(_upload_commit_guard.as_ref(), "put_object_part_commit", &upload_id_path)?;
+            fence_commit_on_lock_loss(_part_commit_guard.as_ref(), "put_object_part_commit", &part_lock_path)?;
+            if opts
+                .namespace_lock_fence
+                .as_ref()
+                .is_some_and(NamespaceLockFence::is_lock_lost)
+            {
+                return Err(StorageError::NamespaceLockQuorumUnavailable {
+                    mode: "put_object_part_outer_lock",
+                    bucket: bucket.to_string(),
+                    object: object.to_string(),
+                    required: 1,
+                    achieved: 0,
+                });
+            }
+            ensure_multipart_bucket_lifecycle_lock_held(bucket, object, opts)?;
+
+            if let Some(limit) = commit_size_limit {
+                let current_size = self
+                    .current_multipart_logical_size(bucket, object, upload_id, &upload_id_path, &commit_fi, part_id)
+                    .await?;
+                let candidate_size = u64::try_from(actual_size).map_err(|_| Error::PartMissingOrCorrupt)?;
+                admitted_multipart_size(current_size, candidate_size, limit)?;
+            }
+
+            let decommission_capacity_guard = if let Some(store) = opts.decommission_capacity_admission.as_ref() {
+                Some(
+                    store
+                        .acquire_external_decommission_capacity_fence(
+                            &[self.pool_index],
+                            DecommissionCapacityAdmission::ExistingMultipart,
+                        )
+                        .await?,
+                )
+            } else {
+                None
+            };
+            #[cfg(test)]
+            pause_multipart_commit(bucket, object, MultipartCommitPause::PutPartAfterCapacityAdmission).await;
+            if decommission_capacity_guard.as_ref().is_some_and(|guard| guard.is_lock_lost()) {
+                return Err(StorageError::NamespaceLockQuorumUnavailable {
+                    mode: "put_object_part_decommission_capacity",
+                    bucket: bucket.to_string(),
+                    object: object.to_string(),
+                    required: 1,
+                    achieved: 0,
+                });
+            }
+            let rename_result = self
+                .rename_part(
+                    &shuffle_disks,
+                    RUSTFS_META_TMP_BUCKET,
+                    &tmp_part_path,
+                    RUSTFS_META_MULTIPART_BUCKET,
+                    &part_path,
+                    part_info_buff.into(),
+                    write_quorum,
+                    Some(MultipartWriteQuorumContext {
+                        stage: MULTIPART_WRITE_QUORUM_RENAME_PART,
                         bucket,
                         object,
                         upload_id,
-                        opts.expected_bucket_incarnation_id,
-                    )
-                    .await?;
-                    #[cfg(test)]
-                    pause_multipart_commit(bucket, object, MultipartCommitPause::PutPartBeforeLockLost).await;
-                    fence_commit_on_lock_loss(_upload_commit_guard.as_ref(), "put_object_part_commit", &upload_id_path)?;
-                    fence_commit_on_lock_loss(_part_commit_guard.as_ref(), "put_object_part_commit", &part_lock_path)?;
-                    if opts
-                        .namespace_lock_fence
-                        .as_ref()
-                        .is_some_and(NamespaceLockFence::is_lock_lost)
-                    {
-                        return Err(StorageError::NamespaceLockQuorumUnavailable {
-                            mode: "put_object_part_outer_lock",
-                            bucket: bucket.to_string(),
-                            object: object.to_string(),
-                            required: 1,
-                            achieved: 0,
-                        });
-                    }
-                    ensure_multipart_bucket_lifecycle_lock_held(bucket, object, opts)?;
-
-                    if let Some(limit) = commit_size_limit {
-                        let current_size = commit_set
-                            .current_multipart_logical_size(bucket, object, upload_id, &upload_id_path, &commit_fi, part_id)
-                            .await?;
-                        let candidate_size = u64::try_from(actual_size).map_err(|_| Error::PartMissingOrCorrupt)?;
-                        admitted_multipart_size(current_size, candidate_size, limit)?;
-                    }
-
-                    let decommission_capacity_guard = if let Some(store) = opts.decommission_capacity_admission.as_ref() {
-                        Some(
-                            store
-                                .acquire_external_decommission_capacity_fence(
-                                    &[commit_set.pool_index],
-                                    DecommissionCapacityAdmission::ExistingMultipart,
-                                )
-                                .await?,
-                        )
-                    } else {
-                        None
-                    };
-                    #[cfg(test)]
-                    pause_multipart_commit(bucket, object, MultipartCommitPause::PutPartAfterCapacityAdmission).await;
-                    if decommission_capacity_guard.as_ref().is_some_and(|guard| guard.is_lock_lost()) {
-                        return Err(StorageError::NamespaceLockQuorumUnavailable {
-                            mode: "put_object_part_decommission_capacity",
-                            bucket: bucket.to_string(),
-                            object: object.to_string(),
-                            required: 1,
-                            achieved: 0,
-                        });
-                    }
-                    let rename_result = commit_set
-                        .rename_part(
-                            &shuffle_disks,
-                            RUSTFS_META_TMP_BUCKET,
-                            &tmp_part_path,
-                            RUSTFS_META_MULTIPART_BUCKET,
-                            &part_path,
-                            part_info_buff.into(),
-                            write_quorum,
-                            Some(MultipartWriteQuorumContext {
-                                stage: MULTIPART_WRITE_QUORUM_RENAME_PART,
-                                bucket,
-                                object,
-                                upload_id,
-                                part_number: Some(part_id),
-                            }),
-                        )
-                        .await;
-                    drop(decommission_capacity_guard);
-                    let _ = rename_result?;
-                    #[cfg(test)]
-                    observe_multipart_commit(bucket, object, MultipartCommitPause::PutPartBeforeLockLost);
-
-                    #[cfg(test)]
-                    pause_multipart_commit(bucket, object, MultipartCommitPause::PutPartAfterRename).await;
-
-                    let ret: PartInfo = PartInfo {
-                        etag: Some(etag.clone()),
-                        part_num: part_id,
-                        last_mod: Some(OffsetDateTime::now_utc()),
-                        size: w_size,
-                        actual_size,
-                    };
-
-                    // error!("put_object_part ret {:?}", &ret);
-
-                    Ok(ret)
-                }
+                        part_number: Some(part_id),
+                    }),
+                )
                 .await;
-                if result.is_err()
-                    && let Err(err) = commit_set.delete_all(RUSTFS_META_TMP_BUCKET, &commit_tmp_part).await
-                {
-                    warn!(
-                        event = EVENT_SET_DISK_MULTIPART,
-                        component = LOG_COMPONENT_ECSTORE,
-                        subsystem = LOG_SUBSYSTEM_SET_DISK,
-                        result = "cleanup_failed",
-                        bucket,
-                        object,
-                        error = ?err,
-                        "failed to clean multipart commit staging"
-                    );
-                }
-                result
+            drop(decommission_capacity_guard);
+            let _ = rename_result?;
+            #[cfg(test)]
+            observe_multipart_commit(bucket, object, MultipartCommitPause::PutPartBeforeLockLost);
+
+            #[cfg(test)]
+            pause_multipart_commit(bucket, object, MultipartCommitPause::PutPartAfterRename).await;
+            drop(_part_commit_guard);
+            drop(_upload_commit_guard);
+
+            let ret: PartInfo = PartInfo {
+                etag: Some(etag.clone()),
+                part_num: part_id,
+                last_mod: Some(OffsetDateTime::now_utc()),
+                size: w_size,
+                actual_size,
             };
-            if detach_commit_owner {
-                tokio::spawn(commit)
-                    .await
-                .map_err(|err| Error::other_with_context("put_object_part commit task failed", err))?
-            } else {
-                commit.await
-            }
+
+            // error!("put_object_part ret {:?}", &ret);
+
+            Ok(ret)
         }
         .await;
 
-        if !commit_owns_cleanup
-            && result.is_err()
+        if result.is_err()
             && let Err(err) = self.delete_all(RUSTFS_META_TMP_BUCKET, &tmp_part).await
         {
             warn!(tmp_part = %tmp_part, error = ?err, "failed to cleanup multipart temporary data");
@@ -4474,300 +4233,6 @@ mod tests {
         }
     }
 
-    #[test]
-    fn orphan_multipart_identity_requires_canonical_upload_id() {
-        let created = OffsetDateTime::UNIX_EPOCH + time::Duration::seconds(10);
-        let id = Uuid::new_v4();
-        let timestamp = (created + time::Duration::seconds(1)).unix_timestamp_nanos();
-        let encode = |raw: &str| runtime_sources::deployment_upload_id(raw);
-        let upload_id = encode(&format!("{id}x{timestamp}"));
-        assert!(orphan_multipart_upload_id_is_canonical(&upload_id));
-        for raw in [
-            format!("{}x{timestamp}", Uuid::nil()),
-            format!("{}x{timestamp}", id.simple()),
-            format!("{id}x0{timestamp}"),
-            format!("{id}x+{timestamp}"),
-            format!("{id}x{timestamp}/../another-upload"),
-            format!("../{id}x{timestamp}"),
-            format!("{id}x{}", i128::MAX),
-        ] {
-            assert!(
-                !orphan_multipart_upload_id_is_canonical(&encode(&raw)),
-                "noncanonical upload identity must be rejected: {raw}"
-            );
-        }
-        assert!(!orphan_multipart_upload_id_is_canonical("not-base64"));
-    }
-
-    #[tokio::test]
-    async fn orphan_multipart_cleanup_requires_every_disk_to_prove_metadata_absence() {
-        let (_temp_dirs, disks, set) = hermetic_set_disks(4).await;
-        let bucket = "orphan-multipart-proof";
-        let object = "object";
-        make_bucket_on_all(&disks, bucket).await;
-        let upload = set
-            .new_multipart_upload(bucket, object, &ObjectOptions::default())
-            .await
-            .expect("create upload");
-        let upload_path = SetDisks::get_upload_id_dir(bucket, object, &upload.upload_id);
-        let metadata_path = format!("{upload_path}/{STORAGE_FORMAT_FILE}");
-        let leftover_path = format!("{upload_path}/part.late");
-        for disk in &disks {
-            disk.write_all(RUSTFS_META_MULTIPART_BUCKET, &leftover_path, Bytes::from_static(b"late part"))
-                .await
-                .expect("write late part fixture");
-        }
-        for disk in &disks[..3] {
-            disk.delete(RUSTFS_META_MULTIPART_BUCKET, &metadata_path, Default::default())
-                .await
-                .expect("remove majority metadata");
-        }
-        let opts = ObjectOptions::default();
-        assert!(
-            set.lock_orphan_multipart_cleanup(bucket, object, &upload.upload_id, &opts, &[])
-                .await
-                .is_err(),
-            "one surviving metadata copy must block orphan recovery"
-        );
-        disks[3]
-            .write_all(RUSTFS_META_MULTIPART_BUCKET, &metadata_path, Bytes::from_static(b"corrupt metadata"))
-            .await
-            .expect("corrupt final metadata copy");
-        assert!(
-            set.lock_orphan_multipart_cleanup(bucket, object, &upload.upload_id, &opts, &[])
-                .await
-                .is_err(),
-            "corrupt metadata must not be treated as absence"
-        );
-        disks[3]
-            .delete(RUSTFS_META_MULTIPART_BUCKET, &metadata_path, Default::default())
-            .await
-            .expect("remove final metadata copy");
-        set.disks.write().await[3] = None;
-        assert!(
-            set.lock_orphan_multipart_cleanup(bucket, object, &upload.upload_id, &opts, &[])
-                .await
-                .is_err(),
-            "an unreachable disk must not be hidden by absence quorum"
-        );
-        set.disks.write().await[3] = Some(disks[3].clone());
-        for disk in &disks {
-            assert_eq!(
-                disk.read_all(RUSTFS_META_MULTIPART_BUCKET, &leftover_path)
-                    .await
-                    .expect("failed preflight must preserve every part"),
-                Bytes::from_static(b"late part")
-            );
-        }
-
-        let guard = set
-            .lock_orphan_multipart_cleanup(bucket, object, &upload.upload_id, &opts, &[])
-            .await
-            .expect("all disks now prove metadata absence");
-        assert!(guard.delete(bucket, object, &opts).await.expect("recover exact staging"));
-        drop(guard);
-        for disk in &disks {
-            assert!(matches!(
-                disk.list_dir(bucket, RUSTFS_META_MULTIPART_BUCKET, &upload_path, -1).await,
-                Err(DiskError::FileNotFound | DiskError::VolumeNotFound)
-            ));
-        }
-        let guard = set
-            .lock_orphan_multipart_cleanup(bucket, object, &upload.upload_id, &opts, &[])
-            .await
-            .expect("already absent staging remains provably absent");
-        assert!(
-            !guard
-                .delete(bucket, object, &opts)
-                .await
-                .expect("absence does not report recovery")
-        );
-    }
-
-    #[tokio::test]
-    async fn orphan_multipart_cleanup_rejects_metadata_directory_on_one_disk() {
-        let (temp_dirs, disks, set) = hermetic_set_disks(4).await;
-        let bucket = "orphan-multipart-directory";
-        let object = "object";
-        make_bucket_on_all(&disks, bucket).await;
-        let upload = set
-            .new_multipart_upload(bucket, object, &ObjectOptions::default())
-            .await
-            .expect("create upload");
-        let upload_path = SetDisks::get_upload_id_dir(bucket, object, &upload.upload_id);
-        let metadata_path = format!("{upload_path}/{STORAGE_FORMAT_FILE}");
-        let leftover_path = format!("{upload_path}/part.late");
-        for disk in &disks {
-            disk.delete(RUSTFS_META_MULTIPART_BUCKET, &metadata_path, Default::default())
-                .await
-                .expect("remove upload metadata");
-            disk.write_all(RUSTFS_META_MULTIPART_BUCKET, &leftover_path, Bytes::from_static(b"late part"))
-                .await
-                .expect("write leftover part");
-        }
-        let metadata_directory = temp_dirs[3].path().join(RUSTFS_META_MULTIPART_BUCKET).join(&metadata_path);
-        tokio::fs::create_dir_all(&metadata_directory)
-            .await
-            .expect("replace one missing metadata file with a directory");
-        let (_, errors) =
-            SetDisks::read_all_raw_file_info(&set.disks.read().await, RUSTFS_META_MULTIPART_BUCKET, &upload_path, false).await;
-        assert!(
-            errors.iter().all(|error| matches!(error, Some(DiskError::FileNotFound))),
-            "raw metadata reads alone must reproduce the directory-as-absence case: {errors:?}"
-        );
-        assert!(
-            set.lock_orphan_multipart_cleanup(bucket, object, &upload.upload_id, &ObjectOptions::default(), &[])
-                .await
-                .is_err(),
-            "one physical metadata directory must block cleanup on every disk"
-        );
-        assert!(metadata_directory.is_dir(), "failed preflight must preserve the metadata directory");
-        for disk in &disks {
-            assert_eq!(
-                disk.read_all(RUSTFS_META_MULTIPART_BUCKET, &leftover_path)
-                    .await
-                    .expect("failed preflight must preserve every leftover part"),
-                Bytes::from_static(b"late part")
-            );
-        }
-    }
-
-    #[derive(Clone, Copy, PartialEq, Eq)]
-    enum OrphanMultipartLockDomainCase {
-        Shared,
-        Independent,
-        PartiallyOverlapping,
-    }
-
-    async fn check_orphan_multipart_distributed_lock_owners(case: OrphanMultipartLockDomainCase) {
-        let ctx = Arc::new(crate::runtime::instance::InstanceContext::new());
-        ctx.update_erasure_type(SetupType::DistErasure).await;
-        let first_client: Arc<dyn LockClient> =
-            Arc::new(LocalClient::with_manager(Arc::new(rustfs_lock::GlobalLockManager::new())));
-        let second_client: Arc<dyn LockClient> =
-            Arc::new(LocalClient::with_manager(Arc::new(rustfs_lock::GlobalLockManager::new())));
-        let second_clients = match case {
-            OrphanMultipartLockDomainCase::Shared => vec![first_client.clone()],
-            OrphanMultipartLockDomainCase::Independent => vec![second_client],
-            OrphanMultipartLockDomainCase::PartiallyOverlapping => vec![first_client.clone(), second_client],
-        };
-        let (first_dirs, first_disks, first_set) =
-            hermetic_set_disks_with_lockers_and_ctx(4, 0, 2, vec![first_client], ctx.clone()).await;
-        let (second_dirs, second_disks, second_set) = hermetic_set_disks_with_lockers_and_ctx(4, 1, 2, second_clients, ctx).await;
-        let bucket = "orphan-multipart-dist-locks";
-        let object = "object";
-        make_bucket_on_all(&first_disks, bucket).await;
-        make_bucket_on_all(&second_disks, bucket).await;
-        let upload_id = runtime_sources::deployment_upload_id(&format!(
-            "{}x{}",
-            Uuid::new_v4(),
-            OffsetDateTime::now_utc().unix_timestamp_nanos()
-        ));
-        let upload_path = SetDisks::get_upload_id_dir(bucket, object, &upload_id);
-        for directory in first_dirs.iter().chain(second_dirs.iter()) {
-            let staging = directory.path().join(RUSTFS_META_MULTIPART_BUCKET).join(&upload_path);
-            tokio::fs::create_dir_all(&staging)
-                .await
-                .expect("create exact orphan staging");
-            tokio::fs::write(staging.join("part.late"), b"late part")
-                .await
-                .expect("write orphan part");
-        }
-        let opts = ObjectOptions::default();
-        let first_guard = first_set
-            .lock_orphan_multipart_cleanup(bucket, object, &upload_id, &opts, &[])
-            .await
-            .expect("first domain has an orphan");
-        let second_result = tokio::time::timeout(
-            Duration::from_secs(10),
-            second_set.lock_orphan_multipart_cleanup(bucket, object, &upload_id, &opts, std::slice::from_ref(&first_guard)),
-        )
-        .await
-        .expect("lock-domain admission must not wait on a guard it already owns");
-        if case == OrphanMultipartLockDomainCase::PartiallyOverlapping {
-            let error = match second_result {
-                Ok(_) => panic!("one existing quorum cannot authorize a partially overlapping domain"),
-                Err(error) => error,
-            };
-            assert!(error.to_string().contains("partially overlapping lock domains"));
-            for directory in first_dirs.iter().chain(second_dirs.iter()) {
-                assert!(
-                    directory
-                        .path()
-                        .join(RUSTFS_META_MULTIPART_BUCKET)
-                        .join(&upload_path)
-                        .join("part.late")
-                        .exists(),
-                    "uncertain lock coverage must preserve every physical part"
-                );
-            }
-            return;
-        }
-        let second_guard = second_result.expect("the second pool has a proven lock domain");
-        assert_eq!(
-            Arc::ptr_eq(&first_guard.lock_guard, &second_guard.lock_guard),
-            case == OrphanMultipartLockDomainCase::Shared,
-            "only the same exact backend may share a real upload guard across pool labels"
-        );
-        let second_namespace = second_set
-            .new_ns_lock(RUSTFS_META_MULTIPART_BUCKET, &upload_path)
-            .await
-            .unwrap();
-        let mut second_writer = Box::pin(second_namespace.get_write_lock(Duration::from_secs(10)));
-        assert!(futures::poll!(&mut second_writer).is_pending());
-        if case == OrphanMultipartLockDomainCase::Shared {
-            drop(first_guard);
-            assert!(
-                futures::poll!(&mut second_writer).is_pending(),
-                "the second pool must retain the shared real owner after the first guard drops"
-            );
-            drop(second_guard);
-            drop(
-                tokio::time::timeout(Duration::from_secs(10), second_writer)
-                    .await
-                    .expect("the shared backend should release after the last owner")
-                    .expect("shared-domain writer resumes"),
-            );
-            return;
-        }
-        let first_namespace = first_set
-            .new_ns_lock(RUSTFS_META_MULTIPART_BUCKET, &upload_path)
-            .await
-            .unwrap();
-        let mut first_writer = Box::pin(first_namespace.get_write_lock(Duration::from_secs(10)));
-        assert!(futures::poll!(&mut first_writer).is_pending());
-        drop(first_guard);
-        drop(
-            tokio::time::timeout(Duration::from_secs(10), first_writer)
-                .await
-                .expect("first domain should release independently")
-                .expect("first writer resumes"),
-        );
-        assert!(futures::poll!(&mut second_writer).is_pending(), "the second real owner must remain held");
-        drop(second_guard);
-        drop(
-            tokio::time::timeout(Duration::from_secs(10), second_writer)
-                .await
-                .expect("second domain should release with its owner")
-                .expect("second writer resumes"),
-        );
-    }
-
-    #[tokio::test]
-    async fn orphan_multipart_cleanup_keeps_separate_distributed_lock_owners() {
-        check_orphan_multipart_distributed_lock_owners(OrphanMultipartLockDomainCase::Independent).await;
-    }
-
-    #[tokio::test]
-    async fn orphan_multipart_cleanup_reuses_shared_distributed_owner_across_pools() {
-        check_orphan_multipart_distributed_lock_owners(OrphanMultipartLockDomainCase::Shared).await;
-    }
-
-    #[tokio::test]
-    async fn orphan_multipart_cleanup_rejects_partially_overlapping_distributed_domains() {
-        check_orphan_multipart_distributed_lock_owners(OrphanMultipartLockDomainCase::PartiallyOverlapping).await;
-    }
-
     async fn stage_upload_with_create_opts(
         set_disks: &Arc<SetDisks>,
         bucket: &str,
@@ -6905,76 +6370,6 @@ mod tests {
             .expect("abort should delete the upload after UploadPart releases the lock");
     }
 
-    #[tokio::test(flavor = "multi_thread")]
-    #[serial]
-    async fn cancelled_put_object_part_keeps_upload_lock_through_commit() {
-        for pause in [
-            MultipartCommitPause::PutPartAfterCapacityAdmission,
-            MultipartCommitPause::PutPartAfterRename,
-        ] {
-            let manager = Arc::new(rustfs_lock::GlobalLockManager::new());
-            let signaling = Arc::new(SignalingLockClient::new(Arc::new(LocalClient::with_manager(manager))));
-            let lockers: Vec<Arc<dyn LockClient>> = vec![signaling.clone()];
-            let (_temp_dirs, disk_stores, set_disks) = hermetic_set_disks_with_lockers(4, 0, 2, lockers).await;
-            let bucket = "multipart-cancelled-part-commit-bucket";
-            let object = "object";
-            make_bucket_on_all(&disk_stores, bucket).await;
-            let upload = set_disks
-                .new_multipart_upload(bucket, object, &ObjectOptions::default())
-                .await
-                .expect("multipart upload should be created");
-            let upload_id = upload.upload_id;
-            let upload_path = SetDisks::get_upload_id_dir(bucket, object, &upload_id);
-            signaling.set_target(rustfs_lock::ObjectKey::new(RUSTFS_META_MULTIPART_BUCKET, upload_path.clone()));
-            let _setup_type_guard = SetupTypeGuard::switch_to(SetupType::DistErasure).await;
-            let barrier = MultipartCommitBarrier::install(bucket, object, pause);
-
-            let put_store = set_disks.clone();
-            let put_upload_id = upload_id.clone();
-            let put = tokio::spawn(async move {
-                let mut reader = PutObjReader::from_vec(vec![0x45; 4096]);
-                put_store
-                    .put_object_part(bucket, object, &put_upload_id, 1, &mut reader, &ObjectOptions::default())
-                    .await
-            });
-            barrier.wait_until_paused().await;
-            put.abort();
-            assert!(
-                put.await
-                    .expect_err("the request waiter should remain cancellable")
-                    .is_cancelled()
-            );
-
-            let abort_store = set_disks.clone();
-            let abort_upload_id = upload_id.clone();
-            let mut abort = tokio::spawn(async move {
-                abort_store
-                    .abort_multipart_upload(bucket, object, &abort_upload_id, &ObjectOptions::default())
-                    .await
-            });
-            signaling.wait_for_attempts(2).await;
-            assert!(
-                tokio::time::timeout(Duration::from_millis(100), &mut abort).await.is_err(),
-                "Abort must not acquire the upload lock while a cancelled request's commit is paused at {pause:?}"
-            );
-            barrier.release();
-            tokio::time::timeout(Duration::from_secs(10), abort)
-                .await
-                .expect("Abort should finish after the commit drains")
-                .expect("Abort task should not panic")
-                .expect("Abort should delete the committed part staging");
-            for disk in &disk_stores {
-                assert!(
-                    matches!(
-                        disk.list_dir(bucket, RUSTFS_META_MULTIPART_BUCKET, &upload_path, -1).await,
-                        Err(DiskError::FileNotFound)
-                    ),
-                    "the upload directory must be absent after the commit and Abort finish"
-                );
-            }
-        }
-    }
-
     #[tokio::test(flavor = "multi_thread")]
     #[serial]
     async fn put_object_part_different_part_numbers_commit_concurrently() {
@@ -10404,18 +9799,6 @@ mod tests {
                 "the post-commit crash must leave the upload listable for reclamation"
             );
 
-            let cleanup = set_disks
-                .lock_orphan_multipart_cleanup(bucket, object, &u_new, &ObjectOptions::default(), &[])
-                .await
-                .expect("the committed upload has no staging metadata on any disk");
-            assert!(
-                cleanup
-                    .delete(bucket, object, &ObjectOptions::default())
-                    .await
-                    .expect("recover completed upload staging")
-            );
-            drop(cleanup);
-
             // The commit rename already consumed the upload's xl.meta, so the
             // retry cannot read staging. It must still return the committed
             // object: AWS CompleteMultipartUpload is idempotent for the same
@@ -10434,6 +9817,21 @@ mod tests {
                 !upload_is_listed(&set_disks, bucket, object, &u_new).await,
                 "the idempotent retry must reclaim staging left behind by the crash"
             );
+
+            // Reclaim the leftover exactly as the production tail does: delete_all
+            // on the upload path (abort_multipart_upload cannot — the upload's
+            // metadata is gone, so it too answers InvalidUploadID).
+            let upload_dir = SetDisks::get_upload_id_dir(bucket, object, &u_new);
+            set_disks
+                .delete_all(RUSTFS_META_MULTIPART_BUCKET, &upload_dir)
+                .await
+                .expect("sweeping the leftover upload staging should succeed");
+            assert!(
+                !upload_is_listed(&set_disks, bucket, object, &u_new).await,
+                "reclaiming the upload must drop it from the listing"
+            );
+            let (body_after, _) = read_object(&set_disks, bucket, object).await;
+            assert_eq!(body_after, new, "reclaiming the leftover upload must not disturb the committed object");
         }
 
         #[tokio::test]
@@ -10480,18 +9878,6 @@ mod tests {
                             .expect("the failed post-commit completion should release its object guard"),
                     );
 
-                    let cleanup = set_disks
-                        .lock_orphan_multipart_cleanup(bucket, object, &u_new, &ObjectOptions::default(), &[])
-                        .await
-                        .expect("committed staging has no upload metadata");
-                    assert!(
-                        cleanup
-                            .delete(bucket, object, &ObjectOptions::default())
-                            .await
-                            .expect("orphan recovery must only remove upload staging")
-                    );
-                    drop(cleanup);
-
                     let (body, _) = read_object(&set_disks, bucket, object).await;
                     assert_eq!(body, new, "the committed replacement must remain readable after the crash");
                     let mut receipts = 0;

crates/ecstore/src/store/bucket.rs

@@ -20,7 +20,6 @@ use crate::bucket::{
 use crate::error::is_err_bucket_not_found;
 use crate::runtime::sources as runtime_sources;
 use crate::set_disk::{BucketInfoQuorum, get_lock_acquire_timeout};
-use crate::storage_api_contracts::admin::DiskSetSelector;
 use crate::storage_api_contracts::bucket::{BUCKET_LIFECYCLE_LOCK_OBJECT, SRBucketDeleteOp};
 use crate::storage_api_contracts::namespace::NamespaceLocking as _;
 use futures::stream::{self, StreamExt};
@@ -1152,7 +1151,7 @@ impl ECStore {
         let mut scoped_results = futures::future::join_all(self.bucket_sets().map(|(pool_index, set_index, set)| async move {
             let result = match quorum {
                 BucketInfoQuorum::Read => set.stat_bucket_with_quorum(bucket, quorum).await,
-                BucketInfoQuorum::Write | BucketInfoQuorum::HealCandidates => set.get_bucket_info(bucket, opts).await,
+                BucketInfoQuorum::Write => set.get_bucket_info(bucket, opts).await,
             };
             (pool_index, set_index, result)
         }))
@@ -1232,70 +1231,34 @@ impl ECStore {
     #[instrument(skip(self))]
     pub(super) async fn handle_list_bucket(&self, opts: &BucketOptions) -> Result<Vec<BucketInfo>> {
         // TODO(backlog): support cached bucket listing via opts.cached
-        Ok(self
-            .list_bucket_from_sets(opts, usize::MAX, BucketInfoQuorum::Write, None)
-            .await?
-            .buckets)
+        Ok(self.list_bucket_from_sets(opts, usize::MAX).await?.buckets)
     }
 
     pub async fn list_bucket_for_scanner(&self, opts: &BucketOptions) -> Result<crate::cluster::rpc::ScannerBucketListing> {
-        self.list_bucket_from_sets(opts, SCANNER_BUCKET_LIST_SET_CONCURRENCY, BucketInfoQuorum::Write, None)
-            .await
-    }
-
-    /// Discover candidates for metadata initialization at each set's namespace
-    /// read quorum. Metadata loading still validates persisted configuration;
-    /// this listing must not be used as a complete scanner or mutation view.
-    pub async fn list_buckets_for_metadata_startup(&self) -> Result<Vec<BucketInfo>> {
-        Ok(self
-            .list_bucket_from_sets(
-                &BucketOptions {
-                    no_metadata: true,
-                    ..Default::default()
-                },
-                SCANNER_BUCKET_LIST_SET_CONCURRENCY,
-                BucketInfoQuorum::Read,
-                None,
-            )
-            .await?
-            .buckets)
-    }
-
-    /// Preserve namespace completeness for administrator traversal. A scoped
-    /// request must not depend on unrelated sets being available.
-    pub async fn list_buckets_for_admin_heal(
-        &self,
-        selector: Option<DiskSetSelector>,
-    ) -> Result<crate::cluster::rpc::ScannerBucketListing> {
-        self.list_bucket_from_sets(
-            &BucketOptions {
-                no_metadata: true,
-                ..Default::default()
-            },
-            SCANNER_BUCKET_LIST_SET_CONCURRENCY,
-            BucketInfoQuorum::HealCandidates,
-            selector,
-        )
-        .await
+        self.list_bucket_from_sets(opts, SCANNER_BUCKET_LIST_SET_CONCURRENCY).await
     }
 
     async fn list_bucket_from_sets(
         &self,
         opts: &BucketOptions,
         max_concurrency: usize,
-        quorum: BucketInfoQuorum,
-        selector: Option<DiskSetSelector>,
     ) -> Result<crate::cluster::rpc::ScannerBucketListing> {
-        let sets = self
-            .bucket_sets()
-            .filter(|(pool, set, _)| selector.is_none_or(|scope| scope.pool_idx == *pool && scope.set_idx == *set))
-            .collect::<Vec<_>>();
-        let set_count = sets.len();
+        let set_count = self.pools.iter().map(|pool| pool.disk_set.len()).sum();
         let concurrency = bucket_list_set_concurrency(set_count, max_concurrency);
-        let mut set_listings = stream::iter(sets.into_iter().map(move |(pool_index, set_index, set)| async move {
-            set.list_bucket_with_quorum(quorum)
-                .await
-                .map(|(buckets, complete)| (pool_index, set_index, buckets, complete))
+        let deleted = opts.deleted;
+        let cached = opts.cached;
+        let no_metadata = opts.no_metadata;
+        let mut set_listings = stream::iter(self.bucket_sets().map(move |(pool_index, set_index, set)| {
+            let opts = BucketOptions {
+                deleted,
+                cached,
+                no_metadata,
+            };
+            async move {
+                set.list_bucket_for_scanner(&opts)
+                    .await
+                    .map(|(buckets, complete)| (pool_index, set_index, buckets, complete))
+            }
         }))
         .buffer_unordered(concurrency);
         let mut topology_complete = set_count != 0;
@@ -2261,64 +2224,6 @@ mod tests {
         }
     }
 
-    #[tokio::test]
-    async fn admin_bucket_discovery_preserves_completeness_and_scope() {
-        use crate::storage_api_contracts::admin::DiskSetSelector;
-        let (temp, store) = setup_bucket_quorum_test_env(&[2, 2], None).await;
-        let empty = store
-            .list_buckets_for_admin_heal(None)
-            .await
-            .expect("empty namespace enumeration");
-        assert!(empty.buckets.is_empty());
-        assert!(empty.topology_complete);
-        for disk in 0..2 {
-            tokio::fs::create_dir(temp.path().join(format!("pool0-disk{disk}/healthy-bucket")))
-                .await
-                .expect("healthy bucket");
-        }
-        tokio::fs::create_dir(temp.path().join("pool1-disk0/partial-bucket"))
-            .await
-            .expect("single surviving bucket directory");
-        let global = store
-            .list_buckets_for_admin_heal(None)
-            .await
-            .expect("online disk enumeration");
-        assert!(global.topology_complete, "every disk answered the candidate inventory");
-        assert_eq!(
-            global.buckets.iter().map(|b| b.name.as_str()).collect::<Vec<_>>(),
-            ["healthy-bucket", "partial-bucket"]
-        );
-        let healthy = store
-            .list_buckets_for_admin_heal(Some(DiskSetSelector::new(0, 0)))
-            .await
-            .expect("healthy scope");
-        assert!(healthy.topology_complete);
-        assert_eq!(healthy.buckets.len(), 1);
-        assert_eq!(healthy.set_buckets.len(), 1);
-        let partial = store
-            .list_buckets_for_admin_heal(Some(DiskSetSelector::new(1, 0)))
-            .await
-            .expect("partial scope");
-        assert!(partial.topology_complete);
-        assert_eq!(partial.buckets[0].name, "partial-bucket");
-        let invalid = store
-            .list_buckets_for_admin_heal(Some(DiskSetSelector::new(0, 1)))
-            .await
-            .expect("unconfigured scope");
-        assert!(!invalid.topology_complete);
-        assert!(invalid.buckets.is_empty());
-        let offline = take_set_disks_offline(&store, &store.pools[1].disk_set[0], &[0]).await;
-        assert!(store.list_buckets_for_admin_heal(None).await.is_err(), "quorum errors must remain errors");
-        assert!(
-            store
-                .list_buckets_for_admin_heal(Some(DiskSetSelector::new(0, 0)))
-                .await
-                .expect("unrelated offline pool")
-                .topology_complete
-        );
-        restore_set_disks(&store, &store.pools[1].disk_set[0], offline).await;
-    }
-
     #[test]
     fn should_not_override_when_metadata_created_is_unix_epoch() {
         assert!(!should_override_created_from_metadata(OffsetDateTime::UNIX_EPOCH));
@@ -3638,79 +3543,6 @@ mod tests {
         assert_eq!(info_err, StorageError::ErasureReadQuorum);
     }
 
-    #[tokio::test]
-    #[serial]
-    async fn metadata_startup_listing_uses_each_sets_read_quorum() {
-        for drives_per_pool in [vec![2], vec![3], vec![4], vec![6], vec![8], vec![4, 6]] {
-            let (_temp_dir, store) = setup_bucket_quorum_test_env(&drives_per_pool, None).await;
-            let bucket = "metadata-startup-read-quorum";
-            for pool in &store.pools {
-                pool.disk_set[0]
-                    .make_bucket(bucket, &MakeBucketOptions::default())
-                    .await
-                    .expect("healthy set should create the startup candidate");
-            }
-            let mut offline = Vec::new();
-            for (pool_index, &width) in drives_per_pool.iter().enumerate() {
-                let set = &store.pools[pool_index].disk_set[0];
-                let indexes = (width.div_ceil(2)..width).collect::<Vec<_>>();
-                offline.push(take_set_disks_offline(&store, set, &indexes).await);
-            }
-            let listing = store
-                .list_buckets_for_metadata_startup()
-                .await
-                .expect("startup must discover buckets at every set's read quorum");
-            assert_eq!(listing.len(), 1);
-            assert_eq!(listing[0].name, bucket);
-            if drives_per_pool.iter().any(|width| width % 2 == 0) {
-                let error = store
-                    .list_bucket_for_scanner(&BucketOptions::default())
-                    .await
-                    .expect_err("startup discovery must not weaken scanner quorum");
-                assert_eq!(error, StorageError::ErasureWriteQuorum);
-                let error = store
-                    .list_bucket(&BucketOptions {
-                        no_metadata: true,
-                        ..Default::default()
-                    })
-                    .await
-                    .expect_err("ordinary listing must retain its existing quorum");
-                assert_eq!(error, StorageError::ErasureWriteQuorum);
-            }
-            // A healthy earlier pool must not hide read-quorum loss in a later pool.
-            let last_pool = store.pools.len() - 1;
-            let set = &store.pools[last_pool].disk_set[0];
-            let extra = take_set_disks_offline(&store, set, &[0]).await;
-            let error = store
-                .list_buckets_for_metadata_startup()
-                .await
-                .expect_err("every set must retain read quorum for startup discovery");
-            assert_eq!(error, StorageError::ErasureReadQuorum);
-            restore_set_disks(&store, set, extra).await;
-            for (pool_index, disks) in offline.into_iter().enumerate() {
-                restore_set_disks(&store, &store.pools[pool_index].disk_set[0], disks).await;
-            }
-        }
-    }
-
-    #[tokio::test]
-    #[serial]
-    async fn metadata_startup_listing_does_not_accept_single_disk_bucket_remnants() {
-        let (temp_dir, store) = setup_bucket_quorum_test_env(&[4], None).await;
-        let bucket = "incomplete-startup-bucket";
-        tokio::fs::create_dir(temp_dir.path().join("pool0-disk0").join(bucket))
-            .await
-            .expect("create one uncommitted directory remnant");
-        let set = &store.pools[0].disk_set[0];
-        let offline = take_set_disks_offline(&store, set, &[2, 3]).await;
-        let listing = store
-            .list_buckets_for_metadata_startup()
-            .await
-            .expect("two responsive disks satisfy discovery quorum");
-        assert!(listing.iter().all(|info| info.name != bucket));
-        restore_set_disks(&store, set, offline).await;
-    }
-
     #[tokio::test]
     #[serial]
     async fn scanner_bucket_listing_unions_every_erasure_set() {

crates/ecstore/src/store/heal.rs

@@ -270,151 +270,13 @@ impl ECStore {
         .map_err(|error| Error::other_with_context("bucket heal owner task failed", error))?
     }
 
-    /// Recover hidden delete journals through the lifecycle-owned heal boundary.
-    pub async fn recover_delete_transactions(self: &Arc<Self>, bucket: &str, opts: &HealOpts) -> Result<HealResultItem> {
-        Ok(self
-            .recover_delete_transactions_at_incarnation(bucket, None, None, opts)
-            .await?
-            .unwrap_or_else(|| HealResultItem {
-                heal_item_type: HealItemType::Bucket.to_string(),
-                bucket: bucket.to_owned(),
-                detail: "delete recovery: discovered=0".to_owned(),
-                ..Default::default()
-            }))
-    }
-
-    pub async fn recover_delete_transactions_at_incarnation(
-        self: &Arc<Self>,
-        bucket: &str,
-        object: Option<&str>,
-        expected: Option<Uuid>,
-        opts: &HealOpts,
-    ) -> Result<Option<HealResultItem>> {
-        self.recover_delete_transactions_in_scope(bucket, object, None, expected, opts)
-            .await
-    }
-
-    pub async fn recover_delete_prefix_at_incarnation(
-        self: &Arc<Self>,
-        bucket: &str,
-        prefix: &str,
-        expected: Option<Uuid>,
-        opts: &HealOpts,
-    ) -> Result<Option<HealResultItem>> {
-        self.recover_delete_transactions_in_scope(bucket, None, Some(prefix), expected, opts)
-            .await
-    }
-
-    async fn recover_delete_transactions_in_scope(
-        self: &Arc<Self>,
-        bucket: &str,
-        object: Option<&str>,
-        prefix: Option<&str>,
-        expected: Option<Uuid>,
-        opts: &HealOpts,
-    ) -> Result<Option<HealResultItem>> {
-        if opts.no_lock && !opts.dry_run {
-            return Err(StorageError::InvalidArgument(
-                "heal".to_string(),
-                "no_lock".to_string(),
-                "delete recovery requires namespace locking".to_string(),
-            ));
-        }
-        let expected = match expected {
-            Some(expected) => expected,
-            None => self.bucket_incarnation_id_from_disk(bucket).await?,
-        };
-        let object = object.map(str::to_owned);
-        let prefix = prefix.map(str::to_owned);
-        self.run_bucket_heal_at_incarnation(bucket, expected, opts, |store, bucket, opts| async move {
-            let scope = bucket_heal_scope(&bucket).ok_or(StorageError::PreconditionFailed)?;
-            let explicit_delete = opts.remove
-                && metadata_sys::has_authoritative_never_versioned_state_in(&store.ctx, &bucket).await?
-                && matches!(
-                    metadata_sys::get_object_lock_config_state_in(&store.ctx, &bucket).await?,
-                    metadata_sys::ObjectLockConfigState::ConfirmedAbsent
-                );
-            let movement_gate = store.ctx.data_movement_operation_gate();
-            let _movement_guard = movement_gate.read().await;
-            let can_commit = || scope.check().is_ok();
-            let recovery_context = crate::set_disk::DeleteRecoveryContext {
-                dry_run: opts.dry_run,
-                explicit_delete,
-                namespace_owner: (!opts.dry_run)
-                    .then(|| Arc::new((scope.clone(), store.ctx.begin_namespace_commit())) as Arc<dyn Send + Sync>),
-                ..crate::set_disk::DeleteRecoveryContext::new(Some(scope.incarnation), &can_commit)
-            };
-            let mut summary = crate::set_disk::DeleteRecoverySummary::default();
-            for pool in store.get_pools_for_heal_object(&opts)? {
-                if store.is_suspended(pool.pool_idx).await || store.is_pool_rebalancing(pool.pool_idx).await {
-                    return Err(StorageError::SlowDown);
-                }
-                for (set_index, set) in pool.disk_set.iter().enumerate() {
-                    if opts.set.is_some_and(|requested| requested != set_index) {
-                        continue;
-                    }
-                    if let Some(object) = &object {
-                        if !Arc::ptr_eq(set, &pool.get_disks_by_key(object)) {
-                            continue;
-                        }
-                        summary.add(set.recover_delete_object(&bucket, object, recovery_context.clone()).await?);
-                    } else if let Some(prefix) = &prefix {
-                        summary.add(set.recover_delete_prefix(&bucket, prefix, recovery_context.clone()).await?);
-                    } else {
-                        summary.add(set.recover_delete_bucket(&bucket, recovery_context.clone()).await?);
-                    }
-                }
-            }
-            if summary.pending > 0 {
-                return Err(StorageError::ErasureReadQuorum);
-            }
-            if object.is_some() && !opts.dry_run && summary.blocked > 0 {
-                return Err(StorageError::SlowDown);
-            }
-            if summary.discovered == 0 {
-                return Ok(None);
-            }
-            Ok(Some(HealResultItem {
-                heal_item_type: if object.is_some() {
-                    HealItemType::Object
-                } else {
-                    HealItemType::Bucket
-                }
-                .to_string(),
-                object: object.or(prefix).unwrap_or_default(),
-                bucket,
-                detail: format!(
-                    "delete recovery{}: discovered={}, committed={}, rolled_back={}, reclaimed={}, blocked={}",
-                    if opts.dry_run { " (dry run)" } else { "" },
-                    summary.discovered,
-                    summary.committed,
-                    summary.rolled_back,
-                    summary.reclaimed,
-                    summary.blocked
-                ),
-                ..Default::default()
-            }))
-        })
-        .await
-    }
-
     pub async fn heal_bucket_at_incarnation(
         self: &Arc<Self>,
         bucket: &str,
         expected: uuid::Uuid,
         opts: &HealOpts,
     ) -> Result<HealResultItem> {
-        self.run_bucket_heal_at_incarnation(bucket, expected, opts, move |store, bucket, mut opts| async move {
-            if !opts.recreate && !opts.remove {
-                let mut missing = false;
-                for (pool, set) in store.heal_erasure_set_scopes(&opts).await? {
-                    missing |= store.pools[pool].get_disks(set).bucket_needs_reconstruction(&bucket).await?;
-                }
-                if missing {
-                    crate::bucket::metadata_sys::validate_bucket_reconstruction_in(&store.ctx, &bucket, expected).await?;
-                    opts.recreate = true;
-                }
-            }
+        self.run_bucket_heal_at_incarnation(bucket, expected, opts, |store, bucket, opts| async move {
             store.heal_bucket(&bucket, &opts).await
         })
         .await
@@ -1461,178 +1323,6 @@ mod tests {
         .expect("minimal pool should build")
     }
 
-    #[tokio::test]
-    #[serial_test::serial]
-    async fn admin_bucket_recovery_recreates_missing_directories_at_current_incarnation() {
-        let (root, store, shutdown) = multi_pool_heal_store().await;
-        let bucket = "admin-bucket-recovery";
-        store
-            .make_bucket(bucket, &MakeBucketOptions::default())
-            .await
-            .expect("create a committed bucket generation");
-        let incarnation = store
-            .bucket_incarnation_id_from_disk(bucket)
-            .await
-            .expect("persisted generation");
-        for disk in 1..4 {
-            tokio::fs::remove_dir(root.path().join(format!("pool0-disk{disk}/{bucket}")))
-                .await
-                .expect("remove only an empty fixture bucket directory");
-        }
-        store
-            .heal_bucket_at_incarnation(bucket, incarnation, &HealOpts::default())
-            .await
-            .expect("default administrator recovery should use committed generation authority");
-        for disk in 0..4 {
-            assert!(root.path().join(format!("pool0-disk{disk}/{bucket}")).is_dir());
-        }
-        assert_eq!(
-            store
-                .bucket_incarnation_id_from_disk(bucket)
-                .await
-                .expect("recovered generation"),
-            incarnation
-        );
-        shutdown.cancel();
-    }
-
-    #[tokio::test]
-    #[serial_test::serial]
-    async fn admin_bucket_recovery_rejects_missing_corrupt_uncommitted_and_retired_authority() {
-        let (root, store, shutdown) = multi_pool_heal_store().await;
-        for state in ["missing", "corrupt", "uncommitted", "mismatched", "retired"] {
-            let bucket = format!("recovery-authority-{state}");
-            store
-                .make_bucket(&bucket, &MakeBucketOptions::default())
-                .await
-                .expect("create fixture generation");
-            let incarnation = store
-                .bucket_incarnation_id_from_disk(&bucket)
-                .await
-                .expect("fixture identity");
-            let (mut metadata, persisted) = metadata_sys::get_config_from_disk_with_presence_in(&store.ctx, &bucket)
-                .await
-                .expect("committed metadata");
-            assert!(persisted && metadata.bucket_creation_committed);
-            let path = metadata.save_file_path();
-            match state {
-                "missing" => delete_config(store.clone(), &path).await.expect("remove metadata authority"),
-                "corrupt" => save_config(store.clone(), &path, b"corrupt bucket metadata".to_vec())
-                    .await
-                    .expect("corrupt authority"),
-                "uncommitted" | "mismatched" => {
-                    if state == "uncommitted" {
-                        metadata.bucket_creation_committed = false;
-                    } else {
-                        metadata.bucket_incarnation_id = Uuid::new_v4();
-                    }
-                    metadata
-                        .save_with_store_committed(store.clone())
-                        .await
-                        .expect("inject inconsistent metadata authority");
-                }
-                "retired" => {
-                    // A returning disk may retain a directory after retirement was committed.
-                    crate::bucket::retirement::commit_retirement(store.clone(), &bucket, incarnation, &ObjectOptions::default())
-                        .await
-                        .expect("persist retirement evidence");
-                }
-                _ => unreachable!("fixed fixture states"),
-            }
-            for disk in 1..4 {
-                tokio::fs::remove_dir(root.path().join(format!("pool0-disk{disk}/{bucket}")))
-                    .await
-                    .expect("lose fixture directories");
-            }
-            assert!(
-                store
-                    .heal_bucket_at_incarnation(&bucket, incarnation, &HealOpts::default())
-                    .await
-                    .is_err(),
-                "must reject {state}"
-            );
-            for disk in 1..4 {
-                assert!(
-                    !root.path().join(format!("pool0-disk{disk}/{bucket}")).exists(),
-                    "{state} cannot authorize reconstruction"
-                );
-            }
-        }
-        shutdown.cancel();
-    }
-
-    #[tokio::test]
-    #[serial_test::serial]
-    async fn admin_bucket_recovery_rejects_offline_disks_and_fully_missing_sets() {
-        for offline in [false, true] {
-            let (root, store, shutdown) = multi_pool_heal_store().await;
-            let bucket = "recovery-unproven-set";
-            store
-                .make_bucket(bucket, &MakeBucketOptions::default())
-                .await
-                .expect("fixture bucket");
-            let incarnation = store.bucket_incarnation_id_from_disk(bucket).await.expect("fixture identity");
-            let first_missing = usize::from(offline);
-            for disk in first_missing..4 {
-                tokio::fs::remove_dir(root.path().join(format!("pool0-disk{disk}/{bucket}")))
-                    .await
-                    .expect("remove fixture directories");
-            }
-            if offline {
-                let disk = store.pools[0].disk_set[0].disks.read().await[0]
-                    .clone()
-                    .expect("online fixture disk");
-                disk.close().await.expect("stop disk monitoring");
-                disk.force_offline_for_test();
-            }
-            assert!(
-                store
-                    .heal_bucket_at_incarnation(bucket, incarnation, &HealOpts::default())
-                    .await
-                    .is_err()
-            );
-            for disk in first_missing..4 {
-                assert!(!root.path().join(format!("pool0-disk{disk}/{bucket}")).exists());
-            }
-            shutdown.cancel();
-        }
-    }
-
-    #[tokio::test]
-    #[serial_test::serial]
-    async fn admin_bucket_recovery_dry_run_preserves_missing_directories() {
-        let (root, store, shutdown) = multi_pool_heal_store().await;
-        let bucket = "recovery-dry-run";
-        store
-            .make_bucket(bucket, &MakeBucketOptions::default())
-            .await
-            .expect("fixture bucket");
-        let incarnation = store.bucket_incarnation_id_from_disk(bucket).await.expect("fixture identity");
-        for disk in 1..4 {
-            tokio::fs::remove_dir(root.path().join(format!("pool0-disk{disk}/{bucket}")))
-                .await
-                .expect("remove fixture directories");
-        }
-        store
-            .heal_bucket_at_incarnation(
-                bucket,
-                incarnation,
-                &HealOpts {
-                    dry_run: true,
-                    ..Default::default()
-                },
-            )
-            .await
-            .expect("dry run can inspect eligible recovery");
-        for disk in 1..4 {
-            assert!(
-                !root.path().join(format!("pool0-disk{disk}/{bucket}")).exists(),
-                "dry run must not reconstruct directories"
-            );
-        }
-        shutdown.cancel();
-    }
-
     #[tokio::test]
     #[serial_test::serial]
     async fn bucket_incarnation_heal_preserves_successor_shards_and_allows_fresh_repair() {

crates/ecstore/src/store/init.rs

@@ -108,10 +108,6 @@ fn should_retry_format_load(err: &Error) -> bool {
     )
 }
 
-fn format_load_retry_exhausted(err: &Error, retries: u32) -> bool {
-    retries >= 10 && !matches!(err, Error::ErasureReadQuorum | Error::FirstDiskWait | Error::NotFirstDisk)
-}
-
 fn should_auto_start_rebalance_after_init(decommission_running: bool, rebalance_resume_required: bool) -> bool {
     rebalance_resume_required && !decommission_running
 }
@@ -488,7 +484,7 @@ impl ECStore {
             check_disk_fatal_errs(&errs)?;
 
             let loaded_format = {
-                let mut times = 0_u32;
+                let mut times = 0;
                 let mut interval = 1;
                 loop {
                     match init_format::connect_load_init_formats_with_instance_ctx(
@@ -504,13 +500,13 @@ impl ECStore {
                         Ok(fm) => break Ok(fm),
                         Err(e) if !should_retry_format_load(&e) => break Err(e),
                         // Wrap the final error if we are giving up
-                        Err(e) if format_load_retry_exhausted(&e, times) => {
+                        Err(e) if times >= 10 => {
                             break Err(Error::other(format!("store init failed to load formats after {times} retries: {e}")));
                         }
                         // Retrying so just drop the error
                         Err(_) => {}
                     }
-                    times = times.saturating_add(1);
+                    times += 1;
                     if interval < 16 {
                         interval *= 2;
                     }
@@ -523,12 +519,6 @@ impl ECStore {
                         "Retrying storage format load"
                     );
                     select! {
-                        _ = ctx.cancelled() => {
-                            return Err(Error::other(std::io::Error::new(
-                                std::io::ErrorKind::Interrupted,
-                                "storage format initialization cancelled",
-                            )));
-                        }
                         _ = tokio::signal::ctrl_c() => {
                             info!(
                                 event = EVENT_STORE_FORMAT_RETRY,
@@ -2211,23 +2201,6 @@ mod tests {
         assert_eq!(attempts.load(Ordering::SeqCst), 2);
     }
 
-    #[test]
-    fn format_peer_wait_does_not_exhaust_the_retry_budget() {
-        for retries in [0, 9, 10, u32::MAX] {
-            for error in [
-                StorageError::ErasureReadQuorum,
-                StorageError::FirstDiskWait,
-                StorageError::NotFirstDisk,
-            ] {
-                assert!(
-                    !super::format_load_retry_exhausted(&error, retries),
-                    "peer availability must remain retryable"
-                );
-            }
-            assert_eq!(super::format_load_retry_exhausted(&StorageError::Timeout, retries), retries >= 10);
-        }
-    }
-
     #[test]
     fn test_should_retry_format_load_rejects_permanent_corruption() {
         assert!(!should_retry_format_load(&StorageError::CorruptedFormat));
@@ -5030,339 +5003,6 @@ mod tests {
         shutdown.cancel();
     }
 
-    #[tokio::test]
-    #[serial_test::serial(storage_class_env)]
-    async fn repeated_abort_recovers_exact_orphan_before_decommission_drains() {
-        use crate::storage_api_contracts::namespace::NamespaceLocking as _;
-        use sha2::Digest;
-
-        let temp_dir = tempfile::tempdir().expect("create orphan recovery store dir");
-        let (_ctx, store, shutdown) =
-            without_storage_class_env(build_isolated_test_store(temp_dir.path(), "orphan-multipart-drain", &[4, 4])).await;
-        crate::bucket::metadata_sys::init_bucket_metadata_sys(store.clone(), Vec::new()).await;
-        let bucket = format!("orphan-multipart-{}", uuid::Uuid::new_v4());
-        let object = "exact-object.bin";
-        store
-            .make_bucket(&bucket, &MakeBucketOptions::default())
-            .await
-            .expect("create bucket");
-        let incarnation = store.bucket_incarnation_id(&bucket).await.expect("bucket incarnation");
-        let lifecycle_guard = store.acquire_bucket_lifecycle_read_lock(&bucket).await.expect("bucket guard");
-        let mut upload_opts = ObjectOptions {
-            expected_bucket_incarnation_id: Some(incarnation),
-            ..Default::default()
-        };
-        upload_opts.add_bucket_lifecycle_lock_guard(&lifecycle_guard);
-        let upload = store.pools[0]
-            .new_multipart_upload(&bucket, object, &upload_opts)
-            .await
-            .expect("create source upload");
-        drop(lifecycle_guard);
-        store
-            .abort_multipart_upload(&bucket, object, &upload.upload_id, &ObjectOptions::default())
-            .await
-            .expect("initial abort");
-
-        let upload_sha =
-            hex_simd::encode_to_string(sha2::Sha256::digest(format!("{bucket}/{object}").as_bytes()), hex_simd::AsciiCase::Lower);
-        let upload_uuid = crate::runtime::sources::upload_uuid_suffix(&upload.upload_id);
-        let staging_dir = |pool_index: usize, disk_index: usize| {
-            temp_dir
-                .path()
-                .join(format!("pool{pool_index}/set0/disk{disk_index}"))
-                .join(crate::disk::RUSTFS_META_MULTIPART_BUCKET)
-                .join(&upload_sha)
-                .join(&upload_uuid)
-        };
-        for disk_index in 0..3 {
-            let path = staging_dir(0, disk_index);
-            tokio::fs::create_dir_all(&path).await.expect("create late part directory");
-            tokio::fs::write(path.join("part.1"), b"late part after abort")
-                .await
-                .expect("write late part fixture");
-        }
-        mark_test_pool_decommissioning(&store, 0).await;
-        assert!(
-            store
-                .list_multipart_uploads(&bucket, "", None, None, None, 100)
-                .await
-                .expect("logical upload listing")
-                .uploads
-                .is_empty()
-        );
-        assert!(store.ensure_decommission_multipart_uploads_drained_for_test(0).await.is_err());
-        assert!(
-            store
-                .abort_multipart_upload(&bucket, "different-key", &upload.upload_id, &ObjectOptions::default())
-                .await
-                .is_err(),
-            "the same UploadID for a different key must not recover this path"
-        );
-
-        // A minority metadata copy in another pool blocks recovery everywhere.
-        let conflicting_metadata = staging_dir(1, 0).join(crate::disk::STORAGE_FORMAT_FILE);
-        tokio::fs::create_dir_all(conflicting_metadata.parent().expect("metadata parent"))
-            .await
-            .expect("create conflicting pool staging");
-        tokio::fs::write(&conflicting_metadata, b"unreadable metadata")
-            .await
-            .expect("write conflicting metadata");
-        assert!(
-            store
-                .abort_multipart_upload(&bucket, object, &upload.upload_id, &ObjectOptions::default())
-                .await
-                .is_err(),
-            "no pool may be cleaned until every pool proves absence"
-        );
-        for disk_index in 0..3 {
-            assert!(staging_dir(0, disk_index).join("part.1").exists());
-        }
-        tokio::fs::remove_file(conflicting_metadata)
-            .await
-            .expect("remove conflicting fixture");
-        let (paused, release) = crate::store::multipart::pause_orphan_multipart_cleanup_for_test(&bucket);
-        let recovery_store = store.clone();
-        let recovery_bucket = bucket.clone();
-        let recovery_upload = upload.upload_id.clone();
-        let mut recovery = tokio::spawn(async move {
-            recovery_store
-                .abort_multipart_upload(&recovery_bucket, object, &recovery_upload, &ObjectOptions::default())
-                .await
-        });
-        tokio::time::timeout(Duration::from_secs(10), async {
-            tokio::select! {
-                paused = paused => paused.expect("cleanup owner announces arrival"),
-                result = &mut recovery => panic!("recovery returned before physical cleanup: {result:?}"),
-            }
-        })
-        .await
-        .expect("recovery should reach physical cleanup");
-        recovery.abort();
-        assert!(recovery.await.expect_err("request waiter is cancellable").is_cancelled());
-        let upload_lock = store.pools[0]
-            .get_disks_by_key(object)
-            .new_ns_lock(crate::disk::RUSTFS_META_MULTIPART_BUCKET, &format!("{upload_sha}/{upload_uuid}"))
-            .await
-            .expect("same upload namespace");
-        let mut upload_writer = Box::pin(upload_lock.get_write_lock(Duration::from_secs(10)));
-        let mut bucket_writer = Box::pin(store.acquire_bucket_lifecycle_write_lock(&bucket));
-        assert!(
-            futures::poll!(&mut upload_writer).is_pending(),
-            "cancelled recovery must retain the upload lock"
-        );
-        assert!(
-            futures::poll!(&mut bucket_writer).is_pending(),
-            "cancelled recovery must retain the real bucket guard"
-        );
-        release.send(()).expect("allow physical cleanup to finish");
-        let (upload_guard, bucket_guard) =
-            tokio::time::timeout(Duration::from_secs(10), async { tokio::join!(upload_writer, bucket_writer) })
-                .await
-                .expect("both guards should release after physical cleanup");
-        drop(upload_guard.expect("upload writer resumes"));
-        drop(bucket_guard.expect("bucket writer resumes"));
-        for pool_index in 0..2 {
-            for disk_index in 0..4 {
-                assert!(!staging_dir(pool_index, disk_index).exists(), "every checked staging path must be absent");
-            }
-        }
-        store
-            .ensure_decommission_multipart_uploads_drained_for_test(0)
-            .await
-            .expect("physical decommission gate should open after recovery");
-        assert!(
-            matches!(
-                store
-                    .abort_multipart_upload(&bucket, object, &upload.upload_id, &ObjectOptions::default())
-                    .await,
-                Err(StorageError::InvalidUploadID(_, _, _))
-            ),
-            "an absent upload must not claim to have recovered anything"
-        );
-        shutdown.cancel();
-    }
-
-    #[tokio::test]
-    #[serial_test::serial(storage_class_env)]
-    async fn repeated_abort_keeps_no_such_upload_while_a_disk_is_offline() {
-        use sha2::Digest;
-
-        let temp_dir = tempfile::tempdir().expect("create offline-disk recovery store dir");
-        let (_ctx, store, shutdown) =
-            without_storage_class_env(build_isolated_test_store(temp_dir.path(), "orphan-multipart-offline", &[4, 4])).await;
-        crate::bucket::metadata_sys::init_bucket_metadata_sys(store.clone(), Vec::new()).await;
-        let bucket = format!("orphan-multipart-{}", uuid::Uuid::new_v4());
-        let object = "offline-object.bin";
-        store
-            .make_bucket(&bucket, &MakeBucketOptions::default())
-            .await
-            .expect("create bucket");
-        let incarnation = store.bucket_incarnation_id(&bucket).await.expect("bucket incarnation");
-        let lifecycle_guard = store.acquire_bucket_lifecycle_read_lock(&bucket).await.expect("bucket guard");
-        let mut upload_opts = ObjectOptions {
-            expected_bucket_incarnation_id: Some(incarnation),
-            ..Default::default()
-        };
-        upload_opts.add_bucket_lifecycle_lock_guard(&lifecycle_guard);
-        let upload = store.pools[0]
-            .new_multipart_upload(&bucket, object, &upload_opts)
-            .await
-            .expect("create source upload");
-        drop(lifecycle_guard);
-        store
-            .abort_multipart_upload(&bucket, object, &upload.upload_id, &ObjectOptions::default())
-            .await
-            .expect("initial abort");
-
-        let upload_sha =
-            hex_simd::encode_to_string(sha2::Sha256::digest(format!("{bucket}/{object}").as_bytes()), hex_simd::AsciiCase::Lower);
-        let upload_uuid = crate::runtime::sources::upload_uuid_suffix(&upload.upload_id);
-        let staging_dir = |disk_index: usize| {
-            temp_dir
-                .path()
-                .join(format!("pool0/set0/disk{disk_index}"))
-                .join(crate::disk::RUSTFS_META_MULTIPART_BUCKET)
-                .join(&upload_sha)
-                .join(&upload_uuid)
-        };
-        for disk_index in 0..3 {
-            let path = staging_dir(disk_index);
-            tokio::fs::create_dir_all(&path).await.expect("create late part directory");
-            tokio::fs::write(path.join("part.1"), b"late part after abort")
-                .await
-                .expect("write late part fixture");
-        }
-
-        // One offline disk means absence cannot be proven. The idempotent Abort
-        // must keep answering NoSuchUpload instead of surfacing a disk error,
-        // and must not delete anything.
-        let set = store.pools[0].get_disks_by_key(object);
-        let offline = set.disks.write().await[3].take();
-        assert!(offline.is_some(), "fixture disk must have been online");
-        assert!(
-            matches!(
-                store
-                    .abort_multipart_upload(&bucket, object, &upload.upload_id, &ObjectOptions::default())
-                    .await,
-                Err(StorageError::InvalidUploadID(_, _, _))
-            ),
-            "unprovable absence must keep NoSuchUpload while a disk is offline"
-        );
-        for disk_index in 0..3 {
-            assert!(
-                staging_dir(disk_index).join("part.1").exists(),
-                "no staging may be deleted while a disk is offline"
-            );
-        }
-
-        set.disks.write().await[3] = offline;
-        store
-            .abort_multipart_upload(&bucket, object, &upload.upload_id, &ObjectOptions::default())
-            .await
-            .expect("recovery resumes once every disk is observable");
-        for disk_index in 0..4 {
-            assert!(!staging_dir(disk_index).exists(), "recovered staging must be absent");
-        }
-        shutdown.cancel();
-    }
-
-    #[tokio::test]
-    #[serial_test::serial(storage_class_env)]
-    async fn orphan_multipart_recovery_preserves_current_bucket_incarnation() {
-        use sha2::Digest;
-
-        let temp_dir = tempfile::tempdir().expect("create bucket lifetime recovery store dir");
-        let (_ctx, store, shutdown) =
-            without_storage_class_env(build_isolated_test_store(temp_dir.path(), "orphan-bucket-lifetime", &[4])).await;
-        crate::bucket::metadata_sys::init_bucket_metadata_sys(store.clone(), Vec::new()).await;
-        let bucket = format!("orphan-lifetime-{}", uuid::Uuid::new_v4());
-        let object = "object.bin";
-        store
-            .make_bucket(&bucket, &MakeBucketOptions::default())
-            .await
-            .expect("create original bucket");
-        let original_incarnation = store.bucket_incarnation_id(&bucket).await.expect("original incarnation");
-        let upload = store
-            .new_multipart_upload(&bucket, object, &ObjectOptions::default())
-            .await
-            .expect("create original upload");
-        store
-            .abort_multipart_upload(&bucket, object, &upload.upload_id, &ObjectOptions::default())
-            .await
-            .expect("abort original upload");
-        store
-            .delete_bucket(&bucket, &crate::storage_api_contracts::bucket::DeleteBucketOptions::default())
-            .await
-            .expect("delete original bucket");
-        store
-            .make_bucket(&bucket, &MakeBucketOptions::default())
-            .await
-            .expect("recreate bucket");
-        assert_ne!(store.bucket_incarnation_id(&bucket).await.expect("new incarnation"), original_incarnation);
-        let mut reader = PutObjReader::from_vec(b"current bucket object".to_vec());
-        store
-            .put_object(&bucket, object, &mut reader, &ObjectOptions::default())
-            .await
-            .expect("write current bucket object");
-        let current_upload = store
-            .new_multipart_upload(&bucket, object, &ObjectOptions::default())
-            .await
-            .expect("create current incarnation upload for the same key");
-
-        let upload_sha =
-            hex_simd::encode_to_string(sha2::Sha256::digest(format!("{bucket}/{object}").as_bytes()), hex_simd::AsciiCase::Lower);
-        let upload_uuid = crate::runtime::sources::upload_uuid_suffix(&upload.upload_id);
-        let leftover = temp_dir
-            .path()
-            .join("pool0/set0/disk0")
-            .join(crate::disk::RUSTFS_META_MULTIPART_BUCKET)
-            .join(upload_sha)
-            .join(upload_uuid)
-            .join("part.1");
-        tokio::fs::create_dir_all(leftover.parent().expect("staging parent"))
-            .await
-            .expect("create old incarnation staging");
-        tokio::fs::write(&leftover, b"previous bucket lifetime")
-            .await
-            .expect("write previous incarnation fixture");
-        for opts in [
-            ObjectOptions {
-                expected_bucket_incarnation_id: Some(original_incarnation),
-                ..Default::default()
-            },
-            ObjectOptions {
-                expected_bucket_incarnation_id: Some(uuid::Uuid::nil()),
-                ..Default::default()
-            },
-        ] {
-            assert!(
-                store
-                    .abort_multipart_upload(&bucket, object, &upload.upload_id, &opts)
-                    .await
-                    .is_err(),
-                "a stale or nil current-bucket identity must fail closed"
-            );
-            assert!(leftover.exists(), "failed bucket lifetime proof must preserve the path");
-        }
-        store
-            .abort_multipart_upload(&bucket, object, &upload.upload_id, &ObjectOptions::default())
-            .await
-            .expect("exact metadata-less staging from an old incarnation is unreferenced");
-        assert!(!leftover.exists());
-        store
-            .get_multipart_info(&bucket, object, &current_upload.upload_id, &ObjectOptions::default())
-            .await
-            .expect("recovery must preserve the current incarnation's active upload");
-        let mut reader = store
-            .get_object_reader(&bucket, object, None, HeaderMap::new(), &ObjectOptions::default())
-            .await
-            .expect("current object must remain readable");
-        let mut body = Vec::new();
-        reader.stream.read_to_end(&mut body).await.expect("read current object");
-        assert_eq!(body, b"current bucket object");
-        shutdown.cancel();
-    }
-
     #[tokio::test]
     #[serial_test::serial(storage_class_env)]
     async fn active_multipart_upload_routes_before_faulted_suspended_source() {
@@ -9927,18 +9567,8 @@ mod tests {
     #[cfg(feature = "test-util")]
     async fn decommission_migrates_and_verifies_registered_durable_ilm_records_scenario() {
         let temp_dir = tempfile::tempdir().expect("create temp store dir");
-        // The explicit recovery below must be the only journal consumer,
-        // including when receipt validation outlasts the background interval.
-        let mut instance_ctx = crate::runtime::instance::InstanceContext::new();
-        instance_ctx.suppress_tier_delete_journal_recovery_for_test();
-        let (ctx, store, _shutdown) = without_storage_class_env(build_isolated_test_store_with_layout(
-            temp_dir.path(),
-            "durable-ilm-decommission",
-            &[(1, 4), (1, 4)],
-            CancellationToken::new(),
-            Some(Arc::new(instance_ctx)),
-        ))
-        .await;
+        let (ctx, store, _shutdown) =
+            without_storage_class_env(build_isolated_test_store(temp_dir.path(), "durable-ilm-decommission", &[4, 4])).await;
 
         let tier_name = "DECOMMISSION-ILM";
         let backend = register_transition_reconcile_test_tier(&ctx.tier_config_mgr(), tier_name).await;
@@ -10410,16 +10040,6 @@ mod tests {
         .await
         .expect("manual task receipt should restore after the corruption check");
 
-        assert!(matches!(
-            com::read_config(store.pools[0].clone(), &tier_path).await,
-            Err(Error::ConfigNotFound)
-        ));
-        assert_eq!(
-            com::read_config(store.pools[1].clone(), &tier_path)
-                .await
-                .expect("manual recovery must still observe the exact migrated tier journal"),
-            tier_bytes
-        );
         let tier_stats = recover_tier_delete_journal_entries(store.clone(), 100, None)
             .await
             .expect("tier journal recovery should consume the migrated record before completion");
@@ -11176,19 +10796,8 @@ mod tests {
         const JOURNAL_COUNT: usize = 65;
 
         let temp_dir = tempfile::tempdir().expect("create bounded rollback store dir");
-        // The test's worker must be the only one to roll back the seeded manifest: if
-        // the setup outlasts TIER_DELETE_JOURNAL_RECOVERY_INTERVAL, a background pass
-        // rolls it back before the slow-delete hook is installed.
-        let mut instance_ctx = crate::runtime::instance::InstanceContext::new();
-        instance_ctx.suppress_tier_delete_journal_recovery_for_test();
-        let (ctx, store, _shutdown) = without_storage_class_env(build_isolated_test_store_with_layout(
-            temp_dir.path(),
-            "bounded-dispatch-rollback",
-            &[(1, 4)],
-            CancellationToken::new(),
-            Some(Arc::new(instance_ctx)),
-        ))
-        .await;
+        let (ctx, store, _shutdown) =
+            without_storage_class_env(build_isolated_test_store(temp_dir.path(), "bounded-dispatch-rollback", &[4])).await;
         crate::bucket::metadata_sys::init_bucket_metadata_sys(store.clone(), Vec::new()).await;
         let bucket = "bounded-dispatch-rollback-bucket";
         store
@@ -11344,19 +10953,8 @@ mod tests {
     #[serial_test::serial(storage_class_env)]
     async fn dispatch_manifest_page_timeout_detaches_one_deduplicated_worker_until_convergence() {
         let temp_dir = tempfile::tempdir().expect("create detached rollback store dir");
-        // The test's own passes must be the only ones to process the seeded manifest:
-        // if the setup outlasts TIER_DELETE_JOURNAL_RECOVERY_INTERVAL, a background
-        // pass rolls it back before the slow-delete hook is installed.
-        let mut instance_ctx = crate::runtime::instance::InstanceContext::new();
-        instance_ctx.suppress_tier_delete_journal_recovery_for_test();
-        let (ctx, store, _shutdown) = without_storage_class_env(build_isolated_test_store_with_layout(
-            temp_dir.path(),
-            "detached-dispatch-rollback",
-            &[(1, 4)],
-            CancellationToken::new(),
-            Some(Arc::new(instance_ctx)),
-        ))
-        .await;
+        let (ctx, store, _shutdown) =
+            without_storage_class_env(build_isolated_test_store(temp_dir.path(), "detached-dispatch-rollback", &[4])).await;
         crate::bucket::metadata_sys::init_bucket_metadata_sys(store.clone(), Vec::new()).await;
         let bucket = "detached-dispatch-rollback-bucket";
         store
@@ -11424,20 +11022,9 @@ mod tests {
         const WORKER_LIMIT: usize = 4;
 
         let temp_dir = tempfile::tempdir().expect("create bounded detached rollback store dir");
-        // The test's passes must be the only ones to process the seeded manifests: if
-        // the setup outlasts TIER_DELETE_JOURNAL_RECOVERY_INTERVAL, a background pass
-        // rolls them back before the pause hook is installed. The restarted store
-        // below keeps background recovery, which its half of the test relies on.
-        let mut instance_ctx = crate::runtime::instance::InstanceContext::new();
-        instance_ctx.suppress_tier_delete_journal_recovery_for_test();
-        let (ctx, store, shutdown) = without_storage_class_env(build_isolated_test_store_with_layout(
-            temp_dir.path(),
-            "bounded-detached-dispatch-rollback",
-            &[(1, 4)],
-            CancellationToken::new(),
-            Some(Arc::new(instance_ctx)),
-        ))
-        .await;
+        let (ctx, store, shutdown) =
+            without_storage_class_env(build_isolated_test_store(temp_dir.path(), "bounded-detached-dispatch-rollback", &[4]))
+                .await;
         crate::bucket::metadata_sys::init_bucket_metadata_sys(store.clone(), Vec::new()).await;
         let tier_name = "BOUNDED-DETACHED-DISPATCH-ROLLBACK";
         let _backend = register_mock_tier(&ctx.tier_config_mgr(), tier_name).await;
@@ -17759,19 +17346,6 @@ mod tests {
     #[tokio::test]
     #[serial_test::serial(storage_class_env)]
     async fn delete_holds_bucket_incarnation_sentinel_through_object_commit() {
-        // The recreation waits for the delete's sentinel on its lifecycle write lock
-        // from before `barrier.release()`, so that lock wait also has to cover the
-        // delete's commit. Raise the lock acquire timeout from the 5 s default to give
-        // the commit room on a starved runner; a kept sentinel still fails the test,
-        // after 30 s instead of 5 s.
-        temp_env::async_with_vars(
-            [(rustfs_config::ENV_OBJECT_LOCK_ACQUIRE_TIMEOUT, Some("30"))],
-            delete_holds_bucket_incarnation_sentinel_through_object_commit_body(),
-        )
-        .await;
-    }
-
-    async fn delete_holds_bucket_incarnation_sentinel_through_object_commit_body() {
         let temp = tempfile::tempdir().expect("create temp store dir");
         let (_ctx, store, _shutdown) =
             without_storage_class_env(build_isolated_test_store(temp.path(), "delete-bucket-incarnation", &[4])).await;
@@ -17844,19 +17418,13 @@ mod tests {
             "bucket delete and same-name recreation must wait for the in-flight object delete"
         );
         barrier.release();
-        // The recreation's lock wait started before the release; give the delete as long.
-        tokio::time::timeout(Duration::from_secs(30), delete)
+        delete
             .await
-            .expect("delete should finish once the barrier is released")
             .expect("delete task should join")
             .expect("old-generation delete should complete");
-        // If the delete kept the sentinel, `delete_bucket` would fail on its lifecycle
-        // write lock after the lock acquire timeout (30 s here). The rest of the
-        // recreation is fsync-bound IO on four disks, so this bound, above the lock
-        // timeout, only stops a hang.
-        tokio::time::timeout(Duration::from_secs(120), recreate)
+        tokio::time::timeout(Duration::from_secs(10), recreate)
             .await
-            .expect("bucket recreation should not hang after delete releases the sentinel")
+            .expect("bucket recreation should finish after delete releases the sentinel")
             .expect("bucket recreation task should join")
             .expect("bucket recreation should succeed");
 
@@ -17909,19 +17477,6 @@ mod tests {
     #[tokio::test]
     #[serial_test::serial(storage_class_env)]
     async fn delete_objects_holds_bucket_incarnation_sentinel_through_commit() {
-        // The recreation waits for the delete's sentinel on its lifecycle write lock
-        // from before `barrier.release()`, so that lock wait also has to cover the
-        // delete's commit. Raise the lock acquire timeout from the 5 s default to give
-        // the commit room on a starved runner; a kept sentinel still fails the test,
-        // after 30 s instead of 5 s.
-        temp_env::async_with_vars(
-            [(rustfs_config::ENV_OBJECT_LOCK_ACQUIRE_TIMEOUT, Some("30"))],
-            delete_objects_holds_bucket_incarnation_sentinel_through_commit_body(),
-        )
-        .await;
-    }
-
-    async fn delete_objects_holds_bucket_incarnation_sentinel_through_commit_body() {
         let temp = tempfile::tempdir().expect("create temp store dir");
         let (_ctx, store, _shutdown) =
             without_storage_class_env(build_isolated_test_store(temp.path(), "batch-delete-bucket-incarnation", &[4])).await;
@@ -17983,22 +17538,14 @@ mod tests {
             "bucket recreation must wait for the in-flight batch delete"
         );
         barrier.release();
-        // The recreation's lock wait started before the release; give the delete as long.
-        let (_deleted, errors) = tokio::time::timeout(Duration::from_secs(30), delete)
-            .await
-            .expect("batch delete should finish once the barrier is released")
-            .expect("batch delete task should join");
+        let (_deleted, errors) = delete.await.expect("batch delete task should join");
         assert!(
             errors.iter().all(Option::is_none),
             "old-generation batch delete should complete: {errors:?}"
         );
-        // If the delete kept the sentinel, `delete_bucket` would fail on its lifecycle
-        // write lock after the lock acquire timeout (30 s here). The rest of the
-        // recreation is fsync-bound IO on four disks, so this bound, above the lock
-        // timeout, only stops a hang.
-        tokio::time::timeout(Duration::from_secs(120), recreate)
+        tokio::time::timeout(Duration::from_secs(10), recreate)
             .await
-            .expect("bucket recreation should not hang after batch delete releases the sentinel")
+            .expect("bucket recreation should finish after batch delete releases the sentinel")
             .expect("bucket recreation task should join")
             .expect("bucket recreation should succeed");
         store
@@ -18924,70 +18471,6 @@ mod tests {
         assert_eq!(body, b"new-generation");
     }
 
-    #[tokio::test(flavor = "multi_thread")]
-    #[serial_test::serial]
-    #[serial_test::serial(storage_class_env)]
-    async fn cancelled_ecstore_put_part_keeps_bucket_lifecycle_lock_through_commit() {
-        for pause in [
-            crate::set_disk::MultipartCommitPause::PutPartAfterCapacityAdmission,
-            crate::set_disk::MultipartCommitPause::PutPartAfterRename,
-        ] {
-            let temp_dir = tempfile::tempdir().expect("create cancelled multipart lifecycle store dir");
-            let (_ctx, store, shutdown) =
-                without_storage_class_env(build_isolated_test_store(temp_dir.path(), "cancelled-multipart-lifecycle", &[4, 4]))
-                    .await;
-            crate::bucket::metadata_sys::init_bucket_metadata_sys(Arc::clone(&store), Vec::new()).await;
-            let bucket = format!("cancelled-part-lifecycle-{}", Uuid::new_v4());
-            let object = "part.bin";
-            store
-                .make_bucket(&bucket, &MakeBucketOptions::default())
-                .await
-                .expect("create bucket for multipart lifecycle cancellation");
-            let upload = store
-                .new_multipart_upload(&bucket, object, &ObjectOptions::default())
-                .await
-                .expect("create upload through the owning ECStore");
-            let barrier = crate::set_disk::MultipartCommitBarrier::install(&bucket, object, pause);
-            let put_store = Arc::clone(&store);
-            let put_bucket = bucket.clone();
-            let put_upload_id = upload.upload_id.clone();
-            let put = tokio::spawn(async move {
-                let mut reader = PutObjReader::from_vec(vec![0x45; 4096]);
-                put_store
-                    .put_object_part(&put_bucket, object, &put_upload_id, 1, &mut reader, &ObjectOptions::default())
-                    .await
-            });
-            barrier.wait_until_paused().await;
-            put.abort();
-            assert!(
-                put.await
-                    .expect_err("the ECStore request waiter must remain cancellable")
-                    .is_cancelled()
-            );
-
-            let mut writer = Box::pin(store.acquire_bucket_lifecycle_write_lock(&bucket));
-            assert!(
-                tokio::time::timeout(Duration::from_millis(100), &mut writer).await.is_err(),
-                "canceling UploadPart at {pause:?} must not release its bucket lifecycle owner before the commit drains"
-            );
-            barrier.release();
-            let lifecycle_guard = tokio::time::timeout(Duration::from_secs(10), writer)
-                .await
-                .expect("the bucket lifecycle writer must resume after the part commit drains")
-                .expect("acquire the bucket lifecycle write lock after cancellation");
-            drop(lifecycle_guard);
-
-            let parts = store
-                .list_object_parts(&bucket, object, &upload.upload_id, None, 100, &ObjectOptions::default())
-                .await
-                .expect("the cancelled request's owned part commit must finish successfully");
-            assert_eq!(parts.parts.len(), 1, "the commit must publish exactly the uploaded part at {pause:?}");
-            assert_eq!(parts.parts[0].part_num, 1);
-            assert_eq!(parts.parts[0].size, 4096);
-            shutdown.cancel();
-        }
-    }
-
     #[tokio::test]
     #[serial_test::serial(storage_class_env)]
     async fn pre_upgrade_multipart_upload_can_complete_and_abort_in_the_same_bucket_lifetime() {
@@ -22540,6 +22023,7 @@ mod tests {
             .make_bucket(bucket, &MakeBucketOptions::default())
             .await
             .expect("kill-point bucket should be created");
+        let _compaction_proof = install_transition_transaction_compaction_fleet_proof_for_test("object-transaction-fencing-test");
 
         for (index, point, expected_state, expected_revision, expected_committed, expected_recovered) in [
             (
@@ -22583,14 +22067,10 @@ mod tests {
                 .await
                 .expect("kill-point source should be written");
             let remote_before = backend.object_count().await;
-            // Earlier cancellation/recovery cases must not consume this
-            // transaction's 30-second homogeneous-fleet proof window.
-            let _compaction_proof =
-                install_transition_transaction_compaction_fleet_proof_for_test("object-transaction-fencing-test");
             let barrier = TransitionTransactionKillPointBarrier::install(bucket, &object, point);
             let transition_store = store.clone();
             let transition_object = object.clone();
-            let mut transition = tokio::spawn(async move {
+            let transition = tokio::spawn(async move {
                 transition_store
                     .transition_object(
                         bucket,
@@ -22609,12 +22089,7 @@ mod tests {
                     )
                     .await
             });
-            tokio::select! {
-                _ = barrier.wait_until_paused() => {}
-                result = &mut transition => {
-                    panic!("transition exited before reaching {point:?}: {result:?}");
-                }
-            }
+            barrier.wait_until_paused().await;
             transition.abort();
             assert!(
                 transition

crates/ecstore/src/store/init_format.rs

@@ -134,18 +134,6 @@ pub(crate) async fn connect_load_init_formats_with_instance_ctx(
     {
         return Err(Error::CorruptedFormat);
     }
-    if matches!(format_quorum, Some(Err(Error::ErasureReadQuorum)))
-        && let Some((format, members)) = select_existing_format_read_quorum(&formats, &errs, set_drive_count)
-    {
-        retain_format_quorum_members(instance_ctx, disks, &format, &members, set_drive_count).await?;
-        // Loading an existing format grants no authority to initialize pool
-        // metadata or migrate formats. The persisted storage classes and each
-        // object's EC metadata still determine service and operation quorums.
-        return Ok(LoadedFormat {
-            format,
-            pool_meta_bootstrap_authority: PoolMetaBootstrapAuthority::None,
-        });
-    }
     let resumable_partial_migration = formats_present > 0
         && errs.iter().any(|error| matches!(error, Some(DiskError::UnformattedDisk)))
         && format_quorum.as_ref().is_some_and(Result::is_err)
@@ -590,43 +578,6 @@ pub fn get_format_erasure_in_quorum(formats: &[Option<FormatV3>], slot_offset: u
     select_format_erasure_in_quorum(formats, slot_offset).map(|(format, _)| format)
 }
 
-/// Load a consistent existing layout with enough members per set to read the
-/// max-parity system metadata containing the persisted storage classes. This
-/// is not object-read admission: STANDARD, RRS and historical objects can have
-/// different EC layouts, which are validated after this bootstrap read.
-/// Format migration and healing still use strict-majority election.
-fn select_existing_format_read_quorum(
-    formats: &[Option<FormatV3>],
-    errors: &[Option<DiskError>],
-    set_drive_count: usize,
-) -> Option<(FormatV3, Vec<bool>)> {
-    if set_drive_count == 0 || formats.is_empty() || formats.len() != errors.len() {
-        return None;
-    }
-    let metadata_parity = set_drive_count / 2;
-    let metadata_read_quorum = set_drive_count - metadata_parity;
-    let reference = formats.iter().flatten().next()?;
-    if !formats_match_reference_slots(formats, reference, 0)
-        || check_format_erasure_value_for_topology(reference, formats.len(), set_drive_count).is_err()
-        || formats.iter().zip(errors).any(|(format, error)| match (format, error) {
-            (Some(_), None) => false,
-            (None, Some(DiskError::DiskNotFound)) => false,
-            (None, Some(error)) => !is_network_like_disk_error(error),
-            _ => true,
-        })
-        || formats
-            .chunks(set_drive_count)
-            .any(|set| set.iter().flatten().count() < metadata_read_quorum)
-    {
-        return None;
-    }
-
-    let mut format = reference.clone();
-    format.erasure.this = Uuid::nil();
-    format.disk_info = None;
-    Some((format, formats.iter().map(Option::is_some).collect()))
-}
-
 pub(crate) fn select_format_erasure_in_quorum(formats: &[Option<FormatV3>], slot_offset: usize) -> Result<(FormatV3, Vec<bool>)> {
     let mut candidates = HashMap::new();
     let formats_present = formats.iter().flatten().count();
@@ -1169,190 +1120,6 @@ mod tests {
         assert!(!formats_match_reference_slots(&formats, &reference, 0));
     }
 
-    fn half_pool_formats() -> (Vec<Option<FormatV3>>, Vec<Option<DiskError>>) {
-        let reference = FormatV3::new(6, 4);
-        let formats = (0..24)
-            .map(|slot| {
-                (slot % 4 < 2).then(|| {
-                    let mut format = reference.clone();
-                    format.erasure.this = reference.erasure.sets[slot / 4][slot % 4];
-                    format
-                })
-            })
-            .collect::<Vec<_>>();
-        let errors = formats
-            .iter()
-            .map(|format| format.is_none().then_some(DiskError::DiskNotFound))
-            .collect();
-        (formats, errors)
-    }
-
-    #[test]
-    fn existing_format_read_quorum_follows_each_sets_metadata_layout() {
-        // These are system-metadata EC boundaries, independent of user-object
-        // storage classes and of how many drives are hosted by each node.
-        for (set_width, read_quorum) in [(1, 1), (2, 1), (3, 2), (4, 2), (6, 3), (8, 4), (12, 6), (16, 8)] {
-            for set_count in [1, 2, 5] {
-                let reference = FormatV3::new(set_count, set_width);
-                let mut formats = (0..set_count * set_width)
-                    .map(|slot| {
-                        let set = slot / set_width;
-                        let drive = slot % set_width;
-                        // Rotate surviving slots between sets so admission
-                        // cannot depend on the first drive or node being up.
-                        ((drive + set) % set_width < read_quorum).then(|| {
-                            let mut format = reference.clone();
-                            format.erasure.this = reference.erasure.sets[set][drive];
-                            format
-                        })
-                    })
-                    .collect::<Vec<_>>();
-                let mut errors = formats
-                    .iter()
-                    .map(|format| format.is_none().then_some(DiskError::DiskNotFound))
-                    .collect::<Vec<_>>();
-                let (loaded, members) = select_existing_format_read_quorum(&formats, &errors, set_width)
-                    .unwrap_or_else(|| panic!("metadata read quorum must load {set_count} sets of {set_width} drives"));
-                assert_eq!(loaded.shared_identity(), reference.shared_identity());
-                assert!(loaded.erasure.this.is_nil());
-                assert_eq!(members.iter().filter(|member| **member).count(), set_count * read_quorum);
-
-                // More members in other sets cannot compensate for one set
-                // falling below its own metadata reconstruction boundary.
-                for slot in 0..(set_count - 1) * set_width {
-                    let mut format = reference.clone();
-                    format.erasure.this = reference.erasure.sets[slot / set_width][slot % set_width];
-                    formats[slot] = Some(format);
-                    errors[slot] = None;
-                }
-                let lost_slot = formats
-                    .iter()
-                    .rposition(Option::is_some)
-                    .expect("at least one surviving drive");
-                formats[lost_slot] = None;
-                errors[lost_slot] = Some(DiskError::DiskNotFound);
-                assert!(
-                    select_existing_format_read_quorum(&formats, &errors, set_width).is_none(),
-                    "one set below read quorum must block {set_count} sets of {set_width} drives"
-                );
-            }
-        }
-    }
-
-    #[test]
-    fn existing_format_read_quorum_preserves_election_and_set_boundaries() {
-        let (mut formats, mut errors) = half_pool_formats();
-        assert!(select_existing_format_read_quorum(&[], &[], 0).is_none());
-        assert!(select_existing_format_read_quorum(&formats, &errors[..23], 4).is_none());
-        assert!(select_existing_format_read_quorum(&formats, &errors, 0).is_none());
-        assert!(select_existing_format_read_quorum(&formats, &errors, 6).is_none());
-        assert!(select_existing_format_read_quorum(&formats, &errors, 4).is_some());
-        assert!(matches!(get_format_erasure_in_quorum(&formats, 0), Err(Error::ErasureReadQuorum)));
-
-        let mut extra = formats[0].as_ref().expect("seed format").clone();
-        extra.erasure.this = extra.erasure.sets[0][2];
-        formats[2] = Some(extra);
-        errors[2] = None;
-        assert!(get_format_erasure_in_quorum(&formats, 0).is_ok(), "13 votes must establish a majority");
-        formats[5] = None;
-        errors[5] = Some(DiskError::DiskNotFound);
-        assert!(
-            select_existing_format_read_quorum(&formats, &errors, 4).is_none(),
-            "12 votes cannot compensate for a set with only one readable drive"
-        );
-        formats[2] = None;
-        errors[2] = Some(DiskError::DiskNotFound);
-        assert!(select_existing_format_read_quorum(&formats, &errors, 4).is_none(), "11 votes must wait");
-    }
-
-    #[test]
-    fn existing_format_read_quorum_rejects_conflicts_invalid_slots_and_blank_disks() {
-        for failure in [
-            "conflict",
-            "wrong_slot",
-            "nil",
-            "duplicate",
-            "blank",
-            "corrupt",
-            "access_denied",
-            "tie",
-        ] {
-            let (mut formats, mut errors) = half_pool_formats();
-            match failure {
-                "conflict" => formats[1].as_mut().expect("second format").id = Uuid::new_v4(),
-                "wrong_slot" => {
-                    let first_id = formats[0].as_ref().expect("first format").erasure.this;
-                    formats[1].as_mut().expect("second format").erasure.this = first_id;
-                }
-                "nil" => {
-                    for format in formats.iter_mut().flatten() {
-                        format.id = Uuid::nil();
-                    }
-                }
-                "duplicate" => {
-                    for format in formats.iter_mut().flatten() {
-                        format.erasure.sets[5][3] = format.erasure.sets[0][0];
-                    }
-                }
-                "blank" => errors[2] = Some(DiskError::UnformattedDisk),
-                "corrupt" => errors[2] = Some(DiskError::CorruptedFormat),
-                "access_denied" => errors[2] = Some(DiskError::FileAccessDenied),
-                "tie" => {
-                    let other = FormatV3::new(6, 4);
-                    for slot in 0..24 {
-                        if formats[slot].is_none() {
-                            let mut format = other.clone();
-                            format.erasure.this = other.erasure.sets[slot / 4][slot % 4];
-                            formats[slot] = Some(format);
-                            errors[slot] = None;
-                        }
-                    }
-                }
-                _ => unreachable!(),
-            }
-            assert!(
-                select_existing_format_read_quorum(&formats, &errors, 4).is_none(),
-                "{failure} must not authorize degraded startup"
-            );
-        }
-    }
-
-    #[tokio::test]
-    async fn existing_format_load_accepts_each_half_of_four_by_six_without_format_writes() {
-        let (_root, all_disks) = local_disks_with_set_width(24, 4).await;
-        let reference = FormatV3::new(6, 4);
-        for (slot, disk) in all_disks.iter().enumerate() {
-            let mut format = reference.clone();
-            format.erasure.this = reference.erasure.sets[slot / 4][slot % 4];
-            save_format_file(disk, &Some(format)).await.expect("persist existing format");
-        }
-        let before = format_bytes(&all_disks).await;
-        for first in 0..4 {
-            for second in first + 1..4 {
-                let mut disks = all_disks.clone();
-                for (slot, disk) in disks.iter_mut().enumerate() {
-                    if slot % 4 != first && slot % 4 != second {
-                        *disk = None;
-                    }
-                }
-                let loaded = connect_load_init_formats_with_instance_ctx(
-                    &Arc::new(InstanceContext::new()),
-                    first == 0,
-                    &mut disks,
-                    6,
-                    4,
-                    None,
-                )
-                .await
-                .expect("two nodes must load the existing format");
-                assert_eq!(loaded.format.shared_identity(), reference.shared_identity());
-                assert!(matches!(loaded.pool_meta_bootstrap_authority, PoolMetaBootstrapAuthority::None));
-                assert_eq!(disks.iter().flatten().count(), 12);
-                assert_eq!(format_bytes(&all_disks).await, before, "read startup must never rewrite formats");
-            }
-        }
-    }
-
     #[tokio::test]
     async fn existing_format_load_succeeds_with_a_strict_majority() {
         let (_temp_dir, mut disks) = two_local_disks_with_missing_third().await;

crates/ecstore/src/store/list_objects.rs

@@ -56,7 +56,7 @@ use rustfs_filemeta::{
 };
 use rustfs_io_metrics::{
     LIST_OBJECTS_GATHER_OUTCOME_INPUT_CLOSED, LIST_OBJECTS_GATHER_OUTCOME_LIMIT_REACHED, LIST_OBJECTS_SOURCE_WALKER,
-    ListObjectsGatherObservation, ListObjectsIndexPageObservation,
+    ListObjectsGatherCaller, ListObjectsGatherObservation, ListObjectsIndexPageObservation,
 };
 use rustfs_utils::path::{self, SLASH_SEPARATOR, base_dir_from_prefix};
 use std::cmp::Reverse;
@@ -109,6 +109,7 @@ fn recursive_walk_list_path_options(bucket: String, prefix: &str, opts: &WalkOpt
         incl_deleted: true,
         recursive: true,
         versioned: true,
+        caller: ListObjectsGatherCaller::S3ListVersions,
         walkdir_timeout: opts.walkdir_timeout,
         walkdir_stall_timeout: opts.walkdir_stall_timeout,
         ..Default::default()
@@ -274,6 +275,7 @@ fn log_list_path_finished(
 
 #[derive(Debug, Default, Clone)]
 pub struct ListPathOptions {
+    pub caller: ListObjectsGatherCaller,
     pub id: Option<String>,
 
     // Bucket of the listing.
@@ -4116,6 +4118,7 @@ impl ECStore {
         let effective_max_keys = if max_keys <= 0 { 0 } else { max_keys_plus_one(max_keys, true) };
         let mut opts = ListPathOptions {
             bucket: bucket.to_owned(),
+            caller: ListObjectsGatherCaller::S3List,
             prefix: prefix.to_owned(),
             separator: delimiter.clone(),
             // Always request max_keys + 1 to detect if there are more results
@@ -4289,6 +4292,11 @@ impl ECStore {
         let effective_max_keys = list_versions_scan_limit(max_keys, has_version_marker);
         let mut opts = ListPathOptions {
             bucket: bucket.to_owned(),
+            caller: if include_version_purge {
+                ListObjectsGatherCaller::Lifecycle
+            } else {
+                ListObjectsGatherCaller::S3ListVersions
+            },
             prefix: prefix.to_owned(),
             separator: delimiter.clone(),
             limit: effective_max_keys,
@@ -4993,11 +5001,45 @@ where
     let _cancel_guard = cancel.clone().drop_guard();
     let (result_tx, mut result_rx) = mpsc::channel(1);
     let mut workers = JoinSet::new();
-    workers.spawn(async move { producer.await.map(|()| None) }.instrument(tracing::Span::current()));
-    let gather_cancel = cancel.clone();
-    workers.spawn(
-        async move { gather_results(gather_cancel, opts, recv, result_tx).await.map(Some) }.instrument(tracing::Span::current()),
+    #[cfg(feature = "diagnostic-read-version-caller")]
+    let mut diagnostic_context = crate::cluster::rpc::runtime_sources::ReadVersionDiagnosticContext::capture_entry(
+        rustfs_io_metrics::internode_metrics::ReadVersionEntrySource::ListObjects,
     );
+    #[cfg(feature = "diagnostic-read-version-caller")]
+    let request_caller = diagnostic_context.caller.or_else(|| {
+        use rustfs_io_metrics::ListObjectsGatherCaller;
+        use rustfs_io_metrics::internode_metrics::ReadVersionRequestCaller;
+
+        match opts.caller {
+            ListObjectsGatherCaller::S3List | ListObjectsGatherCaller::S3ListVersions => {
+                Some(ReadVersionRequestCaller::ListObjects)
+            }
+            ListObjectsGatherCaller::Lifecycle => Some(ReadVersionRequestCaller::Lifecycle),
+            ListObjectsGatherCaller::Other => Some(ReadVersionRequestCaller::Other),
+            ListObjectsGatherCaller::Unattributed => None,
+        }
+    });
+    #[cfg(feature = "diagnostic-read-version-caller")]
+    {
+        diagnostic_context.caller = request_caller;
+    }
+    let producer = async move { producer.await.map(|()| None) };
+    #[cfg(feature = "diagnostic-read-version-caller")]
+    let producer = crate::cluster::rpc::runtime_sources::with_read_version_diagnostic_context(
+        diagnostic_context,
+        rustfs_io_metrics::internode_metrics::ReadVersionContextBoundary::ListProducer,
+        producer,
+    );
+    workers.spawn(producer.instrument(tracing::Span::current()));
+    let gather_cancel = cancel.clone();
+    let gather = async move { gather_results(gather_cancel, opts, recv, result_tx).await.map(Some) };
+    #[cfg(feature = "diagnostic-read-version-caller")]
+    let gather = crate::cluster::rpc::runtime_sources::with_read_version_diagnostic_context(
+        diagnostic_context,
+        rustfs_io_metrics::internode_metrics::ReadVersionContextBoundary::ListGather,
+        gather,
+    );
+    workers.spawn(gather.instrument(tracing::Span::current()));
 
     let mut failure = None;
     let mut stopping = false;
@@ -5133,6 +5175,7 @@ async fn gather_results(
                 let duration_ms = started.elapsed().as_secs_f64() * 1000.0;
                 rustfs_io_metrics::record_list_objects_gather(ListObjectsGatherObservation {
                     source: LIST_OBJECTS_SOURCE_WALKER,
+                    caller: opts.caller,
                     outcome: LIST_OBJECTS_GATHER_OUTCOME_LIMIT_REACHED,
                     limit: opts.limit,
                     scanned_entries,
@@ -5195,6 +5238,7 @@ async fn gather_results(
         let duration_ms = started.elapsed().as_secs_f64() * 1000.0;
         rustfs_io_metrics::record_list_objects_gather(ListObjectsGatherObservation {
             source: LIST_OBJECTS_SOURCE_WALKER,
+            caller: opts.caller,
             outcome: LIST_OBJECTS_GATHER_OUTCOME_INPUT_CLOSED,
             limit: opts.limit,
             scanned_entries,
@@ -9221,7 +9265,7 @@ mod test {
         };
         let snapshot = PersistentListMetadataObject::from_object_info(&object);
         assert!(snapshot.to_object_info("bucket").etag.is_none());
-        let mut old_snapshot = snapshot;
+        let mut old_snapshot = snapshot.clone();
         old_snapshot.etag = object.etag.clone();
         let old_contents = format!(
             "# rustfs-listobjects-key-only-v2\n{}\n",
@@ -12294,6 +12338,44 @@ mod test {
         );
     }
 
+    #[cfg(feature = "diagnostic-read-version-caller")]
+    #[tokio::test]
+    async fn list_path_producer_worker_preserves_s3_and_caller_context() {
+        let mut request = rustfs_io_metrics::s3_http_metrics::S3HttpRequestGuard::new("GET");
+        let api_operation = request
+            .in_scope(|| {
+                rustfs_io_metrics::record_s3_op(rustfs_s3_ops::S3Operation::ListObjectsV2);
+                rustfs_io_metrics::s3_http_metrics::current_s3_http_operation_context()
+            })
+            .expect("dispatched list operation should be captured");
+        request.response(200);
+
+        let (sender, recv) = mpsc::channel(1);
+        let opts = ListPathOptions {
+            caller: rustfs_io_metrics::ListObjectsGatherCaller::S3List,
+            ..Default::default()
+        };
+        let result = rustfs_io_metrics::s3_http_metrics::with_s3_http_operation_context(Some(api_operation), async move {
+            collect_list_path_results(CancellationToken::new(), opts, recv, async move {
+                assert_eq!(
+                    rustfs_io_metrics::s3_http_metrics::current_s3_http_operation_name(),
+                    Some("s3:ListObjectsV2")
+                );
+                assert_eq!(
+                    crate::cluster::rpc::runtime_sources::current_read_version_request_caller(),
+                    Some(rustfs_io_metrics::internode_metrics::ReadVersionRequestCaller::ListObjects)
+                );
+                drop(sender);
+                Ok(())
+            })
+            .await
+        })
+        .await;
+
+        assert!(result.err.is_some(), "the empty page channel remains an error");
+        assert_eq!(rustfs_io_metrics::s3_http_metrics::current_s3_http_operation_name(), None);
+    }
+
     #[tokio::test]
     async fn list_path_workers_preserve_error_after_input_closes() {
         let (sender, recv) = mpsc::channel(1);

crates/ecstore/src/store/mod.rs

@@ -1363,7 +1363,13 @@ impl crate::storage_api_contracts::object::ObjectIO for ECStore {
         h: HeaderMap,
         opts: &ObjectOptions,
     ) -> Result<GetObjectReader> {
-        self.handle_get_object_reader(bucket, object, range, h, opts).await
+        let read = self.handle_get_object_reader(bucket, object, range, h, opts);
+        #[cfg(feature = "diagnostic-read-version-caller")]
+        let read = crate::cluster::rpc::runtime_sources::with_read_version_entry(
+            rustfs_io_metrics::internode_metrics::ReadVersionEntrySource::StoreObjectReader,
+            read,
+        );
+        read.await
     }
     #[instrument(level = "debug", skip(self, data))]
     async fn put_object(&self, bucket: &str, object: &str, data: &mut PutObjReader, opts: &ObjectOptions) -> Result<ObjectInfo> {

crates/ecstore/src/store/multipart.rs

@@ -32,42 +32,6 @@ fn data_movement_multipart_discovery_counts() -> &'static std::sync::Mutex<std::
     DATA_MOVEMENT_MULTIPART_DISCOVERY_COUNTS.get_or_init(|| std::sync::Mutex::new(std::collections::HashMap::new()))
 }
 
-#[cfg(test)]
-type OrphanMultipartCleanupPause = (tokio::sync::oneshot::Sender<()>, tokio::sync::oneshot::Receiver<()>);
-
-#[cfg(test)]
-static ORPHAN_MULTIPART_CLEANUP_PAUSES: std::sync::OnceLock<
-    std::sync::Mutex<std::collections::HashMap<String, OrphanMultipartCleanupPause>>,
-> = std::sync::OnceLock::new();
-
-#[cfg(test)]
-pub(crate) fn pause_orphan_multipart_cleanup_for_test(
-    bucket: &str,
-) -> (tokio::sync::oneshot::Receiver<()>, tokio::sync::oneshot::Sender<()>) {
-    let (arrived_tx, arrived_rx) = tokio::sync::oneshot::channel();
-    let (release_tx, release_rx) = tokio::sync::oneshot::channel();
-    let previous = ORPHAN_MULTIPART_CLEANUP_PAUSES
-        .get_or_init(Default::default)
-        .lock()
-        .expect("orphan cleanup pause mutex should not poison")
-        .insert(bucket.to_owned(), (arrived_tx, release_rx));
-    assert!(previous.is_none(), "each bucket has at most one orphan cleanup pause");
-    (arrived_rx, release_tx)
-}
-
-#[cfg(test)]
-async fn pause_orphan_multipart_cleanup(bucket: &str) {
-    let pause = ORPHAN_MULTIPART_CLEANUP_PAUSES
-        .get_or_init(Default::default)
-        .lock()
-        .expect("orphan cleanup pause mutex should not poison")
-        .remove(bucket);
-    if let Some((arrived, release)) = pause {
-        let _ = arrived.send(());
-        let _ = release.await;
-    }
-}
-
 fn decommission_multipart_target_clear_pending(opts: &ObjectOptions, target: Option<&ObjectInfo>) -> Result<bool> {
     let expected_mod_time = opts.mod_time.ok_or_else(|| Error::DecommissionCapacityBlocked {
         message: "multipart cleanup cannot prove exact target absence without a modification time".to_string(),
@@ -690,10 +654,7 @@ impl ECStore {
         opts: &ObjectOptions,
     ) -> Result<PartInfo> {
         check_put_object_part_args(bucket, object, upload_id)?;
-        let (mut opts, bucket_lifecycle_guard) = self.guard_multipart_bucket_incarnation(bucket, opts).await?;
-        if let Some(guard) = bucket_lifecycle_guard {
-            opts.bucket_lifecycle_lock_owner = Some(Arc::new(guard));
-        }
+        let (mut opts, _bucket_lifecycle_guard) = self.guard_multipart_bucket_incarnation(bucket, opts).await?;
         if !self.single_pool() {
             opts.decommission_capacity_admission = crate::bucket::metadata_sys::object_store_if_initialized_in(&self.ctx).await;
         }
@@ -789,12 +750,19 @@ impl ECStore {
         opts: &ObjectOptions,
     ) -> Result<()> {
         check_abort_multipart_args(bucket, object, upload_id)?;
-        let (opts, bucket_lifecycle_guard) = self.guard_multipart_bucket_incarnation(bucket, opts).await?;
+        let (opts, _bucket_lifecycle_guard) = self.guard_multipart_bucket_incarnation(bucket, opts).await?;
+        let opts = &opts;
+
+        // TODO(backlog): defer DeleteUploadID to background for faster abort response
+
+        if self.single_pool() {
+            return self.pools[0].abort_multipart_upload(bucket, object, upload_id, opts).await;
+        }
 
         for pool_idx in self.existing_multipart_pool_order().await {
             let pool = &self.pools[pool_idx];
 
-            let err = match pool.abort_multipart_upload(bucket, object, upload_id, &opts).await {
+            let err = match pool.abort_multipart_upload(bucket, object, upload_id, opts).await {
                 Ok(_) => return Ok(()),
                 Err(err) => {
                     //
@@ -807,92 +775,7 @@ impl ECStore {
             }
         }
 
-        self.recover_orphan_multipart_upload(bucket, object, upload_id, opts, bucket_lifecycle_guard)
-            .await
-    }
-
-    async fn recover_orphan_multipart_upload(
-        &self,
-        bucket: &str,
-        object: &str,
-        upload_id: &str,
-        opts: ObjectOptions,
-        bucket_lifecycle_guard: Option<rustfs_lock::NamespaceLockGuard>,
-    ) -> Result<()> {
-        let invalid_upload = || StorageError::InvalidUploadID(bucket.to_owned(), object.to_owned(), upload_id.to_owned());
-        if self.ctx.lock_manager().is_disabled() {
-            return Err(invalid_upload());
-        }
-        let Some(bucket_lifecycle_guard) = bucket_lifecycle_guard else {
-            // A signal borrowed from an enclosing operation cannot keep its lock
-            // alive when the request waiter disappears during physical cleanup.
-            return Err(invalid_upload());
-        };
-        let Some(expected_incarnation) = opts.expected_bucket_incarnation_id.filter(|id| !id.is_nil()) else {
-            return Err(invalid_upload());
-        };
-        // Recovery is a side path of an idempotent Abort. Until the first
-        // deletion is issued, anything that stops staging absence from being
-        // proven (an offline disk, lock contention, partially overlapping lock
-        // domains, unreadable bucket metadata) keeps the upload reported as
-        // gone, exactly as before recovery existed; only failures inside the
-        // deletion task are request failures, because a retry can finish them.
-        let (metadata, persisted) =
-            match crate::bucket::metadata_sys::get_config_from_disk_with_presence_in(&self.ctx, bucket).await {
-                Ok(loaded) => loaded,
-                Err(error) => {
-                    debug!(bucket, object, upload_id, %error, "orphan multipart cleanup skipped: bucket metadata unavailable");
-                    return Err(invalid_upload());
-                }
-            };
-        if !persisted
-            || !metadata.bucket_creation_committed
-            || !metadata.bucket_incarnation_sidecar
-            || metadata.bucket_incarnation_id != expected_incarnation
-        {
-            return Err(invalid_upload());
-        }
-
-        // Lock order: bucket lifecycle, then upload locks in pool-index order.
-        // Prove metadata absence across every pool before deleting any staging.
-        let mut cleanup_guards = Vec::with_capacity(self.pools.len());
-        for pool in &self.pools {
-            let guard = match pool
-                .get_disks_by_key(object)
-                .lock_orphan_multipart_cleanup(bucket, object, upload_id, &opts, &cleanup_guards)
-                .await
-            {
-                Ok(guard) => guard,
-                Err(error) => {
-                    if !is_err_invalid_upload_id(&error) {
-                        debug!(bucket, object, upload_id, %error, "orphan multipart cleanup skipped: staging absence not proven");
-                    }
-                    return Err(invalid_upload());
-                }
-            };
-            cleanup_guards.push(guard);
-        }
-        let bucket = bucket.to_owned();
-        let object = object.to_owned();
-        let upload_id = upload_id.to_owned();
-        tokio::spawn(async move {
-            // Own both the real bucket guard and all upload guards until every
-            // disk deletion settles, including after cancellation of the waiter.
-            let _bucket_lifecycle_guard = bucket_lifecycle_guard;
-            #[cfg(test)]
-            pause_orphan_multipart_cleanup(&bucket).await;
-            let mut recovered = false;
-            for guard in &cleanup_guards {
-                recovered |= guard.delete(&bucket, &object, &opts).await?;
-            }
-            if recovered {
-                Ok(())
-            } else {
-                Err(StorageError::InvalidUploadID(bucket, object, upload_id))
-            }
-        })
-        .await
-        .map_err(|error| Error::other_with_context("orphan multipart cleanup task failed", error))?
+        Err(StorageError::InvalidUploadID(bucket.to_owned(), object.to_owned(), upload_id.to_owned()))
     }
 
     pub(crate) async fn abort_multipart_upload_for_data_movement(

crates/ecstore/src/store/object.rs

@@ -4849,9 +4849,7 @@ impl ECStore {
         opts: ObjectOptions,
         guards: impl IntoIterator<Item = rustfs_lock::NamespaceLockGuard>,
     ) -> Result<ObjectInfo> {
-        let guards = Arc::new(guards.into_iter().collect::<Vec<_>>());
-        let mut opts = opts;
-        opts.delete_mutation_owner = Some(Arc::new((opts.delete_mutation_owner.take(), guards.clone())));
+        let guards: Vec<_> = guards.into_iter().collect();
         let store = Arc::clone(self);
         let bucket = bucket.to_owned();
         let object = object.to_owned();
@@ -4863,7 +4861,7 @@ impl ECStore {
                 if result.is_ok() {
                     list_objects::observe_list_objects_mutation(&store, &bucket).await;
                 }
-                // Transient options also retain these guards through late disk I/O.
+                // Keep: the only use of `guards` here, it moves them into the task so the locks outlive a dropped caller.
                 drop(guards);
                 result
             })
@@ -4932,7 +4930,7 @@ impl ECStore {
             let mut opts = opts;
             let receipt_sink = install_tier_free_version_receipt_sink(&mut opts);
             let result = self
-                .handle_delete_object_with_journal_inner_boxed(bucket, object, opts, tier_journal_api)
+                .handle_delete_object_with_journal_inner(bucket, object, opts, tier_journal_api)
                 .await;
             enqueue_recorded_tier_free_versions(self, receipt_sink).await;
             result
@@ -4940,16 +4938,6 @@ impl ECStore {
         .await
     }
 
-    fn handle_delete_object_with_journal_inner_boxed<'a>(
-        &'a self,
-        bucket: &'a str,
-        object: &'a str,
-        opts: ObjectOptions,
-        tier_journal_api: Option<Arc<ECStore>>,
-    ) -> impl std::future::Future<Output = Result<ObjectInfo>> + Send + 'a {
-        Box::pin(self.handle_delete_object_with_journal_inner(bucket, object, opts, tier_journal_api))
-    }
-
     async fn handle_delete_object_with_journal_inner(
         &self,
         bucket: &str,
@@ -4967,9 +4955,9 @@ impl ECStore {
             if is_meta_bucketname(bucket) || (opts.delete_prefix && opts.bucket_lifecycle_lock_fence.is_some()) {
                 None
             } else if opts.delete_prefix {
-                Some(Arc::new(self.acquire_bucket_lifecycle_write_lock(bucket).await?))
+                Some(self.acquire_bucket_lifecycle_write_lock(bucket).await?)
             } else {
-                Some(Arc::new(self.acquire_bucket_lifecycle_read_lock(bucket).await?))
+                Some(self.acquire_bucket_lifecycle_read_lock(bucket).await?)
             };
         let object = if opts.delete_prefix && !opts.delete_prefix_object {
             object.to_owned()
@@ -4987,7 +4975,6 @@ impl ECStore {
         opts.tier_delete_journal_api = tier_journal_api.clone();
         if let Some(guard) = _bucket_lifecycle_guard.as_ref() {
             opts.add_bucket_lifecycle_lock_guard(guard);
-            opts.delete_mutation_owner = Some(Arc::new((opts.delete_mutation_owner.take(), guard.clone())));
         }
 
         if !is_meta_bucketname(bucket) {
@@ -5240,40 +5227,6 @@ impl ECStore {
                 return Ok(obj);
             }
             Err(err) => {
-                if (is_err_object_not_found(&err) || is_err_version_not_found(&err))
-                    && (opts.version_id.is_some() || (!opts.versioned && !opts.version_suspended))
-                    && opts.http_preconditions.is_none()
-                    && !opts.replication_request
-                {
-                    let mut reclaimed = None;
-                    for pool in &self.pools {
-                        if self.is_suspended(pool.pool_idx).await || self.is_pool_rebalancing(pool.pool_idx).await {
-                            return Err(StorageError::SlowDown);
-                        }
-                        let pool_idx = pool.pool_idx;
-                        let pool = pool.clone();
-                        match self
-                            .run_external_decommission_capacity_object_delete(
-                                pool_idx,
-                                bucket,
-                                object,
-                                object,
-                                opts.clone(),
-                                |opts| async move { pool.delete_object(bucket, object, opts).await },
-                            )
-                            .await
-                        {
-                            Ok(result) => reclaimed = Some(result),
-                            Err(missing) if is_err_object_not_found(&missing) || is_err_version_not_found(&missing) => {}
-                            Err(error) => return Err(error),
-                        }
-                    }
-                    if let Some(mut result) = reclaimed {
-                        result.bucket = bucket.to_owned();
-                        result.name = decode_dir_object(object);
-                        return Ok(result);
-                    }
-                }
                 // A folder key (`prefix/`) with no object metadata may still exist on
                 // disk as an orphan empty-directory tree (issue #4189): listings show
                 // it as a common prefix, but no regular delete path can remove it.
@@ -5469,13 +5422,12 @@ impl ECStore {
             None
         } else {
             match self.acquire_bucket_lifecycle_read_lock(bucket).await {
-                Ok(guard) => Some(Arc::new(guard)),
+                Ok(guard) => Some(guard),
                 Err(err) => return return_batch_delete_lock_error_with_accounting(objects.as_slice(), err),
             }
         };
         if let Some(guard) = _bucket_lifecycle_guard.as_ref() {
             opts.add_bucket_lifecycle_lock_guard(guard);
-            opts.delete_mutation_owner = Some(Arc::new((opts.delete_mutation_owner.take(), guard.clone())));
         }
         if opts.delete_replication_config_snapshot.is_none() {
             match ReplicationObjectBridge::delete_request_config_in(&self.ctx, bucket).await {
@@ -5541,11 +5493,10 @@ impl ECStore {
         if current_bucket_incarnation_id.is_some() {
             pause_delete_after_object_lock_snapshot(bucket).await;
         }
-        let _object_lock_guards = Arc::new(match self.acquire_delete_objects_write_locks(bucket, &objects, &mut opts).await {
+        let _object_lock_guards = match self.acquire_delete_objects_write_locks(bucket, &objects, &mut opts).await {
             Ok(guards) => guards,
             Err(err) => return return_batch_delete_lock_error_with_accounting(objects.as_slice(), err),
-        });
-        opts.delete_mutation_owner = Some(Arc::new((opts.delete_mutation_owner.take(), _object_lock_guards.clone())));
+        };
         #[cfg(any(test, feature = "test-util"))]
         if !_object_lock_guards.is_empty() {
             notify_delete_namespace_acquired(bucket);
@@ -9290,31 +9241,8 @@ mod tests {
                 .expect("read original marker");
             assert_eq!(info.delete_marker_incarnation(), Some(old_incarnation));
         }
-        // Save each stale member's inline versions before converging the old
-        // bucket's delete receipts, then reintroduce them after recreation.
-        let mut stale_metadata = Vec::new();
-        for disk in before.iter().skip(12).flatten() {
-            for version in &historical {
-                let info = disk
-                    .read_version(
-                        "",
-                        bucket,
-                        "history.bin",
-                        &version.to_string(),
-                        &crate::disk::ReadOptions {
-                            read_data: true,
-                            ..Default::default()
-                        },
-                    )
-                    .await
-                    .expect("read stale inline data version");
-                assert!(info.inline_data() && info.data.is_some(), "fixture payload must be preserved in xl.meta");
-            }
-            stale_metadata.push((
-                disk.read_all(bucket, "history.bin/xl.meta").await.unwrap(),
-                disk.read_all(bucket, &format!("{object}/xl.meta")).await.unwrap(),
-            ));
-        }
+        // Taking four actual members offline preserves their existing xl.meta.
+        // All deletion/recreation work below goes through the normal store API.
         for slot in 12..16 {
             set.disks.write().await[slot] = None;
         }
@@ -9344,29 +9272,6 @@ mod tests {
             )
             .await
             .expect("delete exact old marker with four drives offline");
-        let pending = store
-            .handle_delete_bucket(bucket, &crate::storage_api_contracts::bucket::DeleteBucketOptions::default())
-            .await
-            .expect_err("offline participants require retained delete receipts");
-        assert!(matches!(
-            pending,
-            StorageError::BucketNotEmpty(_) | StorageError::BucketNotEmptyWithDetails { .. }
-        ));
-        assert!(
-            !crate::bucket::retirement::is_retired(store.clone(), bucket, old_incarnation)
-                .await
-                .unwrap()
-        );
-        *set.disks.write().await = before.clone();
-        store
-            .recover_delete_transactions_at_incarnation(
-                bucket,
-                None,
-                Some(old_incarnation),
-                &rustfs_heal_contracts::heal_channel::HealOpts::default(),
-            )
-            .await
-            .expect("converge old bucket deletes on all participants");
         store
             .handle_delete_bucket(bucket, &crate::storage_api_contracts::bucket::DeleteBucketOptions::default())
             .await
@@ -9376,9 +9281,6 @@ mod tests {
                 .await
                 .expect("durable retirement")
         );
-        for slot in 12..16 {
-            set.disks.write().await[slot] = None;
-        }
         store
             .handle_make_bucket(bucket, &create)
             .await
@@ -9404,13 +9306,6 @@ mod tests {
                 .expect("write new generation");
             new_versions.push((key, info.version_id.unwrap(), payload));
         }
-        for (slot, (history, marker)) in (12..16).zip(stale_metadata) {
-            for (key, bytes) in [("history.bin", history), (object, marker)] {
-                let root = dirs[slot].path().join(bucket).join(key);
-                tokio::fs::create_dir_all(&root).await.unwrap();
-                tokio::fs::write(root.join("xl.meta"), bytes).await.unwrap();
-            }
-        }
         *set.disks.write().await = before.clone();
         let heal = rustfs_heal_contracts::heal_channel::HealOpts {
             remove: true,
@@ -11582,160 +11477,3 @@ mod tests {
         .await;
     }
 }
-
-#[cfg(test)]
-mod missing_key_delete_quorum_tests {
-    use super::*;
-    use crate::bucket::metadata_sys::{init_bucket_metadata_sys, test_support::isolated_store_over_temp_disks};
-    use crate::storage_api_contracts::bucket::{BucketOperations, BucketOptions, MakeBucketOptions};
-
-    async fn degraded_store(bucket: &str) -> (Vec<tempfile::TempDir>, Arc<ECStore>) {
-        let (dirs, store) = isolated_store_over_temp_disks().await;
-        init_bucket_metadata_sys(store.clone(), Vec::new()).await;
-        store
-            .make_bucket(bucket, &MakeBucketOptions::default())
-            .await
-            .expect("create authoritative never-versioned bucket");
-        store.pools[0].disk_set[0].disks.write().await[0] = None;
-        let info = store
-            .get_bucket_info(bucket, &BucketOptions::default())
-            .await
-            .expect("existing bucket must remain readable at quorum");
-        assert_eq!(info.name, bucket);
-        let lookup = store
-            .get_object_info_for_delete(bucket, "never-existed", &ObjectOptions::default())
-            .await;
-        assert!(
-            lookup.as_ref().is_err_and(is_err_object_not_found),
-            "remaining disks must establish absence through production lookup: {lookup:?}"
-        );
-        (dirs, store)
-    }
-
-    #[tokio::test]
-    async fn single_delete_missing_key_preserves_quorum_noop() {
-        let (_dirs, store) = degraded_store("missing-key-quorum-single").await;
-        let result = store
-            .delete_object_with_tier_delete_journal("missing-key-quorum-single", "never-existed", ObjectOptions::default())
-            .await;
-        assert!(
-            result.as_ref().is_err_and(is_err_object_not_found),
-            "single missing-key delete must preserve its idempotent not-found outcome at quorum: {result:?}"
-        );
-    }
-
-    #[tokio::test]
-    async fn batch_delete_missing_key_preserves_quorum_noop() {
-        let (_dirs, store) = degraded_store("missing-key-quorum-batch").await;
-        let (_, errors, _) = store
-            .delete_objects_with_tier_delete_journal_and_accounting(
-                "missing-key-quorum-batch",
-                vec![ObjectToDelete {
-                    object_name: "never-existed".to_owned(),
-                    ..Default::default()
-                }],
-                ObjectOptions::default(),
-            )
-            .await;
-        assert_eq!(errors.len(), 1, "batch DELETE must return one result per key");
-        assert!(
-            errors.iter().all(Option::is_none),
-            "batch missing-key delete must preserve its idempotent success at quorum: {errors:?}"
-        );
-    }
-
-    #[cfg(not(windows))]
-    #[tokio::test]
-    async fn batch_delete_timeout_retains_physical_key_guard() {
-        use crate::disk::os::prepared_publication_test_hooks as hooks;
-        use crate::disk::{DiskAPI, ReadOptions};
-        use crate::storage_api_contracts::object::ObjectIO;
-        temp_env::async_with_vars([("RUSTFS_DELETE_RECOVERY_TIMEOUT_SECS", Some("30"))], async {
-            let (_dirs, store) = isolated_store_over_temp_disks().await;
-            init_bucket_metadata_sys(store.clone(), Vec::new()).await;
-            let bucket = "review-batch-late-owner";
-            let key = "object";
-            store
-                .make_bucket(bucket, &MakeBucketOptions::default())
-                .await
-                .expect("fixture bucket");
-            let mut reader = PutObjReader::from_vec(vec![0x51; 1024 * 1024]);
-            store
-                .put_object(
-                    bucket,
-                    key,
-                    &mut reader,
-                    &ObjectOptions {
-                        write_completion: crate::object_api::WriteCompletion::TailDrained,
-                        ..Default::default()
-                    },
-                )
-                .await
-                .expect("seed real external object");
-            let disks = store.pools[0].disk_set[0].disk_inventory().await;
-            let disk = disks[0].as_ref().expect("online drive");
-            let info = disk
-                .read_version("", bucket, key, "", &ReadOptions::default())
-                .await
-                .expect("seed metadata");
-            assert!(!info.inline_data());
-            let crate::disk::Disk::Local(local) = disk.as_ref() else {
-                panic!("local disk");
-            };
-            let part = local
-                .get_disk()
-                .get_object_path_for_io(bucket, key)
-                .expect("leased key")
-                .join(info.data_dir.expect("external data directory").to_string())
-                .join("part.1");
-            let (entered_tx, entered_rx) = tokio::sync::oneshot::channel();
-            let (release, release_rx) = std::sync::mpsc::channel::<()>();
-            let hook = hooks::install_at(hooks::Stage::Remove, &part, move || {
-                let _ = entered_tx.send(());
-                let _ = release_rx.recv();
-            });
-            let deleting = store.clone();
-            let delete = tokio::spawn(async move {
-                deleting
-                    .delete_objects_with_tier_delete_journal_and_accounting(
-                        bucket,
-                        vec![ObjectToDelete {
-                            object_name: key.to_owned(),
-                            ..Default::default()
-                        }],
-                        ObjectOptions::default(),
-                    )
-                    .await
-            });
-            tokio::time::timeout(Duration::from_secs(10), entered_rx)
-                .await
-                .expect("physical cleanup entry")
-                .expect("entry hook");
-            assert!(part.exists(), "payload unlink has not run");
-            let (_, errors, _) = tokio::time::timeout(Duration::from_secs(40), delete)
-                .await
-                .expect("real settlement timeout must return")
-                .expect("delete task");
-            assert!(errors.iter().all(Option::is_none), "certified batch response: {errors:?}");
-            assert!(
-                store.ctx.namespace_commits_pending(),
-                "background physical operation must retain its owner"
-            );
-            let ns = store.handle_new_ns_lock(bucket, key).await.expect("same production key lock");
-            let probe = ns.get_write_lock(Duration::from_millis(300)).await;
-            let acquired_while_pending = probe.is_ok();
-            drop(probe);
-            drop(release);
-            drop(hook);
-            assert!(
-                !acquired_while_pending,
-                "batch released the exact-key guard while physical cleanup was still paused after its real recovery timeout"
-            );
-            let _released = ns
-                .get_write_lock(Duration::from_secs(10))
-                .await
-                .expect("physical cleanup must eventually release the exact-key guard");
-        })
-        .await;
-    }
-}

crates/ecstore/src/store/peer.rs

@@ -1697,9 +1697,7 @@ mod tests {
     #[serial_test::serial]
     async fn target_ordinary_timeout_keeps_its_physical_namespace_owner() {
         use crate::disk::os::prepared_publication_test_hooks as hooks;
-        // Allow real fixture I/O to finish under load. Expire the operation
-        // with virtual time only after its physical publication is paused.
-        temp_env::async_with_vars([(rustfs_config::ENV_DRIVE_MAX_TIMEOUT_DURATION, Some("60"))], async {
+        temp_env::async_with_vars([(rustfs_config::ENV_DRIVE_MAX_TIMEOUT_DURATION, Some("1"))], async {
             let ctx = Arc::new(InstanceContext::new());
             let store = super::super::tests::build_store_with_ctx(ctx.clone());
             let root = tempfile::tempdir().expect("root");
@@ -1733,7 +1731,7 @@ mod tests {
             .await
             .expect("bounded entry");
             tokio::time::pause();
-            tokio::time::advance(std::time::Duration::from_secs(61)).await;
+            tokio::time::advance(std::time::Duration::from_secs(2)).await;
             tokio::time::resume();
             let result = tokio::time::timeout(std::time::Duration::from_secs(5), &mut rename)
                 .await

crates/heal/src/error.rs

@@ -90,9 +90,6 @@ pub enum Error {
         source: Box<Error>,
     },
 
-    #[error("Bucket discovery incomplete for the requested heal scope; not all buckets could be traversed")]
-    BucketDiscoveryIncomplete,
-
     #[error("Invalid heal type: {heal_type}")]
     InvalidHealType { heal_type: String },
 
@@ -130,7 +127,7 @@ impl Error {
         match self {
             Error::TaskCancelled | Error::TaskTimeout | Error::StaleBucketIncarnation { .. } => false,
             Error::ReplacementTargetNotReady(_) | Error::DanglingDeleteDeferred { .. } => true,
-            Error::TransientSkip { .. } | Error::BucketDiscoveryIncomplete => true,
+            Error::TransientSkip { .. } => true,
             // Lock failures classify by LockError's own taxonomy: only the
             // fatal variants (ResourceNotFound / PermissionDenied /
             // Configuration) are terminal - retrying cannot fix them - while

crates/heal/src/heal/channel.rs

@@ -853,13 +853,6 @@ mod tests {
         async fn get_bucket_info(&self, _bucket: &str) -> crate::Result<Option<crate::heal::storage_api::status::BucketInfo>> {
             Ok(None)
         }
-        async fn list_buckets_for_admin_heal(
-            &self,
-            _selector: Option<crate::heal::storage_api::storage::DiskSetSelector>,
-        ) -> crate::Result<Vec<crate::heal::storage_api::status::BucketInfo>> {
-            self.list_buckets().await
-        }
-
         async fn list_buckets(&self) -> crate::Result<Vec<crate::heal::storage_api::status::BucketInfo>> {
             Ok(vec![])
         }

crates/heal/src/heal/erasure_healer.rs

@@ -1400,15 +1400,6 @@ impl ErasureSetHealer {
                 None => self.storage.heal_bucket(bucket, &bucket_opts).await?,
             };
             task.record_result_item(result).await;
-            // Keep recovery in the checkpoint's admitted incarnation and repair
-            // the publication bucket before adopting a survivor's delete proof.
-            if let Some(result) = self
-                .storage
-                .recover_delete_transactions(bucket, None, incarnation, &self.heal_opts)
-                .await?
-            {
-                task.record_result_item(result).await;
-            }
         }
 
         // 1. get bucket info
@@ -2399,8 +2390,6 @@ mod resume_loop_tests {
     #[derive(Default)]
     struct FakeStorage {
         admin_disk: Mutex<Option<DiskStore>>,
-        bucket_discovery_incomplete: AtomicBool,
-        bucket_discovery_scopes: Mutex<Vec<Option<crate::heal::storage_api::storage::DiskSetSelector>>>,
         bucket_pages: Mutex<HashMap<String, Page>>,
         incarnations: Mutex<HashMap<String, uuid::Uuid>>,
         receipts: Mutex<HashMap<String, Option<crate::heal::outcome::HealObjectDisposition>>>,
@@ -2480,17 +2469,6 @@ mod resume_loop_tests {
                 ..Default::default()
             }))
         }
-        async fn list_buckets_for_admin_heal(
-            &self,
-            selector: Option<crate::heal::storage_api::storage::DiskSetSelector>,
-        ) -> Result<Vec<BucketInfo>> {
-            self.bucket_discovery_scopes.lock().expect("discovery scopes").push(selector);
-            if self.bucket_discovery_incomplete.load(Ordering::SeqCst) {
-                return Err(Error::BucketDiscoveryIncomplete);
-            }
-            self.list_buckets().await
-        }
-
         async fn list_buckets(&self) -> Result<Vec<BucketInfo>> {
             Ok(self
                 .bucket_pages

crates/heal/src/heal/erasure_healer/admin.rs

@@ -51,10 +51,9 @@ impl ErasureSetHealer {
             }
             let resolved;
             let buckets = if buckets.is_empty() {
-                let (pool_idx, set_idx) = crate::heal::utils::parse_set_disk_id(set_disk_id)?;
                 resolved = self
                     .storage
-                    .list_buckets_for_admin_heal(Some(crate::heal::storage_api::storage::DiskSetSelector::new(pool_idx, set_idx)))
+                    .list_buckets()
                     .await?
                     .into_iter()
                     .map(|bucket| bucket.name)

crates/heal/src/heal/erasure_healer/resume_loop_tests/admin_outcome.rs

@@ -483,50 +483,3 @@ async fn admin_erasure_checkpoint_publication_cannot_undo_acknowledged_cancel()
     assert_eq!(outcome.coverage, HealTraversalCoverage::Partial);
     assert_eq!((outcome.counters.processed, outcome.counters.healed), (1, 1));
 }
-
-#[tokio::test]
-async fn admin_bucket_discovery_rejects_incomplete_checkpoint_and_freezes_complete_empty_scope() {
-    let env = make_env().await;
-    env.storage.bucket_discovery_incomplete.store(true, Ordering::SeqCst);
-    let result = env
-        .healer
-        .prepare_admin_checkpoint(&env.checkpoint, &[], "pool_0_set_0")
-        .await;
-    assert!(matches!(result, Err(Error::BucketDiscoveryIncomplete)));
-    assert!(env.checkpoint.get_checkpoint().await.admin.is_none());
-    let reloaded = CheckpointManager::load_from_disk(env.healer.disk.clone(), &env.task_id)
-        .await
-        .expect("reload checkpoint");
-    assert!(reloaded.get_checkpoint().await.admin.is_none(), "incomplete scope must not be persisted");
-    assert_eq!(
-        env.storage.bucket_discovery_scopes.lock().expect("scopes").as_slice(),
-        &[Some(crate::heal::storage_api::storage::DiskSetSelector::new(0, 0))]
-    );
-
-    env.storage.bucket_discovery_incomplete.store(false, Ordering::SeqCst);
-    assert!(
-        env.healer
-            .prepare_admin_checkpoint(&env.checkpoint, &[], "pool_0_set_0")
-            .await
-            .expect("complete empty scope")
-            .is_empty()
-    );
-    env.storage.bucket_discovery_incomplete.store(true, Ordering::SeqCst);
-    objects(&env.storage, "created-after-admission", &["object"]);
-    let reloaded = CheckpointManager::load_from_disk(env.healer.disk.clone(), &env.task_id)
-        .await
-        .expect("reload frozen scope");
-    assert!(
-        env.healer
-            .prepare_admin_checkpoint(&reloaded, &[], "pool_0_set_0")
-            .await
-            .expect("resume frozen scope")
-            .is_empty()
-    );
-    assert_eq!(
-        env.storage.bucket_discovery_scopes.lock().expect("scopes").len(),
-        2,
-        "resume must not rediscover or expand scope"
-    );
-    assert!(env.storage.calls().is_empty());
-}