test(e2e): retain startup and shutdown failure diagnostics

* fix(usage): reconcile stale counters after lifecycle expiration

* fix(usage): account lifecycle expiry during continuous writes

* test(usage): run lifecycle usage scenarios on one scanner store

* test(usage): use a Windows-representable pre-mutation offset

* fix(usage): harden expiry accounting recovery and quota checks

Borrow expiry receipt bucket names and avoid allocating a map key on cache hits. Cover cancelled receipts, durable snapshot recovery, and legacy quota admission after scanner confirmation. Use representable timestamp offsets in the quota regression.

Refs rustfs/backlog#2689

Co-Authored-By: heihutu <heihutu@gmail.com>
Co-Authored-By: zhi22915 <qiuzgang@gmail.com>

* test(usage): recover stale persisted counts through the scanner

Seed incorrect complete usage for empty and retained-object buckets, then run the real scanner and publication consumer without further object mutations. Verify durable and admin usage over two cycles instead of writing a corrected snapshot in the test.

Refs rustfs/backlog#2689
Refs rustfs/backlog#2691

Co-Authored-By: heihutu <heihutu@gmail.com>
Co-Authored-By: zhi22915 <qiuzgang@gmail.com>

* test(e2e): bound delimiter pagination fixture concurrency

The 120-second smoke timeout expired after 1018 of 1200 serial fixture PUTs, before LIST ran. Prepare the same objects with at most eight concurrent requests and await every PUT. Retain the timeout and strengthen exact prefix, KeyCount, empty Contents, and continuation-token assertions with phase diagnostics.

Refs rustfs/backlog#2689

Co-Authored-By: heihutu <heihutu@gmail.com>
Co-Authored-By: zhi22915 <qiuzgang@gmail.com>

* test(upgrade): establish a persisted previous-release baseline

Seed the pinned previous-release cluster and restart it once with its data intact before replacing any node. Require every old writer to pass the strict readiness probe and preserve the seed through both mixed phases and the final current cluster. Keep InternalError fail-fast behavior and all existing compatibility deadlines and assertions.

Refs rustfs/backlog#2689
Refs rustfs/backlog#2384

Co-Authored-By: heihutu <heihutu@gmail.com>
Co-Authored-By: zhi22915 <qiuzgang@gmail.com>

* test(ecstore): bound cancellation metadata persistence waits

Use the system-bucket incarnation boundary now supplied by main PR #8268. Bound the three cancellation waits that previously hung during pool.bin persistence, retaining their remote-generation, target-cohort, and durable-state assertions.

Refs rustfs/backlog#2697
Refs rustfs/backlog#2689

Co-Authored-By: heihutu <heihutu@gmail.com>
Co-Authored-By: zhi22915 <qiuzgang@gmail.com>

---------

Co-authored-by: Chris <anzhengchao@gmail.com>
Co-authored-by: Hauser <housemecn@gmail.com>
Co-authored-by: heihutu <heihutu@gmail.com>
Co-authored-by: zhi22915 <qiuzgang@gmail.com>

.config/e2e-distributed-selection.txt

@@ -1,2 +1,2 @@
-sha256-linux=21ab66fd172018682ce825a53f26aa8f286d57f3f687d8616d1850e56474dcd8
-sha256-darwin=21ab66fd172018682ce825a53f26aa8f286d57f3f687d8616d1850e56474dcd8
+sha256-linux=6fc377fa1f9f06e065f077c7bd53d04efbccf4453f95dde67afaa47778185377
+sha256-darwin=6fc377fa1f9f06e065f077c7bd53d04efbccf4453f95dde67afaa47778185377

.config/e2e-full-selection.txt

@@ -1,2 +1,2 @@
-sha256-darwin=9e9687ded961aa4e619a15def6e1ee0a5153be05423d009ee7a58126624c83e7
-sha256-linux=364666693c2bf258997643c9d87e07e7b7501e58175f190d201d53c81946eb20
+sha256-darwin=14c8cd76b8cbf643329c371daa70a5edd4357d42259a2fd9c653e548675d4a6d
+sha256-linux=80d0fae627007c31f250fb1fb21c8a952861195d53d7f60b6e71636d5ca65b46

.config/e2e-nightly-selection.txt

@@ -1,2 +1,2 @@
-sha256-darwin=83a7dcaffd5a789517ae9f02a224f66a9713937885cff96fca2ad7e216f197ae
-sha256-linux=626c10f8c964507ff987b6c86069e9019dc6d2ae7fb02db9be5df5aa8cc5145b
+sha256-darwin=efe68675f4676a0d2901242c5815b1f7af5cdec8231f7cccd408f6dacf5c75db
+sha256-linux=003ed6480935a0a95680cc061be143722fbd37bd477b8ff98607b1a064f3aef1

.config/e2e-smoke-selection.txt

@@ -1 +1 @@
-sha256=5de6852b2e606eca4ee0d86dbb3b725a3eb8923143fd00df96c549acc36040bd
+sha256=476cb92aa9e385c9bb509ee5df3e5bd07e8c929730040b60d16791b6c0616d00

.config/make/tests.mak

@@ -52,6 +52,7 @@ script-tests: ## Run shell script tests
 	./scripts/check_embedded_secrets.sh --self-test
 	$(RUSTFS_PYTHON_BIN) ./scripts/check_test_wiring.py --self-test
 	$(RUSTFS_PYTHON_BIN) ./scripts/test_e2e_binary.py
+	$(RUSTFS_PYTHON_BIN) ./scripts/test_migration_gate_evidence.py
 	$(RUSTFS_PYTHON_BIN) ./scripts/ci_gate.py --self-test
 	$(RUSTFS_PYTHON_BIN) ./scripts/check_security_coverage.py --self-test
 	$(RUSTFS_PYTHON_BIN) ./scripts/check_scheduled_validation_freshness.py --self-test
@@ -61,6 +62,7 @@ script-tests: ## Run shell script tests
 	$(RUSTFS_PYTHON_BIN) ./scripts/test_functional_chain_health.py
 	$(RUSTFS_PYTHON_BIN) ./scripts/test_ci_timing_report.py
 	$(RUSTFS_PYTHON_BIN) ./scripts/s3-tests/test_report_compat.py
+	$(RUSTFS_PYTHON_BIN) ./scripts/s3-tests/test_runner_tools.py
 	bash -n ./scripts/validate_object_data_cache_cold_stampede.sh
 	$(RUSTFS_PYTHON_BIN) ./scripts/check_object_data_cache_follower_samples.py --self-test
 	./scripts/validate_object_data_cache_cold_stampede.sh --self-test

.config/nextest.toml

@@ -107,6 +107,16 @@ test-group = 'ecstore-serial-flaky'
 filter = 'package(rustfs-ecstore) & test(/^set_disk::ops::heal::heal_result_report_tests::/)'
 test-group = 'ecstore-serial-flaky'
 
+# Serialize the replacement bucket metadata heal test. It builds a real-disk
+# hermetic erasure set, writes bucket metadata, then asserts byte-identical
+# xl.meta readback after a rejected delayed repair. Under parallel load,
+# concurrent ecstore tests mutate the process-wide disk registry and alter
+# the serialized timestamp/disk-identity fields, flipping the assertion.
+# Preventive serialization only, no retries.
+[[profile.default.overrides]]
+filter = 'package(rustfs-ecstore) & test(replacement_bucket_metadata_respects_pool_ownership_and_required_records)'
+test-group = 'ecstore-serial-flaky'
+
 # Serialize the metadata-cache generation-retirement pair. Both carry
 # #[serial(metadata_cache_invalidation_probe)] — a no-op across nextest's
 # process boundary — and assert get_object_metadata_cache generation
@@ -132,11 +142,37 @@ test-group = 'ecstore-serial-flaky'
 filter = 'package(rustfs) & (binary(/^embedded.*_test$/) | binary(admin_diagnostic_capability_e2e))'
 test-group = 'embedded-test-ports'
 
-# Inventory delivery and real drive/object probes include durable filesystem IO
-# in short deadlines. Reserve capacity so unrelated storage fixtures cannot
-# exhaust those budgets. Keep the deadlines and assertions unchanged.
+# Registration, inventory delivery, and real drive/object probes include
+# durable filesystem IO in short deadlines. Reserve capacity so sibling test
+# processes cannot exhaust those budgets. Keep deadlines and assertions intact.
 [[profile.default.overrides]]
-filter = 'package(rustfs) & (binary(connect_inventory) | binary(connect_perf_drive) | (binary(connect_perf_object) & test(=real_rustfs_endpoint_and_production_cli_support_bounded_get_and_put)))'
+filter = 'package(rustfs) & (binary(/^connect_registration$/) | binary(connect_inventory) | binary(connect_perf_drive) | (binary(connect_perf_object) & test(=real_rustfs_endpoint_and_production_cli_support_bounded_get_and_put)))'
+threads-required = "num-test-threads"
+
+# This cancellation probe hashes the serving executable before acquiring its
+# collector lease. Keep competing test processes out of its 30s startup budget.
+[[profile.default.overrides]]
+filter = 'package(rustfs) & test(=connect::diagnostics::trace_runtime::tests::local_runtime_profile_cancellation_waits_for_lease_release)'
+threads-required = "num-test-threads"
+
+# These real IAM/state fixtures keep production lock-acquisition limits while
+# peers complete durable IO. Reserve capacity from unrelated test processes;
+# the shared fixture's durable_iam_state_ prefix keeps the whole family together.
+[[profile.default.overrides]]
+filter = 'package(rustfs) & test(/^admin::handlers::site_replication::tests::durable_iam_state_/)'
+threads-required = "num-test-threads"
+
+# Real target-repair and rebalance-stop fixtures commit durable metadata while
+# peers wait on production locks or bounded cancellation checks. Reserve IO
+# capacity for each fixture; its internal races and deadlines remain intact.
+[[profile.default.overrides]]
+filter = 'package(rustfs) & test(/^admin::handlers::(replication::target_repair_tests::|rebalance::rebalance_handler_tests::real_admin_stop_)/)'
+threads-required = "num-test-threads"
+
+# Reserve capacity for these short heartbeat TLS/fsync probes (rustfs#8129).
+# Their existing deadlines, assertions, and retry policy remain in force.
+[[profile.default.overrides]]
+filter = 'package(rustfs) & binary(connect_heartbeat) & test(/^(restart_replays_pending_request_then_advances_sequence|restart_replays_exact_legacy_capabilities_with_and_without_jobs)$/)'
 threads-required = "num-test-threads"
 
 # Serialize the durable manual-transition checkpoint test across nextest's
@@ -219,6 +255,13 @@ test-group = 'e2e-cluster-nightly'
 filter = 'package(e2e_test) & (test(/^kms::kms_vault_test::/) | test(/^kms::kms_rekey_sweep_test::/) | test(/^kms::configured_roundtrip_test::test_configured_vault_kms_admin_and_versioned_cleanup$/))'
 test-group = 'e2e-vault'
 
+# These server-heavy smoke cases perform 1k+ sequential S3 writes or run the
+# scanner and a remote tier server together. Reserve the full test budget so
+# unrelated E2E processes cannot starve their storage and polling deadlines.
+[[profile.default.overrides]]
+filter = 'package(e2e_test) & (test(/^list_objects_v2_pagination_test::tests::(test_list_objects_v2_delimiter_small_page_traverses_all|test_list_objects_v2_max_keys_above_limit_returns_token|test_list_objects_v2_maxkeys_above_limit_with_delimiter)$/) | test(=reliant::tiering::test_hermetic_transition_restore_failure_expiry_and_retry))'
+threads-required = "num-test-threads"
+
 # This four-disk, 65-member rollback probe already drives up to 32 concurrent
 # durable deletions. Reserve this nextest run's capacity for its progress oracle.
 [[profile.default.overrides]]
@@ -283,13 +326,6 @@ retries = 2
 filter = 'package(rustfs-ecstore) & (test(concurrent_resend_same_part_commits_one_generation) | test(concurrent_config_writes_from_separate_nodes_do_not_lose_writes))'
 test-group = 'ecstore-serial-flaky'
 
-# QUARANTINE: OPEN rustfs#4690 — walk_dir stall-budget accounting test depends
-# on producer/consumer timing windows that stretch past the budget on loaded
-# CI runners (regression test for rustfs#4644; failed on a zero-Rust-diff PR).
-[[profile.ci.overrides]]
-filter = 'package(rustfs-ecstore) & test(walk_dir_does_not_charge_consumer_backpressure_to_the_stall_budget)'
-retries = 2
-
 # Serialize the relocated-pool GET resume regression under the ci profile too
 # (see the matching default-profile override near the top). No longer a
 # quarantine: the fixture race (rustfs#6701/rustfs#6703) was fixed by #6707,
@@ -321,6 +357,11 @@ test-group = 'ecstore-serial-flaky'
 filter = 'package(rustfs-ecstore) & test(/^set_disk::ops::heal::heal_result_report_tests::/)'
 test-group = 'ecstore-serial-flaky'
 
+# Serialize the replacement bucket metadata heal test under the ci profile too.
+[[profile.ci.overrides]]
+filter = 'package(rustfs-ecstore) & test(replacement_bucket_metadata_respects_pool_ownership_and_required_records)'
+test-group = 'ecstore-serial-flaky'
+
 # Serialize the metadata-cache generation-retirement pair under the ci
 # profile too (see the matching default-profile override near the top). Not a
 # quarantine: no retries.
@@ -335,7 +376,30 @@ filter = 'package(rustfs) & (binary(/^embedded.*_test$/) | binary(admin_diagnost
 test-group = 'embedded-test-ports'
 
 [[profile.ci.overrides]]
-filter = 'package(rustfs) & (binary(connect_inventory) | binary(connect_perf_drive) | (binary(connect_perf_object) & test(=real_rustfs_endpoint_and_production_cli_support_bounded_get_and_put)))'
+filter = 'package(rustfs) & (binary(/^connect_registration$/) | binary(connect_inventory) | binary(connect_perf_drive) | (binary(connect_perf_object) & test(=real_rustfs_endpoint_and_production_cli_support_bounded_get_and_put)))'
+threads-required = "num-test-threads"
+
+# Preserve the cancellation probe's startup budget and assertions in CI too.
+[[profile.ci.overrides]]
+filter = 'package(rustfs) & test(=connect::diagnostics::trace_runtime::tests::local_runtime_profile_cancellation_waits_for_lease_release)'
+threads-required = "num-test-threads"
+
+# Keep the same IAM/state fixture capacity reservation in CI without changing the
+# production lock deadline, internal concurrency, assertions, or retry policy.
+[[profile.ci.overrides]]
+filter = 'package(rustfs) & test(/^admin::handlers::site_replication::tests::durable_iam_state_/)'
+threads-required = "num-test-threads"
+
+# Match the real admin fixture capacity reservation without retries or changes
+# to production lock deadlines, cancellation barriers, or assertions.
+[[profile.ci.overrides]]
+filter = 'package(rustfs) & test(/^admin::handlers::(replication::target_repair_tests::|rebalance::rebalance_handler_tests::real_admin_stop_)/)'
+threads-required = "num-test-threads"
+
+# Reserve capacity for these short heartbeat TLS/fsync probes (rustfs#8129).
+# Their existing deadlines, assertions, and retry policy remain in force.
+[[profile.ci.overrides]]
+filter = 'package(rustfs) & binary(connect_heartbeat) & test(/^(restart_replays_pending_request_then_advances_sequence|restart_replays_exact_legacy_capabilities_with_and_without_jobs)$/)'
 threads-required = "num-test-threads"
 
 # Serialize the durable manual-transition checkpoint test under the ci profile
@@ -469,6 +533,7 @@ default-filter = """
         | test(/^replication_extension_test::(test_replication_check_succeeds_with_remote_target|test_replication_check_rejects_target_without_object_lock|test_set_remote_target_rejects_unversioned_source_bucket|test_replication_check_rejects_unversioned_source_bucket|test_replication_check_rejects_missing_replication_config|test_replication_check_rejects_invalid_bucket|test_set_remote_target_rejects_same_bucket_on_same_deployment|test_set_remote_target_rejects_unversioned_target_bucket|test_set_remote_target_update_requires_arn|test_set_remote_target_update_rejects_missing_target|test_set_remote_target_rejects_invalid_target_url|test_set_remote_target_rejects_self_signed_https_target_without_skip_tls_verify|test_set_remote_target_rejects_private_ca_https_target_without_ca_cert_pem|test_list_remote_targets_rejects_empty_bucket|test_list_remote_targets_rejects_invalid_bucket|test_remove_remote_target_rejects_missing_target|test_remove_remote_target_rejects_missing_arn|test_remove_remote_target_rejects_invalid_bucket|test_remove_remote_target_rejects_target_used_by_replication|test_delete_bucket_replication_removes_remote_target)$/)
         | test(/^reliant::lifecycle::/)
         | test(/^reliant::tiering::/)
+        | test(/^upgrade_compatibility_test::upgrade_write_readiness_tests::/)
         | test(/^on_demand_migration::(get_basic_test::(get_miss_pulls_inline_and_serves_locally_afterwards|head_miss_answers_from_the_source_without_persisting)|interaction_test::test_odm_admin_config_is_redacted_and_status_counts_match_the_source)$/)
     )
 """
@@ -477,13 +542,15 @@ fail-fast = false
 [profile.e2e-smoke.junit]
 path = "junit.xml"
 
-# The pagination boundary cases can stall when a server/listing regression
-# prevents the continuation request from completing. Keep the timeout scoped
-# to those known failure modes so legitimate lifecycle/tiering waits retain
-# their test-level timing budget.
+# These storage-heavy fixtures reserve capacity. Pagination tests bound listing
+# requests separately, after their seed objects have been written.
 [[profile.e2e-smoke.overrides]]
 filter = 'package(e2e_test) & test(/^list_objects_v2_pagination_test::tests::(test_list_objects_v2_delimiter_small_page_traverses_all|test_list_objects_v2_max_keys_above_limit_returns_token|test_list_objects_v2_maxkeys_above_limit_with_delimiter)$/)'
-slow-timeout = { period = "60s", terminate-after = 2, grace-period = "10s" }
+threads-required = "num-test-threads"
+
+[[profile.e2e-smoke.overrides]]
+filter = 'package(e2e_test) & test(=reliant::tiering::test_hermetic_transition_restore_failure_expiry_and_retry)'
+threads-required = "num-test-threads"
 
 # ---------------------------------------------------------------------------
 # e2e-repl-nightly profile — scheduled full replication e2e lane (repl-1)
@@ -693,6 +760,7 @@ default-filter = """
     & !test(/^replication_extension_test::/)
     & !test(/^replication_target_matrix_test::/)
     & !test(/^on_demand_migration::(concurrency_test|fault_test|interop_test|real_source_test)::/)
+    & !test(/^upgrade_compatibility_test::upgrade_write_readiness_tests::/)
 """
 fail-fast = false
 
@@ -721,3 +789,9 @@ test-group = 'e2e-cluster-nightly'
 [[profile.e2e-full.overrides]]
 filter = 'package(e2e_test) & (test(/^kms::kms_vault_test::/) | test(/^kms::kms_rekey_sweep_test::/) | test(/^kms::configured_roundtrip_test::test_configured_vault_kms_admin_and_versioned_cleanup$/))'
 test-group = 'e2e-vault'
+
+# Keep the storage-heavy smoke cases isolated in the full suite too. They
+# start multiple server fixtures and issue many sequential S3 operations.
+[[profile.e2e-full.overrides]]
+filter = 'package(e2e_test) & (test(/^list_objects_v2_pagination_test::tests::(test_list_objects_v2_delimiter_small_page_traverses_all|test_list_objects_v2_max_keys_above_limit_returns_token|test_list_objects_v2_maxkeys_above_limit_with_delimiter)$/) | test(=reliant::tiering::test_hermetic_transition_restore_failure_expiry_and_retry))'
+threads-required = "num-test-threads"

.github/actions/quick-checks/action.yml

@@ -100,12 +100,7 @@ runs:
 
     - name: Check test wiring
       shell: bash
-      run: |
-        python3 ./scripts/check_test_wiring.py --self-test
-        python3 ./scripts/check_scheduled_validation_freshness.py --self-test
-        python3 ./scripts/test_security_workflow.py
-        python3 ./scripts/test_nightly_candidate.py
-        python3 ./scripts/check_test_wiring.py
+      run: python3 ./scripts/check_test_wiring.py
 
     - name: Check no planning docs committed
       shell: bash

.github/actions/setup/action.yml

@@ -107,6 +107,8 @@ runs:
     - name: Install cargo-nextest
       if: inputs.install-test-tools == 'true'
       uses: taiki-e/install-action@96c7780c1d8a2b8723e12031def873a434d39d8d # nextest
+      with:
+        tool: cargo-nextest@0.9.138
 
     - name: Setup Rust cache
       uses: Swatinem/rust-cache@6323deb102c322ba6fcbdcafc7e3dddab59af2b6 # v2

.github/workflows/architecture-migration-rules.yml

@@ -1,61 +0,0 @@
-# Copyright 2024 RustFS Team
-#
-# Licensed under the Apache License, Version 2.0 (the "License");
-# you may not use this file except in compliance with the License.
-# You may obtain a copy of the License at
-#
-#      http://www.apache.org/licenses/LICENSE-2.0
-#
-# Unless required by applicable law or agreed to in writing, software
-# distributed under the License is distributed on an "AS IS" BASIS,
-# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
-# See the License for the specific language governing permissions and
-# limitations under the License.
-
-name: Architecture Migration Rules
-
-on:
-  pull_request:
-    types: [ opened, synchronize, reopened, closed ]
-    branches: [ main ]
-    paths:
-      - "ARCHITECTURE.md"
-      - "docs/architecture/**"
-      - "scripts/check_architecture_migration_rules.sh"
-      - ".github/workflows/architecture-migration-rules.yml"
-  workflow_dispatch:
-
-permissions:
-  contents: read
-
-concurrency:
-  group: ${{ github.workflow }}-${{ github.event.pull_request.number || github.ref }}
-  cancel-in-progress: true
-
-jobs:
-  cancel-closed-pr-runs:
-    name: Cancel Closed PR Runs
-    if: github.event_name == 'pull_request' && github.event.action == 'closed'
-    runs-on: sm-standard-2
-    timeout-minutes: 10
-    steps:
-      - name: Explain cancellation run
-        run: echo "PR closed; this run only cancels older runs in the same concurrency group."
-
-  architecture-migration-rules:
-    name: Architecture Migration Rules
-    if: github.event_name != 'pull_request' || github.event.action != 'closed'
-    runs-on: sm-standard-2
-    timeout-minutes: 10
-    steps:
-      - uses: actions/checkout@f548e57e544e1ff5a4c46bf1e1b8685f8e4a348a # v7
-        with:
-          persist-credentials: false
-
-      - name: Install ripgrep
-        uses: taiki-e/install-action@7623a79cdfecb99d681017af368ca353d9f49bb5 # v2
-        with:
-          tool: ripgrep@15.2.0
-
-      - name: Check architecture migration rules
-        run: ./scripts/check_architecture_migration_rules.sh

.github/workflows/audit.yml

@@ -204,7 +204,7 @@ jobs:
     # open the Actions tab. Same ci-8 mechanism coverage.yml and
     # e2e-replication-nightly.yml already use.
     if: always() && github.event_name == 'schedule' && contains(needs.*.result, 'failure')
-    runs-on: sm-standard-2
+    runs-on: ubuntu-latest
     timeout-minutes: 10
     permissions:
       contents: read

.github/workflows/build.yml

@@ -1167,7 +1167,7 @@ jobs:
     if: >-
       always() && github.event_name == 'schedule' &&
       (contains(needs.*.result, 'failure') || contains(needs.*.result, 'cancelled'))
-    runs-on: sm-standard-2
+    runs-on: ubuntu-latest
     timeout-minutes: 10
     permissions:
       contents: read

.github/workflows/cache-warm.yml

@@ -15,16 +15,16 @@
 # Sole writer of the Rust dependency caches that ci.yml restores.
 #
 # Why this is a separate workflow rather than steps inside ci.yml: ci.yml's
-# concurrency group cancels in-progress runs on main pushes, and merges land far
-# faster than its 70-minute pipeline. Measured over 15 consecutive main pushes:
+# concurrency group originally cancelled in-progress main runs, while merges
+# landed faster than its 70-minute pipeline. Over 15 consecutive main pushes:
 # 12 cancelled, 2 failed, 0 succeeded. A cancelled run never reaches
 # Swatinem/rust-cache's post step (cache-on-failure does not cover cancellation),
 # so the writer lanes were saving nothing and every PR paid a cold restore —
 # 11.8-20.9 minutes of "Setup Rust environment" against 0.7-3.4 warm.
 #
-# Splitting cache writing out of the test pipeline lets ci.yml keep cancelling
-# superseded runs (which is correct — nobody needs test results for a commit
-# that is already three merges behind) while the caches still get written.
+# Keep cache writing separate from validation: only these jobs publish the
+# complete feature closure for each key. Main validation now also finishes its
+# running baseline, while superseded PR attempts can still be cancelled.
 #
 # The group below deliberately does NOT cancel in progress; see the comment on
 # it for how that bounds concurrency and why it is scoped by event.
@@ -256,7 +256,7 @@ jobs:
   # belonged to it.
   warm-ci-uring:
     name: Warm ci-uring
-    runs-on: sm-standard-2
+    runs-on: ubuntu-latest
     timeout-minutes: 60
     steps:
       - name: Checkout repository

.github/workflows/ci.yml

@@ -47,12 +47,12 @@ permissions:
   contents: read
 
 # Concurrency groups are scoped per event so different triggers never cancel
-# each other: PR pushes cancel the previous run of that PR, main pushes keep
-# latest-wins semantics among themselves, and scheduled runs always complete
-# (a shared group used to let every merge kill the weekly scheduled run).
+# each other. PR pushes cancel superseded attempts; main and scheduled runs
+# finish so a busy merge stream cannot starve the complete baseline. For main,
+# GitHub retains one running run and replaces the pending run with the latest.
 concurrency:
   group: ${{ github.workflow }}-${{ github.event_name }}-${{ github.event.pull_request.number || github.ref }}
-  cancel-in-progress: ${{ github.event_name != 'schedule' }}
+  cancel-in-progress: ${{ github.event_name == 'pull_request' }}
 
 env:
   CARGO_TERM_COLOR: always
@@ -128,7 +128,7 @@ jobs:
 
   test-and-lint:
     name: Workspace Test and Lint
-    if: needs.classify-changes.outputs.mode == 'full' && (github.event_name != 'pull_request' || github.event.action != 'closed')
+    if: contains(fromJSON('["full", "e2e"]'), needs.classify-changes.outputs.mode) && (github.event_name != 'pull_request' || github.event.action != 'closed')
     needs: [ quick-checks, classify-changes ]
     runs-on: sm-standard-4
     timeout-minutes: 90
@@ -146,11 +146,9 @@ jobs:
         uses: ./.github/actions/setup
         with:
           rust-version: stable
-          # Every lane in this workflow reads its cache and none writes it.
-          # cache-warm.yml is the sole writer for all four keys: this workflow
-          # cancels superseded runs on main, and a cancelled run never reaches
-          # rust-cache's post step, so writing from here saved nothing (12 of 15
-          # consecutive main-push runs were cancelled). See rustfs/backlog#1600.
+          # Every lane reads its cache; cache-warm.yml remains the sole writer
+          # with the matching feature closure. Tests never compete to replace
+          # shared caches, including when a PR attempt is cancelled.
           cache-shared-key: ci-dev
           cache-save-if: 'false'
           install-build-packaging-tools: 'false'
@@ -195,13 +193,10 @@ jobs:
           # Raised to 3 on main pushes and manual dispatches; PRs keep 2 so the
           # merge path is untouched while the experiment runs.
           #
-          # Dispatch is included because push alone cannot supply the samples:
-          # this workflow cancels superseded runs on main, and only 4 of the last
-          # 20 push-triggered Test and Lint jobs reached a terminal state — at
-          # that rate ten samples would take roughly fifty merges. The
-          # concurrency group is scoped by event_name, so a dispatched run has
-          # its own group and is not cancelled by merge traffic, which makes the
-          # sample collectable on demand rather than by waiting.
+          # During baseline collection, main pushes cancelled superseded runs:
+          # only 4 of 20 Test and Lint jobs reached a terminal state. Running
+          # main validations now finish; dispatch remains an independent group
+          # for collecting additional samples on demand.
           #
           # Baseline over 17 samples at 2:
           # median nextest/clippy step ratio 1.95, spread 1.85-2.06. The gate-2
@@ -250,11 +245,11 @@ jobs:
           mkdir -p artifacts/test-and-lint
           set +e
           timeout --verbose --signal=TERM --kill-after=30s 15m \
-            cargo test --all --doc \
+            cargo test --all --exclude e2e_test --doc \
             2>&1 | tee artifacts/test-and-lint/doctest.log
           status=${PIPESTATUS[0]}
           {
-            echo "command=cargo test --all --doc"
+            echo "command=cargo test --all --exclude e2e_test --doc"
             echo "exit_status=${status}"
             echo "finished_at=$(date --utc --iso-8601=seconds)"
             echo
@@ -281,21 +276,10 @@ jobs:
       - name: Check log-analyzer rule anchors
         run: ./scripts/check_log_analyzer_rules.sh
 
-      # Explicit gate for migration-critical suites. These tests already ran in
-      # the full nextest pass above; a single filtered nextest invocation keeps
-      # the named gate without rebuilding or re-running them one package at a time.
-      #
-      # The gate selects tests by name substring (data_movement / rebalance /
-      # decommission / source_cleanup / delete_marker), so renames can silently
-      # thin it. The script owns the filter expression and first verifies the
-      # selected-test count against the committed floor in
-      # .config/migration-gate-floor.txt before running the gate; renames or
-      # removals must update that file consciously (see the script header).
-      # Kept on the default profile (no --profile ci): a second --profile ci run
-      # would clobber target/nextest/ci/junit.xml, and none of these tests are
-      # quarantined so they gain nothing from the ci profile's retry overrides.
-      - name: Run rebalance/decommission migration proofs
-        run: ./scripts/check_migration_gate_count.sh
+      # Preserve the migration floor and require successful execution evidence
+      # from the workspace run, without compiling or executing its subset again.
+      - name: Verify rebalance/decommission migration proofs
+        run: ./scripts/check_migration_gate_count.sh evidence artifacts/test-and-lint/core-test-listing.json target/nextest/ci/junit.xml
 
   # This gate builds into a fresh target directory to isolate the E2E root.
   # Give its cold build a separate budget from workspace tests and migration proofs.
@@ -431,6 +415,7 @@ jobs:
         run: cargo clippy -p rustfs -p rustfs-ecstore --all-targets --features rio-v2 -- -D warnings
 
       - name: Run rio-v2 feature tests
+        shell: bash
         env:
           # Match the main nextest lane's #5394 link-I/O guard. A cold feature
           # cache otherwise fans out enough rust-lld processes to exhaust this
@@ -440,9 +425,25 @@ jobs:
           # --profile ci so the quarantine list (and its junit flaky markers)
           # covers this leg too; the default profile is the local no-retry
           # profile and silently ignored quarantined flakes here (rustfs#6703).
-          cargo nextest run --profile ci -p rustfs -p rustfs-ecstore --features rio-v2
+          mkdir -p artifacts/test-and-lint
+          rm -f target/nextest/ci/junit.xml
+          ./scripts/ci/resource_sampler.sh start nextest-rio-v2
+          trap './scripts/ci/resource_sampler.sh stop' EXIT
+          cargo nextest run --profile ci -p rustfs -p rustfs-ecstore --features rio-v2 \
+            2>&1 | tee artifacts/test-and-lint/nextest.log
           cargo test -p rustfs --doc --features rio-v2
 
+      - name: Upload rio-v2 test reports and diagnostics
+        if: always()
+        uses: actions/upload-artifact@043fb46d1a93c77aae656e7c1c64a875d1fc6a0a # v7.0.1
+        with:
+          name: junit-test-and-lint-rio-v2-${{ github.run_number }}
+          path: |
+            target/nextest/ci/junit.xml
+            artifacts/test-and-lint
+          retention-days: 3
+          if-no-files-found: error
+
   connect-short-credential-boundary:
     name: Connect Short Credential Boundary
     if: needs.classify-changes.outputs.mode == 'full' && (github.event_name != 'pull_request' || github.event.action != 'closed')
@@ -508,13 +509,9 @@ jobs:
     runs-on: sm-standard-4
     timeout-minutes: 90
     strategy:
-      # On a PR, one failing protocol leg is enough to know the PR is not ready,
-      # so stop the sibling leg instead of paying another ~40 minutes for it.
-      # Everywhere else (main pushes, the merge queue, the weekly schedule) keep
-      # the full signal: there we want to know whether swift AND sftp are broken,
-      # not just whichever failed first. This is the only part of the early-stop
-      # work that also covers fork PRs, since it needs no token.
-      fail-fast: ${{ github.event_name == 'pull_request' }}
+      # Preserve both independent results when one protocol fails, so diagnosis
+      # and a focused fix do not require rebuilding an interrupted sibling.
+      fail-fast: false
       matrix:
         features:
           - name: swift
@@ -537,27 +534,50 @@ jobs:
           cache-save-if: 'false'
           install-build-packaging-tools: 'false'
 
-      - name: Protect Connect test home
-        run: chmod go-w "$(realpath "$HOME")"
+      - name: Prepare protocol test state
+        run: |
+          rm -f target/nextest/ci/junit.xml
+          chmod go-w "$(realpath "$HOME")"
 
       - name: Run clippy with ${{ matrix.features.name }}
         run: |
+          ./scripts/ci/resource_sampler.sh start clippy
+          trap './scripts/ci/resource_sampler.sh stop' EXIT
           cargo clippy -p rustfs -p rustfs-protocols --all-targets ${{ matrix.features.flags }} -- -D warnings
 
       - name: Run tests with ${{ matrix.features.name }}
+        id: protocol-tests
+        shell: bash
         env:
           # Keep feature-test linking under the same bounded concurrency as the
           # main nextest lane; Clippy is metadata-only and needs no such limit.
           CARGO_BUILD_JOBS: "2"
+          SAMPLER_INTERVAL_SECS: "15"
         run: |
           # --profile ci so the quarantine list (and its junit flaky markers)
           # covers this leg too; the default profile is the local no-retry
           # profile and silently ignored quarantined flakes here (rustfs#6703).
-          cargo nextest run --profile ci -p rustfs -p rustfs-protocols ${{ matrix.features.flags }}
+          mkdir -p artifacts/protocol-tests
+          ./scripts/ci/resource_sampler.sh start nextest
+          trap './scripts/ci/resource_sampler.sh stop' EXIT
+          cargo nextest run --profile ci -p rustfs -p rustfs-protocols ${{ matrix.features.flags }} \
+            2>&1 | tee artifacts/protocol-tests/nextest.log
+
+      - name: Upload protocol test reports and diagnostics
+        if: always()
+        uses: actions/upload-artifact@043fb46d1a93c77aae656e7c1c64a875d1fc6a0a # v7.0.1
+        with:
+          name: junit-test-and-lint-${{ matrix.features.name }}-${{ github.run_number }}-${{ github.run_attempt }}
+          path: |
+            target/nextest/ci/junit.xml
+            artifacts/protocol-tests
+            artifacts/test-and-lint
+          retention-days: 3
+          if-no-files-found: warn
 
   build-rustfs-debug-binary:
     name: Build RustFS Debug Binary
-    if: needs.classify-changes.outputs.mode == 'full' && (github.event_name != 'pull_request' || github.event.action != 'closed')
+    if: contains(fromJSON('["full", "e2e"]'), needs.classify-changes.outputs.mode) && (github.event_name != 'pull_request' || github.event.action != 'closed')
     needs: [ quick-checks, classify-changes ]
     runs-on: sm-standard-4
     timeout-minutes: 30
@@ -810,6 +830,10 @@ jobs:
           python3 ./scripts/check_test_wiring.py --check-profile e2e-smoke "${NEXTEST_LISTING}"
           ./scripts/check_security_smoke_count.sh check "${NEXTEST_LISTING}"
 
+      # Keep E2E documentation coverage with its already-built package graph.
+      - name: Run E2E documentation tests
+        run: cargo test -p e2e_test --doc
+
       # PR smoke subset of the in-repo e2e suite (backlog#1149 ci-4). The
       # profile.e2e-smoke default-filter in .config/nextest.toml is the single
       # wiring mechanism for e2e tests in CI — extend that filter instead of
@@ -819,18 +843,22 @@ jobs:
         env:
           NEXTEST_ARCHIVE: ${{ runner.temp }}/rustfs-e2e-smoke.tar.zst
           RUSTFS_E2E_LOG_DIR: ${{ runner.temp }}/rustfs-e2e-smoke-logs
+          SAMPLER_INTERVAL_SECS: "15"
         run: |
+          ./scripts/ci/resource_sampler.sh start e2e-smoke
+          trap './scripts/ci/resource_sampler.sh stop' EXIT
           python3 scripts/e2e_binary.py run --features e2e-test-hooks -- cargo nextest run --profile e2e-smoke --archive-file "${NEXTEST_ARCHIVE}" \
             --status-level all --final-status-level all --failure-output final
 
       - name: Upload e2e smoke diagnostics
-        if: failure()
+        if: always()
         uses: actions/upload-artifact@043fb46d1a93c77aae656e7c1c64a875d1fc6a0a # v7.0.1
         with:
           name: e2e-smoke-diagnostics-${{ github.run_number }}
           path: |
             ${{ runner.temp }}/rustfs-e2e-smoke-logs/
             ${{ runner.temp }}/rustfs-e2e-smoke-list.json
+            artifacts/test-and-lint
           if-no-files-found: warn
 
       - name: Upload e2e smoke JUnit report
@@ -1005,6 +1033,7 @@ jobs:
           RUSTFS_E2E_STARTUP_CAS_BINARY: ${{ runner.temp }}/rustfs-startup-cas-input/rustfs
           RUSTFS_E2E_STARTUP_CAS_BUILD_MANIFEST: ${{ runner.temp }}/rustfs-startup-cas-input/rustfs.e2e-startup-cas-build.json
           RUSTFS_E2E_STARTUP_CAS_ARTIFACT_DIR: ${{ runner.temp }}/rustfs-startup-cas-evidence
+          RUSTFS_HEAL_CHAOS_LOG_ROOT: ${{ runner.temp }}/rustfs-heal-chaos-logs
         run: python3 scripts/e2e_binary.py run --binary "$RUSTFS_E2E_STARTUP_CAS_BINARY" --features e2e-test-hooks -- cargo nextest run --profile e2e-full -p e2e_test
 
       - name: Upload junit
@@ -1028,6 +1057,15 @@ jobs:
           if-no-files-found: warn
           retention-days: 7
 
+      - name: Upload heal cluster logs
+        if: always()
+        uses: actions/upload-artifact@043fb46d1a93c77aae656e7c1c64a875d1fc6a0a # v7.0.1
+        with:
+          name: e2e-full-heal-logs-${{ github.run_number }}
+          path: ${{ runner.temp }}/rustfs-heal-chaos-logs
+          if-no-files-found: warn
+          retention-days: 7
+
   e2e-tests-rio-v2:
     name: End-to-End Tests (rio-v2)
     # Inherits the schedule/dispatch-only gate through needs: on every other
@@ -1270,7 +1308,7 @@ jobs:
     if: >-
       always() && github.event_name == 'schedule' &&
       (contains(needs.*.result, 'failure') || contains(needs.*.result, 'cancelled'))
-    runs-on: sm-standard-2
+    runs-on: ubuntu-latest
     timeout-minutes: 10
     permissions:
       contents: read

.github/workflows/connect-health-service-acceptance.yml

@@ -172,7 +172,7 @@ jobs:
               assert outer.testzip() is None
               with zipfile.ZipFile(io.BytesIO(outer.read(package))) as inner:
                   entries = inner.infolist()
-                  assert {entry.filename for entry in entries} == {'rustfs', 'rustfs-cli'} and len(entries) == 2
+                  assert {entry.filename for entry in entries} == {'rustfs', 'rustfs-cli', 'rustfs.cpu-symbol-catalog.json', 'rustfs.cpu-symbol-catalog.sha256'} and len(entries) == 4
                   assert all(regular(entry) and 0 < entry.file_size <= 1073741824 for entry in entries)
                   assert inner.testzip() is None
                   pathlib.Path('binary').mkdir()

.github/workflows/coverage.yml

@@ -128,7 +128,7 @@ jobs:
     # manual workflow_dispatch runs stay quiet so a debugging run never files a
     # spurious alert.
     if: always() && github.event_name == 'schedule' && contains(needs.*.result, 'failure')
-    runs-on: sm-standard-2
+    runs-on: ubuntu-latest
     timeout-minutes: 10
     permissions:
       contents: read

.github/workflows/e2e-distributed.yml

@@ -38,6 +38,7 @@ on:
       - "Cargo.lock"
       - "Cargo.toml"
       - ".config/nextest.toml"
+      - ".config/e2e-distributed-selection.txt"
       - ".github/workflows/e2e-distributed.yml"
       - "crates/audit/**"
       - "crates/common/**"
@@ -200,7 +201,7 @@ jobs:
     name: Alert on scheduled failure
     needs: [distributed]
     if: always() && github.event_name == 'schedule' && contains(needs.*.result, 'failure')
-    runs-on: sm-standard-2
+    runs-on: ubuntu-latest
     timeout-minutes: 10
     permissions:
       contents: read

.github/workflows/e2e-replication-nightly.yml

@@ -230,7 +230,7 @@ jobs:
     # manual workflow_dispatch runs stay quiet so a debugging run never files a
     # spurious alert.
     if: always() && github.event_name == 'schedule' && contains(needs.*.result, 'failure')
-    runs-on: sm-standard-2
+    runs-on: ubuntu-latest
     timeout-minutes: 10
     permissions:
       contents: read

.github/workflows/e2e-s3tests.yml

@@ -453,7 +453,7 @@ jobs:
     # job fails. Alerts only for scheduled runs (backlog#1149 ci-8) — manual
     # dispatch failures are already watched by a human.
     if: always() && github.event_name == 'schedule' && contains(needs.*.result, 'failure')
-    runs-on: sm-standard-2
+    runs-on: ubuntu-latest
     timeout-minutes: 10
     permissions:
       contents: read

.github/workflows/e2e-upgrade.yml

@@ -51,8 +51,53 @@ env:
   UPGRADE_SOURCE_SHA256: 3ee8df71e8edcfada533be452c4135868f697bc515460ae97b027313eade7a3d
 
 jobs:
+  build:
+    name: Build upgrade candidate
+    runs-on: sm-standard-2
+    timeout-minutes: 60
+    env:
+      FORCE_JAVASCRIPT_ACTIONS_TO_NODE24: "true"
+    steps:
+      - name: Checkout repository
+        uses: actions/checkout@f548e57e544e1ff5a4c46bf1e1b8685f8e4a348a # v7
+        with:
+          persist-credentials: false
+
+      - name: Setup Rust environment
+        uses: ./.github/actions/setup
+        with:
+          cache-shared-key: e2e-upgrade-server
+          cache-save-if: ${{ github.ref == 'refs/heads/main' }}
+          install-build-packaging-tools: "false"
+          install-test-tools: "false"
+
+      - name: Download pinned previous release
+        run: |
+          mkdir -p target/debug
+          archive="target/debug/$UPGRADE_SOURCE_ASSET"
+          curl --fail --location --retry 3 --output "$archive" \
+            "https://github.com/${GITHUB_REPOSITORY}/releases/download/${UPGRADE_SOURCE_VERSION}/${UPGRADE_SOURCE_ASSET}"
+          echo "$UPGRADE_SOURCE_SHA256  $archive" | sha256sum --check --strict
+
+      - name: Build current RustFS binary
+        run: python3 scripts/e2e_binary.py build
+
+      - name: Upload verified upgrade candidate
+        uses: actions/upload-artifact@043fb46d1a93c77aae656e7c1c64a875d1fc6a0a # v7.0.1
+        with:
+          name: upgrade-candidate-${{ github.run_id }}
+          path: |
+            target/debug/rustfs
+            target/debug/rustfs.e2e.json
+            target/debug/${{ env.UPGRADE_SOURCE_ASSET }}
+          if-no-files-found: error
+          retention-days: 3
+          compression-level: 0
+          overwrite: true
+
   upgrade:
     name: ${{ matrix.name }}
+    needs: build
     strategy:
       fail-fast: false
       matrix:
@@ -63,31 +108,24 @@ jobs:
           # the CI required-check names. UPGRADE_SOURCE_VERSION above is the
           # single source of truth for which release they actually run against.
           - name: Direct upgrade from the previous release
-            cache_key: e2e-direct-upgrade
             test: direct_upgrade_from_rc2_preserves_object_contracts
             artifact: direct-upgrade
           - name: Mixed-version rolling upgrade from the previous release
-            cache_key: e2e-mixed-version-upgrade
             test: rolling_upgrade_from_rc2_preserves_mixed_version_contracts
             artifact: mixed-version-upgrade
           - name: Bucket configuration survives the upgrade
-            cache_key: e2e-bucket-config-upgrade
             test: direct_upgrade_from_previous_release_preserves_bucket_configuration
             artifact: bucket-config-upgrade
           - name: Rollback reads current bucket metadata
-            cache_key: e2e-bucket-config-rollback
             test: rollback_to_previous_release_reads_current_bucket_metadata
             artifact: bucket-config-rollback
           - name: ODM configuration recovery after rc.5 rollback
-            cache_key: e2e-odm-config-rollback
             test: rc5_rollback_requires_restoring_odm_configuration
             artifact: odm-config-rollback
           - name: Multipart layouts survive the rc.5 upgrade
-            cache_key: e2e-multipart-layout-upgrade
             test: direct_upgrade_from_rc5_preserves_multipart_layouts
             artifact: multipart-layout-upgrade
           - name: rc.5 multipart replication baseline
-            cache_key: e2e-multipart-layout-baseline
             test: rc5_baseline_replicates_multipart_layouts
             artifact: multipart-layout-baseline
     runs-on: sm-standard-2
@@ -103,19 +141,28 @@ jobs:
       - name: Setup Rust environment
         uses: ./.github/actions/setup
         with:
-          cache-shared-key: ${{ matrix.cache_key }}
-          cache-save-if: ${{ github.ref == 'refs/heads/main' }}
+          cache-shared-key: e2e-upgrade-tests
+          # One main-branch consumer warms the shared test-harness dependencies.
+          cache-save-if: ${{ github.ref == 'refs/heads/main' && matrix.artifact == 'direct-upgrade' }}
           install-build-packaging-tools: "false"
+          install-test-tools: "false"
 
-      - name: Download pinned previous release
+      - name: Download verified upgrade candidate
+        uses: actions/download-artifact@3e5f45b2cfb9172054b4087a40e8e0b5a5461e7c # v8.0.1
+        with:
+          name: upgrade-candidate-${{ github.run_id }}
+          path: target/debug
+
+      - name: Restore candidate executable bit
+        run: chmod +x target/debug/rustfs
+
+      - name: Verify and unpack pinned previous release
         env:
           SOURCE_DIR: ${{ runner.temp }}/rustfs-upgrade-source
         run: |
           set -euo pipefail
           mkdir -p "$SOURCE_DIR"
-          archive="$SOURCE_DIR/$UPGRADE_SOURCE_ASSET"
-          curl --fail --location --retry 3 --output "$archive" \
-            "https://github.com/${GITHUB_REPOSITORY}/releases/download/${UPGRADE_SOURCE_VERSION}/${UPGRADE_SOURCE_ASSET}"
+          archive="target/debug/$UPGRADE_SOURCE_ASSET"
           echo "$UPGRADE_SOURCE_SHA256  $archive" | sha256sum --check --strict
           unzip -q "$archive" -d "$SOURCE_DIR"
           chmod +x "$SOURCE_DIR/rustfs"
@@ -123,14 +170,12 @@ jobs:
           echo "RUSTFS_UPGRADE_SOURCE_BINARY=$SOURCE_DIR/rustfs" >> "$GITHUB_ENV"
           echo "RUSTFS_E2E_LOG_DIR=$RUNNER_TEMP/rustfs-upgrade-logs" >> "$GITHUB_ENV"
 
-      - name: Build current RustFS binary
-        run: |
-          python3 scripts/e2e_binary.py build
-
       - name: Run upgrade compatibility test
         env:
           RUSTFS_SCANNER_HEAL_G09_EVIDENCE_DIR: ${{ runner.temp }}/rustfs-upgrade-g09-evidence/${{ matrix.artifact }}
+          NO_PROXY: 127.0.0.1,localhost,::1
         run: |
+          export no_proxy="$NO_PROXY"
           python3 scripts/e2e_binary.py run -- cargo test --locked -p e2e_test \
             "upgrade_compatibility_test::${{ matrix.test }}" \
             -- --ignored --exact --nocapture

.github/workflows/fuzz.yml

@@ -240,7 +240,7 @@ jobs:
     # job fails. Alerts only for scheduled (nightly) runs (backlog#1149
     # ci-8); PR and manual dispatch failures are already watched by a human.
     if: always() && github.event_name == 'schedule' && contains(needs.*.result, 'failure')
-    runs-on: sm-standard-2
+    runs-on: ubuntu-latest
     timeout-minutes: 10
     permissions:
       contents: read

.github/workflows/helm-package.yml

@@ -145,7 +145,9 @@ jobs:
         uses: actions/upload-artifact@043fb46d1a93c77aae656e7c1c64a875d1fc6a0a # v7.0.1
         with:
           name: helm-package
-          path: helm/rustfs/*.tgz
+          path: |
+            helm/rustfs/*.tgz
+            helm/rustfs/README.md
           retention-days: 1
 
   publish-helm-package:

.github/workflows/minio-interop.yml

@@ -60,9 +60,9 @@ permissions:
 jobs:
   minio-interop:
     name: MinIO interop (EC + SSE read parity)
-    # Skip on forks: needs the repo's runners and is not a contributor gate.
+    # Keep this scheduled interoperability lane scoped to the upstream repo.
     if: github.repository == 'rustfs/rustfs'
-    runs-on: sm-standard-2
+    runs-on: ubuntu-latest
     timeout-minutes: 40
     env:
       # Fixed 32-byte test KMS key baked into the fixture lab; not a secret.
@@ -131,7 +131,7 @@ jobs:
     name: Alert on scheduled failure
     needs: [minio-interop]
     if: always() && github.event_name == 'schedule' && contains(needs.*.result, 'failure')
-    runs-on: sm-standard-2
+    runs-on: ubuntu-latest
     timeout-minutes: 10
     permissions:
       contents: read

.github/workflows/mint.yml

@@ -258,7 +258,7 @@ jobs:
     # job fails. Alerts only for scheduled runs (backlog#1149 ci-8) — manual
     # dispatch failures are already watched by a human.
     if: always() && github.event_name == 'schedule' && contains(needs.*.result, 'failure')
-    runs-on: sm-standard-2
+    runs-on: ubuntu-latest
     timeout-minutes: 10
     permissions:
       contents: read

.github/workflows/nightly-gnu.yml

@@ -498,7 +498,7 @@ jobs:
     if: >-
       always() && github.event_name == 'schedule' &&
       (contains(needs.*.result, 'failure') || contains(needs.*.result, 'cancelled'))
-    runs-on: sm-standard-2
+    runs-on: ubuntu-latest
     timeout-minutes: 10
     permissions:
       contents: read

.github/workflows/oidc-keycloak.yml

@@ -51,7 +51,7 @@ concurrency:
 jobs:
   oidc-keycloak-live:
     name: OIDC Keycloak live gate
-    runs-on: sm-standard-2
+    runs-on: ubuntu-latest
     timeout-minutes: 60
     env:
       FORCE_JAVASCRIPT_ACTIONS_TO_NODE24: "true"
@@ -72,6 +72,11 @@ jobs:
       - name: Build RustFS
         run: cargo build --locked -p rustfs --bin rustfs
 
+      - name: Set up Python
+        uses: actions/setup-python@5fda3b95a4ea91299a34e894583c3862153e4b97 # v7.0.0
+        with:
+          python-version: "3.12"
+
       - name: Install pinned request signer
         run: |
           python3 -m pip install --user --upgrade pip "awscurl==0.44"
@@ -95,7 +100,7 @@ jobs:
     if: >-
       always() && github.event_name == 'schedule' &&
       (needs.oidc-keycloak-live.result == 'failure' || needs.oidc-keycloak-live.result == 'cancelled')
-    runs-on: sm-standard-2
+    runs-on: ubuntu-latest
     timeout-minutes: 10
     permissions:
       contents: read

.github/workflows/on-demand-migration-interop.yml

@@ -69,10 +69,9 @@ env:
 jobs:
   minio-source:
     name: MinIO source (read-through, list-through, backfill)
-    # Skip on forks: needs this repository's runners and is not a contributor
-    # gate.
+    # Keep this scheduled interoperability lane scoped to the upstream repo.
     if: github.repository == 'rustfs/rustfs'
-    runs-on: sm-standard-2
+    runs-on: ubuntu-latest
     timeout-minutes: 90
     env:
       NO_PROXY: 127.0.0.1,localhost
@@ -108,11 +107,15 @@ jobs:
         run: |
           set -euo pipefail
           mkdir -p artifacts/odm-interop/minio
+          docker build -t rustfs-odm-interop-minio \
+            -f crates/rio-v2/tests/minio_fixture_lab/Dockerfile \
+            crates/rio-v2/tests/minio_fixture_lab
           docker run -d --name rustfs-odm-interop-minio \
+            --entrypoint /usr/local/bin/minio \
             -e "MINIO_ROOT_USER=${MINIO_ROOT_USER}" \
             -e "MINIO_ROOT_PASSWORD=${MINIO_ROOT_PASSWORD}" \
             -p 9100:9000 \
-            minio/minio:RELEASE.2025-09-07T16-13-09Z server /data
+            rustfs-odm-interop-minio server /data
           for _ in $(seq 1 120); do
             curl -fsS http://127.0.0.1:9100/minio/health/live >/dev/null 2>&1 && break
             sleep 1
@@ -300,7 +303,7 @@ jobs:
     if: >-
       always() && github.event_name == 'schedule' &&
       (contains(needs.*.result, 'failure') || contains(needs.*.result, 'cancelled'))
-    runs-on: sm-standard-2
+    runs-on: ubuntu-latest
     timeout-minutes: 10
     permissions:
       contents: read

.github/workflows/package.yml

@@ -123,7 +123,7 @@ jobs:
           semver_core='(0|[1-9][0-9]*)\.(0|[1-9][0-9]*)\.(0|[1-9][0-9]*)'
           prerelease_id='(alpha|beta|rc)\.(0|[1-9][0-9]*)'
           if [[ -n "$TAG" ]]; then
-            if [[ "$TAG" =~ ^${semver_core}-${prerelease_id}-preview\.(0|[1-9][0-9]*)$ ]]; then
+            if [[ "$TAG" =~ ^${semver_core}(-${prerelease_id})?-preview\.(0|[1-9][0-9]*)$ ]]; then
               BUILD_TYPE=preview
             elif [[ "$TAG" =~ ^${semver_core}-${prerelease_id}$ ]]; then
               BUILD_TYPE=prerelease
@@ -504,7 +504,7 @@ jobs:
           retention-days: 30
 
       - name: Upload packages to Cloudflare R2
-        if: env.R2_ACCESS_KEY_ID != ''
+        if: needs.resolve.outputs.build_type != 'preview' && env.R2_ACCESS_KEY_ID != ''
         env:
           R2_ACCESS_KEY_ID: ${{ secrets.R2_ACCESS_KEY_ID }}
           R2_SECRET_ACCESS_KEY: ${{ secrets.R2_SECRET_ACCESS_KEY }}

.github/workflows/performance-ab.yml

@@ -366,7 +366,7 @@ jobs:
     if: >-
       always() && github.event_name == 'schedule' &&
       (contains(needs.*.result, 'failure') || contains(needs.*.result, 'cancelled'))
-    runs-on: sm-standard-2
+    runs-on: ubuntu-latest
     timeout-minutes: 10
     permissions:
       contents: read

.github/workflows/runner-hygiene.yml

@@ -66,7 +66,7 @@ jobs:
     # scheduled runs file a tracking issue, manual dispatch stays quiet so
     # debugging never produces a spurious alert.
     if: always() && github.event_name == 'schedule' && contains(needs.*.result, 'failure')
-    runs-on: sm-standard-2
+    runs-on: ubuntu-latest
     timeout-minutes: 10
     permissions:
       contents: read

.github/workflows/rustfs-security-test.yml

@@ -106,7 +106,7 @@ jobs:
         if: ${{ inputs.chain_manifest != '' }}
         env:
           CHAIN_MANIFEST: ${{ inputs.chain_manifest }}
-        run: python3 scripts/functional_chain_evidence.py consume
+        run: python3 rustfs-repo/scripts/functional_chain_evidence.py consume
 
       - name: Checkout auto-testing scripts
         uses: actions/checkout@f548e57e544e1ff5a4c46bf1e1b8685f8e4a348a # v7
@@ -389,7 +389,7 @@ jobs:
           CHAIN_TEST_OUTCOME: ${{ steps.test.outcome }}
           CHAIN_REPORT_OUTCOME: ${{ steps.report.outcome }}
         run: >-
-          python3 scripts/functional_chain_evidence.py record --suite security
+          python3 rustfs-repo/scripts/functional_chain_evidence.py record --suite security
           --report "${SECURITY_ARTIFACTS_DIR}/suite-report.md"
           --output "${RUNNER_TEMP}/chain-security-${GITHUB_RUN_ID}-${GITHUB_RUN_ATTEMPT}/security.json"
 

.github/workflows/schedule-failure-alert-drill.yml

@@ -58,7 +58,7 @@ jobs:
     # Mirrors the consumer wiring, minus the schedule-event guard (this
     # workflow is dispatch-only by design).
     if: always() && contains(needs.*.result, 'failure')
-    runs-on: sm-standard-2
+    runs-on: ubuntu-latest
     timeout-minutes: 10
     permissions:
       contents: read

.github/workflows/targets-integration.yml

@@ -177,7 +177,7 @@ jobs:
     if: >-
       always() && github.event_name == 'schedule' &&
       (contains(needs.*.result, 'failure') || contains(needs.*.result, 'cancelled'))
-    runs-on: sm-standard-2
+    runs-on: ubuntu-latest
     timeout-minutes: 10
     permissions:
       contents: read

Cargo.lock

@@ -987,9 +987,9 @@ dependencies = [
 
 [[package]]
 name = "aws-sdk-kms"
-version = "1.121.0"
+version = "1.122.0"
 source = "registry+https://github.com/rust-lang/crates.io-index"
-checksum = "9febf0461c7441895de457dcff49dd7fa8f013793367f6b81d422c64e939e1cc"
+checksum = "ae92186214a60f1316f6e25df2b5bef8f67b5b08b45f8e3eb6e1278b9ddd7b29"
 dependencies = [
  "arc-swap",
  "aws-credential-types",
@@ -1012,9 +1012,9 @@ dependencies = [
 
 [[package]]
 name = "aws-sdk-s3"
-version = "1.149.0"
+version = "1.150.0"
 source = "registry+https://github.com/rust-lang/crates.io-index"
-checksum = "d48257fb0eb1907a02a965ff7d1b910770a6de2c59a2d03461678df989c5cdcd"
+checksum = "2fecaa77c862fa1e9d636d54fa06e0049a19a4f071b0d72ea1b9086887d09359"
 dependencies = [
  "arc-swap",
  "aws-credential-types",
@@ -1048,9 +1048,9 @@ dependencies = [
 
 [[package]]
 name = "aws-sdk-sso"
-version = "1.112.0"
+version = "1.113.0"
 source = "registry+https://github.com/rust-lang/crates.io-index"
-checksum = "d3d59c47556b283bb2d20fc0b416df7d9d9a3d6f5fa0e77867167a3f2efc23b9"
+checksum = "87184f36ade012f46cdc773c7be10031b4545a1aea70a9a22a044e8f36f1e5c5"
 dependencies = [
  "arc-swap",
  "aws-credential-types",
@@ -1073,9 +1073,9 @@ dependencies = [
 
 [[package]]
 name = "aws-sdk-ssooidc"
-version = "1.114.0"
+version = "1.115.0"
 source = "registry+https://github.com/rust-lang/crates.io-index"
-checksum = "eceb9f31b46ba6da21f666e456395624824a2a086caed940b13d78832fa3ee02"
+checksum = "84edc7495d56e006dff0031cbc9782784902456f7024651c2da902a3d2336262"
 dependencies = [
  "arc-swap",
  "aws-credential-types",
@@ -1098,9 +1098,9 @@ dependencies = [
 
 [[package]]
 name = "aws-sdk-sts"
-version = "1.117.0"
+version = "1.118.0"
 source = "registry+https://github.com/rust-lang/crates.io-index"
-checksum = "9291df9ea767ab5e77d08a53f4f22daa917909a34ed7a22c1a4b82d193a423ad"
+checksum = "df7572569d018ca72923073494eac3c4671a0daea90b9e3923c2d27a3a02a2c1"
 dependencies = [
  "arc-swap",
  "aws-credential-types",
@@ -1193,9 +1193,9 @@ dependencies = [
 
 [[package]]
 name = "aws-smithy-http"
-version = "0.64.0"
+version = "0.64.1"
 source = "registry+https://github.com/rust-lang/crates.io-index"
-checksum = "37843d9add67c3aff5856f409c6dc315d3cdff60f9c0cb5b670dab1e9920306d"
+checksum = "639b4d8f8555f24a9be649811c3eb0b4d4616f4d61daf0c32e28873bc1ea9af1"
 dependencies = [
  "aws-smithy-eventstream",
  "aws-smithy-runtime-api",
@@ -1321,9 +1321,9 @@ dependencies = [
 
 [[package]]
 name = "aws-smithy-runtime-api"
-version = "1.17.0"
+version = "1.18.0"
 source = "registry+https://github.com/rust-lang/crates.io-index"
-checksum = "3fbf162725183ec0df77a9dc82ce22d9ba0b6ea7f376e8ec4f6b695727c70698"
+checksum = "9bb0deb40a69de59f7809251cfe1ab16de135f42c49d222d1202b4d0ce19720f"
 dependencies = [
  "aws-smithy-async",
  "aws-smithy-runtime-api-macros",
@@ -1360,9 +1360,9 @@ dependencies = [
 
 [[package]]
 name = "aws-smithy-types"
-version = "1.8.0"
+version = "1.8.1"
 source = "registry+https://github.com/rust-lang/crates.io-index"
-checksum = "16ecf7989e381d0cbb7ca06104140d940bf38d8eb9004bd3a12d23df03e3d363"
+checksum = "69bb407740a197147da48238ecc94498493c9e85445732360cec180296ca45f1"
 dependencies = [
  "base64-simd",
  "bytes",
@@ -1746,7 +1746,7 @@ checksum = "f8b851b75c23ca7873623d612fe49bd1989aeb03d08fb9432187eb253d3d4c6b"
 dependencies = [
  "alloc-no-stdlib 3.0.0",
  "alloc-stdlib 0.3.0",
- "brotli-decompressor 6.0.0",
+ "brotli-decompressor 6.0.1",
 ]
 
 [[package]]
@@ -1761,9 +1761,9 @@ dependencies = [
 
 [[package]]
 name = "brotli-decompressor"
-version = "6.0.0"
+version = "6.0.1"
 source = "registry+https://github.com/rust-lang/crates.io-index"
-checksum = "46087801e40b785410c352bee60e9b6226312709cf52d5fff92776cdf5cf6490"
+checksum = "941cd9bd4ddab83cb46fa5a2d428f1c857b24ac78cb876cf7beb710840934bd7"
 dependencies = [
  "alloc-no-stdlib 3.0.0",
  "alloc-stdlib 0.3.0",
@@ -1971,9 +1971,9 @@ dependencies = [
 
 [[package]]
 name = "cc"
-version = "1.4.7"
+version = "1.5.1"
 source = "registry+https://github.com/rust-lang/crates.io-index"
-checksum = "54413ede23c2daf518f35156dfde027feb2374004d63bd497f983c8db9c0e313"
+checksum = "f360145194ee8e21db5ee7f3fcd4fe52210864c75c985dae33218202c8bbe040"
 dependencies = [
  "find-msvc-tools",
  "jobserver",
@@ -4217,13 +4217,12 @@ checksum = "34aa73646ffb006b8f5147f3dc182bd4bcb190227ce861fc4a4844bf8e3cb2c0"
 
 [[package]]
 name = "encoding_rs"
-version = "0.8.41"
+version = "0.8.42"
 source = "registry+https://github.com/rust-lang/crates.io-index"
-checksum = "7b5ef0006ac9ab233c38522f5ae99cae3625151de8f706cacee1cba4b8e2832a"
+checksum = "8e985e0451871ad22fb8d2b6b076e2028a502a0d3950998c2c5c0a4f9b5d9679"
 dependencies = [
  "cfg-if",
  "core_detect",
- "multiversion",
  "multiversion_no_op",
  "rustversion",
  "scopeguard",
@@ -4406,9 +4405,9 @@ dependencies = [
 
 [[package]]
 name = "find-msvc-tools"
-version = "0.1.13"
+version = "0.1.14"
 source = "registry+https://github.com/rust-lang/crates.io-index"
-checksum = "ef25905e51abafe4dcea6c15fec58c57b601cdbd0ee53d22ea1d3016c587d39b"
+checksum = "aedcfb3409746eddb02b9e19ebda1c3394f759a152e48ee875a0844d1b955484"
 
 [[package]]
 name = "findshlibs"
@@ -4756,9 +4755,9 @@ dependencies = [
 
 [[package]]
 name = "google-cloud-auth"
-version = "1.16.0"
+version = "1.17.0"
 source = "registry+https://github.com/rust-lang/crates.io-index"
-checksum = "ff461519b1a948200f163574be072753bcfb462a323f0eb426629d89872dd685"
+checksum = "ac9870b8ea484519aa8601e14886f2efc148ea490034e43d04bef1d246170cc9"
 dependencies = [
  "async-trait",
  "aws-lc-rs",
@@ -4785,9 +4784,9 @@ dependencies = [
 
 [[package]]
 name = "google-cloud-gax"
-version = "1.14.0"
+version = "1.15.0"
 source = "registry+https://github.com/rust-lang/crates.io-index"
-checksum = "c5615cff28ee59cfe52fbb4c11b8b1e77f650296e2ea4f4c2b7757ac6b19e752"
+checksum = "fc64ee071c0d22cb68662b454f1bea5424506cafeaddc51260fbc585bac1d29d"
 dependencies = [
  "bytes",
  "futures",
@@ -4805,9 +4804,9 @@ dependencies = [
 
 [[package]]
 name = "google-cloud-gax-internal"
-version = "0.7.18"
+version = "0.7.19"
 source = "registry+https://github.com/rust-lang/crates.io-index"
-checksum = "d2766757d877a7a8ac23da9884cb0e3f10ed9b75a0ce59801ce6b19bf9d5819e"
+checksum = "c47c73e95dc5e1de74bebe21b406394b53ff5ccfdfe0e81a0443cdef468e6744"
 dependencies = [
  "bytes",
  "futures",
@@ -4821,9 +4820,9 @@ dependencies = [
  "http-body-util",
  "hyper",
  "lazy_static",
- "opentelemetry",
- "opentelemetry-semantic-conventions",
- "opentelemetry_sdk",
+ "opentelemetry 0.32.0",
+ "opentelemetry-semantic-conventions 0.32.1",
+ "opentelemetry_sdk 0.32.1",
  "percent-encoding",
  "pin-project",
  "prost 0.14.4",
@@ -4839,14 +4838,14 @@ dependencies = [
  "tonic-prost",
  "tower",
  "tracing",
- "tracing-opentelemetry",
+ "tracing-opentelemetry 0.33.0",
 ]
 
 [[package]]
 name = "google-cloud-iam-v1"
-version = "1.12.0"
+version = "1.13.0"
 source = "registry+https://github.com/rust-lang/crates.io-index"
-checksum = "5f962b40234b1531e6ef73f7558871c96e117231e962086c98804323fb8d2c82"
+checksum = "48f0895fa7b7bd70629ae58daaa962677e93a72c251b2f297bc058a035940821"
 dependencies = [
  "async-trait",
  "bytes",
@@ -4862,9 +4861,9 @@ dependencies = [
 
 [[package]]
 name = "google-cloud-longrunning"
-version = "1.13.0"
+version = "1.14.0"
 source = "registry+https://github.com/rust-lang/crates.io-index"
-checksum = "1c0363c5389ffda2b55cd8a86eef4b19a3481a48467c91dc5d77f626a9572766"
+checksum = "e8ea252eee22a4e2187025ea6789c682a0d791a9748ea99de2abec3026a32cb4"
 dependencies = [
  "async-trait",
  "bytes",
@@ -4880,9 +4879,9 @@ dependencies = [
 
 [[package]]
 name = "google-cloud-lro"
-version = "1.10.0"
+version = "1.11.0"
 source = "registry+https://github.com/rust-lang/crates.io-index"
-checksum = "47af3deef75c14a2983c430898d960c765bddbcc9f9188ca0563108e9227cfe7"
+checksum = "dfb938b40ede9ef70c24f49c2687d84f289c9e9104ddfece739985e4b4844b8f"
 dependencies = [
  "google-cloud-gax",
  "google-cloud-gax-internal",
@@ -4909,9 +4908,9 @@ dependencies = [
 
 [[package]]
 name = "google-cloud-storage"
-version = "1.18.0"
+version = "1.19.0"
 source = "registry+https://github.com/rust-lang/crates.io-index"
-checksum = "973399251b245c63f1d02d0768772833dcf1372ee358f593fb9159de8fe9c7d4"
+checksum = "5f7fa1e66d6b47220dbfc5f243b634136e5b8d14d64cb13419bd59f976681da3"
 dependencies = [
  "async-trait",
  "base64 0.23.1",
@@ -4962,11 +4961,11 @@ dependencies = [
 
 [[package]]
 name = "google-cloud-wkt"
-version = "1.7.0"
+version = "1.8.0"
 source = "registry+https://github.com/rust-lang/crates.io-index"
-checksum = "7fccf98cfd5481a5f5a285181ab0c62123d7d47cd2bb7299448440649349e4e7"
+checksum = "ad4b36fefda4485f0922de7a58a3e562e984d513442a1aae2281dce25124be2e"
 dependencies = [
- "base64 0.22.1",
+ "base64 0.23.1",
  "bytes",
  "serde",
  "serde_json",
@@ -5096,9 +5095,9 @@ dependencies = [
 
 [[package]]
 name = "headers"
-version = "0.4.1"
+version = "0.4.2"
 source = "registry+https://github.com/rust-lang/crates.io-index"
-checksum = "b3314d5adb5d94bcdf56771f2e50dbbc80bb4bdf88967526706205ac9eff24eb"
+checksum = "fc2700e3fe555c3310aa7286cac6167449f2c87e0eb58769c9208a1c58a1d106"
 dependencies = [
  "base64 0.22.1",
  "bytes",
@@ -5283,9 +5282,9 @@ dependencies = [
 
 [[package]]
 name = "hotpath"
-version = "0.26.0"
+version = "0.26.1"
 source = "registry+https://github.com/rust-lang/crates.io-index"
-checksum = "f37234f85a8d1bda6b84d486245dd0847c5b33cca10d5f3691375e66704de0f8"
+checksum = "036ff36fa1c4ae8dfb686606537eaa15b5d5da0c51bf94abeaf9dfc6ee54cac7"
 dependencies = [
  "arc-swap",
  "async-channel",
@@ -5297,6 +5296,7 @@ dependencies = [
  "futures-core",
  "futures-sink",
  "hdrhistogram",
+ "hotpath-drain",
  "hotpath-macros",
  "http 1.5.0",
  "libc",
@@ -5315,10 +5315,16 @@ dependencies = [
 ]
 
 [[package]]
-name = "hotpath-macros"
-version = "0.26.0"
+name = "hotpath-drain"
+version = "0.0.2"
 source = "registry+https://github.com/rust-lang/crates.io-index"
-checksum = "2ac39b5517f540cab511bdbd5a4dc8cfc62858d6cd26a095463c1d2acfb3f290"
+checksum = "e149b660b611650bdf3307941ef3e9b81be3c30ed1dfdbe7f670319674f023a6"
+
+[[package]]
+name = "hotpath-macros"
+version = "0.26.1"
+source = "registry+https://github.com/rust-lang/crates.io-index"
+checksum = "1ff68b15fdbcddfbad685f445597580d6fcbf28b49f01ff975e934351079e13b"
 dependencies = [
  "proc-macro2",
  "quote",
@@ -5459,16 +5465,17 @@ dependencies = [
 
 [[package]]
 name = "hyper-util"
-version = "0.1.20"
+version = "0.1.21"
 source = "registry+https://github.com/rust-lang/crates.io-index"
-checksum = "96547c2556ec9d12fb1578c4eaf448b04993e7fb79cbaad930a656880a6bdfa0"
+checksum = "ddc03d96684f9226b8a787cdb71488417b53ab5ea8fdb1dac946cb9431cc8bff"
 dependencies = [
- "base64 0.22.1",
+ "base64 0.23.1",
  "bytes",
  "futures-channel",
  "futures-util",
  "http 1.5.0",
  "http-body",
+ "httparse",
  "hyper",
  "ipnet",
  "libc",
@@ -5905,9 +5912,9 @@ dependencies = [
 
 [[package]]
 name = "js-sys"
-version = "0.3.105"
+version = "0.3.106"
 source = "registry+https://github.com/rust-lang/crates.io-index"
-checksum = "ce57d20d1ea864ce2ac172ab472d409214f4fd359f0b2a2775abdf522e2af99e"
+checksum = "7883d941dae510fb2d978fc3fe018c71c9e2892fd38854de3e8b92c2e5ad9cc5"
 dependencies = [
  "cfg-if",
  "futures-util",
@@ -6202,9 +6209,9 @@ dependencies = [
 
 [[package]]
 name = "libunftp"
-version = "0.23.0"
+version = "0.23.1"
 source = "registry+https://github.com/rust-lang/crates.io-index"
-checksum = "c9f8ff371890db2cf65a0758dba9a79f9cd965de369f6dbdc6581a22780af45e"
+checksum = "3dc54288ecbf8beefa5cab3a9921ab03bf1f18a3a0872cbcdb5313fc0baebaeb"
 dependencies = [
  "async-trait",
  "bitflags 2.13.2",
@@ -6213,7 +6220,7 @@ dependencies = [
  "dashmap",
  "derive_more",
  "futures-util",
- "getrandom 0.3.4",
+ "getrandom 0.4.3",
  "lazy_static",
  "moka",
  "nix 0.30.1",
@@ -6699,27 +6706,6 @@ version = "0.10.1"
 source = "registry+https://github.com/rust-lang/crates.io-index"
 checksum = "1d87ecb2933e8aeadb3e3a02b828fed80a7528047e68b4f424523a0981a3a084"
 
-[[package]]
-name = "multiversion"
-version = "0.9.0"
-source = "registry+https://github.com/rust-lang/crates.io-index"
-checksum = "b4ca4bea16ffc3f443cf7d866912118196bfef4c6a1556ca00f9f9b00bb43f7c"
-dependencies = [
- "multiversion-macros",
-]
-
-[[package]]
-name = "multiversion-macros"
-version = "0.9.0"
-source = "registry+https://github.com/rust-lang/crates.io-index"
-checksum = "0d416831a7317ef4b08bee00b69cbbb9c8763da7959a7026244d6266869f9c83"
-dependencies = [
- "proc-macro2",
- "quote",
- "rustversion",
- "syn 3.0.6",
-]
-
 [[package]]
 name = "multiversion_no_op"
 version = "1.0.0"
@@ -7221,12 +7207,21 @@ version = "0.39.1"
 source = "registry+https://github.com/rust-lang/crates.io-index"
 checksum = "2e5a6c098c7a3b6547378093f5cc30bc54fd361ce711e05293a5cc589562739b"
 dependencies = [
- "crc32fast",
  "flate2",
+ "memchr",
+ "ruzstd",
+]
+
+[[package]]
+name = "object"
+version = "0.40.0"
+source = "registry+https://github.com/rust-lang/crates.io-index"
+checksum = "dd229a0361b9d0d4396176e02d65897f487eebeab7caa6d443855ee152ca0b9c"
+dependencies = [
+ "crc32fast",
  "hashbrown 0.17.1",
  "indexmap 2.14.2",
  "memchr",
- "ruzstd",
 ]
 
 [[package]]
@@ -7338,12 +7333,26 @@ dependencies = [
 ]
 
 [[package]]
-name = "opentelemetry-appender-tracing"
-version = "0.32.0"
+name = "opentelemetry"
+version = "0.33.0"
 source = "registry+https://github.com/rust-lang/crates.io-index"
-checksum = "2c0080f0dc1d7c786f467cd85a4e395fcab11ee852004f39a29a18ab7c25d837"
+checksum = "6cdb0b1b267eb9db3331b434ed9ddab10d50e280a9adf9d13e5233e2002b61b5"
 dependencies = [
- "opentelemetry",
+ "futures-core",
+ "futures-sink",
+ "js-sys",
+ "pin-project-lite",
+ "thiserror 2.0.21",
+ "tracing",
+]
+
+[[package]]
+name = "opentelemetry-appender-tracing"
+version = "0.33.0"
+source = "registry+https://github.com/rust-lang/crates.io-index"
+checksum = "e2b304e8d8f11326ba366a99ca5c7aaf7cd397b67268a332437175aeb42a40db"
+dependencies = [
+ "opentelemetry 0.33.0",
  "tracing",
  "tracing-core",
  "tracing-log",
@@ -7352,29 +7361,30 @@ dependencies = [
 
 [[package]]
 name = "opentelemetry-http"
-version = "0.32.0"
+version = "0.33.0"
 source = "registry+https://github.com/rust-lang/crates.io-index"
-checksum = "5683015d09e2df236ef005b17f6f196f0d5f6313c4fa43a7b6a53b52776e4331"
+checksum = "ee2c3625b8aa04209f7e01e513bc8044da9687fa38a9eacf59628ca5f3b87300"
 dependencies = [
  "async-trait",
  "bytes",
  "http 1.5.0",
- "opentelemetry",
+ "opentelemetry 0.33.0",
  "reqwest",
 ]
 
 [[package]]
 name = "opentelemetry-otlp"
-version = "0.32.0"
+version = "0.33.0"
 source = "registry+https://github.com/rust-lang/crates.io-index"
-checksum = "9966929966d17620d7c316c643ba62631826e10021409357772d5eea84f62c35"
+checksum = "699a67345e21962a231955b9059a157e428b219fa5df8efe11f08346fb23a344"
 dependencies = [
  "flate2",
  "http 1.5.0",
- "opentelemetry",
+ "httpdate",
+ "opentelemetry 0.33.0",
  "opentelemetry-http",
  "opentelemetry-proto",
- "opentelemetry_sdk",
+ "opentelemetry_sdk 0.33.0",
  "prost 0.14.4",
  "reqwest",
  "thiserror 2.0.21",
@@ -7382,12 +7392,12 @@ dependencies = [
 
 [[package]]
 name = "opentelemetry-proto"
-version = "0.32.0"
+version = "0.33.0"
 source = "registry+https://github.com/rust-lang/crates.io-index"
-checksum = "56d658ba1faf63f7b9c492cfbe6e0ec365440a16132d3270c1065f7b33f1b638"
+checksum = "25da1ac11a0aeccf38d7f77ee0348715adaf8340f65ad46c94a02c6b20e2f65d"
 dependencies = [
- "opentelemetry",
- "opentelemetry_sdk",
+ "opentelemetry 0.33.0",
+ "opentelemetry_sdk 0.33.0",
  "prost 0.14.4",
 ]
 
@@ -7398,14 +7408,20 @@ source = "registry+https://github.com/rust-lang/crates.io-index"
 checksum = "c913ac17a6c451661ee255f4625d143e51647ae78ebd969b75e41c4442f4fe47"
 
 [[package]]
-name = "opentelemetry-stdout"
-version = "0.32.0"
+name = "opentelemetry-semantic-conventions"
+version = "0.33.0"
 source = "registry+https://github.com/rust-lang/crates.io-index"
-checksum = "a1b1c6a247d79091f0062a5f4bd058589525cf987a8d4c169440d9c1be72f0ad"
+checksum = "d288b1e1bc3590052fbb8f0f635ef248dd9022cf1fbaeb6582b7ad6081a17b87"
+
+[[package]]
+name = "opentelemetry-stdout"
+version = "0.33.0"
+source = "registry+https://github.com/rust-lang/crates.io-index"
+checksum = "9f71803f481d3f5ccbe477edb42a78f473613712b2327835911a9ea0ab069824"
 dependencies = [
  "chrono",
- "opentelemetry",
- "opentelemetry_sdk",
+ "opentelemetry 0.33.0",
+ "opentelemetry_sdk 0.33.0",
 ]
 
 [[package]]
@@ -7417,7 +7433,23 @@ dependencies = [
  "futures-channel",
  "futures-executor",
  "futures-util",
- "opentelemetry",
+ "opentelemetry 0.32.0",
+ "percent-encoding",
+ "portable-atomic",
+ "rand 0.9.5",
+ "thiserror 2.0.21",
+]
+
+[[package]]
+name = "opentelemetry_sdk"
+version = "0.33.0"
+source = "registry+https://github.com/rust-lang/crates.io-index"
+checksum = "cb39533d9d1c912123efd7d41d7e0c29d16917b60ce15b4c8d87cb1af7f67520"
+dependencies = [
+ "futures-channel",
+ "futures-executor",
+ "futures-util",
+ "opentelemetry 0.33.0",
  "percent-encoding",
  "portable-atomic",
  "rand 0.9.5",
@@ -8946,9 +8978,9 @@ dependencies = [
 
 [[package]]
 name = "redis"
-version = "1.7.0"
+version = "1.7.1"
 source = "registry+https://github.com/rust-lang/crates.io-index"
-checksum = "2acbc41a996f7652b2ddd9dfd98cc4ff602cfd742ae35382f07f608405ab50ed"
+checksum = "ed5b98ae99461e02895c3a96a25dbe4d852a3d5e5b815d31d5d9e4b5344d99e9"
 dependencies = [
  "arc-swap",
  "arcstr",
@@ -9521,9 +9553,10 @@ dependencies = [
  "metrics-util",
  "mime_guess",
  "moka",
- "opentelemetry",
+ "object 0.40.0",
+ "opentelemetry 0.33.0",
  "opentelemetry-proto",
- "opentelemetry_sdk",
+ "opentelemetry_sdk 0.33.0",
  "p256 0.14.0",
  "parking_lot",
  "percent-encoding",
@@ -9600,6 +9633,7 @@ dependencies = [
  "snap",
  "socket2",
  "subtle",
+ "symbolic-demangle",
  "sysinfo",
  "temp-env",
  "tempfile",
@@ -9613,7 +9647,7 @@ dependencies = [
  "tower",
  "tower-http 0.7.1",
  "tracing",
- "tracing-opentelemetry",
+ "tracing-opentelemetry 0.34.0",
  "tracing-subscriber",
  "url",
  "urlencoding",
@@ -9705,7 +9739,7 @@ name = "rustfs-connect-symbol-catalog"
 version = "1.0.1"
 dependencies = [
  "hex-simd",
- "object 0.39.1",
+ "object 0.40.0",
  "serde",
  "serde_json",
  "sha2 0.11.0",
@@ -9803,8 +9837,8 @@ dependencies = [
  "metrics-util",
  "moka",
  "num_cpus",
- "opentelemetry",
- "opentelemetry_sdk",
+ "opentelemetry 0.33.0",
+ "opentelemetry_sdk 0.33.0",
  "parking_lot",
  "path-absolutize",
  "pin-project-lite",
@@ -9865,7 +9899,7 @@ dependencies = [
  "tower",
  "tracing",
  "tracing-core",
- "tracing-opentelemetry",
+ "tracing-opentelemetry 0.34.0",
  "tracing-subscriber",
  "url",
  "urlencoding",
@@ -9964,6 +9998,7 @@ dependencies = [
  "thiserror 2.0.21",
  "tokio",
  "tokio-util",
+ "tonic",
  "tracing",
  "uuid",
  "walkdir",
@@ -10381,12 +10416,12 @@ dependencies = [
  "metrics-util",
  "num_cpus",
  "nvml-wrapper",
- "opentelemetry",
+ "opentelemetry 0.33.0",
  "opentelemetry-appender-tracing",
  "opentelemetry-otlp",
- "opentelemetry-semantic-conventions",
+ "opentelemetry-semantic-conventions 0.33.0",
  "opentelemetry-stdout",
- "opentelemetry_sdk",
+ "opentelemetry_sdk 0.33.0",
  "percent-encoding",
  "pyroscope",
  "rcgen",
@@ -10418,7 +10453,7 @@ dependencies = [
  "tracing",
  "tracing-appender",
  "tracing-error",
- "tracing-opentelemetry",
+ "tracing-opentelemetry 0.34.0",
  "tracing-subscriber",
  "url",
  "zstd 0.14.0",
@@ -11172,9 +11207,9 @@ dependencies = [
 
 [[package]]
 name = "rustls-platform-verifier"
-version = "0.7.0"
+version = "0.7.1"
 source = "registry+https://github.com/rust-lang/crates.io-index"
-checksum = "26d1e2536ce4f35f4846aa13bff16bd0ff40157cdb14cc056c7b14ba41233ba0"
+checksum = "1167586491e2b18b8bfbb293e8180ec17c201c4f076d7cb3070ca964e7598f98"
 dependencies = [
  "core-foundation 0.10.1",
  "core-foundation-sys",
@@ -11193,9 +11228,9 @@ dependencies = [
 
 [[package]]
 name = "rustls-platform-verifier-android"
-version = "0.1.1"
+version = "0.2.0"
 source = "registry+https://github.com/rust-lang/crates.io-index"
-checksum = "f87165f0995f63a9fbeea62b64d10b4d9d8e78ec6d7d51fb2125fda7bb36788f"
+checksum = "eec689c0bc40ff2458a5977b6619cb718087084a18e02a131c599b62d05e1a5f"
 
 [[package]]
 name = "rustls-webpki"
@@ -11656,9 +11691,9 @@ dependencies = [
 
 [[package]]
 name = "serde_with"
-version = "3.23.0"
+version = "3.24.0"
 source = "registry+https://github.com/rust-lang/crates.io-index"
-checksum = "935177bb8c0cd8ca1a4e6d1a2ac8988bea69cab4f9d3a31311e012ad27868ea4"
+checksum = "df9adc193c780ef8f159aee8b61e2d5801aaa555e6eb0947fe45530ec506296f"
 dependencies = [
  "base64 0.23.1",
  "bs58",
@@ -11677,9 +11712,9 @@ dependencies = [
 
 [[package]]
 name = "serde_with_macros"
-version = "3.23.0"
+version = "3.24.0"
 source = "registry+https://github.com/rust-lang/crates.io-index"
-checksum = "1d607aa01a3cb0ad757d6fd216136910db3c97b102fe686585689615a02dbcdc"
+checksum = "3e17bbc68e28663bbbb90df47e058aa7eda4fb445b89fe70457bb94fbccf6e49"
 dependencies = [
  "darling 0.24.1",
  "proc-macro2",
@@ -11914,9 +11949,9 @@ dependencies = [
 
 [[package]]
 name = "siphasher"
-version = "1.0.3"
+version = "1.0.4"
 source = "registry+https://github.com/rust-lang/crates.io-index"
-checksum = "8ee5873ec9cce0195efcb7a4e9507a04cd49aec9c83d0389df45b1ef7ba2e649"
+checksum = "33f4fe9184a62d842c9ef383018f3306d8ba224fd9d836f56d7288308847c256"
 
 [[package]]
 name = "sketches-ddsketch"
@@ -11966,9 +12001,9 @@ dependencies = [
 
 [[package]]
 name = "smallvec"
-version = "1.16.1"
+version = "1.16.2"
 source = "registry+https://github.com/rust-lang/crates.io-index"
-checksum = "ba467056f1b547ed52077911161fc86985becbc60e8e1857c8a144dab0def891"
+checksum = "f9395f0f0eee849a9b707b2f06bb92a6a422090e2123bb2ef8e87a0e61892a8e"
 dependencies = [
  "serde",
 ]
@@ -12248,9 +12283,9 @@ checksum = "13c2bddecc57b384dee18652358fb23172facb8a2c51ccc10d74c157bdea3292"
 
 [[package]]
 name = "suppaftp"
-version = "12.0.1"
+version = "12.1.0"
 source = "registry+https://github.com/rust-lang/crates.io-index"
-checksum = "ba93b140f4d07f1b3f3fb4d7264b2d422d5c8b4e925eb6a2646e1342d5d2070a"
+checksum = "d79f066c76927a775d61b5642e0b712a17582ebe7f62b7e5abba246e619878f7"
 dependencies = [
  "async-trait",
  "chrono",
@@ -12763,9 +12798,9 @@ dependencies = [
 
 [[package]]
 name = "tokio-test"
-version = "0.4.5"
+version = "0.4.6"
 source = "registry+https://github.com/rust-lang/crates.io-index"
-checksum = "3f6d24790a10a7af737693a3e8f1d03faef7e6ca0cc99aae5066f533766de545"
+checksum = "12bc5db7778a8e08ee255090861e0e7aac55e784300077e7ed293c7be36168b2"
 dependencies = [
  "futures-core",
  "tokio",
@@ -13058,7 +13093,21 @@ source = "registry+https://github.com/rust-lang/crates.io-index"
 checksum = "adbc64cba7137545b8044cb1fe9814f7aacf3c6b5f9b45be8bb5db538befdb26"
 dependencies = [
  "js-sys",
- "opentelemetry",
+ "opentelemetry 0.32.0",
+ "tracing",
+ "tracing-core",
+ "tracing-subscriber",
+ "web-time",
+]
+
+[[package]]
+name = "tracing-opentelemetry"
+version = "0.34.0"
+source = "registry+https://github.com/rust-lang/crates.io-index"
+checksum = "0a904802a1b902f43638b677ff2a650847e3b4404101b6c586d648e8c1e3e8fe"
+dependencies = [
+ "js-sys",
+ "opentelemetry 0.33.0",
  "smallvec",
  "tracing",
  "tracing-core",
@@ -13416,9 +13465,9 @@ dependencies = [
 
 [[package]]
 name = "wasm-bindgen"
-version = "0.2.128"
+version = "0.2.129"
 source = "registry+https://github.com/rust-lang/crates.io-index"
-checksum = "aecb87a33d3b0c5e3b7aa46336eaf486cffafbd281b195e4c8b80d50df2351bf"
+checksum = "9bb54f33acc68fd454578d9820b0bde1a1a3d17aa17bb7b6595806d02886d409"
 dependencies = [
  "cfg-if",
  "once_cell",
@@ -13429,19 +13478,20 @@ dependencies = [
 
 [[package]]
 name = "wasm-bindgen-futures"
-version = "0.4.78"
+version = "0.4.79"
 source = "registry+https://github.com/rust-lang/crates.io-index"
-checksum = "6ef4c5d3d2cdf5c54f4231181768f5510842e350db025faf1f7163b1030ed928"
+checksum = "3cbab34de2d982e9b48e18d216d04c4a6f641066ff19ffb699980f591ee3610e"
 dependencies = [
  "js-sys",
+ "tokio",
  "wasm-bindgen",
 ]
 
 [[package]]
 name = "wasm-bindgen-macro"
-version = "0.2.128"
+version = "0.2.129"
 source = "registry+https://github.com/rust-lang/crates.io-index"
-checksum = "a690d511e3c1a8b3a55e33511e3c2c00c78415cd23650f32b808627f5696b9ed"
+checksum = "2e29d0c35b16e224a7eeb5cd2d25e3e1968fbd65604117b44d3b789d00ee8535"
 dependencies = [
  "quote",
  "wasm-bindgen-macro-support",
@@ -13449,9 +13499,9 @@ dependencies = [
 
 [[package]]
 name = "wasm-bindgen-macro-support"
-version = "0.2.128"
+version = "0.2.129"
 source = "registry+https://github.com/rust-lang/crates.io-index"
-checksum = "411e4887f0071ef2d2164a9d5fdf2d20efbef78fccd3a78b0c10a1dc5295e48a"
+checksum = "6f501a8bc3719dba86ef8ae4728879c08001bea749eb1333ac5b91e040e2a6b7"
 dependencies = [
  "bumpalo",
  "proc-macro2",
@@ -13462,9 +13512,9 @@ dependencies = [
 
 [[package]]
 name = "wasm-bindgen-shared"
-version = "0.2.128"
+version = "0.2.129"
 source = "registry+https://github.com/rust-lang/crates.io-index"
-checksum = "81941cd78d0c92026c33e5e01312845a4cb1e9af3407f9134b100dd03144103e"
+checksum = "23f0c9c52aa7cd7d77769a4cfe2a9adb1b331f489a41d912ce14513d5ab995c6"
 dependencies = [
  "unicode-ident",
 ]
@@ -13484,9 +13534,9 @@ dependencies = [
 
 [[package]]
 name = "web-sys"
-version = "0.3.105"
+version = "0.3.106"
 source = "registry+https://github.com/rust-lang/crates.io-index"
-checksum = "9fbddc4a036f00ec4f18c83445bd3115cb306a91da554919a099d9222fe4a7f8"
+checksum = "88261b9deccee56594c11a3460c462c41f58d148598fe70ad77070126a68aba4"
 dependencies = [
  "js-sys",
  "wasm-bindgen",
@@ -14025,9 +14075,9 @@ dependencies = [
 
 [[package]]
 name = "yoke-derive"
-version = "0.8.3"
+version = "0.8.4"
 source = "registry+https://github.com/rust-lang/crates.io-index"
-checksum = "33811428bee40dbceb6d545e95754741d17a6aef9a4849f0fd62e2ba4f412a78"
+checksum = "ec8ebde2db3681e8c9980cc27822030e68752690ddfa9473e739aeb4dbde6d71"
 dependencies = [
  "proc-macro2",
  "quote",
@@ -14037,18 +14087,18 @@ dependencies = [
 
 [[package]]
 name = "zerocopy"
-version = "0.8.58"
+version = "0.8.59"
 source = "registry+https://github.com/rust-lang/crates.io-index"
-checksum = "c17e8fafad82b542ff3717217ecdc736231b59e387768c9630123b4ce4d2db44"
+checksum = "6df92bf3d9227be3d53173901ddbffac2babc27ae50f397776ffd6dc33f800cb"
 dependencies = [
  "zerocopy-derive",
 ]
 
 [[package]]
 name = "zerocopy-derive"
-version = "0.8.58"
+version = "0.8.59"
 source = "registry+https://github.com/rust-lang/crates.io-index"
-checksum = "595f56e044df4f46a0c9a626f65c3d99eb8488f7e8a8baa12dd76326d9710bf2"
+checksum = "ac4f328cf2f05d084e496c3e9c3f33ed0a183656a16e1fcec4d464d8373aec82"
 dependencies = [
  "proc-macro2",
  "quote",

Cargo.toml

@@ -160,7 +160,7 @@ pulsar = { default-features = false, version = "6.9.0" }
 lapin = { default-features = false, version = "4.12.0" }
 hyper = { version = "1.11.1" }
 hyper-rustls = { default-features = false, version = "0.27.10" }
-hyper-util = { version = "0.1.20" }
+hyper-util = { version = "0.1.21" }
 http = "1.5.0"
 http-body = "1.1.0"
 http-body-util = "0.1.5"
@@ -172,7 +172,7 @@ socket2 = { version = "0.6.5" }
 tokio = { version = "1.53.1" }
 tokio-rustls = { default-features = false, version = "0.26.5" }
 tokio-stream = { version = "0.1.19" }
-tokio-test = "0.4.5"
+tokio-test = "0.4.6"
 tokio-util = { version = "0.7.19" }
 tonic = { version = "0.14.6" }
 tonic-prost = { version = "0.14.6" }
@@ -245,13 +245,13 @@ atoi = "3.1.0"
 atomic_enum = "0.3.0"
 aws-config = { version = "1.12.0" }
 aws-credential-types = { version = "1.3.0" }
-aws-sdk-kms = { default-features = false, version = "1.121.0" }
-aws-sdk-s3 = { default-features = false, version = "1.149.0" }
-aws-sdk-sts = { default-features = false, version = "1.117.0" }
+aws-sdk-kms = { default-features = false, version = "1.122.0" }
+aws-sdk-s3 = { default-features = false, version = "1.150.0" }
+aws-sdk-sts = { default-features = false, version = "1.118.0" }
 aws-smithy-async = { version = "1.3.0" }
 aws-smithy-http-client = { default-features = false, version = "1.4.2" }
-aws-smithy-runtime-api = { version = "1.17.0" }
-aws-smithy-types = { version = "1.8.0" }
+aws-smithy-runtime-api = { version = "1.18.0" }
+aws-smithy-types = { version = "1.8.1" }
 base64-simd = "0.8.0"
 brotli = "9.0.0"
 clap = { version = "4.6.7" }
@@ -269,8 +269,8 @@ enumset = "1.1.14"
 faster-hex = "1.0.0"
 flate2 = "1.1.10"
 glob = "0.3.4"
-google-cloud-storage = "1.18.0"
-google-cloud-auth = "1.16.0"
+google-cloud-storage = "1.19.0"
+google-cloud-auth = "1.17.0"
 hashbrown = { version = "0.17.1" }
 # Base32 for RFC 6238 TOTP shared secrets (RFC 4648 unpadded, the alphabet
 # every authenticator app expects). Already in the graph transitively.
@@ -293,6 +293,7 @@ moka = { version = "0.12.16" }
 netif = "0.1.6"
 num_cpus = { version = "1.17.0" }
 nvml-wrapper = "0.13.0"
+object = { version = "=0.40.0", default-features = false }
 parking_lot = "0.12.5"
 path-absolutize = "4.0.1"
 percent-encoding = "2.3.2"
@@ -309,7 +310,7 @@ rustfs-erasure-codec = { version = "9.0.0" }
 reed-solomon-simd = "3.1.0"
 regex = { version = "1.13.1" }
 rumqttc = { package = "rumqttc-next", version = "0.34.0" }
-redis = { version = "1.7.0" }
+redis = { version = "1.7.1" }
 rustify = { version = "0.7", default-features = false }
 rustix = { version = "1.1.5" }
 rust-embed = { version = "8.12.0" }
@@ -318,8 +319,8 @@ s3s = { version = "0.17.0", features = ["minio"] }
 s3s-sigv4 = "0.17.0"
 serial_test = "4.0.1"
 shadow-rs = { default-features = false, version = "2.0.0" }
-siphasher = "1.0.3"
-smallvec = { version = "1.16.1" }
+siphasher = "1.0.4"
+smallvec = { version = "1.16.2" }
 compact_str = "0.10.0"
 snap = "1.1.2"
 starshard = { version = "2.3.0" }
@@ -333,7 +334,7 @@ tracing = { version = "0.1.44" }
 tracing-appender = "0.2.5"
 tracing-core = "0.1.36"
 tracing-error = "0.2.1"
-tracing-opentelemetry = { version = "0.33" }
+tracing-opentelemetry = { version = "0.34" }
 tracing-subscriber = { version = "0.3.23" }
 transform-stream = "0.3.1"
 url = "2.5.8"
@@ -353,19 +354,19 @@ zstd = "0.14.0"
 metrics = "0.24.6"
 metrics-util = "0.20"
 dial9 = { version = "0.5.2", default-features = false }
-opentelemetry = { version = "0.32.0" }
-opentelemetry-appender-tracing = { version = "0.32.0" }
-opentelemetry-otlp = { version = "0.32.0" }
-opentelemetry-proto = { version = "0.32.0", default-features = false, features = ["metrics", "gen-tonic-messages"] }
-opentelemetry_sdk = { version = "0.32.1" }
-opentelemetry-semantic-conventions = { version = "0.32.1" }
-opentelemetry-stdout = { version = "0.32.0" }
+opentelemetry = { version = "0.33.0" }
+opentelemetry-appender-tracing = { version = "0.33.0" }
+opentelemetry-otlp = { version = "0.33.0" }
+opentelemetry-proto = { version = "0.33.0", default-features = false, features = ["metrics", "gen-tonic-messages"] }
+opentelemetry_sdk = { version = "0.33.0" }
+opentelemetry-semantic-conventions = { version = "0.33.0" }
+opentelemetry-stdout = { version = "0.33.0" }
 pyroscope = { version = "2.1.1" }
 
 # FTP and SFTP
-libunftp = { version = "0.23.0" }
+libunftp = { version = "0.23.1" }
 unftp-core = "0.1.0"
-suppaftp = { version = "12.0.1" }
+suppaftp = { version = "12.1.0" }
 rcgen = { version = "0.14.10", default-features = false, features = ["aws_lc_rs", "crypto", "pem"] }
 russh = { version = "0.63.3" }
 russh-sftp = "3.0.0"
@@ -376,7 +377,7 @@ dav-server = "0.11.0"
 # Performance Analysis and Memory Profiling
 rustfs-mimalloc = { version = "0.5.6" }
 # Preserve Unicode focus filters until rustfs/backlog#2302 is resolved.
-hotpath = { version = "0.26.0", default-features = false }
+hotpath = { version = "0.26.1", default-features = false }
 # Snapshot testing for output format regression detection
 insta = { version = "1.48" }
 

Dockerfile

@@ -106,7 +106,9 @@ RUN chmod +x /usr/bin/rustfs /entrypoint.sh
 
 RUN addgroup -g 10001 -S rustfs && \
     adduser -u 10001 -G rustfs -S rustfs -D && \
-    mkdir -p /data /logs && \
+    mkdir -p /data/.rustfs/events /data/.rustfs/audit /logs /opt/rustfs && \
+    ln -s /data/.rustfs/events /opt/rustfs/events && \
+    ln -s /data/.rustfs/audit /opt/rustfs/audit && \
     chown -R rustfs:rustfs /data /logs && \
     chmod 0750 /data /logs
 

Dockerfile.decommission-local

@@ -44,7 +44,9 @@ RUN set -eux; \
 WORKDIR /app
 
 RUN set -eux; \
-    mkdir -p /data /logs; \
+    mkdir -p /data/.rustfs/events /data/.rustfs/audit /logs /opt/rustfs; \
+    ln -s /data/.rustfs/events /opt/rustfs/events; \
+    ln -s /data/.rustfs/audit /opt/rustfs/audit; \
     chown -R rustfs:rustfs /data /logs /app; \
     chmod 0750 /data /logs
 

Dockerfile.glibc

@@ -110,7 +110,9 @@ RUN chmod +x /usr/bin/rustfs /entrypoint.sh
 
 RUN groupadd -g 10001 rustfs && \
     useradd -u 10001 -g rustfs -m -s /sbin/nologin rustfs && \
-    mkdir -p /data /logs && \
+    mkdir -p /data/.rustfs/events /data/.rustfs/audit /logs /opt/rustfs && \
+    ln -s /data/.rustfs/events /opt/rustfs/events && \
+    ln -s /data/.rustfs/audit /opt/rustfs/audit && \
     chown -R rustfs:rustfs /data /logs && \
     chmod 0750 /data /logs
 

Dockerfile.source

@@ -240,7 +240,9 @@ WORKDIR /app
 
 # Prepare data/log directories with sane defaults
 RUN set -eux; \
-    mkdir -p /data /logs; \
+    mkdir -p /data/.rustfs/events /data/.rustfs/audit /logs /opt/rustfs; \
+    ln -s /data/.rustfs/events /opt/rustfs/events; \
+    ln -s /data/.rustfs/audit /opt/rustfs/audit; \
     chown -R rustfs:rustfs /data /logs /app; \
     chmod 0750 /data /logs
 

README.md

@@ -195,7 +195,7 @@ docker run -d --name rustfs -p 9000:9000 \
 Notes:
 - `RUSTFS_NOTIFY_ENABLE=true` enables the global notify module switch.
 - For ARN `arn:rustfs:sqs::primary:webhook`, use instance-scoped env vars with `_PRIMARY`.
-- If queue dir is omitted, default is `/opt/rustfs/events`; ensure it is writable by the container runtime user.
+- If queue dir is omitted, the official image maps `/opt/rustfs/events` into the persistent `/data` volume so it is writable by the runtime user and pending events survive container recreation. Override `queue_dir` when using another deployment layout.
 - `RUSTFS_NOTIFY_WEBHOOK_SKIP_TLS_VERIFY_PRIMARY` defaults to `false`; enabling it skips webhook TLS certificate verification, allows MITM attacks, and emits a startup warning. Prefer `RUSTFS_NOTIFY_WEBHOOK_CLIENT_CA_PRIMARY` for private CAs.
 - Since `1.0.0-beta.11`, webhook endpoints on private or container networks
   (`Docker Compose service names`, `host.docker.internal`, RFC 1918 addresses) are

crates/common/src/mrf_channel.rs

@@ -163,6 +163,15 @@ pub struct MrfDurableRepairAnchor {
     pub bucket_incarnation_id: Uuid,
 }
 
+/// A completed deep check proved that a legacy object is present and healthy,
+/// but the format has no independent identity commitment that can discharge a
+/// durable partial-write responsibility. Keep the journal entry and pause
+/// automatic retries until an operator has repaired the source of truth.
+#[derive(Debug, Clone, PartialEq, Eq)]
+pub struct MrfUnverifiedLegacyEvent {
+    pub anchor: MrfDurableRepairAnchor,
+}
+
 impl MrfDurableRepairAnchor {
     /// Build a dischargeable anchor only when the caller supplies the storage
     /// incarnation and the original ingress lease. Legacy replay records lack
@@ -324,6 +333,9 @@ pub enum MrfDurableAdmissionError {
 /// cancel repair.
 pub struct MrfDurableSubmission {
     pub intent: MrfIntent,
+    /// Bucket incarnation observed by the committed-write caller, when
+    /// available. Old journal records and legacy callers remain unbound.
+    pub source_bucket_incarnation_id: Option<Uuid>,
     pub response: oneshot::Sender<Result<(), MrfDurableAdmissionError>>,
 }
 
@@ -344,6 +356,19 @@ pub async fn persist_partial_write_intent(
     version_id: Option<Uuid>,
     scope: MrfScope,
 ) -> Result<(), MrfDurableAdmissionError> {
+    persist_partial_write_intent_with_incarnation(bucket, object, version_id, scope, None).await
+}
+
+pub async fn persist_partial_write_intent_with_incarnation(
+    bucket: &str,
+    object: &str,
+    version_id: Option<Uuid>,
+    scope: MrfScope,
+    source_bucket_incarnation_id: Option<Uuid>,
+) -> Result<(), MrfDurableAdmissionError> {
+    if source_bucket_incarnation_id.is_some_and(|incarnation| incarnation.is_nil()) {
+        return Err(MrfDurableAdmissionError::InvalidIdentity);
+    }
     let intent = MrfIntent {
         bucket: Arc::from(bucket),
         object: Arc::from(object),
@@ -355,7 +380,7 @@ pub async fn persist_partial_write_intent(
         enqueued_at_ms: unix_now_ms(),
         attempts: 0,
     };
-    persist_durable_intent(intent).await
+    persist_durable_intent(intent, source_bucket_incarnation_id).await
 }
 
 pub async fn persist_delete_marker_purge_intent(
@@ -368,6 +393,7 @@ pub async fn persist_delete_marker_purge_intent(
     if version_id.is_nil() {
         return Err(MrfDurableAdmissionError::InvalidIdentity);
     }
+    let source_bucket_incarnation_id = delete_marker_purge.bucket_incarnation_id;
     let intent = MrfIntent {
         bucket: Arc::from(bucket),
         object: Arc::from(object),
@@ -379,17 +405,23 @@ pub async fn persist_delete_marker_purge_intent(
         enqueued_at_ms: unix_now_ms(),
         attempts: 0,
     };
-    persist_durable_intent_unconditionally(intent).await
+    persist_durable_intent_unconditionally(intent, Some(source_bucket_incarnation_id)).await
 }
 
-async fn persist_durable_intent(intent: MrfIntent) -> Result<(), MrfDurableAdmissionError> {
+async fn persist_durable_intent(
+    intent: MrfIntent,
+    source_bucket_incarnation_id: Option<Uuid>,
+) -> Result<(), MrfDurableAdmissionError> {
     if !mrf_delivery_enabled() {
         return Err(MrfDurableAdmissionError::Disabled);
     }
-    persist_durable_intent_unconditionally(intent).await
+    persist_durable_intent_unconditionally(intent, source_bucket_incarnation_id).await
 }
 
-async fn persist_durable_intent_unconditionally(mut intent: MrfIntent) -> Result<(), MrfDurableAdmissionError> {
+async fn persist_durable_intent_unconditionally(
+    mut intent: MrfIntent,
+    source_bucket_incarnation_id: Option<Uuid>,
+) -> Result<(), MrfDurableAdmissionError> {
     if intent.bucket.is_empty()
         || intent.object.is_empty()
         || intent.bucket.len() > MRF_MAX_IDENTITY_COMPONENT
@@ -404,7 +436,11 @@ async fn persist_durable_intent_unconditionally(mut intent: MrfIntent) -> Result
     }
     let (response, receipt) = oneshot::channel();
     sender
-        .try_send(MrfDurableSubmission { intent, response })
+        .try_send(MrfDurableSubmission {
+            intent,
+            source_bucket_incarnation_id,
+            response,
+        })
         .map_err(|err| match err {
             mpsc::error::TrySendError::Full(_) => MrfDurableAdmissionError::Full,
             mpsc::error::TrySendError::Closed(_) => MrfDurableAdmissionError::Unavailable,
@@ -782,6 +818,8 @@ const MRF_REPAIRED_EVENT_CAP: usize = 4096;
 static MRF_REPAIRED_EVENTS: OnceLock<std::sync::Mutex<std::collections::VecDeque<MrfRepairedEvent>>> = OnceLock::new();
 static MRF_VERIFIED_REPAIR_EVENTS: OnceLock<std::sync::Mutex<std::collections::VecDeque<MrfVerifiedRepairEvent>>> =
     OnceLock::new();
+static MRF_UNVERIFIED_LEGACY_EVENTS: OnceLock<std::sync::Mutex<std::collections::VecDeque<MrfUnverifiedLegacyEvent>>> =
+    OnceLock::new();
 
 /// Record a legacy notification for compatibility. This is not an
 /// acknowledgement of storage verification or durable repair completion.
@@ -859,6 +897,44 @@ pub fn take_mrf_verified_repair_events_for(bucket: &str) -> Vec<MrfVerifiedRepai
     taken
 }
 
+/// Record a bounded, exact-identity notice that a legacy durable partial-write
+/// was checked but cannot be proven from its on-disk format. This is not a
+/// repair proof and must never release the durable responsibility.
+pub fn note_mrf_unverified_legacy(event: MrfUnverifiedLegacyEvent) {
+    let registry = MRF_UNVERIFIED_LEGACY_EVENTS.get_or_init(|| std::sync::Mutex::new(std::collections::VecDeque::new()));
+    let Ok(mut events) = registry.lock() else {
+        return;
+    };
+    if events.contains(&event) {
+        return;
+    }
+    if events.len() >= MRF_REPAIRED_EVENT_CAP {
+        events.pop_front();
+    }
+    events.push_back(event);
+}
+
+/// Take held-legacy notices for one bucket. The durable ledger still verifies
+/// the full anchor before changing the journal entry to its held state.
+pub fn take_mrf_unverified_legacy_events_for(bucket: &str) -> Vec<MrfUnverifiedLegacyEvent> {
+    let Some(registry) = MRF_UNVERIFIED_LEGACY_EVENTS.get() else {
+        return Vec::new();
+    };
+    let Ok(mut events) = registry.lock() else {
+        return Vec::new();
+    };
+    let mut taken = Vec::new();
+    events.retain(|event| {
+        if event.anchor.bucket.as_ref() == bucket {
+            taken.push(event.clone());
+            false
+        } else {
+            true
+        }
+    });
+    taken
+}
+
 #[cfg(test)]
 mod tests {
     use super::*;
@@ -1233,6 +1309,53 @@ mod tests {
         assert!(take_mrf_verified_repair_events_for("verified-bucket-a").is_empty());
         assert_eq!(take_mrf_verified_repair_events_for("verified-bucket-b").len(), 1);
     }
+
+    #[test]
+    fn unverified_legacy_events_are_bucket_scoped_and_bounded() {
+        let bucket = Arc::<str>::from("unverified-legacy-event-bucket");
+        let _ = take_mrf_unverified_legacy_events_for(bucket.as_ref());
+        let event = MrfUnverifiedLegacyEvent {
+            anchor: MrfDurableRepairAnchor {
+                kind: MrfKind::PartialWrite,
+                bucket: bucket.clone(),
+                object: Arc::from("unverified-object"),
+                version_id: None,
+                scope: Some(MrfScope {
+                    pool_index: 1,
+                    set_index: 2,
+                }),
+                delete_marker_purge: None,
+                lease: MrfIngressLease::new(912),
+                bucket_incarnation_id: Uuid::new_v4(),
+            },
+        };
+        note_mrf_unverified_legacy(event.clone());
+        note_mrf_unverified_legacy(event.clone());
+        assert_eq!(take_mrf_unverified_legacy_events_for(bucket.as_ref()), vec![event]);
+        assert!(take_mrf_unverified_legacy_events_for(bucket.as_ref()).is_empty());
+
+        let cap_bucket = Arc::<str>::from("unverified-legacy-event-cap-bucket");
+        for index in 0..MRF_REPAIRED_EVENT_CAP + 4 {
+            note_mrf_unverified_legacy(MrfUnverifiedLegacyEvent {
+                anchor: MrfDurableRepairAnchor {
+                    kind: MrfKind::PartialWrite,
+                    bucket: cap_bucket.clone(),
+                    object: Arc::from(format!("object-{index}")),
+                    version_id: None,
+                    scope: Some(MrfScope {
+                        pool_index: 1,
+                        set_index: 2,
+                    }),
+                    delete_marker_purge: None,
+                    lease: MrfIngressLease::new(u64::try_from(index).expect("test index fits")),
+                    bucket_incarnation_id: Uuid::new_v4(),
+                },
+            });
+        }
+        let events = take_mrf_unverified_legacy_events_for(cap_bucket.as_ref());
+        assert_eq!(events.len(), MRF_REPAIRED_EVENT_CAP);
+        assert_eq!(events.first().expect("bounded ring is non-empty").anchor.object.as_ref(), "object-4");
+    }
 }
 #[tokio::test]
 async fn partial_write_durable_admission_waits_for_persistence_and_renews_lease() {

crates/connect-symbol-catalog/Cargo.toml

@@ -8,7 +8,7 @@ publish = false
 
 [dependencies]
 hex-simd.workspace = true
-object = { version = "=0.39.1", default-features = false, features = ["read", "std", "elf"] }
+object = { workspace = true, features = ["read", "std", "elf"] }
 serde = { workspace = true, features = ["derive"] }
 serde_json.workspace = true
 sha2.workspace = true
@@ -16,7 +16,7 @@ sha2.workspace = true
 symbolic-demangle = { version = "=13.9.0", default-features = false, features = ["rust"] }
 
 [dev-dependencies]
-object = { version = "=0.39.1", default-features = false, features = ["write", "elf"] }
+object = { workspace = true, features = ["write", "elf"] }
 
 [lints]
 workspace = true

crates/e2e_test/src/chaos.rs

@@ -384,17 +384,17 @@ pub(crate) fn is_cluster_heal_coordination_unavailable(error: &(dyn std::error::
     message.contains("500 Internal Server Error") && message.contains("cluster heal coordination unavailable")
 }
 
-/// Wait for the restarted cluster to admit the first root heal request.
+/// Wait for the restarted cluster to admit a root heal request and return its response.
 pub(crate) async fn start_root_heal_when_control_ready(
     heal_url: &str,
     heal_body: &str,
     access_key: &str,
     secret_key: &str,
-) -> ChaosResult<()> {
+) -> ChaosResult<String> {
     let deadline = tokio::time::Instant::now() + std::time::Duration::from_secs(45);
     loop {
         match signed_admin_post(heal_url, Some(heal_body), access_key, secret_key).await {
-            Ok(_) => return Ok(()),
+            Ok(body) => return Ok(body),
             Err(error) if is_cluster_heal_coordination_unavailable(error.as_ref()) && tokio::time::Instant::now() < deadline => {
                 tokio::time::sleep(std::time::Duration::from_millis(250)).await;
             }

crates/e2e_test/src/common.rs

@@ -22,6 +22,8 @@
 //! - Common test constants and utilities
 
 use aws_sdk_s3::config::{Credentials, Region};
+use aws_sdk_s3::error::{ProvideErrorMetadata, SdkError};
+use aws_sdk_s3::operation::list_buckets::ListBucketsError;
 use aws_sdk_s3::{Client, Config};
 use aws_smithy_http_client::Builder as SmithyHttpClientBuilder;
 use http::header::{CONTENT_TYPE, HOST};
@@ -61,6 +63,41 @@ const TEST_PORT_COUNTER_PATH: &str = "/tmp/rustfs_e2e_next_port";
 const TEST_PORT_LOCK_DIR: &str = "/tmp/rustfs_e2e_port_allocator.lock";
 const TEST_PORT_LOCK_STALE_AFTER: Duration = Duration::from_secs(30);
 
+fn list_buckets_readiness_error(err: &SdkError<ListBucketsError>) -> String {
+    // SDK Display reports only the category. Do not expose raw response bodies or headers.
+    let service_code = err.as_service_error().and_then(|error| error.code()).filter(|code| {
+        matches!(
+            *code,
+            "ServiceUnavailable"
+                | "ServerNotInitialized"
+                | "InternalError"
+                | "AccessDenied"
+                | "InvalidAccessKeyId"
+                | "SignatureDoesNotMatch"
+        )
+    });
+    let transport = match err {
+        SdkError::DispatchFailure(failure) if failure.is_timeout() => Some("timeout"),
+        SdkError::DispatchFailure(failure) if failure.is_io() => Some("io"),
+        SdkError::DispatchFailure(failure) if failure.is_user() => Some("user"),
+        SdkError::DispatchFailure(failure) if failure.is_other() => Some("other"),
+        _ => None,
+    };
+    let mut source: Option<&(dyn std::error::Error + 'static)> = Some(err);
+    let mut io_error = None;
+    while let Some(error) = source {
+        if let Some(error) = error.downcast_ref::<std::io::Error>() {
+            io_error = Some((error.kind(), error.raw_os_error()));
+            break;
+        }
+        source = error.source();
+    }
+    format!(
+        "{err}; http_status={:?}; service_code={service_code:?}; transport={transport:?}; io_error={io_error:?}",
+        err.raw_response().map(|response| response.status().as_u16())
+    )
+}
+
 fn capture_log_path(log_dir: &Path, temp_dir: &str) -> Option<PathBuf> {
     let temp_name = Path::new(temp_dir).file_name()?.to_string_lossy();
     Some(log_dir.join(format!("{temp_name}.log")))
@@ -1575,8 +1612,9 @@ impl RustFSTestClusterEnvironment {
     ///
     /// Verifies service availability by calling the S3 `list_buckets` API against the requested node,
     /// retries up to 120 times with a 1-second interval between attempts.
-    async fn wait_for_node_service_ready(&self, node_idx: usize) -> Result<(), Box<dyn std::error::Error + Send + Sync>> {
+    async fn wait_for_node_service_ready(&mut self, node_idx: usize) -> Result<(), Box<dyn std::error::Error + Send + Sync>> {
         let client = self.create_s3_client(node_idx)?;
+        let mut last_error = None;
 
         for attempt in 0..120 {
             match client.list_buckets().send().await {
@@ -1584,13 +1622,28 @@ impl RustFSTestClusterEnvironment {
                     info!("Cluster node {} service ready after {} attempts", node_idx, attempt + 1);
                     return Ok(());
                 }
-                Err(_) => {
+                Err(err) => {
+                    last_error = Some(err);
                     sleep(Duration::from_secs(1)).await;
                 }
             }
         }
 
-        Err(format!("Cluster node {} service failed to become ready", node_idx).into())
+        let last_error = last_error.as_ref().map(list_buckets_readiness_error);
+        // Observe liveness only after the existing readiness budget is exhausted, before cleanup.
+        let process_status = match self.nodes[node_idx].process.as_mut() {
+            Some(process) => match process.try_wait() {
+                Ok(Some(status)) => format!("exited ({status})"),
+                Ok(None) => "running".to_string(),
+                Err(error) => format!("unavailable (kind={:?}, os_code={:?})", error.kind(), error.raw_os_error()),
+            },
+            None => "not started".to_string(),
+        };
+        Err(format!(
+            "Cluster node {node_idx} service failed to become ready; last ListBuckets error={last_error:?}; process_status={process_status}; capture_log_path={:?}",
+            self.node_capture_log_paths[node_idx]
+        )
+        .into())
     }
 
     /// Create an S3 client configured to communicate with a specific cluster node.
@@ -2101,6 +2154,34 @@ mod tests {
         verify_awscurl_path(&executable).expect("an available client with a working help command should pass");
     }
 
+    #[test]
+    fn readiness_error_reports_transport_without_sensitive_source_text() {
+        use aws_sdk_s3::error::ConnectorError;
+
+        let sensitive = "request credentials must not appear in diagnostics";
+        for (connector, category) in [
+            (
+                ConnectorError::io(std::io::Error::new(ErrorKind::ConnectionReset, sensitive).into()),
+                "io",
+            ),
+            (ConnectorError::timeout(sensitive.into()), "timeout"),
+            (ConnectorError::user(sensitive.into()), "user"),
+            (ConnectorError::other(sensitive.into(), None), "other"),
+        ] {
+            let diagnostic = list_buckets_readiness_error(&SdkError::dispatch_failure(connector));
+            assert!(diagnostic.contains(&format!("transport=Some(\"{category}\")")), "{diagnostic}");
+            assert!(diagnostic.contains("http_status=None"), "{diagnostic}");
+            assert!(!diagnostic.contains(sensitive), "{diagnostic}");
+            if category == "io" {
+                assert!(diagnostic.contains("io_error=Some((ConnectionReset, None))"), "{diagnostic}");
+            }
+        }
+        let os_error = std::io::Error::from_raw_os_error(13);
+        let expected = format!("io_error=Some(({:?}, Some(13)))", os_error.kind());
+        let diagnostic = list_buckets_readiness_error(&SdkError::dispatch_failure(ConnectorError::io(os_error.into())));
+        assert!(diagnostic.contains(&expected), "{diagnostic}");
+    }
+
     #[test]
     fn capture_log_path_uses_temp_directory_basename() {
         assert_eq!(

crates/e2e_test/src/delete_authorization_test.rs

@@ -282,7 +282,7 @@ async fn sdk_list_bucket_and_list_bucket_versions_permissions_are_independent()
 }
 
 #[tokio::test]
-async fn console_admin_force_delete_removes_prefix_versions_and_delete_markers() -> TestResult {
+async fn explicit_force_delete_permission_removes_prefix_versions_and_delete_markers() -> TestResult {
     init_logging();
     let mut env = RustFSTestEnvironment::new().await?;
     env.start_rustfs_server(vec![]).await?;
@@ -300,7 +300,14 @@ async fn console_admin_force_delete_removes_prefix_versions_and_delete_markers()
     put(&root, bucket, "folder-sibling/keep.txt").await?;
     let keep = versions(&root, bucket, "keep.txt").await?;
     let sibling = versions(&root, bucket, "folder-sibling/").await?;
-    let user = policy_user(&env, "consoleAdmin", None).await?;
+    let user = policy_user(
+        &env,
+        "explicit-force-delete",
+        Some(json!({"Version":"2012-10-17","Statement":[
+            {"Effect":"Allow","Action":["s3:DeleteObject","s3:DeleteObjectVersion","s3:ForceDeleteObject"],"Resource":format!("arn:aws:s3:::{bucket}/*")}
+        ]})),
+    )
+    .await?;
 
     force_delete(&user, bucket, "folder/").await?;
     assert!(
@@ -322,6 +329,126 @@ async fn console_admin_force_delete_removes_prefix_versions_and_delete_markers()
     Ok(())
 }
 
+#[tokio::test]
+async fn force_delete_header_requires_explicit_permission() -> TestResult {
+    init_logging();
+    let mut env = RustFSTestEnvironment::new().await?;
+    env.start_rustfs_server(vec![]).await?;
+    let root = env.create_s3_client();
+    let admin = policy_user(&env, "consoleAdmin", None).await?;
+
+    let owner_bucket = "force-owner-bucket";
+    root.create_bucket().bucket(owner_bucket).send().await?;
+    put(&root, owner_bucket, "kept-until-force.txt").await?;
+    force_delete_bucket(&root, owner_bucket, "x-rustfs-force-delete").await?;
+    assert!(
+        head_bucket_missing(&root, owner_bucket).await?,
+        "root force-delete must remove a non-empty bucket"
+    );
+
+    let denied_bucket = "force-admin-denied";
+    root.create_bucket().bucket(denied_bucket).send().await?;
+    put(&root, denied_bucket, "folder/child.txt").await?;
+    put(&root, denied_bucket, "plain.txt").await?;
+    assert_boxed_denied(force_delete(&admin, denied_bucket, "folder/").await);
+    assert!(
+        !versions(&root, denied_bucket, "folder/").await?.is_empty(),
+        "consoleAdmin without s3:ForceDeleteObject must not recursively delete"
+    );
+    assert_boxed_denied(force_delete_bucket(&admin, denied_bucket, "x-rustfs-force-delete").await);
+    assert_boxed_denied(force_delete_bucket(&admin, denied_bucket, "x-minio-force-delete").await);
+    assert!(
+        !head_bucket_missing(&root, denied_bucket).await?,
+        "consoleAdmin without s3:ForceDeleteBucket must not force-delete a non-empty bucket"
+    );
+    let not_empty = admin.delete_bucket().bucket(denied_bucket).send().await;
+    let not_empty = not_empty.expect_err("plain delete of a non-empty bucket must fail");
+    assert_eq!(
+        not_empty.as_service_error().and_then(|error| error.code()),
+        Some("BucketNotEmpty"),
+        "plain delete must keep the emptiness check, got {not_empty:?}"
+    );
+    admin.delete_object().bucket(denied_bucket).key("plain.txt").send().await?;
+    assert!(
+        versions(&root, denied_bucket, "plain.txt").await?.is_empty(),
+        "plain DeleteObject without the force header must stay allowed for consoleAdmin"
+    );
+
+    let object_bucket = "force-explicit-object";
+    root.create_bucket().bucket(object_bucket).send().await?;
+    put(&root, object_bucket, "folder/child.txt").await?;
+    put(&root, object_bucket, "keep.txt").await?;
+    let keep = versions(&root, object_bucket, "keep.txt").await?;
+    let object_user = policy_user(
+        &env,
+        "explicit-object-force",
+        Some(json!({"Version":"2012-10-17","Statement":[
+            {"Effect":"Allow","Action":["s3:DeleteObject","s3:DeleteObjectVersion","s3:ForceDeleteObject"],"Resource":format!("arn:aws:s3:::{object_bucket}/*")}
+        ]})),
+    )
+    .await?;
+    force_delete(&object_user, object_bucket, "folder/").await?;
+    assert!(versions(&root, object_bucket, "folder/").await?.is_empty());
+    assert_eq!(versions(&root, object_bucket, "keep.txt").await?, keep);
+
+    let allowed_bucket = "force-explicit-bucket";
+    root.create_bucket().bucket(allowed_bucket).send().await?;
+    put(&root, allowed_bucket, "inside.txt").await?;
+    let bucket_user = policy_user(
+        &env,
+        "explicit-bucket-force",
+        Some(json!({"Version":"2012-10-17","Statement":[
+            {"Effect":"Allow","Action":["s3:DeleteBucket","s3:ForceDeleteBucket"],"Resource":[
+                format!("arn:aws:s3:::{allowed_bucket}"), format!("arn:aws:s3:::{allowed_bucket}/*")
+            ]}
+        ]})),
+    )
+    .await?;
+    force_delete_bucket(&bucket_user, allowed_bucket, "x-minio-force-delete").await?;
+    assert!(
+        head_bucket_missing(&root, allowed_bucket).await?,
+        "an explicit s3:ForceDeleteBucket grant must remove a non-empty bucket"
+    );
+    Ok(())
+}
+
+async fn force_delete_bucket(
+    client: &Client,
+    bucket: &str,
+    header: &str,
+) -> Result<(), Box<SdkError<aws_sdk_s3::operation::delete_bucket::DeleteBucketError>>> {
+    let header: &'static str = match header {
+        "x-rustfs-force-delete" => "x-rustfs-force-delete",
+        "x-minio-force-delete" => "x-minio-force-delete",
+        other => panic!("unexpected force-delete header {other}"),
+    };
+    client
+        .delete_bucket()
+        .bucket(bucket)
+        .customize()
+        .mutate_request(move |request| {
+            request.headers_mut().insert(header, "true");
+        })
+        .send()
+        .await
+        .map(|_| ())
+        .map_err(Box::new)
+}
+
+async fn head_bucket_missing(client: &Client, bucket: &str) -> TestResult<bool> {
+    match client.head_bucket().bucket(bucket).send().await {
+        Ok(_) => Ok(false),
+        Err(error) => {
+            let code = error.as_service_error().and_then(|service| service.code());
+            if matches!(code, Some("NotFound" | "NoSuchBucket")) {
+                Ok(true)
+            } else {
+                Err(error.into())
+            }
+        }
+    }
+}
+
 #[tokio::test]
 async fn force_delete_authorizes_only_its_path_scope_without_list_permissions() -> TestResult {
     init_logging();
@@ -339,7 +466,7 @@ async fn force_delete_authorizes_only_its_path_scope_without_list_permissions()
         &env,
         "delete-only",
         Some(json!({"Version":"2012-10-17","Statement":[
-            {"Effect":"Allow","Action":["s3:DeleteObject","s3:DeleteObjectVersion"],"Resource":[
+            {"Effect":"Allow","Action":["s3:DeleteObject","s3:DeleteObjectVersion","s3:ForceDeleteObject"],"Resource":[
                 format!("arn:aws:s3:::{bucket}/selected.txt"), format!("arn:aws:s3:::{bucket}/selected.txt/*")
             ]},
             {"Effect":"Deny","Action":["s3:DeleteObject","s3:DeleteObjectVersion"],"Resource":format!("arn:aws:s3:::{bucket}/selected.txt-sibling")}
@@ -376,7 +503,7 @@ async fn force_directory_delete_cannot_remove_an_unauthorized_colliding_parent()
         &env,
         "parent-denier",
         Some(json!({"Version":"2012-10-17","Statement":[
-            {"Effect":"Allow","Action":["s3:DeleteObject","s3:DeleteObjectVersion"],"Resource":format!("arn:aws:s3:::{bucket}/*")},
+            {"Effect":"Allow","Action":["s3:DeleteObject","s3:DeleteObjectVersion","s3:ForceDeleteObject"],"Resource":format!("arn:aws:s3:::{bucket}/*")},
             {"Effect":"Deny","Action":"s3:DeleteObjectVersion","Resource":format!("arn:aws:s3:::{bucket}/collision.txt"),
              "Condition":{"StringEquals":{"s3:VersionId":protected_parent_version}}}
         ]})),
@@ -409,7 +536,7 @@ async fn force_unversioned_directory_requires_only_delete_object() -> TestResult
         &env,
         "unversioned-deleter",
         Some(json!({"Version":"2012-10-17","Statement":[
-            {"Effect":"Allow","Action":"s3:DeleteObject","Resource":format!("arn:aws:s3:::{bucket}/*")},
+            {"Effect":"Allow","Action":["s3:DeleteObject","s3:ForceDeleteObject"],"Resource":format!("arn:aws:s3:::{bucket}/*")},
             {"Effect":"Deny","Action":"s3:DeleteObjectVersion","Resource":format!("arn:aws:s3:::{bucket}/*")}
         ]})),
     )
@@ -438,7 +565,7 @@ async fn force_delete_denied_child_preserves_every_object_despite_bucket_allow()
         &env,
         "child-denier",
         Some(json!({"Version":"2012-10-17","Statement":[
-            {"Effect":"Allow","Action":["s3:DeleteObject","s3:DeleteObjectVersion","s3:ReplicateDelete"],"Resource":format!("arn:aws:s3:::{bucket}/*")},
+            {"Effect":"Allow","Action":["s3:DeleteObject","s3:DeleteObjectVersion","s3:ForceDeleteObject","s3:ReplicateDelete"],"Resource":format!("arn:aws:s3:::{bucket}/*")},
             {"Effect":"Deny","Action":["s3:DeleteObject","s3:DeleteObjectVersion","s3:ReplicateDelete"],"Resource":format!("arn:aws:s3:::{bucket}/folder/z-denied.txt")}
         ]})),
     )
@@ -468,7 +595,7 @@ async fn force_delete_denied_child_preserves_every_object_despite_bucket_allow()
         &env,
         "replica-deleter",
         Some(json!({"Version":"2012-10-17","Statement":[
-            {"Effect":"Allow","Action":"s3:DeleteObject","Resource":format!("arn:aws:s3:::{bucket}/*")},
+            {"Effect":"Allow","Action":["s3:DeleteObject","s3:ForceDeleteObject"],"Resource":format!("arn:aws:s3:::{bucket}/*")},
             {"Effect":"Allow","Action":"s3:ReplicateDelete","Resource":format!("arn:aws:s3:::{bucket}/*")}
         ]})),
     )
@@ -500,7 +627,7 @@ async fn force_delete_denied_historical_version_preserves_versions_and_markers()
     put(&root, bucket, "folder/a-allowed.txt").await?;
     let policy = |version: &str| {
         json!({"Version":"2012-10-17","Statement":[
-            {"Effect":"Allow","Action":["s3:DeleteObject","s3:DeleteObjectVersion"],"Resource":format!("arn:aws:s3:::{bucket}/*")},
+            {"Effect":"Allow","Action":["s3:DeleteObject","s3:DeleteObjectVersion","s3:ForceDeleteObject"],"Resource":format!("arn:aws:s3:::{bucket}/*")},
             {"Effect":"Deny","Action":"s3:DeleteObjectVersion","Resource":format!("arn:aws:s3:::{bucket}/folder/*"),
              "Condition":{"StringEquals":{"s3:VersionId":version}}}
         ]})
@@ -628,7 +755,7 @@ async fn force_delete_checks_every_version_page_before_mutation() -> TestResult
         .try_collect::<Vec<_>>()
         .await?;
     put(&root, bucket, "folder/z-denied.txt").await?;
-    let allow = json!({"Effect":"Allow","Action":["s3:DeleteObject","s3:DeleteObjectVersion"],"Resource":format!("arn:aws:s3:::{bucket}/*")});
+    let allow = json!({"Effect":"Allow","Action":["s3:DeleteObject","s3:DeleteObjectVersion","s3:ForceDeleteObject"],"Resource":format!("arn:aws:s3:::{bucket}/*")});
     let user = policy_user(
         &env,
         "paged-deleter",

crates/e2e_test/src/distributed/harness.rs

@@ -34,6 +34,9 @@ use crate::common::{
 };
 use crate::replication_extension_test::LOOPBACK_REPLICATION_TARGET_ENV;
 use aws_sdk_s3::Client;
+use aws_sdk_s3::config::retry::RetryConfig;
+use aws_sdk_s3::error::{ProvideErrorMetadata, SdkError};
+use aws_sdk_s3::operation::{get_object::GetObjectError, put_object::PutObjectError};
 use aws_sdk_s3::primitives::ByteStream;
 use aws_sdk_s3::types::{BucketVersioningStatus, VersioningConfiguration};
 use http::{Method, StatusCode};
@@ -41,7 +44,7 @@ use sha2::{Digest, Sha256};
 use std::collections::BTreeMap;
 use std::path::{Path, PathBuf};
 use std::time::Duration;
-use tokio::time::{Instant, sleep};
+use tokio::time::{Instant, sleep, timeout_at};
 use uuid::Uuid;
 
 pub(crate) type TestResult<T = ()> = Result<T, Box<dyn std::error::Error + Send + Sync>>;
@@ -400,9 +403,9 @@ pub(crate) async fn put_inventory(
     Ok(inventory)
 }
 
-/// Retry only transport-level service availability failures while a data
-/// movement operation changes the pool map. Generic InternalError responses
-/// remain fatal because accepting them would hide server defects.
+/// Retry only explicit S3 availability responses while a data movement
+/// operation changes the pool map. Transport failures and InternalError
+/// responses remain fatal because callers already require a ready endpoint.
 pub(crate) async fn put_inventory_retrying(
     client: &Client,
     bucket: &str,
@@ -449,19 +452,48 @@ where
     let deadline = Instant::now() + timeout;
     let mut delay = Duration::from_millis(50);
     loop {
-        let last_error = match probe().await {
-            Ok(true) => return Ok(()),
-            Ok(false) => format!("{label} still false"),
-            Err(error) => error.to_string(),
-        };
         if Instant::now() >= deadline {
-            return Err(format!("{label} did not become true within {timeout:?}: {last_error}").into());
+            return Err(format!("{label} did not become true within {timeout:?}").into());
         }
-        sleep(delay).await;
+        // A pending request must consume the same budget as unsuccessful probes.
+        match timeout_at(deadline, probe()).await {
+            Err(_) => return Err(format!("{label} did not become true within {timeout:?}").into()),
+            Ok(Err(error)) => return Err(error),
+            Ok(Ok(true)) => return Ok(()),
+            Ok(Ok(false)) => {}
+        }
+        tokio::time::sleep_until((Instant::now() + delay).min(deadline)).await;
         delay = (delay * 2).min(Duration::from_secs(1));
     }
 }
 
+// Boxed SDK errors retain the operation's service code; Display only says
+// "service error" and cannot distinguish convergence from a server defect.
+pub(crate) fn s3_probe_error_code<'a>(error: &'a (dyn std::error::Error + Send + Sync + 'static)) -> Option<&'a str> {
+    error
+        .downcast_ref::<SdkError<GetObjectError>>()
+        .and_then(|error| error.as_service_error())
+        .and_then(ProvideErrorMetadata::code)
+        .or_else(|| {
+            error
+                .downcast_ref::<SdkError<PutObjectError>>()
+                .and_then(|error| error.as_service_error())
+                .and_then(ProvideErrorMetadata::code)
+        })
+}
+
+pub(crate) fn s3_probe_client(client: &Client) -> Client {
+    // The probe owns the retry policy. SDK retries must not hide a response
+    // that the probe would classify as fatal.
+    Client::from_conf(
+        client
+            .config()
+            .to_builder()
+            .retry_config(RetryConfig::standard().with_max_attempts(1))
+            .build(),
+    )
+}
+
 pub(crate) async fn cluster_admin(
     cluster: &RustFSTestClusterEnvironment,
     method: Method,
@@ -598,15 +630,20 @@ pub(crate) async fn wait_for_replicated_bytes(
     expected: &[u8],
     timeout: Duration,
 ) -> TestResult {
+    let client = s3_probe_client(client);
     wait_until(
         timeout,
         || async {
-            match get_object_bytes(client, bucket, key).await {
+            match get_object_bytes(&client, bucket, key).await {
                 Ok(got) if got.as_slice() == expected => Ok(true),
-                Ok(_) => Ok(false),
+                Ok(got) => Err(format!(
+                    "replicated object {bucket}/{key} bytes mismatch: expected sha256={} got sha256={}",
+                    sha256_hex(expected),
+                    sha256_hex(&got)
+                )
+                .into()),
                 Err(error) => {
-                    let message = error.to_string();
-                    if message.contains("NoSuchKey") || message.contains("NotFound") {
+                    if matches!(s3_probe_error_code(error.as_ref()), Some("NoSuchKey" | "NotFound")) {
                         Ok(false)
                     } else {
                         Err(error)
@@ -988,6 +1025,7 @@ pub(crate) async fn list_pools_json(cluster: &RustFSTestClusterEnvironment) -> T
 }
 
 pub(crate) async fn retrying_put(client: &Client, bucket: &str, key: &str, body: Vec<u8>, timeout: Duration) -> TestResult {
+    let client = s3_probe_client(client);
     wait_until(
         timeout,
         || {
@@ -999,8 +1037,7 @@ pub(crate) async fn retrying_put(client: &Client, bucket: &str, key: &str, body:
                 match put_object(&client, &bucket, &key, body).await {
                     Ok(()) => Ok(true),
                     Err(error) => {
-                        let message = error.to_string();
-                        if message.contains("SlowDown") || message.contains("ServiceUnavailable") || message.contains("503") {
+                        if matches!(s3_probe_error_code(error.as_ref()), Some("SlowDown" | "ServiceUnavailable")) {
                             Ok(false)
                         } else {
                             Err(error)
@@ -1021,19 +1058,20 @@ pub(crate) async fn retrying_get_equals(
     expected: &[u8],
     timeout: Duration,
 ) -> TestResult {
+    let client = s3_probe_client(client);
     wait_until(
         timeout,
         || async {
-            match get_object_bytes(client, bucket, key).await {
+            match get_object_bytes(&client, bucket, key).await {
                 Ok(got) if got.as_slice() == expected => Ok(true),
-                Ok(_) => Ok(false),
+                Ok(got) => Err(format!(
+                    "object {bucket}/{key} bytes mismatch: expected sha256={} got sha256={}",
+                    sha256_hex(expected),
+                    sha256_hex(&got)
+                )
+                .into()),
                 Err(error) => {
-                    let message = error.to_string();
-                    if message.contains("NoSuchKey")
-                        || message.contains("SlowDown")
-                        || message.contains("ServiceUnavailable")
-                        || message.contains("503")
-                    {
+                    if matches!(s3_probe_error_code(error.as_ref()), Some("SlowDown" | "ServiceUnavailable")) {
                         Ok(false)
                     } else {
                         Err(error)
@@ -1046,6 +1084,162 @@ pub(crate) async fn retrying_get_equals(
     .await
 }
 
+#[cfg(test)]
+mod retry_tests {
+    use super::*;
+    use crate::fake_s3_target::{FAKE_ACCESS_KEY, FAKE_SECRET_KEY, FakeS3Target, FaultAction, Operation, SeedMetadata};
+    use std::cell::Cell;
+
+    fn client(target: &FakeS3Target) -> Client {
+        Client::from_conf(build_test_s3_config(
+            target.endpoint(),
+            FAKE_ACCESS_KEY,
+            FAKE_SECRET_KEY,
+            None,
+            "distributed-retry-test",
+        ))
+    }
+
+    #[tokio::test]
+    async fn wait_until_preserves_first_error() {
+        let attempts = Cell::new(0);
+        let error = wait_until(
+            Duration::from_secs(5),
+            || {
+                attempts.set(attempts.get() + 1);
+                std::future::ready(if attempts.get() == 1 {
+                    Err(std::io::Error::new(std::io::ErrorKind::PermissionDenied, "permanent probe failure").into())
+                } else {
+                    Ok(true)
+                })
+            },
+            "permanent failure",
+        )
+        .await
+        .expect_err("a later success must not hide a permanent failure");
+        assert_eq!(attempts.get(), 1);
+        assert_eq!(
+            error
+                .downcast_ref::<std::io::Error>()
+                .expect("preserve the original error")
+                .kind(),
+            std::io::ErrorKind::PermissionDenied
+        );
+    }
+
+    #[tokio::test]
+    async fn wait_until_retries_explicit_pending_state() -> TestResult {
+        let attempts = Cell::new(0);
+        wait_until(
+            Duration::from_secs(5),
+            || {
+                attempts.set(attempts.get() + 1);
+                std::future::ready(Ok(attempts.get() == 2))
+            },
+            "eventual readiness",
+        )
+        .await?;
+        assert_eq!(attempts.get(), 2);
+        Ok(())
+    }
+
+    #[tokio::test]
+    async fn wait_until_bounds_a_pending_probe() {
+        let error = tokio::time::timeout(
+            Duration::from_secs(5),
+            wait_until(Duration::from_millis(10), std::future::pending, "hung probe"),
+        )
+        .await
+        .expect("the probe deadline must finish before the test watchdog")
+        .expect_err("a permanently pending probe must time out");
+        assert!(error.to_string().contains("hung probe did not become true"));
+    }
+
+    #[tokio::test]
+    async fn wait_until_does_not_start_a_probe_after_its_deadline() {
+        let attempts = Cell::new(0);
+        wait_until(
+            Duration::ZERO,
+            || {
+                attempts.set(attempts.get() + 1);
+                std::future::ready(Ok(true))
+            },
+            "expired budget",
+        )
+        .await
+        .expect_err("an expired budget must not admit another probe");
+        assert_eq!(attempts.get(), 0);
+    }
+
+    #[tokio::test]
+    async fn retrying_put_does_not_hide_internal_error() -> TestResult {
+        let target = FakeS3Target::start().await?;
+        target.create_bucket("retry-probe");
+        target.inject(Operation::PutObject, FaultAction::ResponseStatus(500), 1);
+        let error = retrying_put(&client(&target), "retry-probe", "key", b"body".to_vec(), Duration::from_secs(5))
+            .await
+            .expect_err("InternalError must remain fatal even if the next PUT would succeed");
+        assert_eq!(s3_probe_error_code(error.as_ref()), Some("InternalError"));
+        assert_eq!(target.requests().len(), 1);
+        Ok(())
+    }
+
+    #[tokio::test]
+    async fn retrying_put_allows_transient_availability_errors() -> TestResult {
+        let target = FakeS3Target::start().await?;
+        target.create_bucket("retry-probe");
+        for status in [429, 503] {
+            target.inject(Operation::PutObject, FaultAction::ResponseStatus(status), 1);
+            retrying_put(&client(&target), "retry-probe", "key", b"body".to_vec(), Duration::from_secs(5)).await?;
+            assert_eq!(target.take_requests().len(), 2);
+        }
+        Ok(())
+    }
+
+    #[tokio::test]
+    async fn object_polls_reject_wrong_object_bytes() -> TestResult {
+        let target = FakeS3Target::start().await?;
+        target.create_bucket("retry-probe");
+        target.put_seed_object("retry-probe", "key", "wrong bytes", &SeedMetadata::default());
+        let error = retrying_get_equals(&client(&target), "retry-probe", "key", b"expected bytes", Duration::from_secs(5))
+            .await
+            .expect_err("a successful response with wrong bytes must fail immediately");
+        assert!(error.to_string().contains("bytes mismatch"));
+        assert_eq!(target.take_requests().len(), 1);
+        let error = wait_for_replicated_bytes(&client(&target), "retry-probe", "key", b"expected bytes", Duration::from_secs(5))
+            .await
+            .expect_err("replication may lag, but it must not return corrupt bytes");
+        assert!(error.to_string().contains("bytes mismatch"));
+        assert_eq!(target.requests().len(), 1);
+        Ok(())
+    }
+
+    #[tokio::test]
+    async fn object_polls_distinguish_committed_and_replicated_keys() -> TestResult {
+        let target = FakeS3Target::start().await?;
+        target.create_bucket("retry-probe");
+        target.put_seed_object("retry-probe", "key", "expected bytes", &SeedMetadata::default());
+        let client = client(&target);
+        for (status, expected_code) in [(404, "NoSuchKey"), (500, "InternalError"), (403, "AccessDenied")] {
+            target.inject(Operation::GetObject, FaultAction::ResponseStatus(status), 1);
+            let error = retrying_get_equals(&client, "retry-probe", "key", b"expected bytes", Duration::from_secs(5))
+                .await
+                .expect_err("an acknowledged object must not disappear or fail before a later successful read");
+            assert_eq!(s3_probe_error_code(error.as_ref()), Some(expected_code));
+            assert_eq!(target.take_requests().len(), 1);
+        }
+        for status in [429, 503] {
+            target.inject(Operation::GetObject, FaultAction::ResponseStatus(status), 1);
+            retrying_get_equals(&client, "retry-probe", "key", b"expected bytes", Duration::from_secs(5)).await?;
+            assert_eq!(target.take_requests().len(), 2);
+        }
+        target.inject(Operation::GetObject, FaultAction::ResponseStatus(404), 1);
+        wait_for_replicated_bytes(&client, "retry-probe", "key", b"expected bytes", Duration::from_secs(5)).await?;
+        assert_eq!(target.requests().len(), 2);
+        Ok(())
+    }
+}
+
 #[tokio::test]
 async fn append_single_node_pool_extends_ellipses_volumes() {
     let mut env = RustFSTestClusterEnvironment::with_topology(ClusterTopology::per_node_pools(2, vec![vec![0], vec![1]]))

crates/e2e_test/src/distributed/upgrade_test.rs

@@ -26,7 +26,7 @@
 
 use super::harness::{
     DistCluster, DistLayout, TestResult, assert_object_bytes, cluster_admin_ok, enable_versioning, get_object_bytes, put_object,
-    unique_bucket, wait_until,
+    s3_probe_client, s3_probe_error_code, sha256_hex, unique_bucket, wait_until,
 };
 use crate::common::{
     AdminTransport, admin_add_canned_policy_via, admin_attach_user_policy_via, admin_create_user_via, init_logging,
@@ -111,6 +111,7 @@ async fn create_iam_user(dist: &DistCluster, user: &str, secret: &str, policy_na
 }
 
 async fn wait_for_put(client: &Client, bucket: &str, key: &str, body: Vec<u8>, label: &str) -> TestResult {
+    let client = s3_probe_client(client);
     wait_until(
         CREDENTIAL_TIMEOUT,
         || {
@@ -119,8 +120,18 @@ async fn wait_for_put(client: &Client, bucket: &str, key: &str, body: Vec<u8>, l
             let key = key.to_string();
             let body = body.clone();
             async move {
-                put_object(&client, &bucket, &key, body).await?;
-                Ok(true)
+                match put_object(&client, &bucket, &key, body).await {
+                    Ok(()) => Ok(true),
+                    Err(error)
+                        if matches!(
+                            s3_probe_error_code(error.as_ref()),
+                            Some("AccessDenied" | "InvalidAccessKeyId" | "SlowDown" | "ServiceUnavailable")
+                        ) =>
+                    {
+                        Ok(false)
+                    }
+                    Err(error) => Err(error),
+                }
             }
         },
         label,
@@ -129,6 +140,7 @@ async fn wait_for_put(client: &Client, bucket: &str, key: &str, body: Vec<u8>, l
 }
 
 async fn wait_for_bytes(client: &Client, bucket: &str, key: &str, expected: &[u8], label: &str) -> TestResult {
+    let client = s3_probe_client(client);
     wait_until(
         CREDENTIAL_TIMEOUT,
         || {
@@ -137,8 +149,24 @@ async fn wait_for_bytes(client: &Client, bucket: &str, key: &str, expected: &[u8
             let key = key.to_string();
             let expected = expected.to_vec();
             async move {
-                let got = get_object_bytes(&client, &bucket, &key).await?;
-                Ok(got == expected)
+                match get_object_bytes(&client, &bucket, &key).await {
+                    Ok(got) if got == expected => Ok(true),
+                    Ok(got) => Err(format!(
+                        "{label}: object {bucket}/{key} bytes mismatch: expected sha256={} got sha256={}",
+                        sha256_hex(&expected),
+                        sha256_hex(&got)
+                    )
+                    .into()),
+                    Err(error)
+                        if matches!(
+                            s3_probe_error_code(error.as_ref()),
+                            Some("AccessDenied" | "InvalidAccessKeyId" | "SlowDown" | "ServiceUnavailable")
+                        ) =>
+                    {
+                        Ok(false)
+                    }
+                    Err(error) => Err(error),
+                }
             }
         },
         label,
@@ -146,6 +174,57 @@ async fn wait_for_bytes(client: &Client, bucket: &str, key: &str, expected: &[u8
     .await
 }
 
+#[cfg(test)]
+mod retry_tests {
+    use super::*;
+    use crate::common::build_test_s3_config;
+    use crate::fake_s3_target::{FAKE_ACCESS_KEY, FAKE_SECRET_KEY, FakeS3Target, FaultAction, Operation};
+
+    #[tokio::test]
+    async fn iam_polls_only_retry_auth_propagation_and_availability() -> TestResult {
+        let target = FakeS3Target::start().await?;
+        target.create_bucket("upgrade-retry-probe");
+        let client = Client::from_conf(build_test_s3_config(
+            target.endpoint(),
+            FAKE_ACCESS_KEY,
+            FAKE_SECRET_KEY,
+            None,
+            "upgrade-retry-test",
+        ));
+
+        for status in [403, 503] {
+            target.inject(Operation::PutObject, FaultAction::ResponseStatus(status), 1);
+            wait_for_put(&client, "upgrade-retry-probe", "key", b"body".to_vec(), "IAM PUT").await?;
+            assert_eq!(target.take_requests().len(), 2);
+
+            target.inject(Operation::GetObject, FaultAction::ResponseStatus(status), 1);
+            wait_for_bytes(&client, "upgrade-retry-probe", "key", b"body", "IAM GET").await?;
+            assert_eq!(target.take_requests().len(), 2);
+        }
+
+        target.inject(Operation::PutObject, FaultAction::ResponseStatus(500), 1);
+        let error = wait_for_put(&client, "upgrade-retry-probe", "key", b"body".to_vec(), "IAM PUT")
+            .await
+            .expect_err("IAM convergence must not hide InternalError on PUT");
+        assert_eq!(s3_probe_error_code(error.as_ref()), Some("InternalError"));
+        assert_eq!(target.take_requests().len(), 1);
+
+        target.inject(Operation::GetObject, FaultAction::ResponseStatus(500), 1);
+        let error = wait_for_bytes(&client, "upgrade-retry-probe", "key", b"body", "IAM GET")
+            .await
+            .expect_err("IAM convergence must not hide InternalError on GET");
+        assert_eq!(s3_probe_error_code(error.as_ref()), Some("InternalError"));
+        assert_eq!(target.take_requests().len(), 1);
+
+        let error = wait_for_bytes(&client, "upgrade-retry-probe", "key", b"wrong bytes", "IAM GET")
+            .await
+            .expect_err("IAM convergence must not hide a payload mismatch");
+        assert!(error.to_string().contains("bytes mismatch"));
+        assert_eq!(target.requests().len(), 1);
+        Ok(())
+    }
+}
+
 async fn seed_history_and_iam(dist: &DistCluster) -> TestResult<UpgradeSeed> {
     let history_bucket = unique_bucket("upg-hist");
     let versioned_bucket = unique_bucket("upg-ver");

crates/e2e_test/src/distributed_startup_regression_test.rs

@@ -215,6 +215,8 @@ mod tests {
         probe_startup_cas_binary(&binary, &nonce, &artifact).await?;
 
         let mut cluster = RustFSTestClusterEnvironment::new(4).await?;
+        // Isolate bootstrap CAS and its S3 readback from periodic usage publication.
+        cluster.set_env("RUSTFS_SCANNER_ENABLED", "false");
         let mut logs = Vec::new();
         let mut disks = Vec::new();
         let mut endpoints = Vec::new();

crates/e2e_test/src/heal_erasure_disk_rebuild_test.rs

@@ -1450,10 +1450,18 @@ mod tests {
             cluster.set_env("RUSTFS_HEAL_PAGE_PARALLEL_ENABLE", "false");
         }
         // Keep every node's Heal runtime enabled for normal disk registration.
-        let server_rust_log = std::env::var("RUSTFS_HEAL_CHAOS_SERVER_RUST_LOG")
-            .unwrap_or_else(|_| "rustfs::heal::task=info,rustfs=error".to_string());
+        // Capture durable handoff and marker cleanup failures during shutdown.
+        let server_rust_log = std::env::var("RUSTFS_HEAL_CHAOS_SERVER_RUST_LOG").unwrap_or_else(|_| {
+            "rustfs::heal::task=info,rustfs::heal::manager=info,rustfs::main::handle_shutdown=info,rustfs_heal::heal=warn,rustfs=error"
+                .to_string()
+        });
         cluster.set_env("RUST_LOG", server_rust_log);
-        let log_dir = std::env::var("RUSTFS_HEAL_CHAOS_LOG_DIR").unwrap_or_else(|_| format!("{}/logs", cluster.temp_dir));
+        let log_dir = std::env::var("RUSTFS_HEAL_CHAOS_LOG_DIR").unwrap_or_else(|_| {
+            // Keep explicit directories compatible and isolate parallel suite captures.
+            std::env::var("RUSTFS_HEAL_CHAOS_LOG_ROOT")
+                .map(|root| format!("{root}/{interruption_kind}-{}", uuid::Uuid::new_v4()))
+                .unwrap_or_else(|_| format!("{}/logs", cluster.temp_dir))
+        });
         std::fs::create_dir_all(&log_dir)?;
         for node_index in 0..cluster.nodes.len() {
             cluster.set_node_capture_log_path(node_index, format!("{log_dir}/node{node_index}.log"))?;
@@ -1799,7 +1807,8 @@ mod tests {
         let (mut client_token, mut task_status_url) = if background_rejoin_heal_evidence {
             (String::new(), String::new())
         } else {
-            let heal_start_body = signed_admin_post(&heal_url, Some(heal_body), &cluster.access_key, &cluster.secret_key).await?;
+            let heal_start_body =
+                start_root_heal_when_control_ready(&heal_url, heal_body, &cluster.access_key, &cluster.secret_key).await?;
             let heal_start: serde_json::Value = serde_json::from_str(&heal_start_body)
                 .map_err(|err| format!("heal start response is not JSON ({err}): {heal_start_body}"))?;
             let client_token = heal_start["clientToken"]
@@ -2140,7 +2149,8 @@ mod tests {
                     && status["summary"].as_str() == Some("failed")
                 {
                     let heal_start_body =
-                        signed_admin_post(&heal_url, Some(heal_body), &cluster.access_key, &cluster.secret_key).await?;
+                        start_root_heal_when_control_ready(&heal_url, heal_body, &cluster.access_key, &cluster.secret_key)
+                            .await?;
                     let heal_start: serde_json::Value = serde_json::from_str(&heal_start_body)
                         .map_err(|err| format!("recovery heal start response is not JSON ({err}): {heal_start_body}"))?;
                     client_token = heal_start["clientToken"]

crates/e2e_test/src/inline_fast_path_cluster_test.rs

@@ -886,6 +886,8 @@ fn configure_reader_metric_cluster(cluster: &mut RustFSTestClusterEnvironment, c
     cluster.set_env("RUSTFS_OBS_METRICS_EXPORT_ENABLED", "true");
     cluster.set_env("RUSTFS_OBS_TRACES_EXPORT_ENABLED", "false");
     cluster.set_env("RUSTFS_OBS_LOGS_EXPORT_ENABLED", "false");
+    // Keep startup diagnostics available when OTLP log export is disabled.
+    cluster.set_env("RUSTFS_OBS_LOG_STDOUT_ENABLED", "true");
     cluster.set_env("RUSTFS_OBS_METER_INTERVAL", "1");
     cluster.set_env("RUSTFS_OBS_USE_STDOUT", "false");
     cluster.set_env("RUSTFS_GET_CODEC_STREAMING_ENABLE", "false");

crates/e2e_test/src/list_objects_v2_pagination_test.rs

@@ -31,8 +31,19 @@ mod tests {
     use aws_sdk_s3::Client;
     use aws_sdk_s3::primitives::ByteStream;
     use std::collections::HashSet;
+    use std::future::Future;
+    use std::time::Duration;
+    use tokio::time::{Instant, error::Elapsed, timeout_at};
     use tracing::info;
 
+    const PAGINATION_TIMEOUT: Duration = Duration::from_secs(120);
+
+    // Fixture population can require thousands of durable PUTs. Bound the
+    // listing traversal itself with one deadline shared by every page.
+    async fn listing_request_before<T>(deadline: Instant, request: impl Future<Output = T>) -> Result<T, Elapsed> {
+        timeout_at(deadline, request).await
+    }
+
     /// Helper function to create an S3 client for testing
     fn create_s3_client(env: &RustFSTestEnvironment) -> Client {
         env.create_s3_client()
@@ -57,6 +68,54 @@ mod tests {
         }
     }
 
+    #[tokio::test]
+    async fn test_list_objects_v2_deadline_rejects_unresponsive_peer() {
+        use tokio::io::AsyncReadExt;
+        use tokio::net::TcpListener;
+        use tokio::sync::oneshot;
+        use tokio::time::timeout;
+
+        let listener = TcpListener::bind("127.0.0.1:0")
+            .await
+            .expect("bind unresponsive listing peer");
+        let endpoint = format!("http://{}", listener.local_addr().expect("listing peer address"));
+        let (received_tx, received_rx) = oneshot::channel();
+        let server = tokio::spawn(async move {
+            let (mut stream, _) = listener.accept().await.expect("accept listing request");
+            let mut bytes = [0; 4096];
+            assert!(stream.read(&mut bytes).await.expect("read listing request") > 0);
+            received_tx.send(()).expect("signal received listing request");
+            std::future::pending::<()>().await;
+            drop(stream);
+        });
+        let client = Client::from_conf(
+            crate::common::build_test_s3_config(&endpoint, "test-access", "test-secret", None, "pagination-deadline")
+                .to_builder()
+                .retry_config(aws_sdk_s3::config::retry::RetryConfig::standard().with_max_attempts(1))
+                .build(),
+        );
+        let request = client.list_objects_v2().bucket("deadline-test").send();
+        tokio::pin!(request);
+        tokio::select! {
+            response = &mut request => panic!("unresponsive listing peer unexpectedly returned: {response:?}"),
+            received = timeout(Duration::from_secs(10), received_rx) => {
+                received.expect("listing request must reach peer").expect("listing peer must remain alive");
+            }
+        }
+
+        // Expire the same absolute deadline after the real request is observed;
+        // this tests a stalled response without relying on transport timing.
+        let result = timeout(Duration::from_secs(1), listing_request_before(Instant::now(), request))
+            .await
+            .expect("expired pagination deadline must stop the in-flight request");
+        server.abort();
+        let _ = server.await;
+        assert!(
+            result.is_err(),
+            "an unresponsive ListObjectsV2 request must exceed the pagination deadline"
+        );
+    }
+
     /// Test for Issue #2775: continuation forwarding must not
     /// skip a child directory when the prefix component repeats in the key.
     #[tokio::test]
@@ -514,12 +573,12 @@ mod tests {
                 .expect("Failed to put object");
         }
 
-        let output = client
-            .list_objects_v2()
-            .bucket(bucket)
-            .max_keys(1001)
-            .send()
+        eprintln!("Seeded {object_count} objects in {bucket}; starting ListObjectsV2 pagination");
+        let deadline = Instant::now() + PAGINATION_TIMEOUT;
+
+        let output = listing_request_before(deadline, client.list_objects_v2().bucket(bucket).max_keys(1001).send())
             .await
+            .expect("ListObjectsV2 pagination exceeded its deadline after fixture population")
             .expect("Failed to list objects");
 
         assert_eq!(output.contents().len(), 1000);
@@ -535,14 +594,18 @@ mod tests {
             .expect("NextContinuationToken should be present when capped response is truncated")
             .to_string();
 
-        let output = client
-            .list_objects_v2()
-            .bucket(bucket)
-            .max_keys(1001)
-            .continuation_token(next_token)
-            .send()
-            .await
-            .expect("Failed to list objects with continuation token");
+        let output = listing_request_before(
+            deadline,
+            client
+                .list_objects_v2()
+                .bucket(bucket)
+                .max_keys(1001)
+                .continuation_token(next_token)
+                .send(),
+        )
+        .await
+        .expect("ListObjectsV2 continuation exceeded the shared pagination deadline")
+        .expect("Failed to list objects with continuation token");
 
         assert_eq!(output.contents().len(), 2);
         assert!(!output.is_truncated().unwrap_or(false));
@@ -765,6 +828,9 @@ mod tests {
             }
         }
 
+        eprintln!("Seeded {} objects in {bucket}; starting ListObjectsV2 pagination", all_keys.len());
+        let deadline = Instant::now() + PAGINATION_TIMEOUT;
+
         // Paginate with delimiter="/" and max_keys=50
         // Visible per page: up to 50 CommonPrefixes
         let mut listed_keys = Vec::new();
@@ -780,7 +846,10 @@ mod tests {
                 request = request.continuation_token(token);
             }
 
-            let output = request.send().await.expect("Failed to list objects");
+            let output = listing_request_before(deadline, request.send())
+                .await
+                .expect("ListObjectsV2 delimiter traversal exceeded the shared pagination deadline")
+                .expect("Failed to list objects");
             last_page_is_truncated = output.is_truncated().unwrap_or(false);
 
             for obj in output.contents() {
@@ -960,6 +1029,8 @@ mod tests {
     /// tests the non-delimiter case.
     #[tokio::test]
     async fn test_list_objects_v2_maxkeys_above_limit_with_delimiter() {
+        use futures::{StreamExt, stream};
+
         init_logging();
         info!("Starting test: ListObjectsV2 MaxKeys above limit with delimiter");
 
@@ -974,29 +1045,44 @@ mod tests {
         // 12 dirs × 100 files = 1200 raw keys
         let dir_count = 12;
         let files_per_dir = 100;
-        for d in 0..dir_count {
-            for f in 0..files_per_dir {
-                let key = format!("dir-{:02}/file{:03}.txt", d, f);
-                client
-                    .put_object()
-                    .bucket(bucket)
-                    .key(&key)
-                    .body(ByteStream::from_static(b"x"))
-                    .send()
-                    .await
-                    .expect("Failed to put object");
-            }
-        }
+        const FIXTURE_CONCURRENCY: usize = 8;
+        let fixture_started = std::time::Instant::now();
+        eprintln!(
+            "Preparing {} objects with at most {FIXTURE_CONCURRENCY} concurrent PUTs",
+            dir_count * files_per_dir
+        );
+        // Object creation order is irrelevant to this listing contract. Bound
+        // fixture I/O and await every PUT before issuing the LIST request.
+        stream::iter((0..dir_count).flat_map(|d| (0..files_per_dir).map(move |f| (d, f))))
+            .for_each_concurrent(FIXTURE_CONCURRENCY, |(d, f)| {
+                let client = &client;
+                async move {
+                    let key = format!("dir-{d:02}/file{f:03}.txt");
+                    client
+                        .put_object()
+                        .bucket(bucket)
+                        .key(&key)
+                        .body(ByteStream::from_static(b"x"))
+                        .send()
+                        .await
+                        .unwrap_or_else(|err| panic!("Failed to prepare fixture object {key}: {err:?}"));
+                }
+            })
+            .await;
+        eprintln!(
+            "Seeded {} objects in {bucket} in {:?}; starting ListObjectsV2 pagination",
+            dir_count * files_per_dir,
+            fixture_started.elapsed()
+        );
+        let deadline = Instant::now() + PAGINATION_TIMEOUT;
 
         // With delimiter: 12 CommonPrefixes visible, all fit within capped 1000
-        let output = client
-            .list_objects_v2()
-            .bucket(bucket)
-            .delimiter("/")
-            .max_keys(2000)
-            .send()
-            .await
-            .expect("Failed to list objects");
+        let list_started = std::time::Instant::now();
+        let output =
+            listing_request_before(deadline, client.list_objects_v2().bucket(bucket).delimiter("/").max_keys(2000).send())
+                .await
+                .expect("ListObjectsV2 delimiter listing exceeded its deadline after fixture population")
+                .expect("Failed to list objects");
 
         assert_eq!(
             output.common_prefixes().len(),
@@ -1005,6 +1091,15 @@ mod tests {
             dir_count,
             output.common_prefixes().len()
         );
+        let prefixes = output
+            .common_prefixes()
+            .iter()
+            .map(|entry| entry.prefix().expect("each CommonPrefix must carry its prefix").to_owned())
+            .collect::<Vec<_>>();
+        let expected_prefixes = (0..dir_count).map(|d| format!("dir-{d:02}/")).collect::<Vec<_>>();
+        assert_eq!(prefixes, expected_prefixes);
+        assert!(output.contents().is_empty(), "all fixture objects must collapse into prefixes");
+        assert_eq!(output.key_count(), Some(i32::try_from(dir_count).expect("fixture count fits i32")));
         assert_eq!(output.max_keys(), Some(1000));
         // 12 visible < 1000 capped MaxKeys → not truncated
         assert!(
@@ -1013,6 +1108,11 @@ mod tests {
             output.common_prefixes().len()
         );
 
+        assert!(
+            output.next_continuation_token().is_none(),
+            "a complete page must not carry a continuation token"
+        );
+        eprintln!("ListObjectsV2 assertions completed in {:?}", list_started.elapsed());
         info!("MaxKeys above limit with delimiter test passed");
 
         env.stop_server();

crates/e2e_test/src/reliant/tiering.rs

@@ -1056,7 +1056,17 @@ async fn test_hermetic_transition_restore_failure_expiry_and_retry() -> TestResu
     cold_client.create_bucket().bucket(TIER_BUCKET).send().await?;
 
     let mut hot = RustFSTestEnvironment::new().await?;
-    start_tier_source(&mut hot, &[("RUSTFS_SCANNER_CYCLE", "1"), ("RUSTFS_ILM_DEBUG_DAY_SECS", "5")]).await?;
+    // Revisit the restored key every cycle; the default 16-cycle sampling is
+    // independent of the accelerated lifecycle clock used by this fixture.
+    start_tier_source(
+        &mut hot,
+        &[
+            ("RUSTFS_SCANNER_CYCLE", "1"),
+            ("RUSTFS_DATA_USAGE_UPDATE_DIR_CYCLES", "1"),
+            ("RUSTFS_ILM_DEBUG_DAY_SECS", "5"),
+        ],
+    )
+    .await?;
     let hot_client = hot.create_s3_client();
     add_rustfs_tier(&hot, &cold).await?;
 

crates/e2e_test/src/replication_extension_test.rs

@@ -7194,13 +7194,30 @@ async fn test_site_replication_state_edit_fresh_and_stale_real_dual_node() -> Re
     assert!(source_info.sites.iter().all(|peer| !peer.replicate_ilm_expiry));
     assert!(target_info.sites.iter().all(|peer| !peer.replicate_ilm_expiry));
 
+    let source_deployment_id = source_info
+        .sites
+        .iter()
+        .find(|peer| peer.endpoint == source_env.url)
+        .map(|peer| peer.deployment_id.clone())
+        .ok_or("source site missing from its own replication info")?;
+    let target_deployment_id = target_info
+        .sites
+        .iter()
+        .find(|peer| peer.endpoint == target_env.url)
+        .map(|peer| peer.deployment_id.clone())
+        .ok_or("target site missing from its own replication info")?;
     let target_status =
         wait_for_site_replication_status(&target_env, "peer-state=true", |status| status.peer_states.len() == 2).await?;
     let current_updated_at = target_status
         .peer_states
-        .values()
-        .find_map(|state| state.updated_at)
+        .get(&target_deployment_id)
+        .and_then(|state| state.updated_at)
         .ok_or("missing target site replication updated_at")?;
+    let source_updated_at = target_status
+        .peer_states
+        .get(&source_deployment_id)
+        .and_then(|state| state.updated_at)
+        .ok_or("missing source site replication updated_at")?;
 
     let mut stale_peers = BTreeMap::new();
     for peer in target_info.sites {
@@ -7244,16 +7261,23 @@ async fn test_site_replication_state_edit_fresh_and_stale_real_dual_node() -> Re
     .await?;
     assert!(target_after_fresh.sites.iter().all(|peer| peer.replicate_ilm_expiry));
 
+    // State edits apply only to the addressed site; status reports each site's own state.
     let target_status_after_fresh = wait_for_site_replication_status(&target_env, "peer-state=true", |status| {
         status.peer_states.len() == 2
-            && status.peer_states.values().all(|state| {
-                state.updated_at == Some(fresh_updated_at) && state.peers.values().all(|peer| peer.replicate_ilm_expiry)
+            && status.peer_states.get(&target_deployment_id).is_some_and(|state| {
+                state.updated_at == Some(fresh_updated_at)
+                    && state.peers.len() == 2
+                    && state.peers.values().all(|peer| peer.replicate_ilm_expiry)
             })
     })
     .await?;
-    assert!(target_status_after_fresh.peer_states.values().all(|state| {
-        state.updated_at == Some(fresh_updated_at) && state.peers.values().all(|peer| peer.replicate_ilm_expiry)
-    }));
+    let source_state_after_fresh = target_status_after_fresh
+        .peer_states
+        .get(&source_deployment_id)
+        .ok_or("missing source site replication state after target edit")?;
+    assert_eq!(source_state_after_fresh.updated_at, Some(source_updated_at));
+    assert_eq!(source_state_after_fresh.peers.len(), 2);
+    assert!(source_state_after_fresh.peers.values().all(|peer| !peer.replicate_ilm_expiry));
 
     let source_after_fresh = site_replication_info(&source_env).await?;
     assert!(source_after_fresh.sites.iter().all(|peer| !peer.replicate_ilm_expiry));

crates/e2e_test/src/upgrade_compatibility_test.rs

@@ -51,6 +51,10 @@ const PLAIN_BUCKET: &str = "upgrade-plain-data";
 const VERSIONED_BUCKET: &str = "upgrade-versioned-data";
 const MIXED_BUCKET: &str = "upgrade-mixed-version-data";
 const MIXED_NODE_COUNT: usize = 4;
+const UPGRADE_READINESS_BODY: &[u8] = b"upgrade write readiness";
+// Written by the "previous-seed" readiness probe through node 0 before any
+// current binary starts; every later phase must still read it.
+const PREVIOUS_RELEASE_SEED_KEY: &str = ".upgrade-readiness/previous-seed/node-0";
 const MULTIPART_WORKERS: usize = 16;
 const MULTIPART_UPLOADS_PER_WORKER: usize = 16;
 // Peers keep a restarted node's drive in Suspect/Returning for roughly
@@ -67,6 +71,7 @@ const CONFIG_REPLICA_BUCKET: &str = "upgrade-config-replica";
 const ROLLBACK_BUCKET: &str = "rollback-config-data";
 const ROLLBACK_REPLICA_BUCKET: &str = "rollback-config-replica";
 const BUCKET_QUOTA_BYTES: u64 = 64 * 1024 * 1024;
+const QUOTA_USAGE_READINESS_TIMEOUT: Duration = Duration::from_secs(90);
 const LIFECYCLE_RULE_ID: &str = "upgrade-expire-logs";
 const LIFECYCLE_PREFIX: &str = "logs/";
 const LIFECYCLE_DAYS: i32 = 30;
@@ -395,6 +400,7 @@ async fn exercise_mixed_cluster(
     previous_node: usize,
 ) -> TestResult {
     let clients = cluster.create_all_clients()?;
+    wait_for_upgrade_write_readiness(&clients, phase, LISTING_CONVERGENCE_TIMEOUT).await?;
     let current_client = &clients[current_node];
     let previous_client = &clients[previous_node];
 
@@ -423,6 +429,11 @@ async fn exercise_mixed_cluster(
     let multipart_keys = write_multipart_load(&clients, phase).await?;
     let expected_count = multipart_keys.len() + 2;
     for (label, client) in [("current", current_client), ("previous", previous_client)] {
+        assert_eq!(
+            read_object(client, MIXED_BUCKET, PREVIOUS_RELEASE_SEED_KEY, None).await?.1,
+            UPGRADE_READINESS_BODY,
+            "{phase}: the {label} node must retain the previous-release seed"
+        );
         wait_for_phase_listing(
             client,
             phase,
@@ -441,6 +452,231 @@ async fn exercise_mixed_cluster(
     Ok(())
 }
 
+fn upgrade_probe_client(client: &Client) -> Client {
+    Client::from_conf(
+        client
+            .config()
+            .to_builder()
+            .retry_config(aws_sdk_s3::config::retry::RetryConfig::standard().with_max_attempts(1))
+            .build(),
+    )
+}
+
+async fn wait_for_upgrade_write_readiness(clients: &[Client], phase: &str, budget: Duration) -> TestResult {
+    // ListBuckets can succeed before peers recover a restarted disk. Cluster
+    // health also accepts Returning disks whose write health is still FAULTY.
+    // Probe every writer outside the asserted phase prefix; compatibility
+    // writes still execute once and retain their original assertions.
+    let deadline = Instant::now() + budget;
+    for (node, client) in clients.iter().enumerate() {
+        let client = upgrade_probe_client(client);
+        let key = format!(".upgrade-readiness/{phase}/node-{node}");
+        let mut last_response = "no response".to_string();
+        loop {
+            if Instant::now() >= deadline {
+                return Err(
+                    format!("{phase}: node {node} write readiness deadline exceeded; last response: {last_response}").into(),
+                );
+            }
+            let response = tokio::time::timeout_at(
+                deadline,
+                client
+                    .put_object()
+                    .bucket(MIXED_BUCKET)
+                    .key(&key)
+                    .body(ByteStream::from_static(UPGRADE_READINESS_BODY))
+                    .send(),
+            )
+            .await;
+            match response {
+                Ok(Ok(_)) => break,
+                Ok(Err(error)) => {
+                    if error.raw_response().map(|response| response.status().as_u16()) != Some(503)
+                        || error.as_service_error().and_then(ProvideErrorMetadata::code) != Some("ServiceUnavailable")
+                    {
+                        return Err(format!("{phase}: node {node} write readiness failed: {error:?}").into());
+                    }
+                    last_response = format!("{error:?}");
+                }
+                Err(_) => {
+                    return Err(format!(
+                        "{phase}: node {node} write readiness deadline exceeded during PutObject; last response: {last_response}"
+                    )
+                    .into());
+                }
+            }
+            tokio::time::sleep_until(deadline.min(Instant::now() + Duration::from_millis(500))).await;
+        }
+    }
+    Ok(())
+}
+
+async fn prepare_previous_release_baseline(cluster: &mut RustFSTestClusterEnvironment, previous_binary: &Path) -> TestResult {
+    let clients: Vec<_> = cluster.create_all_clients()?.iter().map(upgrade_probe_client).collect();
+    // rc.5 can latch a non-elected node's write fence while its first startup
+    // waits for pool metadata (#7473). First prove the elected writer can
+    // persist data, then restart each old process once with three peers still
+    // readable. This preparation ends before any current binary is started.
+    wait_for_upgrade_write_readiness(&clients[..1], "previous-seed", LISTING_CONVERGENCE_TIMEOUT).await?;
+    for node in [1, 2, 3, 0] {
+        cluster.stop_node(node)?;
+        cluster.start_node_from_binary(node, previous_binary).await?;
+        wait_for_upgrade_write_readiness(
+            std::slice::from_ref(&clients[node]),
+            &format!("previous-restart-{node}"),
+            LISTING_CONVERGENCE_TIMEOUT,
+        )
+        .await?;
+    }
+
+    wait_for_upgrade_write_readiness(&clients, "previous-baseline", LISTING_CONVERGENCE_TIMEOUT).await?;
+    for (reader, client) in clients.iter().enumerate() {
+        assert_eq!(
+            read_object(client, MIXED_BUCKET, PREVIOUS_RELEASE_SEED_KEY, None).await?.1,
+            UPGRADE_READINESS_BODY,
+            "previous-release node {reader} must retain the seed across its rolling restart"
+        );
+        for writer in 0..clients.len() {
+            let key = format!(".upgrade-readiness/previous-baseline/node-{writer}");
+            assert_eq!(
+                read_object(client, MIXED_BUCKET, &key, None).await?.1,
+                UPGRADE_READINESS_BODY,
+                "previous-release node {reader} must read baseline data from writer {writer}"
+            );
+        }
+    }
+    Ok(())
+}
+
+#[cfg(test)]
+mod upgrade_write_readiness_tests {
+    use super::*;
+    use crate::fake_s3_target::FaultAction;
+
+    #[tokio::test]
+    async fn waits_for_each_writer_after_metadata_is_ready() -> TestResult {
+        let target = FakeS3Target::start().await?;
+        target.create_bucket(MIXED_BUCKET);
+        let client = fake_source_client(&target);
+        client.head_bucket().bucket(MIXED_BUCKET).send().await?;
+
+        let phase = "one-previous-node";
+        let first_key = format!(".upgrade-readiness/{phase}/node-0");
+        let second_key = format!(".upgrade-readiness/{phase}/node-1");
+        target.inject_for_key(
+            FakeTargetOperation::PutObject,
+            &first_key,
+            FaultAction::Status(StatusCode::SERVICE_UNAVAILABLE),
+            2,
+        );
+        target.inject_for_key(
+            FakeTargetOperation::PutObject,
+            &second_key,
+            FaultAction::Status(StatusCode::SERVICE_UNAVAILABLE),
+            1,
+        );
+        // Metadata readiness does not prove that a data write can succeed.
+        let premature = client
+            .put_object()
+            .bucket(MIXED_BUCKET)
+            .key(&first_key)
+            .body(ByteStream::from_static(b"upgrade write readiness"))
+            .send()
+            .await
+            .expect_err("metadata readiness does not prove write readiness");
+        assert_eq!(premature.raw_response().map(|response| response.status().as_u16()), Some(503));
+
+        wait_for_upgrade_write_readiness(&[client.clone(), client.clone()], phase, Duration::from_secs(5)).await?;
+        assert_eq!(target.count_requests(FakeTargetOperation::PutObject, &first_key), 3);
+        assert_eq!(target.count_requests(FakeTargetOperation::PutObject, &second_key), 2);
+        assert!(target.has_object(MIXED_BUCKET, &first_key));
+        assert!(target.has_object(MIXED_BUCKET, &second_key));
+        assert!(
+            client
+                .list_objects_v2()
+                .bucket(MIXED_BUCKET)
+                .prefix(format!("{phase}/"))
+                .send()
+                .await?
+                .contents()
+                .is_empty()
+        );
+        target.shutdown().await;
+        Ok(())
+    }
+
+    #[tokio::test]
+    async fn rejects_permanent_errors_without_sdk_retries() -> TestResult {
+        let target = FakeS3Target::start().await?;
+        target.create_bucket(MIXED_BUCKET);
+        let client = Client::from_conf(
+            fake_source_client(&target)
+                .config()
+                .to_builder()
+                .retry_config(aws_sdk_s3::config::retry::RetryConfig::standard().with_max_attempts(3))
+                .build(),
+        );
+        for status in [
+            StatusCode::INTERNAL_SERVER_ERROR,
+            StatusCode::FORBIDDEN,
+            StatusCode::NOT_FOUND,
+        ] {
+            let phase = format!("permanent-{}", status.as_u16());
+            let key = format!(".upgrade-readiness/{phase}/node-0");
+            target.inject_for_key(FakeTargetOperation::PutObject, &key, FaultAction::Status(status), 1);
+            let error = wait_for_upgrade_write_readiness(std::slice::from_ref(&client), &phase, Duration::from_secs(5))
+                .await
+                .expect_err("a permanent error must not be retried into success");
+            assert!(error.to_string().contains("node 0 write readiness failed"), "{error}");
+            assert_eq!(target.count_requests(FakeTargetOperation::PutObject, &key), 1);
+            assert!(!target.has_object(MIXED_BUCKET, &key));
+        }
+        target.shutdown().await;
+        Ok(())
+    }
+
+    #[tokio::test]
+    async fn transient_errors_stop_at_the_deadline() -> TestResult {
+        let target = FakeS3Target::start().await?;
+        target.create_bucket(MIXED_BUCKET);
+        let client = fake_source_client(&target);
+        client.head_bucket().bucket(MIXED_BUCKET).send().await?;
+        let phase = "deadline";
+        let key = format!(".upgrade-readiness/{phase}/node-0");
+        target.inject_for_key(
+            FakeTargetOperation::PutObject,
+            &key,
+            FaultAction::Status(StatusCode::SERVICE_UNAVAILABLE),
+            10,
+        );
+        let error = wait_for_upgrade_write_readiness(&[client], phase, Duration::from_secs(1))
+            .await
+            .expect_err("persistent unavailability must exhaust the shared deadline");
+        // Transport scheduling consumes the same budget; do not require a
+        // response to reach the fake target before the deadline on a busy host.
+        assert!(error.to_string().contains("deadline exceeded"), "{error}");
+        target.shutdown().await;
+        Ok(())
+    }
+
+    #[tokio::test]
+    async fn in_flight_requests_are_bounded_by_the_deadline() -> TestResult {
+        let target = FakeS3Target::start().await?;
+        target.create_bucket(MIXED_BUCKET);
+        let client = fake_source_client(&target);
+        client.head_bucket().bucket(MIXED_BUCKET).send().await?;
+        let phase = "stalled";
+        let key = format!(".upgrade-readiness/{phase}/node-0");
+        target.inject_for_key(FakeTargetOperation::PutObject, &key, FaultAction::Stall(Duration::from_secs(30)), 1);
+        let error = wait_for_upgrade_write_readiness(&[client], phase, Duration::from_secs(1))
+            .await
+            .expect_err("a stalled request must not outlive the readiness deadline");
+        assert!(error.to_string().contains("deadline exceeded during PutObject"), "{error}");
+        target.shutdown().await;
+        Ok(())
+    }
+}
+
 /// Pins the published old writer's limitation and the supported recovery
 /// procedure. This is not a promise that mixed-version ODM is supported.
 /// Replace the loss assertion when ODM gains independent persistence;
@@ -690,6 +926,7 @@ async fn rolling_upgrade_from_rc2_preserves_mixed_version_contracts() -> TestRes
     configure_cluster_logs(&mut cluster)?;
     cluster.start_with_binary(&previous_binary).await?;
     cluster.create_test_bucket(MIXED_BUCKET).await?;
+    prepare_previous_release_baseline(&mut cluster, &previous_binary).await?;
 
     cluster.stop_node(0)?;
     cluster.start_node_from_binary(0, &current_binary).await?;
@@ -705,6 +942,11 @@ async fn rolling_upgrade_from_rc2_preserves_mixed_version_contracts() -> TestRes
     cluster.start_node_from_binary(3, &current_binary).await?;
 
     for (node_idx, client) in cluster.create_all_clients()?.iter().enumerate() {
+        assert_eq!(
+            read_object(client, MIXED_BUCKET, PREVIOUS_RELEASE_SEED_KEY, None).await?.1,
+            UPGRADE_READINESS_BODY,
+            "current node {node_idx} must retain the previous-release seed"
+        );
         for phase in ["one-current-node", "one-previous-node"] {
             wait_for_phase_listing(
                 client,
@@ -732,7 +974,8 @@ async fn rolling_upgrade_from_rc2_preserves_mixed_version_contracts() -> TestRes
                 "assertions": [
                     "current node reads objects written through previous-release client",
                     "previous-release node reads objects written through current client",
-                    "all nodes list every mixed-version object after homogeneous-current convergence"
+                    "all nodes list every mixed-version object after homogeneous-current convergence",
+                    "the previous-release seed survives the baseline restart and every upgrade phase"
                 ],
             }),
         )?;
@@ -837,12 +1080,23 @@ async fn put_object_through_quota_warmup(client: &Client, bucket: &str, key: &st
 
 async fn get_bucket_quota(env: &RustFSTestEnvironment, bucket: &str) -> Result<Option<u64>, BoxError> {
     let path = format!("/rustfs/admin/v3/quota/{bucket}");
-    let (status, response) = admin_request(&env.url, Method::GET, &path, None, &env.access_key, &env.secret_key).await?;
-    if status != StatusCode::OK {
-        return Err(format!("reading the quota of {bucket} failed: {status} {response}").into());
+    let deadline = Instant::now() + QUOTA_USAGE_READINESS_TIMEOUT;
+    loop {
+        let (status, response) = tokio::time::timeout_at(
+            deadline,
+            admin_request(&env.url, Method::GET, &path, None, &env.access_key, &env.secret_key),
+        )
+        .await
+        .map_err(|_| format!("reading the quota of {bucket} exceeded the usage-readiness timeout"))??;
+        if status == StatusCode::OK {
+            let quota: serde_json::Value = serde_json::from_str(&response)?;
+            return Ok(quota.get("quota").and_then(serde_json::Value::as_u64));
+        }
+        if status != StatusCode::SERVICE_UNAVAILABLE || Instant::now() >= deadline {
+            return Err(format!("reading the quota of {bucket} failed: {status} {response}").into());
+        }
+        sleep(Duration::from_millis(500)).await;
     }
-    let quota: serde_json::Value = serde_json::from_str(&response)?;
-    Ok(quota.get("quota").and_then(serde_json::Value::as_u64))
 }
 
 /// `GET /rustfs/admin/v3/list-remote-targets?bucket=...`.

crates/ecstore/src/api/mod.rs

@@ -448,7 +448,7 @@ pub mod disk {
 pub mod error {
     pub use crate::error::{
         Error, PoolMetadataError, PoolMetadataFailure, Result, StorageError, classify_system_path_failure_reason,
-        is_err_bucket_not_found, is_err_object_not_found, is_err_version_not_found,
+        is_err_bucket_not_found, is_err_invalid_upload_id, is_err_object_not_found, is_err_version_not_found,
     };
 }
 

crates/ecstore/src/bucket/bandwidth/monitor.rs

@@ -495,6 +495,193 @@ mod tests {
         assert!(rate > 0.0);
     }
 
+    // Tokio's paused clock does not advance ratelimit's StdClock. Keep natural
+    // refill below one whole token for years and supply each exact budget here.
+    fn supply_reader_test_tokens(throttle: &BucketThrottle, available: u64) {
+        let rate = u64::try_from(throttle.node_bandwidth_per_sec).expect("test throttle rate should be positive");
+        let burst = throttle.burst();
+        *throttle.limiter.lock().expect("test throttle should not poison") = Ratelimiter::builder(rate)
+            .max_tokens(burst)
+            .initial_available(available)
+            .period(Duration::from_nanos(u64::MAX))
+            .build()
+            .expect("controlled token budget should be valid");
+    }
+
+    #[tokio::test(start_paused = true)]
+    async fn test_monitored_reader_retains_partial_tokens_across_waits() {
+        use crate::bucket::bandwidth::reader::{MonitorReaderOptions, MonitoredReader};
+        use futures_util::task::noop_waker_ref;
+        use std::pin::Pin;
+        use std::task::{Context, Poll};
+        use tokio::io::{AsyncRead, ReadBuf};
+
+        for header_size in [0, 25] {
+            let monitor = Monitor::new(1);
+            let opts = BucketOptions {
+                name: "reader-budget".to_string(),
+                replication_arn: "reader-budget-arn".to_string(),
+            };
+            monitor.set_bandwidth_limit(&opts.name, &opts.replication_arn, 100);
+            let throttle = monitor.throttle(&opts).expect("test throttle should exist");
+            supply_reader_test_tokens(&throttle, 25);
+            let data = [0xAB; 100];
+            let mut reader = MonitoredReader::new(
+                monitor,
+                data.as_slice(),
+                MonitorReaderOptions {
+                    bucket_options: opts,
+                    header_size,
+                },
+            );
+            let mut output = [0; 100];
+            let mut buf = ReadBuf::new(&mut output);
+            let mut cx = Context::from_waker(noop_waker_ref());
+
+            assert!(Pin::new(&mut reader).poll_read(&mut cx, &mut buf).is_pending());
+            assert!(buf.filled().is_empty(), "partial payment must not authorize payload reads");
+            assert_eq!(throttle.limiter.lock().expect("test throttle").available(), 0);
+
+            supply_reader_test_tokens(&throttle, 75);
+            tokio::time::advance(Duration::from_secs(1)).await;
+            assert!(
+                matches!(Pin::new(&mut reader).poll_read(&mut cx, &mut buf), Poll::Ready(Ok(()))),
+                "25 paid tokens plus 75 later tokens must fund the original read with header_size={header_size}"
+            );
+            assert_eq!(buf.filled(), &data[..100 - header_size]);
+            assert_eq!(throttle.limiter.lock().expect("test throttle").available(), 0);
+        }
+    }
+
+    #[tokio::test(start_paused = true)]
+    async fn test_monitored_reader_preserves_paid_read_across_cancellation() {
+        use crate::bucket::bandwidth::reader::{MonitorReaderOptions, MonitoredReader};
+        use futures_util::task::noop_waker_ref;
+        use std::future::Future;
+        use std::pin::Pin;
+        use std::task::{Context, Poll};
+        use tokio::io::{AsyncRead, AsyncReadExt, AsyncWriteExt, ReadBuf};
+
+        let monitor = Monitor::new(1);
+        let opts = BucketOptions {
+            name: "reader-pending".to_string(),
+            replication_arn: "reader-pending-arn".to_string(),
+        };
+        monitor.set_bandwidth_limit(&opts.name, &opts.replication_arn, 100);
+        let throttle = monitor.throttle(&opts).expect("test throttle should exist");
+        supply_reader_test_tokens(&throttle, 100);
+        let (mut writer, inner) = tokio::io::duplex(100);
+        let mut reader = MonitoredReader::new(
+            monitor,
+            inner,
+            MonitorReaderOptions {
+                bucket_options: opts,
+                header_size: 0,
+            },
+        );
+        let mut cx = Context::from_waker(noop_waker_ref());
+        let mut first_output = [0; 100];
+        let mut first_read = Box::pin(reader.read(&mut first_output));
+        assert!(first_read.as_mut().poll(&mut cx).is_pending());
+        assert_eq!(throttle.limiter.lock().expect("test throttle").available(), 0);
+        drop(first_read);
+        writer
+            .write_all(&[0xAB; 100])
+            .await
+            .expect("duplex writer should supply the pending read");
+
+        // A canceled read future keeps its reader alive. Resume in a smaller
+        // buffer without charging the already-paid allowance a second time.
+        let mut short_output = [0; 40];
+        let mut short_buf = ReadBuf::new(&mut short_output);
+        assert!(matches!(Pin::new(&mut reader).poll_read(&mut cx, &mut short_buf), Poll::Ready(Ok(()))));
+        assert_eq!(short_buf.filled(), &[0xAB; 40]);
+        assert_eq!(throttle.limiter.lock().expect("test throttle").available(), 0);
+
+        // Completing the short read discards unused allowance, matching the
+        // existing precharge policy. The next read must buy a fresh budget.
+        let mut remaining_output = [0; 100];
+        let mut remaining_buf = ReadBuf::new(&mut remaining_output);
+        assert!(Pin::new(&mut reader).poll_read(&mut cx, &mut remaining_buf).is_pending());
+        supply_reader_test_tokens(&throttle, 100);
+        tokio::time::advance(Duration::from_secs(1)).await;
+        assert!(matches!(
+            Pin::new(&mut reader).poll_read(&mut cx, &mut remaining_buf),
+            Poll::Ready(Ok(()))
+        ));
+        assert_eq!(remaining_buf.filled(), &[0xAB; 60]);
+        assert_eq!(throttle.limiter.lock().expect("test throttle").available(), 0);
+
+        drop(writer);
+        let mut eof_output = [0; 100];
+        let mut eof_buf = ReadBuf::new(&mut eof_output);
+        assert!(Pin::new(&mut reader).poll_read(&mut cx, &mut eof_buf).is_pending());
+        supply_reader_test_tokens(&throttle, 100);
+        tokio::time::advance(Duration::from_secs(1)).await;
+        assert!(matches!(Pin::new(&mut reader).poll_read(&mut cx, &mut eof_buf), Poll::Ready(Ok(()))));
+        assert!(eof_buf.filled().is_empty());
+        assert_eq!(throttle.limiter.lock().expect("test throttle").available(), 0);
+        assert!(Pin::new(&mut reader).poll_read(&mut cx, &mut eof_buf).is_pending());
+    }
+
+    #[tokio::test(start_paused = true)]
+    async fn test_monitored_reader_shrinks_unpaid_payload_after_bandwidth_reduction() {
+        use crate::bucket::bandwidth::reader::{MonitorReaderOptions, MonitoredReader};
+        use futures_util::task::noop_waker_ref;
+        use std::pin::Pin;
+        use std::task::{Context, Poll};
+        use tokio::io::{AsyncRead, ReadBuf};
+
+        for header_size in [0, 25] {
+            let monitor = Monitor::new(1);
+            let opts = BucketOptions {
+                name: "reader-reduced-budget".to_string(),
+                replication_arn: "reader-reduced-budget-arn".to_string(),
+            };
+            monitor.set_bandwidth_limit(&opts.name, &opts.replication_arn, 100);
+            let initial_throttle = monitor.throttle(&opts).expect("initial throttle should exist");
+            supply_reader_test_tokens(&initial_throttle, 25);
+            let data = [0xAB; 100];
+            let mut reader = MonitoredReader::new(
+                monitor.clone(),
+                data.as_slice(),
+                MonitorReaderOptions {
+                    bucket_options: opts.clone(),
+                    header_size,
+                },
+            );
+            let mut output = [0; 100];
+            let mut buf = ReadBuf::new(&mut output);
+            let mut cx = Context::from_waker(noop_waker_ref());
+            assert!(Pin::new(&mut reader).poll_read(&mut cx, &mut buf).is_pending());
+            assert!(buf.filled().is_empty());
+            assert_eq!(initial_throttle.limiter.lock().expect("initial throttle").available(), 0);
+
+            monitor.set_bandwidth_limit(&opts.name, &opts.replication_arn, 10);
+            let reduced_throttle = monitor.throttle(&opts).expect("reduced throttle should exist");
+            assert_eq!(reduced_throttle.burst(), 10);
+            supply_reader_test_tokens(&reduced_throttle, 0);
+            tokio::time::advance(Duration::from_secs(1)).await;
+            let result = Pin::new(&mut reader).poll_read(&mut cx, &mut buf);
+            let expected_payload = if header_size == 0 {
+                assert!(
+                    matches!(result, Poll::Ready(Ok(()))),
+                    "already-paid payload must not wait for the old burst's unpaid remainder"
+                );
+                25
+            } else {
+                assert!(result.is_pending(), "the initial 25 tokens paid only the header");
+                assert!(buf.filled().is_empty(), "payload still requires its own reduced budget");
+                supply_reader_test_tokens(&reduced_throttle, 10);
+                tokio::time::advance(Duration::from_secs(1)).await;
+                assert!(matches!(Pin::new(&mut reader).poll_read(&mut cx, &mut buf), Poll::Ready(Ok(()))));
+                10
+            };
+            assert_eq!(buf.filled(), &data[..expected_payload]);
+            assert_eq!(reduced_throttle.limiter.lock().expect("reduced throttle").available(), 0);
+        }
+    }
+
     #[test]
     fn test_consume_refills_continuously() {
         let clock = TestClock::new();

crates/ecstore/src/bucket/bandwidth/reader.rs

@@ -41,6 +41,8 @@ pub struct MonitoredReader<R> {
     m: Arc<Monitor>,
     opts: MonitorReaderOptions,
     wait_state: std::sync::Mutex<Option<WaitState>>,
+    // Payload limit and unpaid tokens for the read retained across Pending polls.
+    pending_read: Option<(usize, u64)>,
     temp_buf: Vec<u8>,
 }
 
@@ -63,6 +65,7 @@ impl<R> MonitoredReader<R> {
             m,
             opts,
             wait_state: std::sync::Mutex::new(None),
+            pending_read: None,
             temp_buf: Vec::new(),
         }
     }
@@ -90,17 +93,30 @@ impl<R: AsyncRead + Unpin> AsyncRead for MonitoredReader<R> {
 
         let throttle = match this.m.throttle(&this.opts.bucket_options) {
             Some(t) => t,
-            None => return Pin::new(&mut this.r).poll_read(cx, buf),
+            None => {
+                this.pending_read = None;
+                return Pin::new(&mut this.r).poll_read(cx, buf);
+            }
         };
 
         let b = throttle.burst();
         debug_assert!(b >= 1, "burst must be at least 1");
-        let (need, tokens) = calc_need_and_tokens(b, buf.remaining(), &mut this.opts.header_size);
-        let (deficit, rate, consumed) = throttle.consume(tokens);
-        let need = need.min(consumed as usize);
+        let (need, tokens) = match this.pending_read {
+            Some(reservation) => reservation,
+            None => calc_need_and_tokens(b, buf.remaining(), &mut this.opts.header_size),
+        };
+        // A lower limit may discard unpaid payload, but must retain paid
+        // payload and any header tokens still owed by this pending read.
+        let excess_payload = need.saturating_sub(usize::try_from(b).unwrap_or(usize::MAX));
+        let reduction = tokens.min(u64::try_from(excess_payload).unwrap_or(u64::MAX));
+        let need = need.saturating_sub(usize::try_from(reduction).unwrap_or(usize::MAX));
+        let tokens = tokens - reduction;
+        let (deficit, rate, _) = throttle.consume(tokens);
+        this.pending_read = Some((need, deficit));
 
         if deficit > 0 && rate > 0.0 {
-            let duration = std::time::Duration::from_secs_f64(deficit as f64 / rate);
+            // A changed limit may reduce the burst while this read is pending.
+            let duration = std::time::Duration::from_secs_f64(deficit.min(b) as f64 / rate);
             debug!(
                 tokens = tokens,
                 deficit = deficit,
@@ -115,14 +131,17 @@ impl<R: AsyncRead + Unpin> AsyncRead for MonitoredReader<R> {
                         warn!("MonitoredReader wait_state mutex poisoned, recovering");
                         e.into_inner()
                     }) = Some(WaitState { sleep });
-                    return Poll::Pending;
                 }
-                Poll::Ready(()) => {}
+                Poll::Ready(()) => cx.waker().wake_by_ref(),
             }
+            return Poll::Pending;
         }
 
         let filled_before = buf.filled().len();
         let result = poll_limited_read(&mut this.r, cx, buf, need, &mut this.temp_buf);
+        if result.is_ready() {
+            this.pending_read = None;
+        }
         if let Poll::Ready(Ok(())) = result {
             let read_bytes = buf.filled().len().saturating_sub(filled_before) as u64;
             if read_bytes > 0 {

crates/ecstore/src/bucket/durability.rs

@@ -64,25 +64,40 @@ impl BucketDurabilityConfig {
     }
 }
 
-/// Default durability tier seeded into a newly created bucket's metadata
-/// (rustfs/backlog#1811). `relaxed` aligns new buckets with MinIO's default
-/// posture: object data is still fdatasynced, while xl.meta and directory-entry
-/// fsyncs follow the relaxed durability gate.
+/// Default durability policy for newly created buckets. An unset value must
+/// inherit the process-wide mode so a default single-node deployment remains
+/// strict. Relaxed and none remain explicit operator choices.
 pub const ENV_NEW_BUCKET_DURABILITY_MODE: &str = "RUSTFS_NEW_BUCKET_DURABILITY_MODE";
-pub const DEFAULT_NEW_BUCKET_DURABILITY_MODE: &str = BUCKET_DURABILITY_MODE_RELAXED;
+pub const DEFAULT_NEW_BUCKET_DURABILITY_MODE: &str = "inherit";
+
+const EVENT_NEW_BUCKET_DURABILITY_MODE: &str = "new_bucket_durability_mode";
+const LOG_COMPONENT_ECSTORE: &str = "ecstore";
+const LOG_SUBSYSTEM_BUCKET_DURABILITY: &str = "bucket_durability";
 
 /// The `durability.json` bytes to seed into a freshly created bucket's metadata.
 /// Empty means "no override" (the bucket then follows the global
-/// `RUSTFS_DURABILITY_MODE`); otherwise the serialized chosen tier. Operators
-/// can set `inherit` to disable the new-bucket override. Invalid values also
-/// fail closed to inherit the global mode instead of seeding a surprising tier.
-pub fn new_bucket_durability_config_json() -> Vec<u8> {
+/// `RUSTFS_DURABILITY_MODE`); otherwise the serialized chosen tier. Invalid
+/// values also fail closed to inherit the global mode instead of seeding a
+/// surprising tier.
+pub fn new_bucket_durability_config_json(bucket: &str) -> Vec<u8> {
     let raw = std::env::var(ENV_NEW_BUCKET_DURABILITY_MODE).unwrap_or_else(|_| DEFAULT_NEW_BUCKET_DURABILITY_MODE.to_string());
     let mode = raw.trim();
     if mode.eq_ignore_ascii_case("inherit") || mode.is_empty() || !BucketDurabilityConfig::is_valid_mode(mode) {
         return Vec::new();
     }
-    serde_json::to_vec(&BucketDurabilityConfig::new(mode)).expect("BucketDurabilityConfig serialization cannot fail")
+    let config = BucketDurabilityConfig::new(mode);
+    if let Some(mode) = config.normalized_mode().filter(|mode| mode != BUCKET_DURABILITY_MODE_STRICT) {
+        tracing::warn!(
+            event = EVENT_NEW_BUCKET_DURABILITY_MODE,
+            component = LOG_COMPONENT_ECSTORE,
+            subsystem = LOG_SUBSYSTEM_BUCKET_DURABILITY,
+            state = "non_strict_new_bucket_override",
+            bucket = %bucket,
+            mode = %mode,
+            "New bucket durability is explicitly configured below strict"
+        );
+    }
+    serde_json::to_vec(&config).expect("BucketDurabilityConfig serialization cannot fail")
 }
 
 #[cfg(test)]
@@ -90,7 +105,7 @@ mod tests {
     use super::*;
 
     fn new_bucket_seeded_mode() -> Option<String> {
-        let json = new_bucket_durability_config_json();
+        let json = new_bucket_durability_config_json("test-bucket");
         if json.is_empty() {
             return None;
         }
@@ -132,9 +147,9 @@ mod tests {
     }
 
     #[test]
-    fn new_bucket_default_seeds_relaxed_when_unset() {
+    fn new_bucket_default_inherits_when_unset() {
         temp_env::with_var_unset(ENV_NEW_BUCKET_DURABILITY_MODE, || {
-            assert_eq!(new_bucket_seeded_mode().as_deref(), Some(BUCKET_DURABILITY_MODE_RELAXED));
+            assert_eq!(new_bucket_seeded_mode(), None);
         });
     }
 

crates/ecstore/src/bucket/lifecycle/bucket_lifecycle_ops.rs

@@ -5606,6 +5606,11 @@ async fn apply_expiry_on_non_transitioned_objects_with_lock_lost_signal(
 
     //debug!("lc_event.action: {:?}", lc_event.action);
     debug!("expiry_on_non_transitioned_objects opts: {:?}", opts);
+    let usage_accounting = if lc_event.action == IlmAction::DeleteAction && !versioned && !version_suspended && !oi.is_dir {
+        Some(crate::data_usage::begin_expiry_usage_accounting(&oi.bucket).await)
+    } else {
+        None
+    };
     let mut dobj = match api
         .delete_object_with_tier_delete_journal(&oi.bucket, &encode_dir_object(&oi.name), opts)
         .await
@@ -5625,6 +5630,14 @@ async fn apply_expiry_on_non_transitioned_objects_with_lock_lost_signal(
             return false;
         }
     };
+    if let Some(accounting) = usage_accounting
+        && !dobj.name.is_empty()
+        && !dobj.delete_marker
+        && !dobj.is_dir
+        && let Ok(size) = crate::data_usage::quota_object_size(&dobj)
+    {
+        accounting.commit(size).await;
+    }
     schedule_lifecycle_replication_delete_if_needed(oi, &dobj).await;
 
     // The object (or all its versions, for delete_all) was expired; evict any
@@ -12891,6 +12904,130 @@ mod tests {
         );
     }
 
+    #[tokio::test]
+    #[serial]
+    async fn ordinary_expiry_accounts_the_committed_receipt_once() {
+        use crate::bucket::quota::{QuotaOperation, checker::QuotaChecker};
+        use crate::data_usage::{apply_bucket_usage_memory_overlay, replace_bucket_usage_memory_from_info};
+        use rustfs_data_usage::{BucketUsageInfo, DataUsageInfo};
+
+        let (_disk_paths, ecstore) = setup_test_env().await;
+        for size in [0usize, 42] {
+            let bytes = u64::try_from(size).expect("fixture size fits");
+            let bucket = format!("expiry-accounting-{}", Uuid::new_v4().simple());
+            create_test_bucket(&ecstore, &bucket).await;
+            let mut reader = PutObjReader::from_vec(vec![0; size]);
+            let mut queued = ecstore
+                .put_object(&bucket, "expire", &mut reader, &ObjectOptions::default())
+                .await
+                .expect("create expiring object");
+            let mut reader = PutObjReader::from_vec(vec![0; size]);
+            ecstore
+                .put_object(&bucket, "keep", &mut reader, &ObjectOptions::default())
+                .await
+                .expect("create retained object");
+            let mut baseline = DataUsageInfo {
+                last_update: Some(std::time::SystemTime::now()),
+                usage_snapshot_complete: true,
+                buckets_count: 1,
+                ..Default::default()
+            };
+            baseline.buckets_usage.insert(
+                bucket.clone(),
+                BucketUsageInfo {
+                    objects_count: 2,
+                    versions_count: 2,
+                    size: bytes * 2,
+                    ..Default::default()
+                },
+            );
+            baseline.bucket_sizes.insert(bucket.clone(), bytes * 2);
+            baseline.calculate_totals();
+            replace_bucket_usage_memory_from_info(&baseline).await;
+            // Legacy quota admission consumes the same conservative overlay.
+            // Durable reservations have their own storage-commit accounting.
+            metadata_sys::update(
+                &bucket,
+                rustfs_config::QUOTA_CONFIG_FILE,
+                format!(r#"{{"quota":{},"quota_type":"Hard"}}"#, bytes * 2).into_bytes(),
+            )
+            .await
+            .expect("configure the legacy quota");
+            let checker =
+                QuotaChecker::new(Arc::new(tokio::sync::RwLock::new(metadata_sys::BucketMetadataSys::new(ecstore.clone()))));
+            let attempted_size = bytes.max(1);
+            let quota = checker
+                .check_quota(&bucket, QuotaOperation::PutObject, attempted_size)
+                .await
+                .expect("check quota before expiration");
+            assert!(!quota.allowed);
+            assert!(!quota.uses_durable_reservations);
+            assert_eq!(quota.current_usage, Some(bytes * 2));
+            // Queued metadata is not the accounting receipt from the storage commit.
+            queued.size = 1;
+            let event = lifecycle::Event {
+                action: IlmAction::DeleteAction,
+                ..Default::default()
+            };
+            let incarnation = ecstore
+                .bucket_incarnation_id_from_disk(&bucket)
+                .await
+                .expect("bucket incarnation");
+            for (attempt_incarnation, expected) in [(Uuid::new_v4(), false), (incarnation, true), (incarnation, false)] {
+                assert_eq!(
+                    super::apply_expiry_on_non_transitioned_objects(
+                        ecstore.clone(),
+                        &queued,
+                        &event,
+                        &LcEventSrc::Scanner,
+                        attempt_incarnation,
+                    )
+                    .await,
+                    expected
+                );
+                let mut response = baseline.clone();
+                apply_bucket_usage_memory_overlay(&mut response).await;
+                let remaining = if attempt_incarnation == incarnation { 1 } else { 2 };
+                assert_eq!(response.buckets_usage[&bucket].objects_count, remaining);
+                assert_eq!(response.buckets_usage[&bucket].size, remaining * bytes);
+                let quota = checker
+                    .check_quota(&bucket, QuotaOperation::PutObject, attempted_size)
+                    .await
+                    .expect("check quota before scanner confirmation");
+                assert!(!quota.allowed, "expiry must not release unconfirmed quota credit");
+                assert_eq!(quota.current_usage, Some(bytes * 2));
+            }
+            let mut confirmed = baseline.clone();
+            confirmed.last_update = Some(std::time::SystemTime::now());
+            confirmed.scanner_epoch = Some(7);
+            confirmed.scanner_cycle = Some(10);
+            let usage = confirmed.buckets_usage.get_mut(&bucket).expect("bucket is in the snapshot");
+            usage.size = bytes;
+            usage.objects_count = 1;
+            usage.versions_count = 0; // The scanner does not count unversioned objects as versions.
+            confirmed.bucket_sizes.insert(bucket.clone(), bytes);
+            confirmed.calculate_totals();
+            replace_bucket_usage_memory_from_info(&confirmed).await;
+            let quota = checker
+                .check_quota(&bucket, QuotaOperation::PutObject, attempted_size)
+                .await
+                .expect("check quota after scanner confirmation");
+            assert_eq!(quota.allowed, bytes > 0);
+            assert_eq!(quota.current_usage, Some(bytes));
+            let too_large = checker
+                .check_quota(&bucket, QuotaOperation::PutObject, bytes + 1)
+                .await
+                .expect("check the first byte above the quota limit");
+            assert!(!too_large.allowed);
+            assert!(
+                ecstore
+                    .get_object_info(&bucket, "keep", &ObjectOptions::default())
+                    .await
+                    .is_ok()
+            );
+        }
+    }
+
     #[tokio::test]
     #[serial]
     async fn queued_delete_all_rechecks_a_same_id_rule_moved_into_the_future() {

crates/ecstore/src/bucket/metadata.rs

@@ -593,7 +593,7 @@ impl BucketMetadata {
     /// durability posture.
     pub fn new_with_default_durability(name: &str) -> Self {
         let mut metadata = Self::new(name);
-        metadata.durability_config_json = super::durability::new_bucket_durability_config_json();
+        metadata.durability_config_json = super::durability::new_bucket_durability_config_json(name);
         metadata
     }
 
@@ -1732,21 +1732,15 @@ mod test {
     }
 
     #[test]
-    fn new_bucket_metadata_constructor_seeds_default_durability() {
+    fn new_bucket_metadata_constructor_inherits_default_durability() {
         temp_env::with_var_unset(crate::bucket::durability::ENV_NEW_BUCKET_DURABILITY_MODE, || {
             let metadata = BucketMetadata::new_with_default_durability("new-user-bucket");
-            assert_eq!(
-                metadata.durability_config().and_then(|cfg| cfg.normalized_mode()).as_deref(),
-                Some(crate::bucket::durability::BUCKET_DURABILITY_MODE_RELAXED)
-            );
+            assert!(metadata.durability_config().is_none());
 
             let encoded = metadata.marshal_msg().expect("marshal metadata");
             let decoded = BucketMetadata::unmarshal(&encoded).expect("unmarshal metadata");
             assert_eq!(decoded.durability_config_json, metadata.durability_config_json);
-            assert_eq!(
-                decoded.durability_config().and_then(|cfg| cfg.normalized_mode()).as_deref(),
-                Some(crate::bucket::durability::BUCKET_DURABILITY_MODE_RELAXED)
-            );
+            assert!(decoded.durability_config().is_none());
         });
     }
 

crates/ecstore/src/cache_value/metacache_set.rs

@@ -1206,6 +1206,17 @@ mod tests {
         assert_eq!(mixed_timeout, DiskError::Timeout);
     }
 
+    #[test]
+    fn mixed_missing_and_body_io_errors_remain_an_actionable_quorum_failure() {
+        let failures = [
+            DiskError::FileNotFound,
+            DiskError::Io(std::io::Error::other("remote body stream aborted")),
+        ];
+
+        assert!(!is_benign_not_found_listing_failure(&failures));
+        assert_eq!(classify_listing_quorum_failure(&failures), DiskError::ErasureReadQuorum);
+    }
+
     #[test]
     fn missing_path_error_classification_excludes_actionable_failures() {
         assert!(is_missing_path_error(&DiskError::FileNotFound));

crates/ecstore/src/cluster/rpc/internode_data_transport.rs

@@ -901,6 +901,7 @@ pub fn build_internode_data_transport_from_env() -> Result<Arc<dyn InternodeData
 mod tests {
     use super::*;
     use std::sync::atomic::{AtomicBool, AtomicUsize, Ordering};
+    use tokio::io::AsyncWriteExt;
     use tokio::sync::{Barrier, Notify};
 
     async fn wait_for_capability_flight_waiters(entry: &PutFileCapabilityCacheEntry, waiters: usize) {
@@ -1026,6 +1027,45 @@ mod tests {
         assert!(matches!(open_err, Error::MethodNotAllowed));
     }
 
+    #[tokio::test]
+    async fn walk_dir_restores_explicit_remote_missing_error() {
+        let _ = rustfs_credentials::set_global_rpc_secret("walk-dir-error-test-secret".to_string());
+        let listener = match tokio::net::TcpListener::bind("127.0.0.1:0").await {
+            Ok(listener) => listener,
+            Err(err) if err.kind() == io::ErrorKind::PermissionDenied => return,
+            Err(err) => panic!("test listener should bind: {err}"),
+        };
+        let addr = listener.local_addr().expect("test listener address should be available");
+        let server = tokio::spawn(async move {
+            let (mut socket, _) = listener.accept().await.expect("test server should accept the request");
+            let mut request = [0_u8; 2048];
+            let _ = socket.read(&mut request).await.expect("request should be readable");
+            socket
+                .write_all(
+                    b"HTTP/1.1 500 Internal Server Error\r\nx-rustfs-disk-error: file-not-found\r\ncontent-length: 0\r\nconnection: close\r\n\r\n",
+                )
+                .await
+                .expect("typed disk error response should be writable");
+        });
+
+        let endpoint = format!("http://{addr}");
+        let error = match TcpHttpInternodeDataTransport
+            .open_walk_dir(WalkDirStreamRequest {
+                endpoint: endpoint.clone(),
+                disk: "disk-a".to_string(),
+                body: b"{}".to_vec(),
+                stall_timeout: Some(Duration::from_secs(2)),
+            })
+            .await
+        {
+            Ok(_) => panic!("an explicit remote missing-path response must fail the open"),
+            Err(error) => error,
+        };
+
+        assert!(matches!(error, Error::FileNotFound));
+        server.await.expect("test server task should finish");
+    }
+
     #[test]
     fn tcp_http_capabilities_are_behavior_preserving() {
         let transport = TcpHttpInternodeDataTransport;

crates/ecstore/src/config/com.rs

@@ -3154,6 +3154,7 @@ mod tests {
                 version_purge_status: Default::default(),
                 replication_decision: String::new(),
                 checksum: None,
+                multipart_completion_replayed: false,
             }
         }
     }

crates/ecstore/src/core/pools.rs

@@ -19542,7 +19542,10 @@ mod tests {
             .await
             .expect("second-node activation should not panic")
             .expect_err("the second reservation must observe and reject the committed first reservation");
-        assert!(err.to_string().contains("requires 60 bytes, but 40 bytes are available"));
+        assert!(
+            err.to_string().contains("requires 60 bytes, but 40 bytes are available"),
+            "the second reservation must report the committed capacity rejection, got {err:?}"
+        );
 
         let mut persisted = PoolMeta::default();
         persisted
@@ -19855,9 +19858,9 @@ mod tests {
                 .reload_pool_meta()
                 .await
                 .expect("load the active generation on the remote node");
-            other_store
-                .decommission_cancel(0)
+            tokio::time::timeout(std::time::Duration::from_secs(30), other_store.decommission_cancel(0))
                 .await
+                .expect("remote cancel must persist pool metadata without re-entering its state locks")
                 .expect("the remote node should cancel the old generation");
             other_store
                 .clear_decommission(0)
@@ -20122,9 +20125,9 @@ mod tests {
             .expect("the stale operation should have a valid terminal fence plan")
             .expect("the stale operation should have an active reservation");
 
-        first_node
-            .decommission_cancel(0)
+        tokio::time::timeout(std::time::Duration::from_secs(30), first_node.decommission_cancel(0))
             .await
+            .expect("cancel must persist pool metadata without re-entering its state locks")
             .expect("cancel the first durable operation");
         first_node
             .clear_decommission(0)
@@ -20627,8 +20630,9 @@ mod tests {
             .await
             .expect("second target mutation task should not panic")
             .expect("second target mutation should finalize");
-        cancel
+        tokio::time::timeout(std::time::Duration::from_secs(30), cancel)
             .await
+            .expect("cancel must finish after target mutations release their gates")
             .expect("decommission cancellation task should not panic")
             .expect("decommission cancellation should finish after target mutations");
 

crates/ecstore/src/core/pools_test.rs

@@ -2288,7 +2288,7 @@ mod decommission_lock_order_tests {
         let (lossy_store, refresh_calls) = store_with_capacity_lease_loss(&other_store).await;
         set_decommission_capacity_info_overrides_for_test(lossy_store.id, (0..40).map(|_| capacity_snapshot()).collect());
         let barrier = MultipartCommitBarrier::install(&bucket, object, MultipartCommitPause::AfterObjectPublication);
-        let migration = tokio::spawn({
+        let mut migration = tokio::spawn({
             let migration_store = Arc::clone(&lossy_store);
             let migration_bucket = bucket.clone();
             async move {
@@ -2304,7 +2304,12 @@ mod decommission_lock_order_tests {
                 .await
             }
         });
-        barrier.wait_until_paused().await;
+        tokio::select! {
+            () = barrier.wait_until_paused() => {}
+            result = &mut migration => {
+                panic!("initial multipart migration completed before reaching the publication barrier: {result:?}");
+            }
+        }
         tokio::time::pause();
         let published = lossy_store.pools[2]
             .get_object_info(
@@ -2321,6 +2326,18 @@ mod decommission_lock_order_tests {
             .expect("the multipart target must publish before capacity progress save");
         assert!(published.is_multipart(), "published target must retain multipart identity");
         assert_eq!(published.version_id.map(|version| version.to_string()), Some(source_version.clone()));
+        for prefix in [
+            rustfs_utils::http::RUSTFS_INTERNAL_PREFIX,
+            rustfs_utils::http::MINIO_INTERNAL_PREFIX,
+        ] {
+            assert_eq!(
+                published
+                    .user_defined
+                    .get(&format!("{prefix}{}", rustfs_utils::http::SUFFIX_MULTIPART_UPLOAD_ID)),
+                Some(&source_upload.upload_id),
+                "migration must preserve the source completion identity under both internal prefixes"
+            );
+        }
         let mut published_reader = lossy_store.pools[2]
             .get_object_reader(
                 &bucket,
@@ -6851,7 +6868,7 @@ mod decommission_lock_order_tests {
             let migration_store = Arc::clone(&store);
             let migration_bucket = bucket.clone();
             async move {
-                data_movement::migrate_decommission_object(
+                let result = data_movement::migrate_decommission_object(
                     migration_store,
                     0,
                     migration_bucket,
@@ -6860,16 +6877,19 @@ mod decommission_lock_order_tests {
                     "decommission_object_lock_order",
                     Some(owner),
                 )
-                .await
+                .await;
+                (result, std::time::Instant::now())
             }
         });
         barrier.wait_until_owner_paused().await;
 
+        let ordinary_launched_at = std::time::Instant::now();
         let mut ordinary_mutation = tokio::spawn({
             let ordinary_store = Arc::clone(&other_store);
             let ordinary_bucket = bucket.clone();
             async move {
-                match mutation {
+                let ordinary_started_at = std::time::Instant::now();
+                let result = match mutation {
                     ExternalObjectMutation::Put => {
                         let mut data = PutObjReader::from_vec(b"ordinary replacement".to_vec());
                         ordinary_store
@@ -6941,7 +6961,8 @@ mod decommission_lock_order_tests {
                             .restore_transitioned_object(&ordinary_bucket, object, &opts)
                             .await
                     }
-                }
+                };
+                (result, ordinary_started_at, std::time::Instant::now())
             }
         });
         if let Some(get_barrier) = restore_get_barrier.as_ref() {
@@ -6989,8 +7010,10 @@ mod decommission_lock_order_tests {
         } else {
             barrier.wait_until_external_capacity_released().await;
         }
+        let ordinary_lock_phase_observed_at = std::time::Instant::now();
         assert!(!ordinary_mutation.is_finished());
-        let migration_result = if pause_restore_tail {
+        let owner_released_at = std::time::Instant::now();
+        let (migration_result, migration_finished_at) = if pause_restore_tail {
             barrier.release_owner();
             tokio::time::timeout(std::time::Duration::from_secs(30), &mut migration)
                 .await
@@ -7005,7 +7028,7 @@ mod decommission_lock_order_tests {
         };
         migration_result.expect("migration should commit to its reserved target");
 
-        let ordinary_result = if pause_restore_tail {
+        let (ordinary_result, ordinary_started_at, ordinary_finished_at) = if pause_restore_tail {
             let handoff_barrier = PutObjectCommitBarrier::install(&bucket, object, PutObjectCommitPause::AfterRenameHandoff);
             let tail_barrier =
                 crate::set_disk::rename_fanout_barrier::arm(object, 0, crate::set_disk::rename_fanout_barrier::PHASE_RENAME);
@@ -7077,7 +7100,20 @@ mod decommission_lock_order_tests {
                 .expect("ordinary object mutation task should join")
         };
         drop(barrier);
-        ordinary_result.expect("ordinary object mutation should commit to the unreserved target");
+        ordinary_result.unwrap_or_else(|error| {
+            panic!(
+                "ordinary object mutation should commit to the unreserved target: {error:?}; \
+                 ordinary_start_after_launch={:?}, lock_phase_observed_after_launch={:?}, \
+                 owner_release_after_launch={:?}, migration_after_owner_release={:?}, \
+                 ordinary_finish_after_launch={:?}, ordinary_finished_before_migration={}",
+                ordinary_started_at.duration_since(ordinary_launched_at),
+                ordinary_lock_phase_observed_at.duration_since(ordinary_launched_at),
+                owner_released_at.duration_since(ordinary_launched_at),
+                migration_finished_at.duration_since(owner_released_at),
+                ordinary_finished_at.duration_since(ordinary_launched_at),
+                ordinary_finished_at < migration_finished_at,
+            );
+        });
         store.pools[1]
             .get_object_info(
                 &bucket,
@@ -7252,7 +7288,7 @@ mod decommission_lock_order_tests {
             let migration_store = Arc::clone(&store);
             let migration_bucket = bucket.clone();
             async move {
-                data_movement::migrate_decommission_object(
+                let result = data_movement::migrate_decommission_object(
                     migration_store,
                     0,
                     migration_bucket,
@@ -7261,11 +7297,13 @@ mod decommission_lock_order_tests {
                     "same_object_copy_lock_order",
                     Some(owner),
                 )
-                .await
+                .await;
+                (result, std::time::Instant::now())
             }
         });
         barrier.wait_until_owner_paused().await;
 
+        let ordinary_launched_at = std::time::Instant::now();
         let copy_task = tokio::spawn({
             let copy_store = Arc::clone(&other_store);
             let copy_bucket = bucket.clone();
@@ -7280,7 +7318,8 @@ mod decommission_lock_order_tests {
                 ..Default::default()
             };
             async move {
-                copy_store
+                let ordinary_started_at = std::time::Instant::now();
+                let result = copy_store
                     .copy_object(
                         &copy_bucket,
                         object,
@@ -7291,27 +7330,44 @@ mod decommission_lock_order_tests {
                         &destination_opts,
                     )
                     .await
-                    .map(|_| ())
+                    .map(|_| ());
+                (result, ordinary_started_at, std::time::Instant::now())
             }
         });
         barrier.wait_until_external_capacity_released().await;
+        let ordinary_lock_phase_observed_at = std::time::Instant::now();
         assert!(
             !copy_task.is_finished(),
             "historical same-object CopyObject must wait behind the migration object fence"
         );
+        let owner_released_at = std::time::Instant::now();
         barrier.release_owner();
         drop(barrier);
 
-        tokio::time::timeout(std::time::Duration::from_secs(30), migration)
+        let (migration_result, migration_finished_at) = tokio::time::timeout(std::time::Duration::from_secs(30), migration)
             .await
             .expect("migration must not deadlock with historical same-object CopyObject")
-            .expect("migration task should join")
-            .expect("migration should commit to its reserved target");
-        tokio::time::timeout(std::time::Duration::from_secs(30), copy_task)
-            .await
-            .expect("historical same-object CopyObject must finish after the migration releases its object fence")
-            .expect("CopyObject task should join")
-            .expect("CopyObject should preserve the expected current-version precondition");
+            .expect("migration task should join");
+        migration_result.expect("migration should commit to its reserved target");
+        let (ordinary_result, ordinary_started_at, ordinary_finished_at) =
+            tokio::time::timeout(std::time::Duration::from_secs(30), copy_task)
+                .await
+                .expect("historical same-object CopyObject must finish after the migration releases its object fence")
+                .expect("CopyObject task should join");
+        ordinary_result.unwrap_or_else(|error| {
+            panic!(
+                "CopyObject should preserve the expected current-version precondition: {error:?}; \
+                 ordinary_start_after_launch={:?}, lock_phase_observed_after_launch={:?}, \
+                 owner_release_after_launch={:?}, migration_after_owner_release={:?}, \
+                 ordinary_finish_after_launch={:?}, ordinary_finished_before_migration={}",
+                ordinary_started_at.duration_since(ordinary_launched_at),
+                ordinary_lock_phase_observed_at.duration_since(ordinary_launched_at),
+                owner_released_at.duration_since(ordinary_launched_at),
+                migration_finished_at.duration_since(owner_released_at),
+                ordinary_finished_at.duration_since(ordinary_launched_at),
+                ordinary_finished_at < migration_finished_at,
+            );
+        });
     }
 
     #[tokio::test]

crates/ecstore/src/data_usage/mod.rs

@@ -43,7 +43,7 @@ use std::{
     future::Future,
     sync::{
         Arc, LazyLock, OnceLock,
-        atomic::{AtomicU64, Ordering},
+        atomic::{AtomicU64, AtomicUsize, Ordering},
     },
     time::{Duration, SystemTime},
 };
@@ -83,6 +83,79 @@ struct CachedBucketUsage {
     // necessarily a complete scanner reconciliation and therefore creates a
     // fresh cache entry with no pending hold.
     pending_negative_delta: u64,
+    pending_negative_updated_at: Option<SystemTime>,
+    pending_negative_reconciliation: Option<PendingDeleteReconciliation>,
+    in_flight_expirations: Arc<AtomicUsize>,
+}
+
+#[derive(Debug, Clone)]
+struct PendingDeleteReconciliation {
+    bytes: u64,
+    deleted_at: SystemTime,
+    observed: Option<((u64, u64), SystemTime)>,
+}
+
+impl CachedBucketUsage {
+    fn reconcile_pending_deletes(&mut self, position: Option<(u64, u64)>, updated_at: SystemTime) {
+        let Some(position) = position else { return };
+        let Some(pending) = self.pending_negative_reconciliation.as_mut() else { return };
+        match pending.observed {
+            None if updated_at >= pending.deleted_at => pending.observed = Some((position, updated_at)),
+            Some((previous, observed_at)) if position.0 >= previous.0 && position.1 > previous.1 && updated_at >= observed_at => {
+                self.pending_negative_delta = self.pending_negative_delta.saturating_sub(pending.bytes);
+                self.pending_negative_reconciliation = self
+                    .pending_negative_updated_at
+                    .filter(|_| self.pending_negative_delta > 0)
+                    .map(|deleted_at| PendingDeleteReconciliation {
+                        bytes: self.pending_negative_delta,
+                        deleted_at,
+                        observed: (updated_at >= deleted_at).then_some((position, updated_at)),
+                    });
+            }
+            _ => {}
+        }
+    }
+}
+
+/// Keeps a scanner refresh from including a delete before its memory receipt.
+pub(crate) struct ExpiryUsageAccounting<'a> {
+    bucket: &'a str,
+    in_flight: Arc<AtomicUsize>,
+}
+
+impl ExpiryUsageAccounting<'_> {
+    pub(crate) async fn commit(self, deleted_size: u64) {
+        let mut cache = memory_cache().write().await;
+        if let Some(entry) = cache.get_mut(self.bucket)
+            && Arc::ptr_eq(&entry.in_flight_expirations, &self.in_flight)
+        {
+            apply_bucket_object_delete_memory(entry, deleted_size, true);
+        }
+    }
+}
+
+impl Drop for ExpiryUsageAccounting<'_> {
+    fn drop(&mut self) {
+        self.in_flight.fetch_sub(1, Ordering::AcqRel);
+    }
+}
+
+pub(crate) async fn begin_expiry_usage_accounting(bucket: &str) -> ExpiryUsageAccounting<'_> {
+    ensure_bucket_usage_cached(bucket).await;
+    let mut cache = memory_cache().write().await;
+    // The caller owns the bucket name through commit. Avoid allocating either
+    // a receipt name or a map key for every expiry in an already cached bucket.
+    let in_flight = match cache.get(bucket) {
+        Some(entry) => Arc::clone(&entry.in_flight_expirations),
+        None => {
+            let entry = cached_bucket_usage_now(BucketUsageInfo::default());
+            let in_flight = Arc::clone(&entry.in_flight_expirations);
+            cache.insert(bucket.to_owned(), entry);
+            in_flight
+        }
+    };
+    in_flight.fetch_add(1, Ordering::AcqRel);
+    ExpiryUsageAccounting { bucket, in_flight }
 }
 
 type UsageMemoryCache = Arc<RwLock<HashMap<String, CachedBucketUsage>>>;
@@ -1945,6 +2018,9 @@ fn cached_bucket_usage_from_backend(usage: BucketUsageInfo, updated_at: SystemTi
         stale_snapshot_pending: false,
         pending_scanner_position: None,
         pending_negative_delta: 0,
+        pending_negative_updated_at: None,
+        pending_negative_reconciliation: None,
+        in_flight_expirations: Arc::new(AtomicUsize::new(0)),
     }
 }
 
@@ -1959,6 +2035,9 @@ fn cached_bucket_usage_now(usage: BucketUsageInfo) -> CachedBucketUsage {
         stale_snapshot_pending: false,
         pending_scanner_position: None,
         pending_negative_delta: 0,
+        pending_negative_updated_at: None,
+        pending_negative_reconciliation: None,
+        in_flight_expirations: Arc::new(AtomicUsize::new(0)),
     }
 }
 
@@ -2110,6 +2189,10 @@ pub async fn record_bucket_object_delete_memory(bucket: &str, deleted_size: u64,
         .entry(bucket.to_string())
         .or_insert_with(|| cached_bucket_usage_now(BucketUsageInfo::default()));
 
+    apply_bucket_object_delete_memory(entry, deleted_size, removed_current_object);
+}
+
+fn apply_bucket_object_delete_memory(entry: &mut CachedBucketUsage, deleted_size: u64, removed_current_object: bool) {
     entry.usage.size = entry.usage.size.saturating_sub(deleted_size);
     entry.pending_negative_delta = entry.pending_negative_delta.saturating_add(deleted_size);
     if removed_current_object {
@@ -2118,6 +2201,16 @@ pub async fn record_bucket_object_delete_memory(bucket: &str, deleted_size: u64,
     }
 
     let now = SystemTime::now();
+    entry.pending_negative_updated_at = Some(now);
+    // Freeze this tranche even before its first observation. Later deletes
+    // must not keep moving the timestamp that the scanner needs to catch up to.
+    if entry.pending_negative_reconciliation.is_none() && entry.pending_negative_delta > 0 {
+        entry.pending_negative_reconciliation = Some(PendingDeleteReconciliation {
+            bytes: entry.pending_negative_delta,
+            deleted_at: now,
+            observed: None,
+        });
+    }
     entry.refreshed_at = now;
     entry.usage_updated_at = now;
     entry.dirty = true;
@@ -2257,7 +2350,19 @@ async fn replace_bucket_usage_memory_from_info_if_generation(data_usage_info: &D
     if expected_generation.is_some_and(|expected| usage_memory_generation() != expected) {
         return;
     }
-    for (bucket, existing) in cache.iter() {
+    for (bucket, existing) in cache.iter_mut() {
+        existing.reconcile_pending_deletes(snapshot_position, usage_updated_at);
+        if existing.in_flight_expirations.load(Ordering::Acquire) > 0 {
+            next_cache.insert(bucket.clone(), existing.clone());
+            continue;
+        }
+        // Leadership fencing can advance the epoch without scanning. Only a
+        // later cycle can reconcile a dirty overlay that disagreed with the
+        // first complete observation after its last request mutation.
+        let reconciled_dirty_usage = existing
+            .pending_scanner_position
+            .zip(snapshot_position)
+            .is_some_and(|(previous, current)| current.0 >= previous.0 && current.1 > previous.1);
         match next_cache.entry(bucket.clone()) {
             Entry::Occupied(mut candidate) => {
                 if let Some(preserved) = preserve_unknown_dirty_usage(existing, snapshot_position, usage_updated_at) {
@@ -2275,7 +2380,10 @@ async fn replace_bucket_usage_memory_from_info_if_generation(data_usage_info: &D
                     candidate.insert(existing.clone());
                     continue;
                 }
-                if existing.authoritative && existing.dirty && !bucket_usage_counts_match(&existing.usage, &candidate.get().usage)
+                if existing.authoritative
+                    && existing.dirty
+                    && !reconciled_dirty_usage
+                    && !bucket_usage_counts_match(&existing.usage, &candidate.get().usage)
                 {
                     // A scanner snapshot can be saved after newer writes but still miss them if it listed the bucket earlier.
                     let mut preserved = existing.clone();
@@ -2285,8 +2393,10 @@ async fn replace_bucket_usage_memory_from_info_if_generation(data_usage_info: &D
                         // counts until they also converge, but release the
                         // conservative quota hold.
                         preserved.pending_negative_delta = 0;
+                        preserved.pending_negative_reconciliation = None;
                     }
                     preserved.stale_snapshot_pending = true;
+                    preserved.pending_scanner_position = preserved.pending_scanner_position.or(snapshot_position);
                     candidate.insert(preserved);
                 }
             }
@@ -2295,14 +2405,16 @@ async fn replace_bucket_usage_memory_from_info_if_generation(data_usage_info: &D
                     candidate.insert(preserved);
                 } else if existing.authoritative && existing.usage_updated_at > usage_updated_at {
                     candidate.insert(existing.clone());
-                } else if existing.authoritative && existing.dirty {
+                } else if existing.authoritative && existing.dirty && !reconciled_dirty_usage {
                     let mut preserved = existing.clone();
                     if preserved.pending_negative_delta > 0 && preserved.usage.size == 0 {
                         // Omitting the bucket from a complete snapshot confirms
                         // that its post-delete byte total is zero.
                         preserved.pending_negative_delta = 0;
+                        preserved.pending_negative_reconciliation = None;
                     }
                     preserved.stale_snapshot_pending = true;
+                    preserved.pending_scanner_position = preserved.pending_scanner_position.or(snapshot_position);
                     candidate.insert(preserved);
                 }
             }
@@ -5405,11 +5517,12 @@ mod tests {
             .expect("delete usage should remain cached")
             .usage_updated_at;
 
-        let stale = data_usage_info_for_test("bucket-a", 2, 768, mutation_update + Duration::from_nanos(1));
+        // Windows SystemTime has 100 ns resolution; retain strict ordering there.
+        let stale = data_usage_info_for_test("bucket-a", 2, 768, mutation_update + Duration::from_micros(1));
         replace_bucket_usage_memory_from_info(&stale).await;
         assert_eq!(get_bucket_usage_memory("bucket-a").await, Some(768));
 
-        let mut too_old_matching = data_usage_info_for_test("bucket-a", 1, 256, mutation_update - Duration::from_nanos(1));
+        let mut too_old_matching = data_usage_info_for_test("bucket-a", 1, 256, mutation_update - Duration::from_micros(1));
         too_old_matching
             .buckets_usage
             .get_mut("bucket-a")
@@ -5423,7 +5536,7 @@ mod tests {
             "a matching byte total from before the delete cannot release the hold"
         );
 
-        let mut reconciled = data_usage_info_for_test("bucket-a", 1, 256, mutation_update + Duration::from_nanos(2));
+        let mut reconciled = data_usage_info_for_test("bucket-a", 1, 256, mutation_update + Duration::from_micros(2));
         reconciled
             .buckets_usage
             .get_mut("bucket-a")
@@ -5759,6 +5872,333 @@ mod tests {
         );
     }
 
+    #[tokio::test]
+    #[serial]
+    async fn complete_scanner_cycles_reconcile_expired_dirty_usage() {
+        for (remaining_objects, omit_bucket) in [(0, false), (1, false), (0, true)] {
+            clear_usage_memory_cache_for_test().await;
+            let bucket = "expired-dirty-usage";
+            let baseline = data_usage_info_for_test(bucket, 0, 0, SystemTime::now());
+            replace_bucket_usage_memory_from_info(&baseline).await;
+            record_bucket_object_write_memory(bucket, None, 42).await;
+            record_bucket_object_write_memory(bucket, None, 42).await;
+
+            // Lifecycle deletion bypasses request-layer memory accounting.
+            // Repeated complete scans must eventually replace that dirty overlay.
+            for cycle in 1..=3 {
+                let mut snapshot = data_usage_info_for_test(bucket, remaining_objects, remaining_objects * 42, SystemTime::now());
+                snapshot.scanner_epoch = Some(7);
+                snapshot.scanner_cycle = Some(cycle);
+                if omit_bucket {
+                    snapshot.buckets_usage.clear();
+                    snapshot.bucket_sizes.clear();
+                    snapshot.buckets_count = 0;
+                    snapshot.calculate_totals();
+                }
+                replace_bucket_usage_memory_from_info(&snapshot).await;
+                if cycle == 3 {
+                    apply_bucket_usage_memory_overlay_if_authoritative(&mut snapshot, true).await;
+                    assert_eq!(snapshot.objects_total_count, remaining_objects);
+                    assert_eq!(snapshot.objects_total_size, remaining_objects * 42);
+                    if omit_bucket {
+                        assert!(!snapshot.buckets_usage.contains_key(bucket));
+                    } else {
+                        assert_eq!(snapshot.buckets_usage[bucket].objects_count, remaining_objects);
+                        assert_eq!(snapshot.buckets_usage[bucket].size, remaining_objects * 42);
+                    }
+                }
+            }
+        }
+    }
+
+    #[tokio::test]
+    #[serial]
+    async fn expiry_receipts_exclude_refresh_until_committed_or_cancelled() {
+        clear_usage_memory_cache_for_test().await;
+        let bucket = "expiry-receipt";
+        replace_bucket_usage_memory_from_info(&data_usage_info_for_test(bucket, 2, 84, SystemTime::now())).await;
+        let first = begin_expiry_usage_accounting(bucket).await;
+        let cancelled = begin_expiry_usage_accounting(bucket).await;
+        let after_delete = data_usage_info_for_test(bucket, 1, 42, SystemTime::now());
+        replace_bucket_usage_memory_from_info(&after_delete).await;
+        assert_eq!(memory_cache().read().await[bucket].usage.size, 84);
+        first.commit(42).await;
+        assert_eq!(memory_cache().read().await[bucket].usage.size, 42);
+
+        let mut omitted = data_usage_info_for_test(bucket, 0, 0, SystemTime::now());
+        omitted.buckets_usage.clear();
+        omitted.bucket_sizes.clear();
+        omitted.buckets_count = 0;
+        omitted.calculate_totals();
+        replace_bucket_usage_memory_from_info(&omitted).await;
+        assert_eq!(memory_cache().read().await[bucket].usage.size, 42);
+        drop(cancelled);
+        assert_eq!(
+            memory_cache().read().await[bucket]
+                .in_flight_expirations
+                .load(Ordering::Acquire),
+            0
+        );
+
+        replace_bucket_usage_memory_from_info(&data_usage_info_for_test(bucket, 1, 42, SystemTime::now())).await;
+        assert_eq!(get_bucket_usage_memory(bucket).await, Some(42));
+    }
+
+    #[tokio::test]
+    #[serial]
+    async fn cancelled_expiry_receipt_reconciles_without_another_mutation() {
+        clear_usage_memory_cache_for_test().await;
+        let bucket = "expiry-cancelled-after-delete";
+        replace_bucket_usage_memory_from_info(&data_usage_info_for_test(bucket, 0, 0, SystemTime::now())).await;
+        record_bucket_object_write_memory(bucket, None, 42).await;
+        record_bucket_object_write_memory(bucket, None, 42).await;
+        let receipt = begin_expiry_usage_accounting(bucket).await;
+
+        // Storage has deleted one object, but the task has not committed its
+        // memory receipt. Even two complete scans must not replace its baseline.
+        let mut scanned = data_usage_info_for_test(bucket, 1, 42, SystemTime::now());
+        scanned.scanner_epoch = Some(7);
+        for cycle in [10, 11] {
+            scanned.scanner_cycle = Some(cycle);
+            replace_bucket_usage_memory_from_info(&scanned).await;
+            assert_eq!(get_bucket_usage_memory(bucket).await, Some(84));
+        }
+        drop(receipt);
+
+        for (cycle, expected) in [(12, 84), (13, 42)] {
+            scanned.scanner_cycle = Some(cycle);
+            replace_bucket_usage_memory_from_info(&scanned).await;
+            assert_eq!(get_bucket_usage_memory(bucket).await, Some(expected));
+            let mut displayed = scanned.clone();
+            apply_bucket_usage_memory_overlay(&mut displayed).await;
+            assert_eq!(displayed.buckets_usage[bucket].size, expected);
+            assert_eq!(displayed.buckets_usage[bucket].objects_count, expected / 42);
+        }
+    }
+
+    #[tokio::test]
+    #[serial]
+    async fn expiry_restart_reloads_durable_usage_without_persisting_the_overlay() {
+        use crate::bucket::metadata_sys::test_support::isolated_store_over_temp_disks;
+
+        let (_dirs, store) = isolated_store_over_temp_disks().await;
+        let bucket = "expiry-restart";
+        clear_usage_memory_cache_for_test().await;
+        let baseline = data_usage_info_for_test(bucket, 2, 84, SystemTime::now());
+        store_data_usage_in_backend(baseline.clone(), store.clone())
+            .await
+            .expect("persist the pre-expiration scanner snapshot");
+        replace_bucket_usage_memory_from_info(&baseline).await;
+        begin_expiry_usage_accounting(bucket).await.commit(42).await;
+        let mut displayed = baseline.clone();
+        apply_bucket_usage_memory_overlay(&mut displayed).await;
+        assert_eq!(displayed.buckets_usage[bucket].size, 42);
+        assert_eq!(get_bucket_usage_memory(bucket).await, Some(84));
+
+        // Recreate only process-local usage state, then use the real backend
+        // loader. This is a restart-state regression, not a process restart E2E.
+        clear_usage_memory_cache_for_test().await;
+        invalidate_data_usage_snapshot_cache().await;
+        let recovered = load_data_usage_from_backend_cached(store.clone())
+            .await
+            .expect("reload the durable pre-expiration snapshot");
+        assert_eq!(recovered.buckets_usage[bucket].size, 84);
+        replace_bucket_usage_memory_from_info(&recovered).await;
+        assert_eq!(get_bucket_usage_memory(bucket).await, Some(84));
+
+        let confirmed = data_usage_info_for_test(bucket, 1, 42, SystemTime::now());
+        store_data_usage_in_backend(confirmed, store.clone())
+            .await
+            .expect("persist scanner confirmation without another object mutation");
+        let recovered = load_data_usage_from_backend_cached(store.clone())
+            .await
+            .expect("reload the confirmed snapshot");
+        replace_bucket_usage_memory_from_info(&recovered).await;
+        assert_eq!(get_bucket_usage_memory(bucket).await, Some(42));
+
+        // Historical overlay drift must not contaminate the persisted scan.
+        record_bucket_object_write_memory(bucket, None, 42).await;
+        clear_usage_memory_cache_for_test().await;
+        invalidate_data_usage_snapshot_cache().await;
+        let mut recovered = load_data_usage_from_backend_cached(store)
+            .await
+            .expect("reload the confirmed snapshot after losing the stale overlay");
+        replace_bucket_usage_memory_from_info(&recovered).await;
+        apply_bucket_usage_memory_overlay(&mut recovered).await;
+        assert_eq!(recovered.buckets_usage[bucket].objects_count, 1);
+        assert_eq!(recovered.buckets_usage[bucket].size, 42);
+        assert_eq!(get_bucket_usage_memory(bucket).await, Some(42));
+        clear_usage_memory_cache_for_test().await;
+        invalidate_data_usage_snapshot_cache().await;
+    }
+
+    #[tokio::test]
+    #[serial]
+    async fn expiry_receipt_cannot_credit_a_replaced_bucket_cache() {
+        clear_usage_memory_cache_for_test().await;
+        let bucket = "expiry-recreated";
+        replace_bucket_usage_memory_from_info(&data_usage_info_for_test(bucket, 1, 42, SystemTime::now())).await;
+        let receipt = begin_expiry_usage_accounting(bucket).await;
+        clear_usage_memory_cache_for_test().await;
+        replace_bucket_usage_memory_from_info(&data_usage_info_for_test(bucket, 2, 84, SystemTime::now())).await;
+        receipt.commit(42).await;
+        assert_eq!(memory_cache().read().await[bucket].usage.objects_count, 2);
+        assert_eq!(get_bucket_usage_memory(bucket).await, Some(84));
+    }
+
+    #[tokio::test]
+    #[serial]
+    async fn quota_delete_tranches_advance_despite_newer_mutations() {
+        clear_usage_memory_cache_for_test().await;
+        let bucket = "expiry-busy-quota";
+        replace_bucket_usage_memory_from_info(&data_usage_info_for_test(bucket, 10, 420, SystemTime::now())).await;
+        record_bucket_object_delete_memory(bucket, 42, true).await;
+        let mut observation = data_usage_info_for_test(bucket, 9, 378, SystemTime::now());
+        observation.scanner_epoch = Some(7);
+        observation.scanner_cycle = Some(10);
+        record_bucket_object_write_memory(bucket, None, 42).await;
+        replace_bucket_usage_memory_from_info(&observation).await;
+        assert_eq!(get_bucket_usage_memory(bucket).await, Some(462));
+
+        for (epoch, cycle, partial, old_time) in [
+            (8, 10, false, false),
+            (6, 11, false, false),
+            (7, 11, true, false),
+            (7, 11, false, true),
+        ] {
+            let mut rejected = observation.clone();
+            rejected.scanner_epoch = Some(epoch);
+            rejected.scanner_cycle = Some(cycle);
+            rejected.usage_snapshot_complete = !partial;
+            rejected.usage_snapshot_partial = partial;
+            if old_time {
+                rejected.last_update = Some(SystemTime::UNIX_EPOCH);
+            }
+            replace_bucket_usage_memory_from_info(&rejected).await;
+            assert_eq!(memory_cache().read().await[bucket].pending_negative_delta, 42);
+        }
+
+        observation.last_update = Some(SystemTime::now());
+        observation.scanner_cycle = Some(11);
+        // A newer delete must stay charged, without delaying the older receipt.
+        record_bucket_object_delete_memory(bucket, 42, true).await;
+        record_bucket_object_write_memory(bucket, None, 42).await;
+        replace_bucket_usage_memory_from_info(&observation).await;
+        assert_eq!(memory_cache().read().await[bucket].pending_negative_delta, 42);
+        assert_eq!(get_bucket_usage_memory(bucket).await, Some(462));
+
+        for cycle in [12, 13] {
+            observation.last_update = Some(SystemTime::now());
+            observation.scanner_cycle = Some(cycle);
+            record_bucket_object_write_memory(bucket, None, 42).await;
+            replace_bucket_usage_memory_from_info(&observation).await;
+        }
+        assert_eq!(memory_cache().read().await[bucket].pending_negative_delta, 0);
+        assert_eq!(get_bucket_usage_memory(bucket).await, Some(504));
+    }
+
+    #[tokio::test]
+    #[serial]
+    async fn quota_delete_hold_stays_bounded_with_deletes_after_every_observation() {
+        clear_usage_memory_cache_for_test().await;
+        let bucket = "expiry-continuous-quota";
+        replace_bucket_usage_memory_from_info(&data_usage_info_for_test(bucket, 10, 420, SystemTime::now())).await;
+        record_bucket_object_delete_memory(bucket, 42, true).await;
+        for cycle in 1..=12 {
+            let mut observed = data_usage_info_for_test(bucket, 9, 378, SystemTime::now());
+            observed.scanner_epoch = Some(7);
+            observed.scanner_cycle = Some(cycle);
+            record_bucket_object_write_memory(bucket, None, 42).await;
+            record_bucket_object_delete_memory(bucket, 42, true).await;
+            replace_bucket_usage_memory_from_info(&observed).await;
+            let cache = memory_cache().read().await;
+            assert_eq!(cache[bucket].usage.size, 378);
+            assert!(
+                cache[bucket].pending_negative_delta <= 126,
+                "confirmed older deletes must leave the quota hold"
+            );
+        }
+    }
+
+    #[tokio::test]
+    #[serial]
+    async fn dirty_usage_reconciliation_requires_a_later_complete_observation() {
+        clear_usage_memory_cache_for_test().await;
+        let bucket = "dirty-observation-fence";
+        let baseline = data_usage_info_for_test(bucket, 0, 0, SystemTime::now());
+        replace_bucket_usage_memory_from_info(&baseline).await;
+        record_bucket_object_write_memory(bucket, None, 42).await;
+        let mutation_at = memory_cache().read().await[bucket].usage_updated_at;
+        let mut first = data_usage_info_for_test(bucket, 0, 0, mutation_at);
+        first.scanner_epoch = Some(7);
+        first.scanner_cycle = Some(10);
+        let mut partial = first.clone();
+        partial.usage_snapshot_partial = true;
+        partial.usage_snapshot_complete = false;
+        replace_bucket_usage_memory_from_info(&partial).await;
+        assert_eq!(memory_cache().read().await[bucket].pending_scanner_position, None);
+        replace_bucket_usage_memory_from_info(&first).await;
+
+        for (epoch, cycle, partial, update) in [
+            (Some(7), Some(10), false, mutation_at),
+            (Some(8), Some(10), false, mutation_at),
+            (Some(6), Some(11), false, mutation_at),
+            (Some(7), Some(9), false, mutation_at),
+            (Some(7), Some(11), true, mutation_at),
+            (None, Some(11), false, mutation_at),
+            (Some(7), None, false, mutation_at),
+            // Windows `SystemTime` stores 100 ns intervals, so a 1 ns offset is a no-op there.
+            (Some(7), Some(11), false, mutation_at - Duration::from_micros(1)),
+        ] {
+            let mut snapshot = first.clone();
+            snapshot.scanner_epoch = epoch;
+            snapshot.scanner_cycle = cycle;
+            snapshot.usage_snapshot_partial = partial;
+            snapshot.usage_snapshot_complete = !partial;
+            snapshot.last_update = Some(update);
+            replace_bucket_usage_memory_from_info(&snapshot).await;
+            apply_bucket_usage_memory_overlay_if_authoritative(&mut snapshot, true).await;
+            assert_eq!(snapshot.objects_total_count, 1, "epoch={epoch:?}, cycle={cycle:?}, partial={partial}");
+            assert_eq!(snapshot.objects_total_size, 42);
+        }
+
+        let mut successor = first;
+        successor.scanner_epoch = Some(8);
+        successor.scanner_cycle = Some(11);
+        replace_bucket_usage_memory_from_info(&successor).await;
+        apply_bucket_usage_memory_overlay_if_authoritative(&mut successor, true).await;
+        assert_eq!(successor.objects_total_count, 0);
+        assert_eq!(successor.objects_total_size, 0);
+    }
+
+    #[tokio::test]
+    #[serial]
+    async fn a_new_authoritative_mutation_restarts_reconciliation() {
+        clear_usage_memory_cache_for_test().await;
+        let bucket = "dirty-mutation-fence";
+        replace_bucket_usage_memory_from_info(&data_usage_info_for_test(bucket, 0, 0, SystemTime::now())).await;
+        record_bucket_object_write_memory(bucket, None, 42).await;
+        let mut snapshot = data_usage_info_for_test(bucket, 0, 0, SystemTime::now());
+        snapshot.scanner_epoch = Some(7);
+        snapshot.scanner_cycle = Some(10);
+        replace_bucket_usage_memory_from_info(&snapshot).await;
+
+        record_bucket_object_write_memory(bucket, None, 42).await;
+        snapshot.last_update = Some(SystemTime::now());
+        snapshot.scanner_cycle = Some(11);
+        replace_bucket_usage_memory_from_info(&snapshot).await;
+        let mut response = snapshot.clone();
+        apply_bucket_usage_memory_overlay_if_authoritative(&mut response, true).await;
+        assert_eq!(response.objects_total_count, 2);
+        assert_eq!(response.objects_total_size, 84);
+
+        snapshot.scanner_cycle = Some(12);
+        replace_bucket_usage_memory_from_info(&snapshot).await;
+        apply_bucket_usage_memory_overlay_if_authoritative(&mut snapshot, true).await;
+        assert_eq!(snapshot.objects_total_count, 0);
+        assert_eq!(snapshot.objects_total_size, 0);
+    }
+
     #[tokio::test]
     #[serial]
     async fn authoritative_refresh_can_reduce_memory_usage() {

crates/ecstore/src/disk/disk_store.rs

@@ -2085,7 +2085,9 @@ impl DiskAPI for LocalDiskWrapper {
                     }
                     let result = self.disk.disk_info(opts).await?;
 
-                    if let Some(current_disk_id) = *self.disk_id.read().await
+                    // Fresh capacity snapshots omit the disk ID; the stale-disk precheck above already verified it.
+                    if !opts.fresh_capacity
+                        && let Some(current_disk_id) = *self.disk_id.read().await
                         && Some(current_disk_id) != result.id
                     {
                         return Err(DiskError::DiskNotFound);
@@ -2571,7 +2573,9 @@ impl DiskAPI for LocalDiskWrapper {
 mod tests {
     use super::*;
     use crate::disk::endpoint::Endpoint;
+    use crate::disk::format::FormatV3;
     use crate::disk::health_state::RuntimeDriveHealthState;
+    use crate::disk::{FORMAT_CONFIG_FILE, RUSTFS_META_BUCKET};
     use std::{
         io,
         panic::{AssertUnwindSafe, catch_unwind},
@@ -2926,6 +2930,56 @@ mod tests {
         assert_eq!(snapshot.total_errors_availability, 1);
     }
 
+    #[tokio::test]
+    async fn fresh_capacity_keeps_disk_identity_precheck_without_requiring_id_in_snapshot() {
+        let dir = tempfile::tempdir().expect("temp dir should be created");
+        let mut endpoint =
+            Endpoint::try_from(dir.path().to_str().expect("temp dir should be valid UTF-8")).expect("endpoint should parse");
+        endpoint.set_pool_index(0);
+        endpoint.set_set_index(0);
+        endpoint.set_disk_index(0);
+        let meta_dir = dir.path().join(RUSTFS_META_BUCKET);
+        tokio::fs::create_dir_all(&meta_dir)
+            .await
+            .expect("metadata directory should be created");
+        let mut format = FormatV3::new(1, 1);
+        format.erasure.this = format.erasure.sets[0][0];
+        tokio::fs::write(meta_dir.join(FORMAT_CONFIG_FILE), format.to_json().expect("format should serialize"))
+            .await
+            .expect("format should be written");
+
+        let disk = Arc::new(LocalDisk::new(&endpoint, false).await.expect("formatted disk should open"));
+        let disk_id = disk
+            .get_disk_id()
+            .await
+            .expect("disk ID should be readable")
+            .expect("formatted disk should have an ID");
+        let wrapper = LocalDiskWrapper::new(Arc::clone(&disk), false);
+        wrapper.set_disk_id_state(Some(disk_id)).await;
+
+        let snapshot = wrapper
+            .disk_info(&DiskInfoOptions {
+                fresh_capacity: true,
+                ..Default::default()
+            })
+            .await
+            .expect("fresh capacity should pass when the disk identity matches");
+        assert!(snapshot.fresh_capacity);
+        assert!(snapshot.total > 0);
+        assert!(snapshot.id.is_none(), "capacity-only snapshots should not expose the disk ID");
+
+        let stale_wrapper = LocalDiskWrapper::new(disk, false);
+        stale_wrapper.set_disk_id_state(Some(Uuid::nil())).await;
+        let error = stale_wrapper
+            .disk_info(&DiskInfoOptions {
+                fresh_capacity: true,
+                ..Default::default()
+            })
+            .await
+            .expect_err("a stale disk identity must still be rejected before the capacity probe");
+        assert_eq!(error, DiskError::DiskNotFound);
+    }
+
     #[tokio::test]
     async fn delete_versions_counts_returned_batch_error_classes() {
         let dir = tempfile::tempdir().expect("temp dir should be created");

crates/ecstore/src/disk/error.rs

@@ -428,6 +428,19 @@ impl DiskError {
         }
     }
 
+    /// Whether an internode RPC was cancelled, without classifying the peer as offline.
+    pub fn io_error_is_rpc_cancelled(error: &io::Error) -> bool {
+        error
+            .get_ref()
+            .and_then(|source| source.downcast_ref::<RpcStatusError>())
+            .is_some_and(|error| error.status().code() == tonic::Code::Cancelled)
+    }
+
+    pub(crate) fn clone_rpc_status_io_error(error: &io::Error) -> Option<io::Error> {
+        let status = error.get_ref()?.downcast_ref::<RpcStatusError>()?;
+        Some(io::Error::new(error.kind(), RpcStatusError(status.0.clone())))
+    }
+
     pub fn internode_http_error_kind(&self) -> Option<InternodeHttpErrorKind> {
         match self {
             DiskError::Io(io_error) => io_error
@@ -701,8 +714,8 @@ impl Clone for DiskError {
                 DiskError::conditional_file_not_committed(io::Error::new(io_error.kind(), io_error.to_string())),
             ),
             DiskError::Io(io_error) => {
-                if let Some(status) = io_error.get_ref().and_then(|source| source.downcast_ref::<RpcStatusError>()) {
-                    return DiskError::Io(io::Error::new(io_error.kind(), RpcStatusError(status.0.clone())));
+                if let Some(error) = Self::clone_rpc_status_io_error(io_error) {
+                    return DiskError::Io(error);
                 }
                 DiskError::Io(
                     Self::clone_dangling_delete_grace(io_error)
@@ -900,6 +913,25 @@ mod tests {
     use super::*;
     use std::collections::HashMap;
 
+    #[test]
+    fn rpc_status_survives_disk_and_storage_clones() {
+        for code in [tonic::Code::Cancelled, tonic::Code::PermissionDenied] {
+            let status = tonic::Status::new(code, "operation was canceled");
+            let original = DiskError::Io(io::Error::new(io::ErrorKind::Interrupted, RpcStatusError::from(status)));
+            let display = original.to_string();
+            let disk = original.clone();
+            let storage = crate::error::StorageError::from(disk);
+            let cloned = storage.clone();
+            for error in [io::Error::from(original), io::Error::from(storage), io::Error::from(cloned)] {
+                assert_eq!(error.kind(), io::ErrorKind::Interrupted);
+                let status = error.get_ref().unwrap().downcast_ref::<RpcStatusError>().unwrap().status();
+                assert_eq!(status.code(), code);
+                assert_eq!(status.message(), "operation was canceled");
+                assert_eq!(DiskError::from(error).to_string(), display);
+            }
+        }
+    }
+
     #[test]
     fn retired_marker_deferral_survives_disk_and_storage_clones() {
         let original = DiskError::retired_marker_deferred("missing retirement record");

crates/ecstore/src/disk/local.rs

@@ -1372,13 +1372,24 @@ pub(crate) fn durability_mode() -> DurabilityMode {
             rustfs_utils::get_env_opt_str(ENV_RUSTFS_DURABILITY_MODE),
             rustfs_utils::get_env_bool(ENV_RUSTFS_DRIVE_SYNC_ENABLE, DEFAULT_RUSTFS_DRIVE_SYNC_ENABLE),
         );
-        info!(
-            event = EVENT_DISK_LOCAL_DURABILITY_MODE,
-            component = LOG_COMPONENT_ECSTORE,
-            subsystem = LOG_SUBSYSTEM_DISK_LOCAL,
-            mode = mode.as_str(),
-            "Storage durability mode resolved"
-        );
+        if mode == DurabilityMode::Strict {
+            info!(
+                event = EVENT_DISK_LOCAL_DURABILITY_MODE,
+                component = LOG_COMPONENT_ECSTORE,
+                subsystem = LOG_SUBSYSTEM_DISK_LOCAL,
+                mode = mode.as_str(),
+                "Storage durability mode resolved"
+            );
+        } else {
+            warn!(
+                event = EVENT_DISK_LOCAL_DURABILITY_MODE,
+                component = LOG_COMPONENT_ECSTORE,
+                subsystem = LOG_SUBSYSTEM_DISK_LOCAL,
+                state = "non_strict_mode_configured",
+                mode = mode.as_str(),
+                "Storage durability mode does not provide strict power-loss durability"
+            );
+        }
         mode
     })
 }
@@ -6915,7 +6926,9 @@ impl LocalDisk {
 
         let (buf, mtime) = res?;
         if buf.is_empty() {
-            return Err(DiskError::FileNotFound);
+            // A missing xl.meta is mapped by the open/read error above. A file
+            // that exists but has no metadata bytes is corruption, not absence.
+            return Err(DiskError::FileCorrupt);
         }
 
         Ok((buf, mtime))
@@ -11697,6 +11710,39 @@ mod test {
         assert_eq!(err, DiskError::FileCorrupt);
     }
 
+    #[tokio::test]
+    async fn read_version_reports_empty_xl_meta_as_corrupt_not_missing() {
+        use tempfile::tempdir;
+
+        let dir = tempdir().expect("test directory should be created");
+        let endpoint = Endpoint::try_from(dir.path().to_str().expect("test path should be utf8")).expect("endpoint should parse");
+        let disk = LocalDisk::new(&endpoint, false).await.expect("local disk should be created");
+        let bucket = "bucket";
+        let object = "empty-metadata";
+        ensure_test_volume(&disk, bucket).await;
+
+        let missing_err = disk
+            .read_version("", bucket, "missing-object", "", &ReadOptions::default())
+            .await
+            .expect_err("a missing xl.meta remains not found");
+        assert_eq!(missing_err, DiskError::FileNotFound);
+
+        let object_dir = dir.path().join(bucket).join(object);
+        fs::create_dir_all(&object_dir)
+            .await
+            .expect("object directory should be created");
+        fs::write(object_dir.join(STORAGE_FORMAT_FILE), b"")
+            .await
+            .expect("empty metadata fixture should be written");
+
+        let err = disk
+            .read_version("", bucket, object, "", &ReadOptions::default())
+            .await
+            .expect_err("an existing zero-length xl.meta is corrupt, not an absent object");
+
+        assert_eq!(err, DiskError::FileCorrupt);
+    }
+
     #[tokio::test]
     async fn read_version_delete_marker_never_enters_inline_shard_math() {
         use tempfile::tempdir;
@@ -13382,24 +13428,24 @@ mod test {
         );
     }
 
-    /// A writer that stalls on every write, standing in for a slow listing
-    /// consumer (quorum merge, a lagging peer drive).
+    /// A writer that advances paused time on every write, standing in for a
+    /// slow listing consumer without depending on host scheduling.
     struct SlowWriter {
         delay: Duration,
-        sleep: Option<Pin<Box<Sleep>>>,
+        advance: Option<Pin<Box<dyn std::future::Future<Output = ()> + Send>>>,
     }
 
     impl AsyncWrite for SlowWriter {
         fn poll_write(mut self: Pin<&mut Self>, cx: &mut Context<'_>, buf: &[u8]) -> Poll<io::Result<usize>> {
-            if self.sleep.is_none() {
+            if self.advance.is_none() {
                 let delay = self.delay;
-                self.sleep = Some(Box::pin(tokio::time::sleep(delay)));
+                self.advance = Some(Box::pin(tokio::time::advance(delay)));
             }
 
-            let sleep = self.sleep.as_mut().expect("sleep was just installed");
-            match sleep.as_mut().poll(cx) {
+            let advance = self.advance.as_mut().expect("clock advance was just installed");
+            match advance.as_mut().poll(cx) {
                 Poll::Ready(()) => {
-                    self.sleep = None;
+                    self.advance = None;
                     Poll::Ready(Ok(buf.len()))
                 }
                 Poll::Pending => Poll::Pending,
@@ -13434,6 +13480,7 @@ mod test {
 
         let endpoint = Endpoint::try_from(dir.path().to_str().expect("temp dir should be utf8")).expect("endpoint should parse");
         let disk = LocalDisk::new(&endpoint, false).await.expect("local disk should be created");
+        disk.wait_for_startup_cleanup().await;
 
         let stall = Duration::from_millis(300);
         let write_delay = Duration::from_millis(150);
@@ -13447,12 +13494,21 @@ mod test {
 
         let mut writer = SlowWriter {
             delay: write_delay,
-            sleep: None,
+            advance: None,
         };
 
-        let started = std::time::Instant::now();
+        // A live blocking task inhibits Tokio's automatic clock advance while
+        // real filesystem reads are pending. Only consumer writes advance time.
+        let (clock_guard_tx, clock_guard_rx) = std::sync::mpsc::channel::<()>();
+        let clock_guard = tokio::task::spawn_blocking(move || clock_guard_rx.recv());
+        tokio::time::pause();
+        let started = Instant::now();
         let result = disk.walk_dir(opts, &mut writer).await;
         let elapsed = started.elapsed();
+        drop(clock_guard_tx);
+        let _ = clock_guard
+            .await
+            .expect("clock guard should exit after its sender is dropped");
 
         assert!(result.is_ok(), "a walk making steady progress must not time out, got {result:?}");
         assert!(

crates/ecstore/src/ecstore_validation_blackbox.rs

@@ -377,13 +377,21 @@ async fn blackbox_heal_requests_preserve_repair_scope() {
     // Without a durable MRF consumer, partial PUTs fall back to the heal
     // admission channel and wait for its receipt before acknowledging the write.
     // Drive the test receiver alongside the PUT so neither waits on the other.
-    let (_put_dirs, put_set) = make_local_set_disks(4, 2).await;
+    let (_put_dirs, put_store) = crate::bucket::metadata_sys::test_support::isolated_store_over_temp_disks().await;
+    crate::bucket::metadata_sys::init_bucket_metadata_sys(Arc::clone(&put_store), Vec::new()).await;
+    let put_set = Arc::clone(&put_store.pools[0].disk_set[0]);
+    assert_eq!(put_set.set_drive_count, 4);
+    assert_eq!(put_set.default_parity_count, 2);
     let put_bucket = "bb-put-partial-convergence";
     let put_object = "object.bin";
-    put_set
+    put_store
         .make_bucket(put_bucket, &MakeBucketOptions::default())
         .await
         .expect("PUT bucket should be created");
+    let put_incarnation = put_store
+        .bucket_incarnation_id_from_disk(put_bucket)
+        .await
+        .expect("partial repair must bind the real persisted bucket generation");
     let offline_disk = {
         let mut disks = put_set.disks.write().await;
         disks[0].take()
@@ -400,6 +408,7 @@ async fn blackbox_heal_requests_preserve_repair_scope() {
                         &ObjectOptions {
                             no_lock: true,
                             versioned: true,
+                            expected_bucket_incarnation_id: Some(put_incarnation),
                             ..Default::default()
                         },
                     )
@@ -419,6 +428,8 @@ async fn blackbox_heal_requests_preserve_repair_scope() {
     assert_eq!(request.object_version_id.as_deref(), Some(committed_version.as_str()));
     assert_eq!(request.pool_index, Some(0));
     assert_eq!(request.set_index, Some(0));
+    assert_eq!(request.source, HealRequestSource::Mrf);
+    assert_eq!(request.expected_bucket_incarnation_id, Some(put_incarnation));
 
     let duplicate_request = tokio::time::timeout(std::time::Duration::from_millis(100), async {
         loop {
@@ -447,7 +458,7 @@ async fn blackbox_heal_requests_preserve_repair_scope() {
     }
 
     let healthy_bucket = "bb-put-healthy-convergence";
-    put_set
+    put_store
         .make_bucket(healthy_bucket, &MakeBucketOptions::default())
         .await
         .expect("healthy PUT bucket should be created");

crates/ecstore/src/error/mod.rs

@@ -623,8 +623,9 @@ impl Clone for StorageError {
     fn clone(&self) -> Self {
         match self {
             StorageError::Io(e) => {
-                if let Some(error) =
-                    DiskError::clone_dangling_delete_grace(e).or_else(|| DiskError::clone_retired_marker_deferred(e))
+                if let Some(error) = DiskError::clone_rpc_status_io_error(e)
+                    .or_else(|| DiskError::clone_dangling_delete_grace(e))
+                    .or_else(|| DiskError::clone_retired_marker_deferred(e))
                 {
                     return StorageError::Io(error);
                 }

crates/ecstore/src/object_api/types.rs

@@ -1444,6 +1444,10 @@ pub struct ObjectInfo {
     pub version_purge_status: VersionPurgeStatusType,
     pub replication_decision: String,
     pub checksum: Option<Bytes>,
+    /// True when this `CompleteMultipartUpload` result is the object already
+    /// published by the same upload. Callers must not repeat quota, replication,
+    /// lifecycle, or object-created side effects for that response.
+    pub multipart_completion_replayed: bool,
 }
 
 impl Clone for ObjectInfo {
@@ -1484,6 +1488,7 @@ impl Clone for ObjectInfo {
             version_purge_status: self.version_purge_status.clone(),
             replication_decision: self.replication_decision.clone(),
             checksum: self.checksum.clone(),
+            multipart_completion_replayed: self.multipart_completion_replayed,
             expires: self.expires,
         }
     }

crates/ecstore/src/runtime/sources.rs

@@ -602,9 +602,18 @@ pub(crate) async fn initialize_local_disk_maps(
 }
 
 pub(crate) async fn init_tier_config_mgr(store: Arc<ECStore>) -> Result<()> {
-    let handle = get_global_tier_config_mgr();
-    TierConfigMgr::reload_handle(&handle, store.clone()).await?;
-    tokio::spawn(TierConfigMgr::refresh_tier_config_handle(handle, store));
+    init_tier_config_mgr_handle(get_global_tier_config_mgr(), store).await
+}
+
+pub(crate) async fn init_tier_config_mgr_handle(handle: Arc<RwLock<TierConfigMgr>>, store: Arc<ECStore>) -> Result<()> {
+    let initial_reload = TierConfigMgr::reload_handle(&handle, store.clone()).await;
+    // Startup may continue after a transient recovery failure. Its retained
+    // mutation fences still need a worker to consume later commit notifications.
+    tokio::spawn(TierConfigMgr::refresh_tier_config_handle(handle.clone(), store));
+    if initial_reload.is_err() {
+        TierConfigMgr::request_committed_mutation_refresh(&handle).await;
+    }
+    initial_reload?;
     Ok(())
 }
 

crates/ecstore/src/services/tier/tier_mutation_peer.rs

@@ -107,7 +107,9 @@ async fn handle_prepare(
     }
     let tier_config_mgr = api.tier_config_mgr();
 
-    for _ in 0..3 {
+    // Peers can race to create the same durable intent. Re-read after bounded
+    // lock contention so the winning publication follows the normal identity checks.
+    for attempt in 0..3 {
         let (stored, applied) = match load_tier_mutation_intent_record(api.clone(), mutation_id).await {
             Ok(existing) => {
                 if !existing.same_identity_as(&intent) {
@@ -123,9 +125,11 @@ async fn handle_prepare(
                 match save_tier_mutation_intent_record_if_absent(api.clone(), &intent).await {
                     Ok(()) => (intent.clone(), true),
                     Err(Error::PreconditionFailed) => continue,
+                    Err(Error::Lock(rustfs_lock::LockError::Timeout { .. })) if attempt < 2 => continue,
                     Err(err) => return Err(err.into()),
                 }
             }
+            Err(Error::Lock(rustfs_lock::LockError::Timeout { .. })) if attempt < 2 => continue,
             Err(err) => return Err(err.into()),
         };
 

crates/ecstore/src/set_disk/core/io_primitives.rs

@@ -2884,7 +2884,16 @@ impl SetDisks {
             let result = if let Some(deadline) = single_pending_hedge_deadline.take() {
                 tokio::select! {
                     result = join_set.join_next() => result,
-                    _ = tokio::time::sleep_until(deadline) => {
+                    _ = async {
+                        tokio::time::sleep_until(deadline).await;
+                        #[cfg(test)]
+                        rename_fanout_barrier::checkpoint(
+                            object.as_ref(),
+                            0,
+                            rename_fanout_barrier::PHASE_NON_INLINE_HEDGE_TIMER,
+                        )
+                        .await;
+                    } => {
                         if bounded_fanout
                             && !force_full_wait
                             && join_set.len() == 1
@@ -6516,7 +6525,15 @@ impl SetDisks {
 
         let disks = self.get_disks_internal().await;
 
-        // Phase 1: classify every online disk. A directory that holds object
+        // An unresolved slot may hold the only copy of readable object
+        // metadata or uncommitted data under this prefix. Do not classify or
+        // remove orphan residue from the online subset while any drive is
+        // unavailable.
+        if disks.iter().any(Option::is_none) {
+            return Ok(false);
+        }
+
+        // Phase 1: classify every disk. A directory that holds object
         // data or uncommitted residue on ANY disk blocks itself and its
         // ancestors on every disk, so a degraded/healable object is never
         // destroyed; purgeable subtrees beside it are still reclaimed.
@@ -6570,6 +6587,47 @@ impl SetDisks {
         Ok(purged)
     }
 
+    /// Reclaim metadata-less directory trees discovered by a recursive empty
+    /// bucket listing. The initial directory read fails closed if any slot is
+    /// unavailable; per-prefix purges then reuse the full cross-disk scan.
+    pub(crate) async fn purge_orphan_dir_objects_in_bucket(&self, bucket: &str) -> bool {
+        let disks = self.get_disks_internal().await;
+        if disks.is_empty() || disks.iter().any(Option::is_none) {
+            return false;
+        }
+
+        let mut prefixes = HashSet::new();
+        for disk in disks.iter().flatten() {
+            let entries = match disk.list_dir("", bucket, "", 0).await {
+                Ok(entries) => entries,
+                Err(_) => return false,
+            };
+            prefixes.extend(
+                entries
+                    .into_iter()
+                    .filter(|entry| entry.ends_with(SLASH_SEPARATOR) && is_safe_orphan_dir_entry(entry)),
+            );
+        }
+
+        let mut prefixes = prefixes.into_iter().collect::<Vec<_>>();
+        prefixes.sort_unstable();
+        let mut purged = false;
+        for prefix in prefixes {
+            match self.purge_orphan_dir_object(bucket, &prefix).await {
+                Ok(prefix_purged) => purged |= prefix_purged,
+                Err(_) => return false,
+            }
+        }
+        purged
+    }
+
+    /// Orphan cleanup must not infer a complete scan from only the online
+    /// subset of a set's disks.
+    pub(crate) async fn orphan_purge_has_complete_disk_set(&self) -> bool {
+        let disks = self.get_disks_internal().await;
+        !disks.is_empty() && disks.iter().all(Option::is_some)
+    }
+
     fn orphan_purge_in_backoff(&self, key: &str) -> bool {
         let backoff = self
             .orphan_purge_backoff
@@ -7322,6 +7380,9 @@ pub(crate) mod rename_fanout_barrier {
         CLEANUP as PHASE_CLEANUP, READ_VERSION as PHASE_READ_VERSION, RENAME as PHASE_RENAME, ROLLBACK as PHASE_ROLLBACK,
     };
 
+    /// Object-scoped hedge timer checkpoint; slot zero identifies the timer, not a disk.
+    pub const PHASE_NON_INLINE_HEDGE_TIMER: &str = "non_inline_hedge_timer";
+
     /// One armed barrier: the fan-out task matching `(disk_index, phase)` pauses.
     struct Armed {
         disk_index: usize,

crates/ecstore/src/set_disk/mod.rs

@@ -1593,6 +1593,8 @@ mod prepared_get_object_metadata_tests {
                 ("RUSTFS_GET_METADATA_EARLY_STOP_BOUNDED_FANOUT", Some("true")),
             ],
             async {
+                // Hold this object's hedge timer so real-disk latency cannot add speculative fanout.
+                let _hedge_timer = rename_fanout_barrier::arm(&object, 0, rename_fanout_barrier::PHASE_NON_INLINE_HEDGE_TIMER);
                 let calls = disk_call_counters::observe(&object);
                 let mut reader = set_disks
                     .get_object_reader(bucket, &object, None, HeaderMap::new(), &opts)
@@ -4337,9 +4339,15 @@ impl SetDisks {
         }
     }
 
-    pub(in crate::set_disk) async fn persist_partial_write(&self, bucket: &str, object: &str, version_id: Option<&str>) -> bool {
+    pub(in crate::set_disk) async fn persist_partial_write(
+        &self,
+        bucket: &str,
+        object: &str,
+        version_id: Option<&str>,
+        source_bucket_incarnation_id: Option<Uuid>,
+    ) -> bool {
         use rustfs_common::mrf_channel::{
-            MrfDurableAdmissionError, MrfScope, mrf_delivery_enabled, persist_partial_write_intent,
+            MrfDurableAdmissionError, MrfScope, mrf_delivery_enabled, persist_partial_write_intent_with_incarnation,
         };
 
         if !mrf_delivery_enabled() {
@@ -4358,7 +4366,9 @@ impl SetDisks {
             Ok::<_, MrfDurableAdmissionError>((version, scope))
         })();
         let result = match identity {
-            Ok((version, scope)) => persist_partial_write_intent(bucket, object, version, scope).await,
+            Ok((version, scope)) => {
+                persist_partial_write_intent_with_incarnation(bucket, object, version, scope, source_bucket_incarnation_id).await
+            }
             Err(err) => Err(err),
         };
         match result {
@@ -4394,15 +4404,24 @@ impl SetDisks {
 
     pub(in crate::set_disk) async fn submit_rename_tail_heal(
         &self,
-        request: rustfs_heal_contracts::heal_channel::HealChannelRequest,
+        mut request: rustfs_heal_contracts::heal_channel::HealChannelRequest,
     ) {
         if let Some(object) = request.object_prefix.as_deref()
             && self
-                .persist_partial_write(&request.bucket, object, request.object_version_id.as_deref())
+                .persist_partial_write(
+                    &request.bucket,
+                    object,
+                    request.object_version_id.as_deref(),
+                    request.expected_bucket_incarnation_id,
+                )
                 .await
         {
             return;
         }
+        if request.expected_bucket_incarnation_id.is_none() {
+            return;
+        }
+        request.source = rustfs_heal_contracts::heal_channel::HealRequestSource::Mrf;
         #[cfg(test)]
         {
             let capture = self
@@ -4790,6 +4809,10 @@ impl SetDisks {
     /// Read the persisted bucket identity through this set's metadata owner.
     /// Missing or non-authoritative legacy identities remain errors.
     pub async fn bucket_incarnation_id_from_disk(&self, bucket: &str) -> Result<Uuid> {
+        if crate::bucket::utils::is_meta_bucketname(bucket) {
+            // Metadata writes can already hold the pool metadata write lock.
+            return Err(Error::other("system metadata bucket has no bucket incarnation"));
+        }
         metadata_sys::get_bucket_incarnation_id_in(&self.ctx, bucket).await
     }
 
@@ -7415,6 +7438,35 @@ mod tests {
         make_test_set_disks_with_ctx(lockers, bootstrap_ctx()).await
     }
 
+    #[tokio::test]
+    #[serial_test::serial]
+    async fn system_metadata_incarnation_lookup_does_not_reenter_pool_metadata() {
+        let (_temp_dirs, store, _other_store) =
+            crate::services::rebalance::test_three_pool_stores_with_isolated_node_contexts(None).await;
+        let _pool_meta_guard = store.pool_meta.write().await;
+        let set = &store.pools[0].disk_set[0];
+
+        for bucket in [RUSTFS_META_BUCKET, RUSTFS_META_TMP_BUCKET, crate::disk::MIGRATING_META_BUCKET] {
+            tokio::time::timeout(Duration::from_secs(30), set.bucket_incarnation_id_from_disk(bucket))
+                .await
+                .expect("system metadata identity lookup must not reacquire the held pool metadata lock")
+                .expect_err("system metadata buckets have no user bucket incarnation");
+        }
+
+        drop(_pool_meta_guard);
+        let bucket = "user-incarnation-boundary";
+        let incarnation = Uuid::new_v4();
+        crate::bucket::metadata::save_bucket_incarnation(Arc::clone(&store), bucket, incarnation)
+            .await
+            .expect("persist the user bucket identity through the metadata owner");
+        assert_eq!(
+            set.bucket_incarnation_id_from_disk(bucket)
+                .await
+                .expect("user bucket identities must still load from the metadata owner"),
+            incarnation,
+        );
+    }
+
     async fn make_test_set_disks_with_ctx(
         lockers: Vec<Arc<dyn LockClient>>,
         instance_ctx: Arc<InstanceContext>,
@@ -9418,6 +9470,31 @@ mod tests {
         assert!(!purged, "a missing prefix should report nothing to purge");
     }
 
+    #[tokio::test]
+    async fn orphan_directory_purge_preserves_tree_when_a_disk_slot_is_offline() {
+        let (dir, disk) = make_single_local_disk().await;
+        let prefix_dir = dir.path().join("bucket").join("pfx");
+        fs::create_dir_all(prefix_dir.join("nested").join("leaf"))
+            .await
+            .expect("orphan directory tree should be created");
+
+        let set = make_set_disks_with(vec![Some(disk), None]).await;
+        let purged = set
+            .purge_orphan_dir_object("bucket", "pfx/")
+            .await
+            .expect("an unavailable slot should fail closed without a scan error");
+
+        assert!(!purged, "an incomplete disk scan must not claim the tree is an orphan");
+        assert!(prefix_dir.join("nested/leaf").exists(), "online disk contents must remain untouched");
+
+        let bucket_purged = set.purge_orphan_dir_objects_in_bucket("bucket").await;
+        assert!(!bucket_purged, "bucket-wide cleanup must fail closed with an unavailable disk slot");
+        assert!(
+            prefix_dir.join("nested/leaf").exists(),
+            "bucket-wide cleanup must preserve the online tree"
+        );
+    }
+
     // Cross-disk safety: if any drive still holds object data under the prefix, refuse
     // to purge on every drive so a degraded/healable object is never destroyed.
     #[tokio::test]

crates/ecstore/src/set_disk/ops/heal.rs

@@ -1166,7 +1166,7 @@ impl SetDisks {
 
                         if !latest_meta.deleted && !latest_meta.is_remote() && !protected {
                             result.detail =
-                                "Legacy object uses standard repair; independent object identity remains unverified".to_owned();
+                                rustfs_heal_contracts::heal_channel::LEGACY_OBJECT_IDENTITY_UNVERIFIED_DETAIL.to_owned();
                         }
 
                         if disks_to_heal_count == 0 {
@@ -4862,18 +4862,33 @@ mod heal_result_report_tests {
             },
         )
         .await
-        .expect("commit all source shards");
+        .expect("commit source object");
 
-        let mut files = Vec::new();
+        // PUT can succeed at quorum while minority shards remain unavailable.
+        // Require every source disk before modifying any shard in this fixture.
+        let mut source_metadata = Vec::with_capacity(disks.len());
+        for disk in &disks {
+            source_metadata.push(disk.read_version("", bucket, object, "", &ReadOptions::default()).await);
+        }
+        assert!(
+            source_metadata.iter().all(|source| source.is_ok()),
+            "source fixture must populate all {} disks before truncation in {mode:?} mode; per-disk errors: {:?}",
+            disks.len(),
+            source_metadata
+                .iter()
+                .enumerate()
+                .map(|(slot, source)| (slot, source.as_ref().err()))
+                .collect::<Vec<_>>()
+        );
+        let files: Vec<_> = source_metadata
+            .into_iter()
+            .map(|source| source.expect("source metadata checked above"))
+            .collect();
         let mut paths = Vec::new();
         let mut original_shards = Vec::new();
         let mut original_metadata = Vec::new();
         let mut damaged = Vec::new();
-        for (slot, disk) in disks.iter().enumerate() {
-            let source = disk
-                .read_version("", bucket, object, "", &ReadOptions::default())
-                .await
-                .expect("read source metadata");
+        for (slot, source) in files.iter().enumerate() {
             assert_eq!((source.erasure.data_blocks, source.erasure.parity_blocks), (12, 4));
             assert_eq!(source.parts[0].integrity.is_some(), mode == ShardIntegrityWriteMode::Protected);
             let directory = temp_dirs[slot].path().join(bucket).join(object);
@@ -4905,7 +4920,6 @@ mod heal_result_report_tests {
             }
             paths.push(path);
             original_shards.push(original);
-            files.push(source);
         }
         assert_eq!(damaged.len(), coding_indexes.len());
         if mode == ShardIntegrityWriteMode::Protected {

crates/ecstore/src/set_disk/ops/multipart.rs

@@ -1451,6 +1451,103 @@ impl SetDisks {
             delimiter: delimiter.to_owned(),
         })
     }
+
+    /// Answer a CompleteMultipartUpload whose staging upload is already gone.
+    ///
+    /// AWS keeps Complete idempotent for the same upload id and part list while
+    /// the completed object is still the result of that upload (the first 200
+    /// can be lost, and SDK retries depend on a second 200 with the same ETag).
+    /// Returns `Ok(None)` when this key has no such object, so the caller keeps
+    /// `InvalidUploadID`. A matching upload id with a different part list is
+    /// `InvalidPart` and must not fall through to `NoSuchUpload`.
+    async fn replay_completed_multipart_upload(
+        &self,
+        bucket: &str,
+        object: &str,
+        upload_id: &str,
+        uploaded_parts: &[CompletePart],
+        opts: &ObjectOptions,
+    ) -> Result<Option<ObjectInfo>> {
+        if upload_id.is_empty() {
+            return Ok(None);
+        }
+
+        let read_opts = ObjectOptions {
+            no_lock: true,
+            metadata_cache_safe: false,
+            versioned: opts.versioned,
+            version_suspended: opts.version_suspended,
+            ..Default::default()
+        };
+        match self.get_object_info(bucket, object, &read_opts).await {
+            Ok(info) => {
+                match completed_multipart_upload_matches(
+                    &info.user_defined,
+                    info.parts.as_ref(),
+                    info.delete_marker,
+                    upload_id,
+                    uploaded_parts,
+                    bucket,
+                    object,
+                ) {
+                    Ok(true) => return Ok(Some(mark_multipart_completion_replayed(info))),
+                    Ok(false) => {}
+                    Err(err) => return Err(err),
+                }
+            }
+            Err(err) if is_err_object_not_found(&err) || is_err_version_not_found(&err) => {}
+            Err(err) => return Err(err),
+        }
+
+        // Unversioned overwrite replaces the only slot. A versioned bucket can
+        // keep the completed version under a newer one; that version is still
+        // the result of this upload until it is deleted.
+        if !(opts.versioned || opts.version_suspended) {
+            return Ok(None);
+        }
+
+        let Some(versions) = self.load_file_info_versions_exact(bucket, object).await? else {
+            return Ok(None);
+        };
+        for fi in versions.versions {
+            if fi.deleted || fi.tier_free_version() || fi.is_canonical_delete_marker() {
+                continue;
+            }
+            match completed_multipart_upload_matches(&fi.metadata, &fi.parts, false, upload_id, uploaded_parts, bucket, object) {
+                Ok(true) => {
+                    let info = ObjectInfo::from_file_info(&fi, bucket, object, true);
+                    return Ok(Some(mark_multipart_completion_replayed(info)));
+                }
+                Ok(false) => {}
+                Err(err) => return Err(err),
+            }
+        }
+        Ok(None)
+    }
+
+    /// Best-effort removal of staging left behind when commit succeeded and the
+    /// process stopped before `delete_all`. A cleanup miss must not turn the
+    /// idempotent 200 into an error; the object is already durable.
+    async fn reclaim_replayed_multipart_staging(&self, bucket: &str, object: &str, upload_id: &str, upload_id_path: &str) {
+        if let Err(err) = self
+            .delete_all_with_quorum(RUSTFS_META_MULTIPART_BUCKET, upload_id_path, self.default_write_quorum())
+            .await
+        {
+            warn!(
+                target: "rustfs_ecstore::set_disk",
+                event = EVENT_SET_DISK_MULTIPART,
+                component = LOG_COMPONENT_ECSTORE,
+                subsystem = LOG_SUBSYSTEM_SET_DISK,
+                op = "complete_multipart_upload",
+                result = "cleanup_quorum_missed",
+                bucket = %bucket,
+                object = %object,
+                upload_id = %upload_id,
+                error = %err,
+                "replayed multipart completion left staging behind"
+            );
+        }
+    }
 }
 
 #[async_trait::async_trait]
@@ -2337,6 +2434,7 @@ impl crate::storage_api_contracts::multipart::MultipartOperations for SetDisks {
         }
         result
     }
+
     // complete_multipart_upload finished
     #[tracing::instrument(skip(self))]
     async fn complete_multipart_upload(
@@ -2400,9 +2498,27 @@ impl crate::storage_api_contracts::multipart::MultipartOperations for SetDisks {
             .await?;
 
         let expected_restore_operation_id = restore_commit_operation_id_from_metadata(&opts.user_defined)?;
-        let (mut fi, files_metas) = self
+        let (mut fi, files_metas) = match self
             .check_upload_id_exists_with_opts(bucket, object, upload_id, true, opts)
-            .await?;
+            .await
+        {
+            Ok(found) => found,
+            Err(err) if crate::error::is_err_invalid_upload_id(&err) => {
+                match self
+                    .replay_completed_multipart_upload(bucket, object, upload_id, &uploaded_parts, opts)
+                    .await
+                {
+                    Ok(Some(existing)) => {
+                        self.reclaim_replayed_multipart_staging(bucket, object, upload_id, &upload_id_path)
+                            .await;
+                        return Ok(existing);
+                    }
+                    Ok(None) => return Err(err),
+                    Err(replay_err) => return Err(replay_err),
+                }
+            }
+            Err(err) => return Err(err),
+        };
         ensure_data_movement_upload_access(&fi, bucket, object, upload_id, opts)?;
         ensure_multipart_bucket_incarnation(&self.ctx, &fi, bucket, object, upload_id, opts.expected_bucket_incarnation_id)
             .await?;
@@ -2988,6 +3104,16 @@ impl crate::storage_api_contracts::multipart::MultipartOperations for SetDisks {
         rustfs_utils::http::remove_str(&mut fi.metadata, rustfs_filemeta::shard_integrity::SUFFIX_UPLOAD_INTEGRITY);
         fi.persist_shard_integrity()?;
 
+        // The staging directory is removed after commit. Recording the upload id
+        // on the object is what lets a retried CompleteMultipartUpload return this
+        // version instead of NoSuchUpload. insert_str writes both internal prefixes;
+        // a later conflicting pair fails closed in the replay matcher.
+        // Internal migration must preserve the source completion identity rather
+        // than replace it (or invent one) with the temporary transfer upload id.
+        if !opts.data_movement && !upload_id.is_empty() {
+            insert_str(&mut fi.metadata, rustfs_utils::http::SUFFIX_MULTIPART_UPLOAD_ID, upload_id.to_owned());
+        }
+
         for meta in parts_metadatas.iter_mut() {
             if meta.has_valid_erasure_geometry() {
                 meta.size = fi.size;
@@ -3522,6 +3648,58 @@ impl crate::storage_api_contracts::multipart::MultipartOperations for SetDisks {
     }
 }
 
+fn mark_multipart_completion_replayed(mut info: ObjectInfo) -> ObjectInfo {
+    info.multipart_completion_replayed = true;
+    info
+}
+
+/// `Ok(true)` when `upload_id` is the completion that published this version and
+/// the requested parts are that version's part list. `Ok(false)` means this
+/// version is not that completion. `Err` is a definite part-list or metadata
+/// failure and must not be reported as `NoSuchUpload`.
+fn completed_multipart_upload_matches(
+    metadata: &HashMap<String, String>,
+    parts: &[ObjectPartInfo],
+    delete_marker: bool,
+    upload_id: &str,
+    uploaded_parts: &[CompletePart],
+    bucket: &str,
+    object: &str,
+) -> Result<bool> {
+    if delete_marker || upload_id.is_empty() {
+        return Ok(false);
+    }
+    let Some(stored_upload_id) = rustfs_utils::http::get_consistent_str(metadata, rustfs_utils::http::SUFFIX_MULTIPART_UPLOAD_ID)
+    else {
+        if rustfs_utils::http::contains_key_str(metadata, rustfs_utils::http::SUFFIX_MULTIPART_UPLOAD_ID) {
+            return Err(Error::FileCorrupt);
+        }
+        return Ok(false);
+    };
+    if stored_upload_id.is_empty() || stored_upload_id != upload_id {
+        return Ok(false);
+    }
+    if parts.len() != uploaded_parts.len() {
+        let part_num = uploaded_parts.first().map(|part| part.part_num).unwrap_or(0);
+        return Err(Error::InvalidPart(part_num, bucket.to_owned(), object.to_owned()));
+    }
+    for (stored, requested) in parts.iter().zip(uploaded_parts) {
+        if stored.number != requested.part_num {
+            return Err(Error::InvalidPart(requested.part_num, bucket.to_owned(), object.to_owned()));
+        }
+        let stored_etag = rustfs_utils::path::trim_etag(&stored.etag);
+        let client_etag = requested.etag.as_deref().map(rustfs_utils::path::trim_etag);
+        if client_etag.as_deref() != Some(stored_etag.as_str()) {
+            return Err(Error::InvalidPart(
+                requested.part_num,
+                stored.etag.clone(),
+                requested.etag.clone().unwrap_or_default(),
+            ));
+        }
+    }
+    Ok(true)
+}
+
 /// Final ETag for a completed multipart object. An authorized replication
 /// request preserves the source ETag so the replication HEAD comparison
 /// converges even when the source ETag is not derivable from the uploaded
@@ -4939,6 +5117,10 @@ mod tests {
             )
             .await
             .expect("completed data movement object should be readable");
+        assert!(
+            !rustfs_utils::http::contains_key_str(&completed.user_defined, rustfs_utils::http::SUFFIX_MULTIPART_UPLOAD_ID),
+            "migration of a source without a completion identity must not invent one"
+        );
         assert!(!rustfs_utils::http::contains_key_str(
             &completed.user_defined,
             rustfs_utils::http::SUFFIX_DATA_MOVEMENT_UPLOAD
@@ -8630,6 +8812,216 @@ mod tests {
         assert_eq!(paged, vec!["u0", "u1", "u2", "u3"]);
     }
 
+    #[test]
+    fn completed_multipart_upload_match_requires_same_upload_and_parts() {
+        let upload_id = "upload-1";
+        let mut metadata = HashMap::new();
+        insert_str(&mut metadata, rustfs_utils::http::SUFFIX_MULTIPART_UPLOAD_ID, upload_id.to_string());
+        let parts = vec![ObjectPartInfo {
+            number: 1,
+            etag: "abc".to_string(),
+            ..Default::default()
+        }];
+        let requested = vec![CompletePart {
+            part_num: 1,
+            etag: Some("\"abc\"".to_string()),
+            ..Default::default()
+        }];
+
+        assert!(
+            completed_multipart_upload_matches(&metadata, &parts, false, upload_id, &requested, "bucket", "object")
+                .expect("quoted ETag must match the stored part")
+        );
+        let wrong_etag = vec![CompletePart {
+            part_num: 1,
+            etag: Some("def".to_string()),
+            ..Default::default()
+        }];
+        assert!(matches!(
+            completed_multipart_upload_matches(&metadata, &parts, false, upload_id, &wrong_etag, "bucket", "object"),
+            Err(StorageError::InvalidPart(1, _, _))
+        ));
+        assert!(
+            !completed_multipart_upload_matches(&metadata, &parts, false, "other-upload", &requested, "bucket", "object")
+                .expect("a different upload id is not this completion")
+        );
+        assert!(
+            !completed_multipart_upload_matches(&metadata, &parts, true, upload_id, &requested, "bucket", "object")
+                .expect("a delete marker is not the completed object")
+        );
+
+        metadata.insert(
+            format!(
+                "{}{}",
+                rustfs_utils::http::MINIO_INTERNAL_PREFIX,
+                rustfs_utils::http::SUFFIX_MULTIPART_UPLOAD_ID
+            ),
+            "disagrees".to_string(),
+        );
+        assert!(
+            matches!(
+                completed_multipart_upload_matches(&metadata, &parts, false, upload_id, &requested, "bucket", "object"),
+                Err(StorageError::FileCorrupt)
+            ),
+            "conflicting internal upload ids must fail closed"
+        );
+    }
+
+    #[tokio::test(flavor = "multi_thread", worker_threads = 2)]
+    async fn retried_complete_multipart_upload_returns_the_committed_object() {
+        let (_temp_dirs, disk_stores, set_disks) = hermetic_set_disks(4).await;
+        let bucket = "multipart-complete-retry";
+        let object = "object";
+        make_bucket_on_all(&disk_stores, bucket).await;
+
+        let payload = vec![0x11; 4096];
+        let (upload_id, parts) =
+            stage_upload_with_create_opts(&set_disks, bucket, object, &payload, &ObjectOptions::default()).await;
+        let first = set_disks
+            .clone()
+            .complete_multipart_upload(bucket, object, &upload_id, parts.clone(), &ObjectOptions::default())
+            .await
+            .expect("the first completion should publish the object");
+        assert!(!first.multipart_completion_replayed, "the first completion publishes a new object");
+        assert_eq!(
+            rustfs_utils::http::get_consistent_str(&first.user_defined, rustfs_utils::http::SUFFIX_MULTIPART_UPLOAD_ID),
+            Some(upload_id.as_str()),
+            "the completed object must record the upload id"
+        );
+
+        let mut quoted_parts = parts.clone();
+        quoted_parts[0].etag = quoted_parts[0].etag.as_ref().map(|etag| format!("\"{etag}\""));
+        let retried = set_disks
+            .clone()
+            .complete_multipart_upload(bucket, object, &upload_id, quoted_parts, &ObjectOptions::default())
+            .await
+            .expect("a retry with the same upload id and parts must succeed");
+        assert!(retried.multipart_completion_replayed);
+        assert_eq!(retried.etag, first.etag);
+        assert_eq!(retried.size, first.size);
+        assert_eq!(retried.version_id, first.version_id);
+
+        let mismatched = parts.clone();
+        let mut mismatched = mismatched;
+        mismatched[0].etag = Some("not-the-part".to_string());
+        let mismatch_err = set_disks
+            .clone()
+            .complete_multipart_upload(bucket, object, &upload_id, mismatched, &ObjectOptions::default())
+            .await
+            .expect_err("a retry with a different part ETag must not pretend the upload is missing");
+        assert!(matches!(mismatch_err, StorageError::InvalidPart(..)));
+
+        let unknown = set_disks
+            .clone()
+            .complete_multipart_upload(bucket, object, "not-a-real-upload", parts.clone(), &ObjectOptions::default())
+            .await
+            .expect_err("an unknown upload id must stay InvalidUploadID");
+        assert!(matches!(unknown, StorageError::InvalidUploadID(..)));
+
+        let mut reader = PutObjReader::from_vec(b"replaced".to_vec());
+        set_disks
+            .put_object(bucket, object, &mut reader, &ObjectOptions::default())
+            .await
+            .expect("an overwrite should replace the completed object");
+        let overwritten = set_disks
+            .clone()
+            .complete_multipart_upload(bucket, object, &upload_id, parts, &ObjectOptions::default())
+            .await
+            .expect_err("a retry after the object is no longer that upload must stay InvalidUploadID");
+        assert!(matches!(overwritten, StorageError::InvalidUploadID(..)));
+        let current = set_disks
+            .get_object_info(bucket, object, &ObjectOptions::default())
+            .await
+            .expect("the overwrite must remain readable");
+        assert_eq!(current.size, b"replaced".len() as i64);
+    }
+
+    #[tokio::test(flavor = "multi_thread", worker_threads = 2)]
+    async fn concurrent_complete_multipart_upload_returns_one_object() {
+        let (_temp_dirs, disk_stores, set_disks) = hermetic_set_disks(4).await;
+        let bucket = "multipart-complete-concurrent";
+        let object = "object";
+        make_bucket_on_all(&disk_stores, bucket).await;
+
+        let (upload_id, parts) =
+            stage_upload_with_create_opts(&set_disks, bucket, object, &[0x22; 4096], &ObjectOptions::default()).await;
+        let first_set = set_disks.clone();
+        let second_set = set_disks.clone();
+        let first_parts = parts.clone();
+        let second_parts = parts.clone();
+        let first_upload = upload_id.clone();
+        let second_upload = upload_id.clone();
+        let (first, second) = tokio::join!(
+            async move {
+                first_set
+                    .complete_multipart_upload(bucket, object, &first_upload, first_parts, &ObjectOptions::default())
+                    .await
+            },
+            async move {
+                second_set
+                    .complete_multipart_upload(bucket, object, &second_upload, second_parts, &ObjectOptions::default())
+                    .await
+            },
+        );
+        let first = first.expect("one completion must succeed");
+        let second = second.expect("the duplicate completion must succeed");
+        assert_eq!(first.etag, second.etag, "both completions must report the same ETag");
+        assert_eq!(first.version_id, second.version_id);
+        assert_eq!(
+            usize::from(first.multipart_completion_replayed) + usize::from(second.multipart_completion_replayed),
+            1
+        );
+        let info = set_disks
+            .get_object_info(bucket, object, &ObjectOptions::default())
+            .await
+            .expect("the object must be readable after duplicate completion");
+        assert_eq!(info.etag, first.etag);
+        assert_eq!(info.size, 4096);
+    }
+
+    #[tokio::test]
+    async fn versioned_complete_retry_returns_the_completed_version() {
+        let (_temp_dirs, disk_stores, set_disks) = hermetic_set_disks(4).await;
+        let bucket = "multipart-complete-versioned";
+        let object = "object";
+        make_bucket_on_all(&disk_stores, bucket).await;
+        let versioned = ObjectOptions {
+            versioned: true,
+            ..Default::default()
+        };
+
+        let (upload_id, parts) = stage_upload_with_create_opts(&set_disks, bucket, object, &[0x33; 4096], &versioned).await;
+        let completed = set_disks
+            .clone()
+            .complete_multipart_upload(bucket, object, &upload_id, parts.clone(), &versioned)
+            .await
+            .expect("versioned completion should publish a version");
+        let completed_version = completed.version_id.expect("versioned completion assigns a version id");
+
+        let mut reader = PutObjReader::from_vec(b"newer-version".to_vec());
+        let successor = set_disks
+            .put_object(bucket, object, &mut reader, &versioned)
+            .await
+            .expect("a later versioned put should add a version");
+        assert_ne!(successor.version_id, Some(completed_version));
+
+        let retried = set_disks
+            .clone()
+            .complete_multipart_upload(bucket, object, &upload_id, parts, &versioned)
+            .await
+            .expect("the retry must still find the version this upload published");
+        assert!(retried.multipart_completion_replayed);
+        assert_eq!(retried.etag, completed.etag);
+        assert_eq!(retried.version_id, Some(completed_version));
+
+        let latest = set_disks
+            .get_object_info(bucket, object, &versioned)
+            .await
+            .expect("the latest version must stay the successor put");
+        assert_eq!(latest.version_id, successor.version_id);
+        assert_ne!(latest.etag, completed.etag);
+    }
+
     /// Crash-consistency for the two `complete_multipart_upload` commit windows.
     ///
     /// rustfs/backlog#864: a fault that interrupts a completion must never mutate
@@ -9299,18 +9691,24 @@ mod tests {
                 "the post-commit crash must leave the upload listable for reclamation"
             );
 
-            // A retried CompleteMultipartUpload is answered deterministically: the
-            // commit rename already consumed the upload's metadata, so the retry
-            // resolves to InvalidUploadID (NoSuchUpload to the S3 client, the
-            // standard answer for a completed-then-retried upload) — never a torn
-            // state, and the committed object is untouched by the retry.
-            let retried = complete(&set_disks, bucket, object, &u_new, parts_retry).await;
+            // The commit rename already consumed the upload's xl.meta, so the
+            // retry cannot read staging. It must still return the committed
+            // object: AWS CompleteMultipartUpload is idempotent for the same
+            // upload id and part list, and the retry must not tear that object.
+            let (body_before_retry, etag_before_retry) = read_object(&set_disks, bucket, object).await;
+            assert_eq!(body_before_retry, new, "a post-commit crash must leave the whole new version readable");
+            let retried = complete(&set_disks, bucket, object, &u_new, parts_retry)
+                .await
+                .expect("a retried complete after the commit landed must return the committed object");
+            assert!(retried.multipart_completion_replayed, "the retry must not publish a second object");
+            assert_eq!(retried.etag, etag_before_retry, "the retry must return the committed ETag");
+            let (body_after_retry, etag_after_retry) = read_object(&set_disks, bucket, object).await;
+            assert_eq!(body_after_retry, new, "the retry must not disturb the committed object");
+            assert_eq!(etag_after_retry, etag_before_retry, "the retry must not rewrite the committed ETag");
             assert!(
-                matches!(retried, Err(StorageError::InvalidUploadID(..))),
-                "a retried complete after the commit landed must resolve to InvalidUploadID, got {retried:?}"
+                !upload_is_listed(&set_disks, bucket, object, &u_new).await,
+                "the idempotent retry must reclaim staging left behind by the crash"
             );
-            let (body_after_retry, _) = read_object(&set_disks, bucket, object).await;
-            assert_eq!(body_after_retry, new, "the failed retry must not disturb the committed object");
 
             // Reclaim the leftover exactly as the production tail does: delete_all
             // on the upload path (abort_multipart_upload cannot — the upload's

crates/ecstore/src/set_disk/ops/object.rs

@@ -3500,6 +3500,13 @@ impl SetDisks {
         mut publication_fence: Option<RemoteTuplePublicationFence>,
     ) -> Result<(ObjectInfo, Option<OldCurrentSize>)> {
         let protect_write = opts.shard_integrity_write_enabled();
+        let source_bucket_incarnation_id = match opts.expected_bucket_incarnation_id {
+            Some(incarnation_id) => Some(incarnation_id),
+            // Internal metadata writes may already hold the pool metadata
+            // lock; resolving a user bucket identity would re-enter it.
+            None if !is_meta_bucketname(bucket) => self.bucket_incarnation_id_from_disk(bucket).await.ok(),
+            None => None,
+        };
         if publication_fence.is_none()
             && opts.data_movement
             && rustfs_utils::http::metadata_compat::contains_key_str(
@@ -4031,7 +4038,7 @@ impl SetDisks {
             {
                 #[cfg(any(test, feature = "test-util"))]
                 pause_put_object_commit(bucket, object, PutObjectCommitPause::BeforeNamespace).await;
-                if let Some(expected_incarnation_id) = opts.expected_bucket_incarnation_id
+                if let Some(expected_incarnation_id) = source_bucket_incarnation_id
                     && opts.bucket_lifecycle_lock_fence.is_none()
                 {
                     bucket_lifecycle_guard = Some(
@@ -4642,6 +4649,7 @@ impl SetDisks {
                         request.object_version_id = committed_version_id
                             .or_else(|| commit_version_suspended.then(Uuid::nil))
                             .map(|version_id| version_id.to_string());
+                        request.expected_bucket_incarnation_id = source_bucket_incarnation_id;
                         let object_lock_guard = _object_lock_guard.take();
                         let publication_guard = _publication_guard.take();
                         let bucket_lifecycle_guard = _bucket_lifecycle_guard.take();
@@ -4790,6 +4798,7 @@ impl SetDisks {
                     request.object_version_id = committed_version_id
                         .or_else(|| commit_version_suspended.then(Uuid::nil))
                         .map(|version_id| version_id.to_string());
+                    request.expected_bucket_incarnation_id = source_bucket_incarnation_id;
                     commit_set.submit_rename_tail_heal(request).await;
                 }
 
@@ -7492,8 +7501,15 @@ impl SetDisks {
             ensure_delete_commit_locks_held(None, bucket, &encoded_object, opts)?;
             authorization.authorized_journal_name(&candidate)?;
             begin_scanner_publication_delete_mutation(opts.scanner_publication_commit_scope.as_ref())?;
-            self.delete_object_version(bucket, &encoded_object, &delete_request, false)
-                .await?;
+            self.delete_object_version_with_purge(
+                bucket,
+                &encoded_object,
+                &delete_request,
+                false,
+                None,
+                opts.expected_bucket_incarnation_id,
+            )
+            .await?;
             if let Some((_, deleted_object)) = replication_delete {
                 ReplicationLifecycleBridge::schedule_delete(bucket.to_string(), deleted_object).await;
             }
@@ -7573,6 +7589,7 @@ impl SetDisks {
         fi: &FileInfo,
         force_del_marker: bool,
         delete_marker_purge: Option<MrfDeleteMarkerPurge>,
+        source_bucket_incarnation_id: Option<Uuid>,
     ) -> Result<()> {
         let transported = delete_file_info_with_replication_transport_metadata(fi);
         let fi = &transported;
@@ -7727,13 +7744,55 @@ impl SetDisks {
         {
             let version_id = fi.version_id.map(|version| version.to_string());
             let _ = self
-                .add_partial(bucket, object, version_id.as_deref().unwrap_or_default())
+                .add_partial_with_source_incarnation(
+                    bucket,
+                    object,
+                    version_id.as_deref().unwrap_or_default(),
+                    source_bucket_incarnation_id,
+                )
                 .await;
         }
         quorum_result
     }
 }
 
+impl SetDisks {
+    async fn add_partial_with_source_incarnation(
+        &self,
+        bucket: &str,
+        object: &str,
+        version_id: &str,
+        source_bucket_incarnation_id: Option<Uuid>,
+    ) -> Result<()> {
+        if self
+            .persist_partial_write(bucket, object, Some(version_id), source_bucket_incarnation_id)
+            .await
+        {
+            return Ok(());
+        }
+        let Some(source_bucket_incarnation_id) = source_bucket_incarnation_id else {
+            // The fallback is best-effort. Without the source generation it
+            // cannot safely target the object name after bucket recreation.
+            return Ok(());
+        };
+        let mut request = rustfs_heal_contracts::heal_channel::create_heal_request_with_options(
+            bucket.to_string(),
+            Some(object.to_string()),
+            false,
+            Some(HealChannelPriority::Normal),
+            Some(self.pool_index),
+            Some(self.set_index),
+        );
+        request.object_version_id = (!version_id.is_empty()).then(|| version_id.to_string());
+        request.source = rustfs_heal_contracts::heal_channel::HealRequestSource::Mrf;
+        request.expected_bucket_incarnation_id = Some(source_bucket_incarnation_id);
+        if let Err(error) = rustfs_heal_contracts::heal_channel::send_heal_request(request).await {
+            warn!(bucket, object, version_id, error = %error, "Failed to enqueue heal request for partial object");
+        }
+        Ok(())
+    }
+}
+
 #[async_trait::async_trait]
 impl crate::storage_api_contracts::object::ObjectOperations for SetDisks {
     type Error = Error;
@@ -8021,7 +8080,12 @@ impl crate::storage_api_contracts::object::ObjectOperations for SetDisks {
     }
     #[tracing::instrument(skip(self))]
     async fn delete_object_version(&self, bucket: &str, object: &str, fi: &FileInfo, force_del_marker: bool) -> Result<()> {
-        self.delete_object_version_with_purge(bucket, object, fi, force_del_marker, None)
+        let source_bucket_incarnation_id = if is_meta_bucketname(bucket) {
+            None
+        } else {
+            self.bucket_incarnation_id_from_disk(bucket).await.ok()
+        };
+        self.delete_object_version_with_purge(bucket, object, fi, force_del_marker, None, source_bucket_incarnation_id)
             .await
     }
 
@@ -8843,7 +8907,15 @@ impl crate::storage_api_contracts::object::ObjectOperations for SetDisks {
                                 delete_request.set_skip_tier_free_version();
                             }
                             begin_scanner_publication_delete_mutation(scanner_publication_commit_scope.as_ref())?;
-                            self.delete_object_version(bucket, object, &delete_request, false).await?;
+                            self.delete_object_version_with_purge(
+                                bucket,
+                                object,
+                                &delete_request,
+                                false,
+                                None,
+                                opts.expected_bucket_incarnation_id,
+                            )
+                            .await?;
                             if let Some((_, deleted_object)) = replication_delete {
                                 ReplicationLifecycleBridge::schedule_delete(bucket.to_string(), deleted_object).await;
                             }
@@ -8858,7 +8930,15 @@ impl crate::storage_api_contracts::object::ObjectOperations for SetDisks {
                             };
                             delete_request.set_tier_free_version_id(&Uuid::new_v4().to_string());
                             begin_scanner_publication_delete_mutation(scanner_publication_commit_scope.as_ref())?;
-                            self.delete_object_version(bucket, object, &delete_request, false).await?;
+                            self.delete_object_version_with_purge(
+                                bucket,
+                                object,
+                                &delete_request,
+                                false,
+                                None,
+                                opts.expected_bucket_incarnation_id,
+                            )
+                            .await?;
                         }
                         for version in &versions.free_versions {
                             ensure_delete_commit_locks_held(_lock_guard.as_ref(), bucket, object, &opts)?;
@@ -8870,7 +8950,15 @@ impl crate::storage_api_contracts::object::ObjectOperations for SetDisks {
                             };
                             delete_request.set_tier_free_version();
                             begin_scanner_publication_delete_mutation(scanner_publication_commit_scope.as_ref())?;
-                            self.delete_object_version(bucket, object, &delete_request, false).await?;
+                            self.delete_object_version_with_purge(
+                                bucket,
+                                object,
+                                &delete_request,
+                                false,
+                                None,
+                                opts.expected_bucket_incarnation_id,
+                            )
+                            .await?;
                         }
                     }
                 }
@@ -8978,7 +9066,7 @@ impl crate::storage_api_contracts::object::ObjectOperations for SetDisks {
             };
             ensure_delete_commit_locks_held(_lock_guard.as_ref(), bucket, object, &opts)?;
             begin_scanner_publication_delete_mutation(scanner_publication_commit_scope.as_ref())?;
-            self.delete_object_version(bucket, object, &dfi, false)
+            self.delete_object_version_with_purge(bucket, object, &dfi, false, None, opts.expected_bucket_incarnation_id)
                 .await
                 .map_err(|e| to_object_err(e, vec![bucket, object]))?;
             self.invalidate_get_object_metadata_cache(bucket, object).await;
@@ -9072,6 +9160,7 @@ impl crate::storage_api_contracts::object::ObjectOperations for SetDisks {
                 &fi,
                 should_force_delete_marker_for_missing_version(&opts),
                 delete_marker_purge.clone(),
+                opts.expected_bucket_incarnation_id,
             )
             .await
             .map_err(|e| to_object_err(e, vec![bucket, object]))?;
@@ -9124,9 +9213,16 @@ impl crate::storage_api_contracts::object::ObjectOperations for SetDisks {
         if opts.skip_free_version {
             dfi.set_skip_tier_free_version();
         }
-        self.delete_object_version_with_purge(bucket, object, &dfi, opts.delete_marker, delete_marker_purge)
-            .await
-            .map_err(|e| to_object_err(e, vec![bucket, object]))?;
+        self.delete_object_version_with_purge(
+            bucket,
+            object,
+            &dfi,
+            opts.delete_marker,
+            delete_marker_purge,
+            opts.expected_bucket_incarnation_id,
+        )
+        .await
+        .map_err(|e| to_object_err(e, vec![bucket, object]))?;
         #[cfg(test)]
         pause_delete_object_commit_after_publish(bucket, object).await;
 
@@ -9188,28 +9284,13 @@ impl crate::storage_api_contracts::object::ObjectOperations for SetDisks {
 
     #[tracing::instrument(skip(self))]
     async fn add_partial(&self, bucket: &str, object: &str, version_id: &str) -> Result<()> {
-        if self.persist_partial_write(bucket, object, Some(version_id)).await {
-            return Ok(());
-        }
-        let mut request = rustfs_heal_contracts::heal_channel::create_heal_request_with_options(
-            bucket.to_string(),
-            Some(object.to_string()),
-            false,
-            Some(HealChannelPriority::Normal),
-            Some(self.pool_index),
-            Some(self.set_index),
-        );
-        request.object_version_id = (!version_id.is_empty()).then(|| version_id.to_string());
-        if let Err(e) = rustfs_heal_contracts::heal_channel::send_heal_request(request).await {
-            warn!(
-                bucket,
-                object,
-                version_id,
-                error = %e,
-                "Failed to enqueue heal request for partial object"
-            );
-        }
-        Ok(())
+        let source_bucket_incarnation_id = if is_meta_bucketname(bucket) {
+            None
+        } else {
+            self.bucket_incarnation_id_from_disk(bucket).await.ok()
+        };
+        self.add_partial_with_source_incarnation(bucket, object, version_id, source_bucket_incarnation_id)
+            .await
     }
 
     #[tracing::instrument(skip(self))]
@@ -9768,7 +9849,10 @@ impl crate::storage_api_contracts::object::ObjectOperations for SetDisks {
         #[cfg(all(test, feature = "test-util"))]
         pause_transition_transaction_at(bucket, object, TransitionTransactionKillPoint::CommitFenceBeforeLocalCommit).await;
         upload_cleanup.disarm();
-        if let Err(err) = self.delete_object_version(bucket, object, &fi, false).await {
+        if let Err(err) = self
+            .delete_object_version_with_purge(bucket, object, &fi, false, None, opts.expected_bucket_incarnation_id)
+            .await
+        {
             warn!(
                 bucket = bucket,
                 object = object,
@@ -9839,7 +9923,12 @@ impl crate::storage_api_contracts::object::ObjectOperations for SetDisks {
                 continue;
             }
             let _ = self
-                .add_partial(bucket, object, opts.version_id.as_deref().unwrap_or_default())
+                .add_partial_with_source_incarnation(
+                    bucket,
+                    object,
+                    opts.version_id.as_deref().unwrap_or_default(),
+                    opts.expected_bucket_incarnation_id,
+                )
                 .await;
             break;
         }
@@ -19558,15 +19647,11 @@ mod put_object_tmp_cleanup_tests {
                 expected.is_subset(&after_tail),
                 "the detached tail must re-mark capacity after the first scope was drained"
             );
-            let request = tokio::time::timeout(Duration::from_secs(30), heal_requests.recv())
-                .await
-                .expect("a failed tail should submit heal")
-                .expect("the per-set heal capture should stay connected");
-            assert_eq!(request.bucket, bucket);
-            assert_eq!(request.object_prefix.as_deref(), Some(object));
-            assert_eq!(request.object_version_id.as_deref(), Some(expected_version_id.as_str()));
-            assert_eq!(request.pool_index, Some(set_disks.pool_index));
-            assert_eq!(request.set_index, Some(set_disks.set_index));
+            assert!(
+                heal_requests.try_recv().is_err(),
+                "a legacy fixture without a bucket-generation record must not enqueue an unfenced fallback heal"
+            );
+            assert_eq!(expected_version_id, Uuid::nil().to_string());
         })
         .await;
     }
@@ -19687,7 +19772,7 @@ mod put_object_tmp_cleanup_tests {
 
     #[tokio::test]
     #[serial_test::serial(capacity_dirty_scope)]
-    async fn tail_drained_put_preserves_quorum_success_and_heals_failed_tail() {
+    async fn tail_drained_put_preserves_quorum_success_without_unfenced_fallback_heal() {
         let (_dirs, disks, set) = hermetic_set_disks(4).await;
         let bucket = "put-full-tail-heal";
         let object = "full-tail-heal-object";
@@ -19723,12 +19808,10 @@ mod put_object_tmp_cleanup_tests {
             .expect("PUT task should join")
             .expect("a minority tail error must not negate committed quorum");
         assert_eq!(tasks.running(), 0);
-        let heal = tokio::time::timeout(Duration::from_secs(30), heals.recv())
-            .await
-            .expect("failed tail must schedule heal")
-            .expect("heal capture must remain connected");
-        assert_eq!(heal.bucket, bucket);
-        assert_eq!(heal.object_prefix.as_deref(), Some(object));
+        assert!(
+            heals.try_recv().is_err(),
+            "a legacy fixture without a bucket-generation record must not enqueue an unfenced fallback heal"
+        );
         let info = set
             .get_object_info(bucket, object, &ObjectOptions::default())
             .await
@@ -22371,6 +22454,61 @@ mod single_delete_namespace_owner_tests {
         .expect("seed a complete real object version");
     }
 
+    #[tokio::test]
+    #[serial_test::serial]
+    async fn internal_metadata_mutations_do_not_reenter_pool_metadata() {
+        let (_dirs, store, _other_store) =
+            crate::services::rebalance::test_three_pool_stores_with_isolated_node_contexts(None).await;
+        let set = &store.pools[0].disk_set[0];
+        let disks = set.disk_inventory().await.into_iter().flatten().collect::<Vec<_>>();
+
+        for bucket in [RUSTFS_META_BUCKET, crate::disk::MIGRATING_META_BUCKET] {
+            for disk in &disks {
+                if let Err(err) = disk.make_volume(bucket).await {
+                    assert_eq!(err, DiskError::VolumeExists, "prepare the internal metadata volume");
+                }
+            }
+            let version = Uuid::new_v4();
+            seed_version(set, bucket, "delete-under-pool-lock", version, b"metadata version").await;
+            let request = FileInfo {
+                name: "delete-under-pool-lock".to_string(),
+                version_id: Some(version),
+                mod_time: Some(OffsetDateTime::now_utc()),
+                ..Default::default()
+            };
+            let mut reader = PutObjReader::from_vec(b"metadata under lock".to_vec());
+            let opts = ObjectOptions {
+                write_completion: WriteCompletion::TailDrained,
+                ..Default::default()
+            };
+
+            // Pool metadata transactions retain this write guard while saving
+            // internal objects. None of these paths may look up a user bucket.
+            let _pool_meta_guard = store.pool_meta.write().await;
+            let limit = Duration::from_secs(30);
+            let (put, delete, partial) = tokio::join!(
+                tokio::time::timeout(limit, set.put_object(bucket, "put-under-pool-lock", &mut reader, &opts)),
+                tokio::time::timeout(limit, set.delete_object_version(bucket, &request.name, &request, false)),
+                tokio::time::timeout(limit, set.add_partial(bucket, "partial-under-pool-lock", "")),
+            );
+            assert!(
+                matches!(&put, Ok(Ok(_))) && matches!(&delete, Ok(Ok(()))) && matches!(&partial, Ok(Ok(()))),
+                "internal operations must finish while pool metadata is locked: bucket={bucket}, put={put:?}, delete={delete:?}, partial={partial:?}"
+            );
+            for disk in &disks {
+                let written = disk
+                    .read_version("", bucket, "put-under-pool-lock", "", &ReadOptions::default())
+                    .await
+                    .expect("the internal PUT must reach every disk");
+                assert_eq!(written.size, b"metadata under lock".len() as i64);
+                let deleted = disk
+                    .read_version("", bucket, &request.name, &version.to_string(), &ReadOptions::default())
+                    .await;
+                assert!(matches!(deleted, Err(DiskError::FileNotFound | DiskError::FileVersionNotFound)));
+            }
+        }
+    }
+
     #[tokio::test]
     #[serial_test::serial(capacity_dirty_scope)]
     async fn single_delete_advances_namespace_generation_through_cleanup() {

crates/ecstore/src/store/bucket.rs

@@ -1569,12 +1569,27 @@ mod tests {
         disk_indexes: &[usize],
     ) -> Vec<(usize, crate::disk::DiskStore)> {
         let offline = {
-            let mut disks = set.disks.write().await;
+            let disks = set.disks.read().await;
             disk_indexes
                 .iter()
-                .map(|index| (*index, disks[*index].take().expect("fault-injection disk should start online")))
+                .map(|index| (*index, disks[*index].clone().expect("fault-injection disk should start online")))
                 .collect::<Vec<_>>()
         };
+        for (_, disk) in &offline {
+            disk.close().await.expect("fault injection should stop per-disk monitoring");
+            disk.force_offline_for_test();
+        }
+        // Empty slots can be renewed from the still-present disk paths by the endpoint monitor.
+        // Keep the original handles and prove reconnect cannot clear their injected IO failure.
+        set.connect_disks().await;
+        {
+            let disks = set.disks.read().await;
+            for (index, disk) in &offline {
+                let current = disks[*index].as_ref().expect("fault-injection slot should remain populated");
+                assert!(Arc::ptr_eq(current, disk), "reconnect must preserve the fault-injection handle");
+                assert_eq!(disk.runtime_state(), crate::disk::health_state::RuntimeDriveHealthState::Offline);
+            }
+        }
         let local_disk_map = ecstore.ctx.local_disk_map();
         let mut local_disks = local_disk_map.write().await;
         for (_, disk) in &offline {
@@ -1595,9 +1610,12 @@ mod tests {
                 local_disks.insert(disk.endpoint().to_string(), Some(Arc::clone(disk)));
             }
         }
-        let mut disks = set.disks.write().await;
+        let disks = set.disks.read().await;
         for (index, disk) in offline {
-            assert!(disks[index].replace(disk).is_none(), "fault-injection disk slot should remain empty");
+            let current = disks[index].as_ref().expect("fault-injection slot should remain populated");
+            assert!(Arc::ptr_eq(current, &disk), "fault-injection disk handle should remain unchanged");
+            assert_eq!(disk.runtime_state(), crate::disk::health_state::RuntimeDriveHealthState::Offline);
+            disk.reset_health_for_store_init_retry();
         }
     }
 
@@ -1854,8 +1872,12 @@ mod tests {
         assert_eq!(residue.diagnostic_bytes_read, 0);
     }
 
-    #[tokio::test]
+    #[tokio::test(start_paused = true)]
     async fn bucket_residue_scan_distinguishes_visible_and_tier_free_xlmeta() {
+        // Classification keeps the production budget, but real filesystem
+        // scheduling must not advance the clock before classification finishes.
+        let (clock_guard_tx, clock_guard_rx) = std::sync::mpsc::channel::<()>();
+        let clock_guard = tokio::task::spawn_blocking(move || clock_guard_rx.recv());
         let root = tempfile::tempdir().expect("temporary bucket root should be created");
         let bucket_path = root.path().join("bucket");
         let visible_path = bucket_path.join("visible").join(STORAGE_FORMAT_FILE);
@@ -1877,6 +1899,7 @@ mod tests {
         let visible_scan = scan_metadata_less_residue(&bucket_path)
             .await
             .expect("visible xl.meta scan should succeed");
+        assert!(!visible_scan.diagnostic_truncated, "{visible_scan:?}");
         assert_eq!(visible_scan.xlmeta_blocker, Some(BucketDeleteBlockerKind::VisibleVersion));
 
         tokio::fs::remove_dir_all(bucket_path.join("visible"))
@@ -1913,6 +1936,7 @@ mod tests {
         let free_scan = scan_metadata_less_residue(&bucket_path)
             .await
             .expect("free-version xl.meta scan should succeed");
+        assert!(!free_scan.diagnostic_truncated, "{free_scan:?}");
         assert_eq!(free_scan.xlmeta_blocker, Some(BucketDeleteBlockerKind::TierFreeVersion));
 
         tokio::fs::remove_dir_all(bucket_path.join("free"))
@@ -1933,6 +1957,7 @@ mod tests {
         let exact_limit_scan = scan_metadata_less_residue(&bucket_path)
             .await
             .expect("exact-limit xl.meta scan should remain fail closed");
+        assert!(!exact_limit_scan.diagnostic_truncated, "{exact_limit_scan:?}");
         assert_eq!(exact_limit_scan.xlmeta_blocker, Some(BucketDeleteBlockerKind::UnknownXlMeta));
         assert_eq!(exact_limit_scan.diagnostic_bytes_read, BUCKET_DELETE_XLMETA_DIAGNOSTIC_MAX_BYTES);
         tokio::fs::remove_dir_all(bucket_path.join("exact-limit"))
@@ -1953,9 +1978,31 @@ mod tests {
         let oversized_scan = scan_metadata_less_residue(&bucket_path)
             .await
             .expect("oversized xl.meta scan should remain fail closed");
+        assert!(!oversized_scan.diagnostic_truncated, "{oversized_scan:?}");
         assert_eq!(oversized_scan.xlmeta_blocker, Some(BucketDeleteBlockerKind::UnknownXlMeta));
         assert_eq!(oversized_scan.diagnostic_bytes_read, 0);
         assert!(oversized_scan.diagnostic_bytes_read <= BUCKET_DELETE_XLMETA_DIAGNOSTIC_MAX_BYTES);
+
+        drop(clock_guard_tx);
+        let _ = clock_guard
+            .await
+            .expect("classification clock guard should exit after its sender is dropped");
+
+        // An exhausted diagnostic budget may return before observing xl.meta.
+        // It must report truncation rather than inventing a classification.
+        let first_io_started = Arc::new(AtomicBool::new(false));
+        let mut budget = BucketDeleteDiagnosticBudget::new().with_first_io_delay(
+            BUCKET_DELETE_DIAGNOSTIC_MAX_ELAPSED + Duration::from_millis(100),
+            first_io_started.clone(),
+        );
+        let truncated = scan_metadata_less_residue_with_budget(&bucket_path, &mut budget)
+            .await
+            .expect("a diagnostic timeout should return a partial scan");
+        assert!(first_io_started.load(Ordering::SeqCst));
+        assert!(truncated.diagnostic_truncated, "{truncated:?}");
+        assert_eq!(truncated.xlmeta_blocker, None);
+        assert_eq!(truncated.entries_scanned, 0);
+        assert_eq!(truncated.diagnostic_bytes_read, 0);
     }
 
     #[tokio::test]
@@ -3376,7 +3423,7 @@ mod tests {
 
     #[tokio::test(flavor = "multi_thread")]
     #[serial]
-    async fn make_bucket_seeds_new_bucket_durability_override() {
+    async fn make_bucket_inherits_default_durability() {
         temp_env::async_with_vars([(crate::bucket::durability::ENV_NEW_BUCKET_DURABILITY_MODE, None::<&str>)], async {
             let (_disk_paths, ecstore) = setup_bucket_delete_test_env().await;
             let bucket = format!("bucket-default-durability-{}", Uuid::new_v4().simple());
@@ -3389,10 +3436,7 @@ mod tests {
             let metadata = metadata_sys::get_in(&ecstore.ctx, &bucket)
                 .await
                 .expect("metadata should load for the new bucket");
-            assert_eq!(
-                metadata.durability_config().and_then(|cfg| cfg.normalized_mode()).as_deref(),
-                Some(crate::bucket::durability::BUCKET_DURABILITY_MODE_RELAXED)
-            );
+            assert!(metadata.durability_config().is_none());
         })
         .await;
     }

crates/ecstore/src/store/init.rs

@@ -11176,9 +11176,17 @@ mod tests {
         const ADMITTED_READS: usize = 32;
 
         let temp_dir = tempfile::tempdir().expect("create validation-read shutdown store dir");
-        let (ctx, store, shutdown) =
-            without_storage_class_env(build_isolated_test_store(temp_dir.path(), "dispatch-validation-read-shutdown", &[4]))
-                .await;
+        // The manual worker must own the seeded records and shutdown progress.
+        let mut instance_ctx = crate::runtime::instance::InstanceContext::new();
+        instance_ctx.suppress_tier_delete_journal_recovery_for_test();
+        let (ctx, store, shutdown) = without_storage_class_env(build_isolated_test_store_with_layout(
+            temp_dir.path(),
+            "dispatch-validation-read-shutdown",
+            &[(1, 4)],
+            CancellationToken::new(),
+            Some(Arc::new(instance_ctx)),
+        ))
+        .await;
         crate::bucket::metadata_sys::init_bucket_metadata_sys(store.clone(), Vec::new()).await;
         let bucket = "dispatch-validation-read-shutdown-bucket";
         store
@@ -11710,8 +11718,17 @@ mod tests {
         const ADMITTED_BATCH: usize = 32;
 
         let temp_dir = tempfile::tempdir().expect("create delete-batch shutdown store dir");
-        let (ctx, store, shutdown) =
-            without_storage_class_env(build_isolated_test_store(temp_dir.path(), "dispatch-delete-batch-shutdown", &[4])).await;
+        // The manual worker must own the seeded records and shutdown progress.
+        let mut instance_ctx = crate::runtime::instance::InstanceContext::new();
+        instance_ctx.suppress_tier_delete_journal_recovery_for_test();
+        let (ctx, store, shutdown) = without_storage_class_env(build_isolated_test_store_with_layout(
+            temp_dir.path(),
+            "dispatch-delete-batch-shutdown",
+            &[(1, 4)],
+            CancellationToken::new(),
+            Some(Arc::new(instance_ctx)),
+        ))
+        .await;
         crate::bucket::metadata_sys::init_bucket_metadata_sys(store.clone(), Vec::new()).await;
         let bucket = "dispatch-delete-batch-shutdown-bucket";
         store
@@ -20362,6 +20379,259 @@ mod tests {
         }
     }
 
+    #[cfg(feature = "test-util")]
+    #[tokio::test]
+    #[serial_test::serial(storage_class_env)]
+    async fn tier_config_initial_reload_lock_failure_keeps_recovery_worker() {
+        use crate::storage_api_contracts::namespace::NamespaceLocking as _;
+
+        let temp_dir = tempfile::tempdir().expect("create contended tier startup store dir");
+        let (ctx, store, shutdown) =
+            without_storage_class_env(build_isolated_test_store(temp_dir.path(), "tier-startup-contention", &[4])).await;
+        let manager = ctx.tier_config_mgr();
+        register_mock_tier(&manager, "COLD-A").await;
+        let candidate = TierConfigMgr::new();
+        let candidate = candidate.read().await;
+        let candidate_digest = tier_config_candidate_digest(&candidate).expect("empty candidate digest should resolve");
+        candidate
+            .save_tiering_config(store.clone())
+            .await
+            .expect("committed removal config should persist");
+        let config_info = store
+            .get_object_info(
+                RUSTFS_META_BUCKET,
+                &format!("{}/{TIER_CONFIG_FILE}", com::CONFIG_PREFIX),
+                &ObjectOptions::default(),
+            )
+            .await
+            .expect("committed removal config metadata should load");
+        let mutation_id = uuid::Uuid::new_v4();
+        let mut intent = tier_mutation_peer_test_intent(mutation_id, "COLD-A", candidate_digest);
+        intent.kind = TierMutationIntentKind::Remove;
+        intent.affected_targets[0].new_backend_identity = None;
+        intent
+            .advance(
+                TierMutationIntentState::Committed,
+                Some(config_info.etag.expect("committed config should have an ETag")),
+            )
+            .expect("removal intent should commit");
+        save_tier_mutation_intent_record(store.clone(), &intent)
+            .await
+            .expect("committed removal intent should persist");
+
+        let config_lock = format!("{}/{TIER_CONFIG_FILE}.lock", com::CONFIG_PREFIX);
+        let namespace = store
+            .new_ns_lock(RUSTFS_META_BUCKET, &config_lock)
+            .await
+            .expect("tier config namespace should resolve");
+        let owner = namespace
+            .get_write_lock(crate::set_disk::get_lock_acquire_timeout())
+            .await
+            .expect("competing recovery should hold the config lock");
+        let initial_error = runtime_sources::init_tier_config_mgr_handle(manager.clone(), store.clone())
+            .await
+            .expect_err("startup must still report the initial recovery lock failure");
+        assert!(initial_error.to_string().contains(&config_lock));
+        match TierConfigMgr::acquire_operation_lease(&manager, "COLD-A").await {
+            Err(err) => assert_eq!(err.message, "Remote tier configuration is being replaced"),
+            Ok(_) => panic!("failed initial recovery must retain its mutation fence"),
+        }
+        drop(owner);
+
+        tokio::time::timeout(Duration::from_secs(30), async {
+            loop {
+                if manager.read().await.tiers.is_empty()
+                    && matches!(
+                        TierConfigMgr::acquire_operation_lease(&manager, "COLD-A").await,
+                        Err(err) if err.code == crate::services::tier::tier_handlers::ERR_TIER_NOT_FOUND.code
+                    )
+                {
+                    break;
+                }
+                tokio::time::sleep(Duration::from_millis(10)).await;
+            }
+        })
+        .await
+        .expect("recovery worker must publish and clear its fence without another admin mutation");
+        assert_eq!(
+            load_tier_mutation_intent_record(store.clone(), mutation_id)
+                .await
+                .expect("unexpired committed tombstone must remain durable"),
+            intent
+        );
+        shutdown.cancel();
+    }
+
+    #[cfg(feature = "test-util")]
+    #[tokio::test]
+    #[serial_test::serial(storage_class_env)]
+    async fn tier_mutation_peer_prepare_reloads_after_save_and_load_lock_timeouts() {
+        use crate::storage_api_contracts::namespace::NamespaceLocking as _;
+        use std::sync::atomic::Ordering;
+
+        let temp_dir = tempfile::tempdir().expect("create contended tier prepare store dir");
+        let (ctx, store, shutdown) =
+            without_storage_class_env(build_isolated_test_store(temp_dir.path(), "tier-prepare-contention", &[4])).await;
+        let mutation_id = uuid::Uuid::new_v4();
+        let intent = tier_mutation_peer_test_intent(mutation_id, "COLD-A", [9; 32]);
+        let object = crate::services::tier::tier_mutation_intent::tier_mutation_intent_record_object_name(mutation_id)
+            .expect("intent record object should resolve");
+        let barrier = crate::set_disk::PutObjectCommitBarrier::install(
+            RUSTFS_META_BUCKET,
+            &object,
+            crate::set_disk::PutObjectCommitPause::BeforeNamespace,
+        );
+        let prepare_store = store.clone();
+        let payload = intent.encode().expect("prepare intent should encode");
+        let mut prepare = tokio::spawn(async move {
+            handle_tier_mutation_peer_request(
+                prepare_store,
+                TIER_MUTATION_RPC_PROTOCOL_VERSION,
+                TierMutationRpcPhase::Prepare,
+                mutation_id,
+                &payload,
+            )
+            .await
+        });
+        barrier.wait_until_paused().await;
+        let namespace = store.pools[0].disk_set[0]
+            .new_ns_lock(RUSTFS_META_BUCKET, &object)
+            .await
+            .expect("intent record namespace should resolve");
+        let owner = namespace
+            .get_write_lock(crate::set_disk::get_lock_acquire_timeout())
+            .await
+            .expect("competing publisher should hold the intent namespace");
+        // The competing publisher already owns the namespace. Bypass only its
+        // recursive acquisition and drain the durable tail before releasing it.
+        com::save_config_with_opts(
+            store.clone(),
+            &object,
+            intent.encode().expect("winning intent should encode"),
+            &ObjectOptions {
+                no_lock: true,
+                max_parity: true,
+                write_completion: crate::object_api::WriteCompletion::TailDrained,
+                ..Default::default()
+            },
+        )
+        .await
+        .expect("competing publisher should persist the matching intent under its lock");
+        let manager = ctx
+            .lock_manager()
+            .as_fast_lock_manager()
+            .expect("fixture should use real locks");
+        let key = rustfs_lock::fast_lock::types::ObjectKey {
+            bucket: Arc::from(RUSTFS_META_BUCKET),
+            object: Arc::from(object.as_str()),
+            version: None,
+        };
+        let timeouts = &manager.get_shard(&key).metrics().timeouts;
+        let before = timeouts.load(Ordering::Relaxed);
+        barrier.release_and_wait_until_namespace_pending().await;
+        drop(barrier);
+        tokio::select! {
+            result = &mut prepare => panic!("Prepare must retry the contended save and load before completing: {result:?}"),
+            result = tokio::time::timeout(Duration::from_secs(30), async {
+                while timeouts.load(Ordering::Relaxed) - before < 2 {
+                    tokio::time::sleep(Duration::from_millis(10)).await;
+                }
+            }) => result.expect("the save and subsequent load should each exhaust their lock wait"),
+        }
+        assert_eq!(timeouts.load(Ordering::Relaxed) - before, 2);
+        drop(owner);
+        let outcome = tokio::time::timeout(Duration::from_secs(30), prepare)
+            .await
+            .expect("third prepare round should complete after the competing publisher releases its lock")
+            .expect("prepare task should join")
+            .expect("third prepare round should reload the matching durable intent");
+        assert_eq!(outcome.state, TierMutationPeerState::Prepared);
+        assert!(!outcome.applied, "the retry must reuse the winning record rather than recreate it");
+        assert_eq!(
+            load_tier_mutation_intent_record(store.clone(), mutation_id)
+                .await
+                .expect("winning record should remain readable"),
+            intent
+        );
+        assert_eq!(timeouts.load(Ordering::Relaxed) - before, 2);
+        shutdown.cancel();
+    }
+
+    #[cfg(feature = "test-util")]
+    #[tokio::test]
+    #[serial_test::serial(storage_class_env)]
+    async fn tier_mutation_peer_prepare_preserves_lock_timeout_after_three_rounds() {
+        use crate::storage_api_contracts::namespace::NamespaceLocking as _;
+        use std::sync::atomic::Ordering;
+
+        let temp_dir = tempfile::tempdir().expect("create exhausted tier prepare store dir");
+        let (ctx, store, shutdown) =
+            without_storage_class_env(build_isolated_test_store(temp_dir.path(), "tier-prepare-exhaustion", &[4])).await;
+        let mutation_id = uuid::Uuid::new_v4();
+        let intent = tier_mutation_peer_test_intent(mutation_id, "COLD-A", [9; 32]);
+        let object = crate::services::tier::tier_mutation_intent::tier_mutation_intent_record_object_name(mutation_id)
+            .expect("intent record object should resolve");
+        let barrier = crate::set_disk::PutObjectCommitBarrier::install(
+            RUSTFS_META_BUCKET,
+            &object,
+            crate::set_disk::PutObjectCommitPause::BeforeNamespace,
+        );
+        let prepare_store = store.clone();
+        let payload = intent.encode().expect("prepare intent should encode");
+        let prepare = tokio::spawn(async move {
+            handle_tier_mutation_peer_request(
+                prepare_store,
+                TIER_MUTATION_RPC_PROTOCOL_VERSION,
+                TierMutationRpcPhase::Prepare,
+                mutation_id,
+                &payload,
+            )
+            .await
+        });
+        barrier.wait_until_paused().await;
+        let namespace = store.pools[0].disk_set[0]
+            .new_ns_lock(RUSTFS_META_BUCKET, &object)
+            .await
+            .expect("intent record namespace should resolve");
+        let lock_timeout = crate::set_disk::get_lock_acquire_timeout();
+        let owner = namespace
+            .get_write_lock(lock_timeout)
+            .await
+            .expect("competing publisher should hold the intent namespace");
+        let manager = ctx
+            .lock_manager()
+            .as_fast_lock_manager()
+            .expect("fixture should use real locks");
+        let key = rustfs_lock::fast_lock::types::ObjectKey {
+            bucket: Arc::from(RUSTFS_META_BUCKET),
+            object: Arc::from(object.as_str()),
+            version: None,
+        };
+        let timeouts = &manager.get_shard(&key).metrics().timeouts;
+        let before = timeouts.load(Ordering::Relaxed);
+        barrier.release_and_wait_until_namespace_pending().await;
+        drop(barrier);
+        let error = tokio::time::timeout(Duration::from_secs(30), prepare)
+            .await
+            .expect("three contended prepare rounds should exhaust the bounded budget")
+            .expect("prepare task should join")
+            .expect_err("Prepare must fail while the competing publisher keeps the namespace");
+        match error {
+            TierMutationPeerError::Store(Error::Lock(rustfs_lock::LockError::Timeout { resource, timeout })) => {
+                assert_eq!(resource, key.to_string());
+                assert_eq!(timeout, lock_timeout);
+            }
+            other => panic!("Prepare must preserve the final typed lock timeout: {other:?}"),
+        }
+        assert_eq!(timeouts.load(Ordering::Relaxed) - before, 3, "save and load must share three rounds");
+        drop(owner);
+        assert!(matches!(
+            load_tier_mutation_intent_record(store, mutation_id).await,
+            Err(Error::ConfigNotFound)
+        ));
+        shutdown.cancel();
+    }
+
     #[cfg(feature = "test-util")]
     #[tokio::test]
     #[serial_test::serial(storage_class_env)]

crates/ecstore/src/store/list_objects.rs

@@ -387,6 +387,24 @@ fn should_purge_empty_directory_listing(
         && result.prefixes.is_empty()
 }
 
+fn should_purge_empty_recursive_bucket_listing(
+    prefix: &str,
+    delimiter: Option<&str>,
+    marker: Option<&str>,
+    max_keys: i32,
+    incl_deleted: bool,
+    result: &ListObjectsInfo,
+) -> bool {
+    prefix.is_empty()
+        && delimiter.is_none()
+        && marker.is_none()
+        && max_keys > 0
+        && !incl_deleted
+        && !result.is_truncated
+        && result.objects.is_empty()
+        && result.prefixes.is_empty()
+}
+
 const MARKER_TAG_VERSION: &str = "v2";
 const LEGACY_MARKER_TAG_VERSIONS: &[&str] = &["v1", MARKER_TAG_VERSION];
 const LIST_CACHE_MARKER_PREFIX: &str = "[rustfs_cache:";
@@ -4006,82 +4024,91 @@ impl ECStore {
         // key sorting between `<marker>` and `<marker>[` is silently skipped on
         // the continuation page (backlog#1047).
         opts.parse_marker();
-        if let Some(mode) = list_objects_index_mode_from_env()
-            && let Some(result) = self
-                .clone()
+        let key_only_result = if let Some(mode) = list_objects_index_mode_from_env() {
+            self.clone()
                 .list_objects_from_opt_in_key_only_provider(&opts, mode, max_keys, incl_deleted)
                 .await?
-        {
-            if should_purge_empty_directory_listing(prefix, opts.marker.as_deref(), max_keys, incl_deleted, &result)
-                && has_authoritative_never_versioned_state_in(&self.ctx, bucket)
-                    .await
-                    .unwrap_or(false)
-            {
-                self.purge_orphan_dir_object(bucket, prefix).await;
-            }
-            return Ok(result);
-        }
-
-        let mut list_result = self
-            .clone()
-            .list_path(&opts)
-            .await
-            .unwrap_or_else(|err| MetaCacheEntriesSortedResult {
-                err: Some(to_filemeta_err(err)),
-                ..Default::default()
-            });
-        let next_cache_id = list_result.entries.as_ref().and_then(|entries| entries.list_id.clone());
-
-        // err=None means gather_results filled its limit → disk has more data
-        let disk_has_more = list_result.err.is_none();
-
-        if let Some(err) = list_result.err.take()
-            && err != rustfs_filemeta::Error::Unexpected
-        {
-            return Err(to_object_err(err.into(), vec![bucket, prefix]));
-        }
-
-        if let Some(result) = list_result.entries.as_mut() {
-            result.forward_past(opts.marker.clone());
-        }
-
-        // contextCanceled
-
-        // Last RAW scanned key, captured before folding, so `list_objects_paginate`
-        // can advance past a fully-collapsed common-prefix page (ECA-03 / #944).
-        let last_scanned_key = last_scanned_entry_name(list_result.entries.as_ref());
-
-        let get_objects = ObjectInfo::from_meta_cache_entries_sorted_infos(
-            &list_result.entries.unwrap_or_default(),
-            bucket,
-            prefix,
-            delimiter.clone(),
-        )
-        .await;
-
-        let (objects, prefixes, is_truncated, next_marker, next_version_idmarker) = list_objects_paginate(
-            get_objects,
-            &delimiter,
-            max_keys,
-            disk_has_more,
-            next_cache_id.as_deref(),
-            false,
-            last_scanned_key.as_deref(),
-        );
-        let _ = next_version_idmarker;
-
-        let result = ListObjectsInfo {
-            is_truncated,
-            next_marker,
-            objects,
-            prefixes,
+        } else {
+            None
         };
-        if should_purge_empty_directory_listing(prefix, opts.marker.as_deref(), max_keys, incl_deleted, &result)
+
+        let result = if let Some(result) = key_only_result {
+            result
+        } else {
+            let mut list_result = self
+                .clone()
+                .list_path(&opts)
+                .await
+                .unwrap_or_else(|err| MetaCacheEntriesSortedResult {
+                    err: Some(to_filemeta_err(err)),
+                    ..Default::default()
+                });
+            let next_cache_id = list_result.entries.as_ref().and_then(|entries| entries.list_id.clone());
+
+            // err=None means gather_results filled its limit → disk has more data
+            let disk_has_more = list_result.err.is_none();
+
+            if let Some(err) = list_result.err.take()
+                && err != rustfs_filemeta::Error::Unexpected
+            {
+                return Err(to_object_err(err.into(), vec![bucket, prefix]));
+            }
+
+            if let Some(result) = list_result.entries.as_mut() {
+                result.forward_past(opts.marker.clone());
+            }
+
+            // Last RAW scanned key, captured before folding, so `list_objects_paginate`
+            // can advance past a fully-collapsed common-prefix page (ECA-03 / #944).
+            let last_scanned_key = last_scanned_entry_name(list_result.entries.as_ref());
+
+            let get_objects = ObjectInfo::from_meta_cache_entries_sorted_infos(
+                &list_result.entries.unwrap_or_default(),
+                bucket,
+                prefix,
+                delimiter.clone(),
+            )
+            .await;
+
+            let (objects, prefixes, is_truncated, next_marker, next_version_idmarker) = list_objects_paginate(
+                get_objects,
+                &delimiter,
+                max_keys,
+                disk_has_more,
+                next_cache_id.as_deref(),
+                false,
+                last_scanned_key.as_deref(),
+            );
+            let _ = next_version_idmarker;
+
+            ListObjectsInfo {
+                is_truncated,
+                next_marker,
+                objects,
+                prefixes,
+            }
+        };
+
+        let purge_exact_prefix =
+            should_purge_empty_directory_listing(prefix, opts.marker.as_deref(), max_keys, incl_deleted, &result);
+        let purge_empty_bucket = should_purge_empty_recursive_bucket_listing(
+            prefix,
+            delimiter.as_deref(),
+            opts.marker.as_deref(),
+            max_keys,
+            incl_deleted,
+            &result,
+        );
+        if (purge_exact_prefix || purge_empty_bucket)
             && has_authoritative_never_versioned_state_in(&self.ctx, bucket)
                 .await
                 .unwrap_or(false)
         {
-            self.purge_orphan_dir_object(bucket, prefix).await;
+            if purge_exact_prefix {
+                self.purge_orphan_dir_object(bucket, prefix).await;
+            } else {
+                self.purge_orphan_dir_objects_in_bucket(bucket).await;
+            }
         }
         Ok(result)
     }
@@ -9507,6 +9534,38 @@ mod test {
         assert!(!should_purge_empty_directory_listing("ghost/", None, 1, false, &truncated));
     }
 
+    #[test]
+    fn recursive_bucket_orphan_purge_requires_an_empty_complete_root_scan() {
+        let empty = ListObjectsInfo::default();
+        assert!(super::should_purge_empty_recursive_bucket_listing("", None, None, 1, false, &empty));
+        assert!(!super::should_purge_empty_recursive_bucket_listing(
+            "ghost/", None, None, 1, false, &empty
+        ));
+        assert!(!super::should_purge_empty_recursive_bucket_listing("", Some("/"), None, 1, false, &empty));
+        assert!(!super::should_purge_empty_recursive_bucket_listing(
+            "",
+            None,
+            Some("marker"),
+            1,
+            false,
+            &empty
+        ));
+        assert!(!super::should_purge_empty_recursive_bucket_listing("", None, None, 0, false, &empty));
+        assert!(!super::should_purge_empty_recursive_bucket_listing("", None, None, 1, true, &empty));
+
+        let live = ListObjectsInfo {
+            objects: vec![ObjectInfo::default()],
+            ..Default::default()
+        };
+        assert!(!super::should_purge_empty_recursive_bucket_listing("", None, None, 1, false, &live));
+
+        let incomplete = ListObjectsInfo {
+            is_truncated: true,
+            ..Default::default()
+        };
+        assert!(!super::should_purge_empty_recursive_bucket_listing("", None, None, 1, false, &incomplete));
+    }
+
     #[tokio::test]
     async fn empty_recursive_listing_purges_committed_delete_residue() {
         use crate::bucket::metadata_sys::{init_bucket_metadata_sys, test_support::isolated_store_over_temp_disks};
@@ -9558,6 +9617,72 @@ mod test {
         }
     }
 
+    #[tokio::test]
+    async fn empty_recursive_bucket_listing_purges_orphan_directory_prefixes() {
+        use crate::bucket::metadata_sys::{init_bucket_metadata_sys, test_support::isolated_store_over_temp_disks};
+        use crate::storage_api_contracts::bucket::{BucketOperations as _, MakeBucketOptions};
+
+        let (dirs, store) = isolated_store_over_temp_disks().await;
+        let bucket = "recursive-bucket-orphan-purge";
+        init_bucket_metadata_sys(store.clone(), Vec::new()).await;
+        store
+            .make_bucket(bucket, &MakeBucketOptions::default())
+            .await
+            .expect("bucket should be created with authoritative metadata");
+
+        for dir in &dirs {
+            tokio::fs::create_dir_all(dir.path().join(bucket).join("ghost").join("nested").join("leaf"))
+                .await
+                .expect("metadata-less orphan directory tree should be created");
+        }
+
+        let result = store
+            .clone()
+            .list_objects_generic(bucket, "", None, None, 1000, false)
+            .await
+            .expect("recursive bucket listing should succeed");
+
+        assert!(result.objects.is_empty(), "orphan directories are not S3 objects");
+        assert!(result.prefixes.is_empty(), "a delimiter-less listing has no CommonPrefixes");
+        for dir in &dirs {
+            assert!(
+                !dir.path().join(bucket).join("ghost").exists(),
+                "an empty recursive bucket scan should reclaim its metadata-less orphan prefix"
+            );
+        }
+
+        let versioned_bucket = "recursive-bucket-versioned-orphan";
+        store
+            .make_bucket(versioned_bucket, &MakeBucketOptions::default())
+            .await
+            .expect("versioned test bucket should be created");
+        store
+            .update_bucket_metadata_config(
+                versioned_bucket,
+                crate::bucket::metadata::BUCKET_VERSIONING_CONFIG,
+                b"<VersioningConfiguration><Status>Enabled</Status></VersioningConfiguration>".to_vec(),
+            )
+            .await
+            .expect("bucket versioning should be enabled");
+        for dir in &dirs {
+            tokio::fs::create_dir_all(dir.path().join(versioned_bucket).join("ghost").join("nested").join("leaf"))
+                .await
+                .expect("versioned bucket orphan tree should be created");
+        }
+
+        store
+            .clone()
+            .list_objects_generic(versioned_bucket, "", None, None, 1000, false)
+            .await
+            .expect("recursive versioned bucket listing should succeed");
+        for dir in &dirs {
+            assert!(
+                dir.path().join(versioned_bucket).join("ghost").exists(),
+                "root recursive LIST must not purge orphan residue in a versioned bucket"
+            );
+        }
+    }
+
     #[tokio::test]
     async fn list_objects_exact_prefix_paginates_across_storage_layers() {
         use crate::bucket::metadata_sys::{init_bucket_metadata_sys, test_support::isolated_store_over_temp_disks};

crates/ecstore/src/store/mod.rs

@@ -403,7 +403,10 @@ async fn enqueue_transition_after_write(
 ) -> Result<ObjectInfo> {
     match result {
         Ok(oi) => {
-            if should_enqueue_transition_immediately(&oi) {
+            // A retried completion did not publish a new object. Immediate
+            // transition/expiry already ran for the original commit; the scanner
+            // still sees the object if that commit never reached this hook.
+            if !oi.multipart_completion_replayed && should_enqueue_transition_immediately(&oi) {
                 enqueue_transition_immediate(&oi, src.clone()).await;
                 if let Ok(api) = metadata_sys::object_store_in(&store.ctx).await {
                     enqueue_immediate_expiry(api, &oi, src, opts).await;

crates/ecstore/src/store/multipart.rs

@@ -993,10 +993,32 @@ impl ECStore {
                 .await;
         }
 
-        let pool_idx = self.multipart_upload_pool_idx(bucket, object, upload_id, opts).await?;
-        let pool = self.pools[pool_idx].clone();
-        pool.complete_multipart_upload(bucket, object, upload_id, uploaded_parts, opts)
-            .await
+        match self.multipart_upload_pool_idx(bucket, object, upload_id, opts).await {
+            Ok(pool_idx) => {
+                self.pools[pool_idx]
+                    .clone()
+                    .complete_multipart_upload(bucket, object, upload_id, uploaded_parts, opts)
+                    .await
+            }
+            // The staging upload is already gone. The completed object lives on
+            // the pool that published it; other pools answer InvalidUploadID.
+            Err(err) if is_err_invalid_upload_id(&err) => {
+                let mut missing = err;
+                for pool_idx in self.existing_multipart_pool_order().await {
+                    match self.pools[pool_idx]
+                        .clone()
+                        .complete_multipart_upload(bucket, object, upload_id, uploaded_parts.clone(), opts)
+                        .await
+                    {
+                        Ok(info) => return Ok(info),
+                        Err(err) if is_err_invalid_upload_id(&err) => missing = err,
+                        Err(err) => return Err(err),
+                    }
+                }
+                Err(missing)
+            }
+            Err(err) => Err(err),
+        }
     }
 
     #[cfg(all(test, feature = "test-util"))]

crates/ecstore/src/store/object.rs

@@ -4693,6 +4693,14 @@ impl ECStore {
     /// the caller falls back to surfacing the original NotFound.
     pub(super) async fn purge_orphan_dir_object(&self, bucket: &str, object: &str) -> bool {
         let prefix = decode_dir_object(object);
+        for pool in self.pools.iter() {
+            for set in pool.disk_set.iter() {
+                if !set.orphan_purge_has_complete_disk_set().await {
+                    return false;
+                }
+            }
+        }
+
         let mut purged = false;
         for pool in self.pools.iter() {
             for set in pool.disk_set.iter() {
@@ -4713,6 +4721,27 @@ impl ECStore {
         purged
     }
 
+    pub(super) async fn purge_orphan_dir_objects_in_bucket(&self, bucket: &str) -> bool {
+        if is_meta_bucketname(bucket) {
+            return false;
+        }
+        for pool in self.pools.iter() {
+            for set in pool.disk_set.iter() {
+                if !set.orphan_purge_has_complete_disk_set().await {
+                    return false;
+                }
+            }
+        }
+
+        let mut purged = false;
+        for pool in self.pools.iter() {
+            for set in pool.disk_set.iter() {
+                purged |= set.purge_orphan_dir_objects_in_bucket(bucket).await;
+            }
+        }
+        purged
+    }
+
     pub async fn delete_object_with_tier_delete_journal(
         self: &Arc<Self>,
         bucket: &str,

crates/heal-contracts/src/heal_channel.rs

@@ -24,6 +24,12 @@ pub const HEAL_DELETE_DANGLING: bool = true;
 pub const RUSTFS_RESERVED_BUCKET: &str = "rustfs";
 pub const RUSTFS_RESERVED_BUCKET_PATH: &str = "/rustfs";
 
+/// Detail attached to a completed deep heal when a healthy legacy object has
+/// no independent identity commitment. Durable MRF handling uses this exact
+/// reason to pause proofless retries without treating the object as repaired.
+pub const LEGACY_OBJECT_IDENTITY_UNVERIFIED_DETAIL: &str =
+    "Legacy object uses standard repair; independent object identity remains unverified";
+
 #[derive(Clone, Copy, Debug, Serialize, Deserialize)]
 pub enum HealItemType {
     Metadata,
@@ -355,6 +361,9 @@ pub struct HealChannelRequest {
     pub object_prefix: Option<String>,
     /// Object version ID (optional)
     pub object_version_id: Option<String>,
+    /// Bucket incarnation observed by the object write that produced a local
+    /// durable partial-write responsibility.
+    pub expected_bucket_incarnation_id: Option<Uuid>,
     /// Force start heal
     pub force_start: bool,
     /// Priority
@@ -592,6 +601,7 @@ pub fn create_heal_request(
         bucket,
         object_prefix,
         object_version_id: None,
+        expected_bucket_incarnation_id: None,
         force_start,
         priority: priority.unwrap_or_default(),
         pool_index: None,
@@ -624,6 +634,7 @@ pub fn create_heal_request_with_options(
         bucket,
         object_prefix,
         object_version_id: None,
+        expected_bucket_incarnation_id: None,
         force_start,
         priority: priority.unwrap_or_default(),
         pool_index,
@@ -655,6 +666,7 @@ fn create_auto_heal_disk_request(set_disk_id: String, priority: Option<HealChann
         disk: Some(set_disk_id),
         heal_endpoints: Vec::new(),
         object_version_id: None,
+        expected_bucket_incarnation_id: None,
         force_start: false,
         priority: priority.unwrap_or(HealChannelPriority::Low),
         pool_index: None,

crates/heal/Cargo.toml

@@ -99,6 +99,7 @@ crc-fast = { workspace = true }
 sha2 = { workspace = true }
 
 [dev-dependencies]
+tonic = { workspace = true }
 rustfs-filemeta = { workspace = true }
 serde_json = { workspace = true, features = ["raw_value"] }
 rustfs-test-utils = { workspace = true }

crates/heal/src/error.rs

@@ -151,6 +151,9 @@ impl Error {
                     )
                     || is_recoverable_heal_error_message(&err.to_string())
             }
+            // Nonblocking local CAS and replacement leases report lock
+            // contention as WouldBlock; retain the existing task retry budget.
+            Error::Disk(DiskError::Io(error)) if error.kind() == std::io::ErrorKind::WouldBlock => true,
             Error::Disk(err) => {
                 if err.is_dangling_delete_grace() {
                     return true;
@@ -198,6 +201,11 @@ impl Error {
 }
 
 fn is_recoverable_internode_error(error: &std::io::Error) -> bool {
+    // A peer restart can cancel an RPC after a partial repair. Replay it within
+    // the existing heal retry budget; task cancellation remains terminal.
+    if DiskError::io_error_is_rpc_cancelled(error) {
+        return true;
+    }
     let Some(error) = error.get_ref().and_then(|source| source.downcast_ref::<InternodeHttpError>()) else {
         return false;
     };
@@ -259,6 +267,39 @@ mod tests {
     use super::Error;
     use crate::heal::{DiskError, EcstoreError};
 
+    #[test]
+    fn cancelled_rpc_is_recoverable_across_storage_error_conversions() {
+        let status = tonic::Status::cancelled("operation was canceled");
+        let disk = DiskError::from(status.clone());
+        let storage = EcstoreError::from(status.clone());
+        let disk_storage = EcstoreError::from(DiskError::from(status.clone()));
+        for error in [
+            Error::Disk(disk.clone()),
+            Error::Disk(disk),
+            Error::Storage(storage.clone()),
+            Error::Storage(storage),
+            Error::Storage(disk_storage.clone()),
+            Error::Storage(disk_storage),
+            Error::Io(std::io::Error::from(DiskError::from(status))),
+        ] {
+            assert!(error.is_recoverable_heal(), "typed RPC cancellation must be recoverable: {error:?}");
+        }
+        for status in [
+            tonic::Status::permission_denied("operation was canceled"),
+            tonic::Status::unauthenticated("operation was canceled"),
+            tonic::Status::invalid_argument("operation was canceled"),
+            tonic::Status::internal("operation was canceled"),
+        ] {
+            assert!(
+                !Error::Storage(EcstoreError::from(status).clone()).is_recoverable_heal(),
+                "cancellation text alone must not change application error classification"
+            );
+        }
+        assert!(!Error::Io(std::io::Error::other("operation was canceled")).is_recoverable_heal());
+        assert!(!Error::TaskCancelled.is_recoverable_heal());
+        assert!(!Error::TaskTimeout.is_recoverable_heal());
+    }
+
     #[test]
     fn internode_transport_errors_keep_their_recovery_classification() {
         use rustfs_rio::{InternodeHttpErrorKind as Kind, new_test_internode_http_io_error};

crates/heal/src/heal/channel.rs

@@ -766,6 +766,9 @@ impl HealChannelProcessor {
         let mut heal_request = HealRequest::new(heal_type, options, priority);
         heal_request.id = request.id;
         heal_request.source = request.source;
+        if request.source == HealRequestSource::Mrf {
+            heal_request.expected_mrf_bucket_incarnation_id = request.expected_bucket_incarnation_id;
+        }
         heal_request.heal_endpoints = request.heal_endpoints;
         // force_start controls admission/queue semantics only. Do not reinterpret it as
         // destructive heal options: admin clients commonly pass forceStart=true together
@@ -1143,6 +1146,7 @@ mod tests {
             bucket: "test-bucket".to_string(),
             object_prefix: None,
             object_version_id: None,
+            expected_bucket_incarnation_id: None,
             disk: None,
             heal_endpoints: Vec::new(),
             priority: HealChannelPriority::Normal,
@@ -1176,6 +1180,7 @@ mod tests {
             bucket: String::new(),
             object_prefix: None,
             object_version_id: None,
+            expected_bucket_incarnation_id: None,
             disk: None,
             heal_endpoints: Vec::new(),
             priority: HealChannelPriority::High,
@@ -1209,6 +1214,7 @@ mod tests {
             bucket: "test-bucket".to_string(),
             object_prefix: Some("test-object".to_string()),
             object_version_id: None,
+            expected_bucket_incarnation_id: None,
             disk: None,
             heal_endpoints: Vec::new(),
             priority: HealChannelPriority::High,
@@ -1236,6 +1242,27 @@ mod tests {
         assert!(heal_request.options.no_lock);
     }
 
+    #[tokio::test]
+    async fn test_convert_mrf_fallback_preserves_source_bucket_incarnation() {
+        let incarnation = uuid::Uuid::new_v4();
+        let processor = HealChannelProcessor::new(create_test_heal_manager());
+        let request = HealChannelRequest {
+            id: "mrf-fallback".to_string(),
+            bucket: "test-bucket".to_string(),
+            object_prefix: Some("test-object".to_string()),
+            expected_bucket_incarnation_id: Some(incarnation),
+            source: HealRequestSource::Mrf,
+            ..Default::default()
+        };
+
+        let heal_request = processor
+            .convert_to_heal_request(request)
+            .expect("MRF fallback request converts");
+
+        assert_eq!(heal_request.source, HealRequestSource::Mrf);
+        assert_eq!(heal_request.expected_mrf_bucket_incarnation_id, Some(incarnation));
+    }
+
     #[tokio::test]
     async fn test_convert_to_heal_request_admin_cannot_bypass_object_lock() {
         let heal_manager = create_test_heal_manager();
@@ -1263,6 +1290,7 @@ mod tests {
             bucket: "test-bucket".to_string(),
             object_prefix: Some("test-object".to_string()),
             object_version_id: None,
+            expected_bucket_incarnation_id: None,
             disk: None,
             heal_endpoints: Vec::new(),
             priority: HealChannelPriority::Low,
@@ -1301,6 +1329,7 @@ mod tests {
                 bucket: "test-bucket".to_string(),
                 object_prefix: Some("test-object".to_string()),
                 object_version_id: None,
+                expected_bucket_incarnation_id: None,
                 disk: None,
                 heal_endpoints: Vec::new(),
                 priority: HealChannelPriority::Normal,
@@ -1341,6 +1370,7 @@ mod tests {
                 bucket: "test-bucket".to_string(),
                 object_prefix: Some("test-object".to_string()),
                 object_version_id: None,
+                expected_bucket_incarnation_id: None,
                 disk: None,
                 heal_endpoints: Vec::new(),
                 priority: HealChannelPriority::Normal,
@@ -1374,6 +1404,7 @@ mod tests {
             bucket: "test-bucket".to_string(),
             object_prefix: Some("logs/".to_string()),
             object_version_id: None,
+            expected_bucket_incarnation_id: None,
             disk: None,
             heal_endpoints: Vec::new(),
             priority: HealChannelPriority::High,
@@ -1410,6 +1441,7 @@ mod tests {
             bucket: "test-bucket".to_string(),
             object_prefix: None,
             object_version_id: None,
+            expected_bucket_incarnation_id: None,
             disk: Some("pool_0_set_1".to_string()),
             heal_endpoints: vec!["http://node0:9000/drive1".to_string()],
             priority: HealChannelPriority::Critical,
@@ -1445,6 +1477,7 @@ mod tests {
             bucket: "test-bucket".to_string(),
             object_prefix: None,
             object_version_id: None,
+            expected_bucket_incarnation_id: None,
             disk: Some("invalid-disk-id".to_string()),
             heal_endpoints: Vec::new(),
             priority: HealChannelPriority::Normal,
@@ -1484,6 +1517,7 @@ mod tests {
                 bucket: "test-bucket".to_string(),
                 object_prefix: None,
                 object_version_id: None,
+                expected_bucket_incarnation_id: None,
                 disk: None,
                 heal_endpoints: Vec::new(),
                 priority: channel_priority,
@@ -1516,6 +1550,7 @@ mod tests {
             bucket: "test-bucket".to_string(),
             object_prefix: None,
             object_version_id: None,
+            expected_bucket_incarnation_id: None,
             disk: None,
             heal_endpoints: Vec::new(),
             priority: HealChannelPriority::Normal,
@@ -1550,6 +1585,7 @@ mod tests {
             bucket: "test-bucket".to_string(),
             object_prefix: Some("".to_string()), // Empty prefix should be treated as bucket heal
             object_version_id: None,
+            expected_bucket_incarnation_id: None,
             disk: None,
             heal_endpoints: Vec::new(),
             priority: HealChannelPriority::Normal,
@@ -1588,6 +1624,7 @@ mod tests {
             bucket: "bucket".to_string(),
             object_prefix: Some("object".to_string()),
             object_version_id: None,
+            expected_bucket_incarnation_id: None,
             disk: None,
             heal_endpoints: Vec::new(),
             priority: HealChannelPriority::Low,
@@ -2006,6 +2043,7 @@ mod tests {
             bucket: "bucket".to_string(),
             object_prefix: None,
             object_version_id: None,
+            expected_bucket_incarnation_id: None,
             disk: Some("invalid".to_string()),
             heal_endpoints: Vec::new(),
             priority: HealChannelPriority::Normal,

crates/heal/src/heal/erasure_healer.rs

@@ -2254,6 +2254,8 @@ mod resume_loop_tests {
         /// A transient infrastructure condition (offline disk / unmet quorum):
         /// the version must be recorded as skipped and retried on a later pass.
         Transient,
+        RpcCancelled(u32),
+        Cancelled,
         Timeout,
     }
 
@@ -2456,14 +2458,31 @@ mod resume_loop_tests {
                 .unwrap()
                 .push((object.to_string(), version_id.map(str::to_string)));
             let key = compose_key(object, version_id);
-            let outcome = self.outcomes.lock().unwrap().get(&key).cloned().unwrap_or(HealOutcome::Ok);
+            let outcome = {
+                let mut outcomes = self.outcomes.lock().unwrap();
+                let outcome = outcomes.get(&key).cloned().unwrap_or(HealOutcome::Ok);
+                if let HealOutcome::RpcCancelled(remaining) = &outcome {
+                    outcomes.insert(key.clone(), HealOutcome::RpcCancelled(remaining.saturating_sub(1)));
+                }
+                outcome
+            };
             match outcome {
-                HealOutcome::Ok => Ok((self.results.lock().unwrap().get(&key).cloned().unwrap_or_default(), None)),
+                HealOutcome::Ok | HealOutcome::RpcCancelled(0) => {
+                    Ok((self.results.lock().unwrap().get(&key).cloned().unwrap_or_default(), None))
+                }
                 HealOutcome::FileNotFound => Ok((HealResultItem::default(), Some(Error::Storage(EcstoreError::FileNotFound)))),
                 HealOutcome::VersionNotFound => {
                     Ok((HealResultItem::default(), Some(Error::Storage(EcstoreError::FileVersionNotFound))))
                 }
                 HealOutcome::Transient => Ok((HealResultItem::default(), Some(Error::Storage(EcstoreError::DiskNotFound)))),
+                HealOutcome::RpcCancelled(_) => {
+                    // Pool aggregation clones the selected error before returning it to heal.
+                    let error = EcstoreError::from(tonic::Status::cancelled("injected peer cancellation"));
+                    let cloned = error.clone();
+                    assert_eq!(cloned, error, "pool error clone must preserve its kind and message");
+                    Err(Error::Storage(cloned))
+                }
+                HealOutcome::Cancelled => Err(Error::TaskCancelled),
                 HealOutcome::Timeout => Err(Error::TaskTimeout),
             }
         }

crates/heal/src/heal/erasure_healer/resume_loop_tests/admin_outcome.rs

@@ -13,8 +13,8 @@
 // limitations under the License.
 
 use super::*;
-use crate::heal::outcome::{HealObjectDisposition, HealTaskOutcome};
-use crate::heal::task::{HealOptions, HealPriority, HealRequest, HealTask, HealType};
+use crate::heal::outcome::{HealDeferredReason, HealObjectDisposition, HealObjectIdentity, HealObjectKind, HealTaskOutcome};
+use crate::heal::task::{HealOptions, HealPriority, HealRequest, HealTask, HealType, MAX_BUCKET_OBJECT_HEAL_RETRIES};
 use rustfs_heal_contracts::heal_channel::HealScanMode;
 
 fn request() -> HealRequest {
@@ -61,6 +61,144 @@ fn assert_partition(outcome: &HealTaskOutcome) {
     assert!(c.unknown <= c.skipped);
 }
 
+fn admin_object_identity() -> HealObjectIdentity {
+    HealObjectIdentity {
+        kind: HealObjectKind::Object,
+        bucket: "a".to_owned(),
+        object: "object".to_owned(),
+        version_id: Some("null".to_owned()),
+        bucket_incarnation_id: Some(uuid::Uuid::from_u128(42)),
+        pool_index: None,
+        set_index: None,
+    }
+}
+
+#[tokio::test(start_paused = true)]
+async fn admin_object_rpc_cancelled_retries_require_verified_receipt() {
+    for failures in [1, MAX_BUCKET_OBJECT_HEAL_RETRIES] {
+        for verified in [true, false] {
+            let storage = FakeStorage::default();
+            storage.set_outcome("object", Some("null"), HealOutcome::RpcCancelled(failures));
+            storage
+                .receipts
+                .lock()
+                .expect("receipt fixture")
+                .insert(compose_key("object", Some("null")), verified.then_some(HealObjectDisposition::Repaired));
+            let options = HealOpts::default();
+            let cancel = CancellationToken::new();
+            let identity = admin_object_identity();
+            let heal = super::super::admin::heal_object(&storage, &options, identity.clone(), "rpc-retry", &cancel);
+            tokio::pin!(heal);
+            for attempt in 1..=failures {
+                assert!(matches!(futures::poll!(heal.as_mut()), std::task::Poll::Pending));
+                assert_eq!(storage.calls().len(), usize::try_from(attempt).expect("attempt fits"));
+                tokio::time::advance(HealTask::bucket_object_retry_delay("rpc-retry", attempt)).await;
+            }
+            let ((_, result), record) = heal.await;
+            assert!(matches!(result, Ok(true)));
+            assert_eq!(storage.calls().len(), usize::try_from(failures + 1).expect("attempts fit"));
+            let (outcome, attempt_failures) = record.expect("completed attempt records its disposition");
+            assert_eq!(outcome.identity, identity);
+            assert_eq!(attempt_failures, failures);
+            assert_eq!(
+                outcome.disposition,
+                if verified {
+                    HealObjectDisposition::Repaired
+                } else {
+                    HealObjectDisposition::Unknown
+                },
+                "only a matching positive receipt proves repair after retries"
+            );
+            assert!(outcome.detail.is_none());
+        }
+    }
+}
+
+#[tokio::test(start_paused = true)]
+async fn admin_object_rpc_cancelled_exhausts_bounded_retries_as_deferred() {
+    let storage = FakeStorage::default();
+    storage.set_outcome("object", Some("null"), HealOutcome::RpcCancelled(MAX_BUCKET_OBJECT_HEAL_RETRIES + 1));
+    let options = HealOpts::default();
+    let cancel = CancellationToken::new();
+    let identity = admin_object_identity();
+    let heal = super::super::admin::heal_object(&storage, &options, identity.clone(), "rpc-exhausted", &cancel);
+    tokio::pin!(heal);
+    for attempt in 1..=MAX_BUCKET_OBJECT_HEAL_RETRIES {
+        assert!(matches!(futures::poll!(heal.as_mut()), std::task::Poll::Pending));
+        assert_eq!(storage.calls().len(), usize::try_from(attempt).expect("attempt fits"));
+        tokio::time::advance(HealTask::bucket_object_retry_delay("rpc-exhausted", attempt)).await;
+    }
+    let ((_, result), record) = heal.await;
+    assert!(matches!(result, Err(Error::TransientSkip { .. })));
+    assert_eq!(
+        storage.calls().len(),
+        usize::try_from(MAX_BUCKET_OBJECT_HEAL_RETRIES + 1).expect("attempts fit")
+    );
+    let (outcome, attempt_failures) = record.expect("exhausted retry records a deferred outcome");
+    assert_eq!(outcome.identity, identity);
+    assert_eq!(attempt_failures, MAX_BUCKET_OBJECT_HEAL_RETRIES + 1);
+    assert_eq!(
+        outcome.disposition,
+        HealObjectDisposition::Deferred {
+            reason: HealDeferredReason::TransientExistenceCheck,
+            retry_not_before: None,
+        }
+    );
+    assert!(outcome.detail.is_some());
+}
+
+#[tokio::test(start_paused = true)]
+async fn admin_object_rpc_cancelled_backoff_honors_task_cancellation() {
+    let storage = FakeStorage::default();
+    storage.set_outcome("object", Some("null"), HealOutcome::RpcCancelled(MAX_BUCKET_OBJECT_HEAL_RETRIES + 1));
+    let options = HealOpts::default();
+    let cancel = CancellationToken::new();
+    let started = tokio::time::Instant::now();
+    let heal = super::super::admin::heal_object(&storage, &options, admin_object_identity(), "rpc-cancel", &cancel);
+    tokio::pin!(heal);
+    assert!(matches!(futures::poll!(heal.as_mut()), std::task::Poll::Pending));
+    assert_eq!(storage.calls().len(), 1, "the first peer cancellation has reached retry backoff");
+    cancel.cancel();
+    let ((_, result), record) = heal.await;
+    assert!(matches!(result, Err(Error::TaskCancelled)));
+    assert!(record.is_none(), "task cancellation must not publish an object disposition");
+    assert_eq!(storage.calls().len(), 1, "cancellation must prevent another storage call");
+    assert_eq!(tokio::time::Instant::now(), started, "cancellation must not wait for backoff");
+}
+
+#[tokio::test(start_paused = true)]
+async fn admin_object_task_cancellation_and_timeout_remain_terminal() {
+    for timeout in [false, true] {
+        let storage = FakeStorage::default();
+        storage.set_outcome(
+            "object",
+            Some("null"),
+            if timeout {
+                HealOutcome::Timeout
+            } else {
+                HealOutcome::Cancelled
+            },
+        );
+        let started = tokio::time::Instant::now();
+        let ((_, result), record) = super::super::admin::heal_object(
+            &storage,
+            &HealOpts::default(),
+            admin_object_identity(),
+            "task-terminal",
+            &CancellationToken::new(),
+        )
+        .await;
+        if timeout {
+            assert!(matches!(result, Err(Error::TaskTimeout)));
+        } else {
+            assert!(matches!(result, Err(Error::TaskCancelled)));
+        }
+        assert!(record.is_none());
+        assert_eq!(storage.calls().len(), 1, "terminal task errors must not retry");
+        assert_eq!(tokio::time::Instant::now(), started);
+    }
+}
+
 #[tokio::test]
 async fn admin_erasure_two_buckets_have_five_exact_receipts() {
     let (_temp, storage) = fixture().await;

crates/heal/src/heal/manager.rs

@@ -174,6 +174,7 @@ pub(super) struct MrfRepairNoticeTarget {
     pub(super) scope: Option<rustfs_common::mrf_channel::MrfScope>,
     pub(super) delete_marker_purge: Option<rustfs_common::mrf_channel::MrfDeleteMarkerPurgeIdentity>,
     pub(super) lease: Option<rustfs_common::mrf_channel::MrfIngressLease>,
+    pub(super) durable_anchor: Option<rustfs_common::mrf_channel::MrfDurableRepairAnchor>,
 }
 
 #[derive(Debug, Clone)]
@@ -562,6 +563,7 @@ fn active_heal_for_dedup_key(active_heals: &HashMap<String, Arc<HealTask>>, key:
 fn request_matches_task(request: &HealRequest, task: &HealTask) -> bool {
     request.heal_type == task.heal_type
         && request.bucket_incarnation_id == task.bucket_incarnation_id
+        && request.expected_mrf_bucket_incarnation_id == task.expected_mrf_bucket_incarnation_id
         && request.options == task.options
         && request.priority == task.priority
         && request.source == task.source
@@ -572,6 +574,7 @@ fn request_matches_task(request: &HealRequest, task: &HealTask) -> bool {
 fn request_matches_request(request: &HealRequest, existing: &HealRequest) -> bool {
     request.heal_type == existing.heal_type
         && request.bucket_incarnation_id == existing.bucket_incarnation_id
+        && request.expected_mrf_bucket_incarnation_id == existing.expected_mrf_bucket_incarnation_id
         && request.options == existing.options
         && request.priority == existing.priority
         && request.source == existing.source
@@ -1808,6 +1811,7 @@ impl HealManager {
                 scope: None,
                 delete_marker_purge: None,
                 lease: None,
+                durable_anchor: None,
             },
         )
         .await

crates/heal/src/heal/manager/scheduler.rs

@@ -438,6 +438,7 @@ impl HealManager {
                             release_mrf_repair_notice_targets(&notice_targets);
                         }
                         publish_verified_mrf_repair_events(&notice_targets, &completed_status_for_verified_events);
+                        publish_unverified_legacy_mrf_events(&notice_targets, &completed_status_for_verified_events);
                         // update statistics
                         let mut stats = statistics_clone.write().await;
                         match completed_status {
@@ -880,6 +881,66 @@ pub(super) fn publish_verified_mrf_repair_events(targets: &[MrfRepairNoticeTarge
     }
 }
 
+pub(super) fn unverified_legacy_mrf_event_for_target(
+    target: &MrfRepairNoticeTarget,
+    outcome: &crate::heal::outcome::HealObjectOutcome,
+) -> Option<rustfs_common::mrf_channel::MrfUnverifiedLegacyEvent> {
+    use crate::heal::outcome::{HealObjectDisposition, HealObjectKind};
+    use rustfs_common::mrf_channel::MrfKind;
+
+    let anchor = target.durable_anchor.as_ref()?;
+    if target.kind != MrfKind::PartialWrite
+        || outcome.disposition != HealObjectDisposition::Unknown
+        || outcome.detail.as_deref() != Some(rustfs_heal_contracts::heal_channel::LEGACY_OBJECT_IDENTITY_UNVERIFIED_DETAIL)
+    {
+        return None;
+    }
+    let version_id = target.version_id.filter(|bytes| *bytes != [0; 16]);
+    let expected_version = version_id.map(|bytes| uuid::Uuid::from_bytes(bytes).to_string());
+    let expected_pool = target.scope.and_then(|scope| usize::try_from(scope.pool_index).ok());
+    let expected_set = target.scope.and_then(|scope| usize::try_from(scope.set_index).ok());
+    if anchor.kind != target.kind
+        || anchor.bucket != target.bucket
+        || anchor.object != target.object
+        || anchor.version_id != version_id
+        || anchor.scope != target.scope
+        || Some(anchor.lease) != target.lease
+        || outcome.identity.kind != HealObjectKind::Object
+        || outcome.identity.bucket != target.bucket.as_ref()
+        || outcome.identity.object != target.object.as_ref()
+        || outcome.identity.version_id != expected_version
+        || outcome.identity.pool_index != expected_pool
+        || outcome.identity.set_index != expected_set
+        || outcome.identity.bucket_incarnation_id != Some(anchor.bucket_incarnation_id)
+    {
+        return None;
+    }
+    Some(rustfs_common::mrf_channel::MrfUnverifiedLegacyEvent { anchor: anchor.clone() })
+}
+
+pub(super) fn publish_unverified_legacy_mrf_events(targets: &[MrfRepairNoticeTarget], completed: &CompletedHealStatus) {
+    use crate::heal::outcome::HealExecutionOutcome;
+
+    if !matches!(completed.status, HealTaskStatus::Completed | HealTaskStatus::Failed { .. }) {
+        return;
+    }
+    let Some(outcome) = completed.outcome.as_ref() else {
+        return;
+    };
+    if outcome.execution != HealExecutionOutcome::Completed {
+        return;
+    }
+    for target in targets {
+        if let Some(event) = outcome
+            .objects
+            .iter()
+            .find_map(|object| unverified_legacy_mrf_event_for_target(target, object))
+        {
+            rustfs_common::mrf_channel::note_mrf_unverified_legacy(event);
+        }
+    }
+}
+
 pub(super) fn heal_request_set_key_for_task(task: &HealTask) -> Option<String> {
     match &task.heal_type {
         HealType::ErasureSet { set_disk_id, .. } => Some(set_disk_id.clone()),

crates/heal/src/heal/manager/tests.rs

@@ -1228,6 +1228,7 @@ fn mrf_verified_repair_event_requires_positive_exact_identity() {
         }),
         delete_marker_purge: None,
         lease: None,
+        durable_anchor: None,
     };
     let matching = HealObjectOutcome {
         identity: HealObjectIdentity {
@@ -1335,6 +1336,102 @@ fn mrf_verified_repair_event_requires_positive_exact_identity() {
     }
 }
 
+#[test]
+fn unverified_legacy_notice_requires_the_exact_partial_write_target() {
+    use crate::heal::outcome::{HealObjectIdentity, HealObjectKind, HealObjectOutcome};
+    use rustfs_common::mrf_channel::{
+        MrfDurableRepairAnchor, MrfIngressResult, MrfIntent, MrfKind, MrfScope, try_rearm_mrf_replay_intent,
+    };
+
+    let bucket = Arc::<str>::from("legacy-held-bucket");
+    let object = Arc::<str>::from("legacy-held-object");
+    let scope = MrfScope {
+        pool_index: 1,
+        set_index: 2,
+    };
+    let mut intent = MrfIntent {
+        bucket: bucket.clone(),
+        object: object.clone(),
+        version_id: None,
+        kind: MrfKind::PartialWrite,
+        delete_marker_purge: None,
+        scope: Some(scope),
+        lease: None,
+        enqueued_at_ms: 1,
+        attempts: 0,
+    };
+    assert_eq!(try_rearm_mrf_replay_intent(&mut intent), MrfIngressResult::Enqueued);
+    let lease = intent.lease.expect("replayed intent should own a generation");
+    let anchor = MrfDurableRepairAnchor {
+        kind: MrfKind::PartialWrite,
+        bucket: bucket.clone(),
+        object: object.clone(),
+        version_id: None,
+        scope: Some(scope),
+        delete_marker_purge: None,
+        lease,
+        bucket_incarnation_id: uuid::Uuid::new_v4(),
+    };
+    let target = MrfRepairNoticeTarget {
+        bucket: bucket.clone(),
+        object: object.clone(),
+        version_id: None,
+        kind: MrfKind::PartialWrite,
+        scope: Some(scope),
+        delete_marker_purge: None,
+        lease: Some(lease),
+        durable_anchor: Some(anchor.clone()),
+    };
+    let incarnation = anchor.bucket_incarnation_id;
+    let outcome = HealObjectOutcome {
+        identity: HealObjectIdentity {
+            kind: HealObjectKind::Object,
+            bucket: bucket.to_string(),
+            object: object.to_string(),
+            version_id: None,
+            bucket_incarnation_id: Some(incarnation),
+            pool_index: Some(1),
+            set_index: Some(2),
+        },
+        disposition: HealObjectDisposition::Unknown,
+        detail: Some(rustfs_heal_contracts::heal_channel::LEGACY_OBJECT_IDENTITY_UNVERIFIED_DETAIL.to_string()),
+    };
+
+    let event = super::scheduler::unverified_legacy_mrf_event_for_target(&target, &outcome)
+        .expect("exact legacy partial-write outcome should park only its runtime retry");
+    assert_eq!(event.anchor, anchor);
+
+    let wrong_identity = HealObjectOutcome {
+        identity: HealObjectIdentity {
+            object: "other-object".to_string(),
+            ..outcome.identity.clone()
+        },
+        ..outcome.clone()
+    };
+    assert!(super::scheduler::unverified_legacy_mrf_event_for_target(&target, &wrong_identity).is_none());
+    let missing_incarnation = HealObjectOutcome {
+        identity: HealObjectIdentity {
+            bucket_incarnation_id: None,
+            ..outcome.identity.clone()
+        },
+        ..outcome.clone()
+    };
+    assert!(super::scheduler::unverified_legacy_mrf_event_for_target(&target, &missing_incarnation).is_none());
+    let wrong_incarnation = HealObjectOutcome {
+        identity: HealObjectIdentity {
+            bucket_incarnation_id: Some(uuid::Uuid::new_v4()),
+            ..outcome.identity.clone()
+        },
+        ..outcome.clone()
+    };
+    assert!(super::scheduler::unverified_legacy_mrf_event_for_target(&target, &wrong_incarnation).is_none());
+    let wrong_reason = HealObjectOutcome {
+        detail: Some("object was readable".to_string()),
+        ..outcome
+    };
+    assert!(super::scheduler::unverified_legacy_mrf_event_for_target(&target, &wrong_reason).is_none());
+}
+
 #[test]
 fn completed_mrf_notice_publishes_only_verified_positive_events() {
     use crate::heal::outcome::{HealObjectIdentity, HealObjectKind, HealObjectOutcome, HealTaskOutcome};
@@ -1355,6 +1452,7 @@ fn completed_mrf_notice_publishes_only_verified_positive_events() {
         }),
         delete_marker_purge: None,
         lease: None,
+        durable_anchor: None,
     };
     let mismatch_target = MrfRepairNoticeTarget {
         object: Arc::from("object-b"),
@@ -2471,6 +2569,70 @@ fn test_retry_request_for_recoverable_lock_timeout() {
     assert!(retry_error.contains("Lock acquisition timeout"));
 }
 
+#[cfg(unix)]
+#[tokio::test]
+async fn contended_healing_marker_cas_retains_bounded_task_retries() {
+    let temp = TempDir::new().expect("marker contention directory");
+    let disk = make_manager_resume_disk(&temp, "marker-contention").await;
+    let metadata = temp.path().join("marker-contention").join(super::super::RUSTFS_META_BUCKET);
+    let lock = std::fs::OpenOptions::new()
+        .create(true)
+        .truncate(false)
+        .read(true)
+        .write(true)
+        .open(metadata.join(".rustfs-cas.lock"))
+        .expect("marker CAS lock should open");
+    lock.lock().expect("fixture should own the marker CAS lock");
+    let result = super::super::apply_healing_markers_to_targets(vec![disk.clone()], Some("owner"), None, false).await;
+    assert!(
+        matches!(&result, Err(Error::Disk(DiskError::Io(error))) if error.kind() == std::io::ErrorKind::WouldBlock),
+        "the real contended marker CAS must return WouldBlock: {result:?}"
+    );
+    assert!(
+        !metadata.join(super::super::HEALING_MARKER_PATH).exists(),
+        "lock contention must not publish a healing marker"
+    );
+
+    let mut request = HealRequest::new(
+        HealType::ErasureSet {
+            buckets: vec!["bucket".to_string()],
+            set_disk_id: "pool_0_set_0".to_string(),
+        },
+        HealOptions {
+            timeout: Some(Duration::from_secs(60)),
+            ..HealOptions::default()
+        },
+        HealPriority::Low,
+    );
+    request.source = HealRequestSource::AutoHeal;
+    let original = request.clone();
+    let storage: Arc<dyn HealStorageAPI> = Arc::new(MockStorage);
+    for attempt in 1..=MAX_RECOVERABLE_HEAL_RETRIES {
+        let task = HealTask::from_request(request, storage.clone());
+        let (retry, delay, _) = retry_request_for_result_with_budget(&task, &result)
+            .await
+            .expect("marker CAS contention should retain the existing task retry budget");
+        assert_eq!(retry.id, original.id);
+        assert_eq!(retry.heal_type, original.heal_type);
+        assert_eq!(retry.source, original.source);
+        assert_eq!(retry.options.timeout, original.options.timeout);
+        assert_eq!(retry.retry_attempts, attempt);
+        assert!(delay > Duration::ZERO);
+        request = retry;
+    }
+    let exhausted = HealTask::from_request(request, storage);
+    assert!(retry_request_for_result_with_budget(&exhausted, &result).await.is_none());
+
+    drop(lock);
+    super::super::apply_healing_markers_to_targets(vec![disk], Some("owner"), None, false)
+        .await
+        .expect("the marker CAS should succeed once contention ends");
+    assert_eq!(
+        std::fs::read(metadata.join(super::super::HEALING_MARKER_PATH)).expect("published marker should be readable"),
+        b"owner"
+    );
+}
+
 #[tokio::test]
 async fn retry_request_for_result_preserves_remaining_timeout_budget() {
     let storage: Arc<dyn HealStorageAPI> = Arc::new(MockStorage);

crates/heal/src/heal/mrf_queue/snapshot.rs

@@ -365,6 +365,19 @@ pub async fn publish_committed_snapshot(
     sequence: u64,
     payload: &[u8],
     limit: usize,
+) -> Result<SnapshotPublication, SnapshotError> {
+    publish_committed_snapshot_with_companion(disks, owner, sequence, payload, limit, None).await
+}
+
+/// Publish a companion lifecycle record on the same disk before the commit
+/// manifest. A replica is committed only after both payloads have been stored.
+pub async fn publish_committed_snapshot_with_companion(
+    disks: &[EcstoreDiskStore],
+    owner: Uuid,
+    sequence: u64,
+    payload: &[u8],
+    limit: usize,
+    companion: Option<(&[&str; 2], &[u8], usize)>,
 ) -> Result<SnapshotPublication, SnapshotError> {
     if disks.is_empty() {
         return Err(SnapshotError::NoWritableReplica);
@@ -413,6 +426,18 @@ pub async fn publish_committed_snapshot(
                 continue;
             }
         }
+        if let Some((companion_paths, companion_bytes, companion_limit)) = companion {
+            match cas_replace(disk, companion_paths[slot], companion_bytes, companion_limit).await {
+                Ok(EcstoreConditionalFileUpdate::Updated) => {}
+                Ok(EcstoreConditionalFileUpdate::Missing | EcstoreConditionalFileUpdate::Mismatch) => continue,
+                Err(error) => {
+                    if first_error.is_none() {
+                        first_error = Some(error);
+                    }
+                    continue;
+                }
+            }
+        }
         match cas_replace_expected(disk, MANIFEST_PATHS[slot], expected_manifest.map(EcstoreDiskBytes::from), &manifest).await {
             Ok(EcstoreConditionalFileUpdate::Updated) => manifest_replicas += 1,
             Ok(EcstoreConditionalFileUpdate::Missing | EcstoreConditionalFileUpdate::Mismatch) => {}
@@ -436,6 +461,80 @@ pub async fn publish_committed_snapshot(
     })
 }
 
+/// Read a companion only from replicas that also contain its matching
+/// committed checkpoint. Differing companions for the same checkpoint fail
+/// closed rather than selecting one replica arbitrarily.
+pub async fn read_committed_companion(
+    disks: &[EcstoreDiskStore],
+    owner: Uuid,
+    sequence: u64,
+    companion_paths: &[&str; 2],
+    checkpoint_limit: usize,
+    companion_limit: usize,
+) -> Result<Option<Vec<u8>>, SnapshotError> {
+    let mut selected: Option<Vec<u8>> = None;
+    let mut first_error = None;
+    for disk in disks {
+        let mut checkpoint_slot = None;
+        for (slot, (manifest_path, payload_path)) in MANIFEST_PATHS.into_iter().zip(PAYLOAD_PATHS).enumerate() {
+            let manifest_bytes = match read_bounded(disk, manifest_path, MANIFEST_LEN).await {
+                Ok(Some(bytes)) => bytes,
+                Ok(None) => continue,
+                Err(error @ SnapshotError::Unsupported) => return Err(error),
+                Err(error) => {
+                    if first_error.is_none() {
+                        first_error = Some(error);
+                    }
+                    continue;
+                }
+            };
+            let manifest = match Manifest::decode(&manifest_bytes, checkpoint_limit) {
+                Ok(manifest) if manifest.owner == owner && manifest.sequence == sequence => manifest,
+                Ok(_) | Err(SnapshotError::Corrupt) | Err(SnapshotError::TooLarge) => continue,
+                Err(error) => return Err(error),
+            };
+            let payload = match read_bounded(disk, payload_path, manifest.payload_len).await {
+                Ok(Some(payload)) => payload,
+                Ok(None) => continue,
+                Err(error) => {
+                    if first_error.is_none() {
+                        first_error = Some(error);
+                    }
+                    continue;
+                }
+            };
+            if CommittedSnapshot::decode(slot, &manifest_bytes, payload, checkpoint_limit).is_ok() {
+                checkpoint_slot = Some(slot);
+                break;
+            }
+        }
+        let Some(checkpoint_slot) = checkpoint_slot else {
+            continue;
+        };
+        let companion = match read_bounded(disk, companion_paths[checkpoint_slot], companion_limit).await {
+            Ok(Some(companion)) => companion,
+            Ok(None) => continue,
+            Err(error) => {
+                if first_error.is_none() {
+                    first_error = Some(error);
+                }
+                continue;
+            }
+        };
+        if selected.as_ref().is_some_and(|existing| existing != &companion) {
+            return Err(SnapshotError::Conflict);
+        }
+        selected = Some(companion);
+    }
+    if selected.is_some() {
+        Ok(selected)
+    } else if let Some(error) = first_error {
+        Err(error)
+    } else {
+        Ok(None)
+    }
+}
+
 /// Reclaim the slot superseded by an already committed checkpoint.
 ///
 /// This is a narrow cleanup primitive: it first reads back the current
@@ -842,6 +941,81 @@ mod tests {
         }
     }
 
+    #[tokio::test]
+    async fn lifecycle_companion_is_read_only_with_its_matching_checkpoint_replica() {
+        let root = TempDir::new().expect("test directory");
+        let checkpoint_disk = disk(&root, "checkpoint").await;
+        let sidecar_disk = disk(&root, "sidecar").await;
+        let owner = Uuid::new_v4();
+        let checkpoint_payload = payload("responsibility");
+        let sidecar = b"lifecycle state";
+        commit(&checkpoint_disk, 0, owner, 7, &checkpoint_payload).await;
+        install(&sidecar_disk, ".heal-mrf-lifecycle.0.bin", sidecar).await;
+
+        let absent = read_committed_companion(
+            &[checkpoint_disk.clone(), sidecar_disk.clone()],
+            owner,
+            7,
+            &[".heal-mrf-lifecycle.0.bin", ".heal-mrf-lifecycle.1.bin"],
+            4096,
+            4096,
+        )
+        .await
+        .expect("unpaired lifecycle sidecar is ignored");
+        assert!(absent.is_none());
+
+        publish_committed_snapshot_with_companion(
+            &[checkpoint_disk.clone(), sidecar_disk.clone()],
+            owner,
+            8,
+            &checkpoint_payload,
+            4096,
+            Some((&[".heal-mrf-lifecycle.0.bin", ".heal-mrf-lifecycle.1.bin"], sidecar, 4096)),
+        )
+        .await
+        .expect("paired checkpoint and lifecycle state publish");
+        let paired = read_committed_companion(
+            &[checkpoint_disk, sidecar_disk],
+            owner,
+            8,
+            &[".heal-mrf-lifecycle.0.bin", ".heal-mrf-lifecycle.1.bin"],
+            4096,
+            4096,
+        )
+        .await
+        .expect("read paired lifecycle sidecar")
+        .expect("paired sidecar exists");
+        assert_eq!(paired, sidecar);
+    }
+
+    #[tokio::test]
+    async fn lifecycle_companion_recovers_from_a_healthy_replica_after_peer_read_error() {
+        let root = TempDir::new().expect("test directory");
+        let failing_disk = disk(&root, "failing").await;
+        let healthy_disk = disk(&root, "healthy").await;
+        let owner = Uuid::new_v4();
+        let checkpoint_payload = payload("responsibility");
+        let sidecar = b"operator audit and source incarnation";
+        commit(&healthy_disk, 0, owner, 9, &checkpoint_payload).await;
+        install(&healthy_disk, ".heal-mrf-lifecycle.0.bin", sidecar).await;
+        std::fs::create_dir(root.path().join("failing").join(RUSTFS_META_BUCKET).join(MANIFEST_PATHS[0]))
+            .expect("simulate one replica read error");
+
+        let recovered = read_committed_companion(
+            &[failing_disk, healthy_disk],
+            owner,
+            9,
+            &[".heal-mrf-lifecycle.0.bin", ".heal-mrf-lifecycle.1.bin"],
+            4096,
+            4096,
+        )
+        .await
+        .expect("one unreadable replica must not hide a healthy paired sidecar")
+        .expect("healthy paired sidecar must be recovered");
+
+        assert_eq!(recovered, sidecar);
+    }
+
     #[tokio::test]
     async fn divergent_commits_at_same_sequence_fail_closed() {
         let root = TempDir::new().expect("test directory");

crates/heal/src/heal/task.rs

@@ -323,6 +323,10 @@ pub struct HealRequest {
     pub heal_type: HealType,
     /// Admission identity for an explicit administrator bucket heal. Never rebound on replay.
     pub bucket_incarnation_id: Option<Uuid>,
+    /// Source bucket generation captured for a durable MRF object repair.
+    /// Unlike the admin admission fence above, this must survive queueing and
+    /// retries so execution cannot target a later bucket incarnation.
+    pub expected_mrf_bucket_incarnation_id: Option<Uuid>,
     /// Heal options
     pub options: HealOptions,
     /// Priority
@@ -351,6 +355,7 @@ impl HealRequest {
             id: Uuid::new_v4().to_string(),
             heal_type,
             bucket_incarnation_id: None,
+            expected_mrf_bucket_incarnation_id: None,
             options,
             priority,
             source: HealRequestSource::Internal,
@@ -417,6 +422,7 @@ pub struct HealTask {
     /// Heal type
     pub heal_type: HealType,
     pub bucket_incarnation_id: Option<Uuid>,
+    pub expected_mrf_bucket_incarnation_id: Option<Uuid>,
     /// Heal options
     pub options: HealOptions,
     /// Priority inherited from the request
@@ -494,6 +500,7 @@ impl HealTask {
             id: request.id,
             heal_type: request.heal_type,
             bucket_incarnation_id: request.bucket_incarnation_id,
+            expected_mrf_bucket_incarnation_id: request.expected_mrf_bucket_incarnation_id,
             options: request.options,
             priority: request.priority,
             source: request.source,
@@ -530,6 +537,7 @@ impl HealTask {
             id: self.id.clone(),
             heal_type: self.heal_type.clone(),
             bucket_incarnation_id: self.bucket_incarnation_id,
+            expected_mrf_bucket_incarnation_id: self.expected_mrf_bucket_incarnation_id,
             options: self.options.clone(),
             priority: self.priority,
             source: self.source,

crates/heal/src/heal/task/heal_object.rs

@@ -93,6 +93,12 @@ impl HealTask {
             "Heal object stage entered"
         );
         self.check_control_flags().await?;
+        if self.source == HealRequestSource::Mrf {
+            // Durable partial writes always use the incarnation-fenced storage
+            // path, including for present objects. A normal heal can inspect a
+            // recreated bucket after the replay anchor was captured.
+            return self.heal_mrf_partial_write_object(bucket, object, version_id).await;
+        }
         let mut object_exists = match self.await_with_control(self.storage.object_exists(bucket, object)).await {
             Ok(exists) => exists,
             Err(err @ Error::TransientSkip { .. }) => {
@@ -429,10 +435,6 @@ impl HealTask {
 
     /// Recreate missing object (for EC decode scenarios)
     async fn recreate_missing_object(&self, bucket: &str, object: &str, version_id: Option<&str>) -> Result<()> {
-        if self.source == HealRequestSource::Mrf {
-            return self.recreate_missing_mrf_object(bucket, object, version_id).await;
-        }
-
         debug!(
             target: "rustfs::heal::task",
             event = EVENT_HEAL_OBJECT_STAGE,
@@ -534,8 +536,8 @@ impl HealTask {
         }
     }
 
-    /// Durable MRF responsibilities may complete only with an exact storage proof.
-    async fn recreate_missing_mrf_object(&self, bucket: &str, object: &str, version_id: Option<&str>) -> Result<()> {
+    /// Durable MRF partial writes may complete only with an incarnation-fenced storage proof.
+    async fn heal_mrf_partial_write_object(&self, bucket: &str, object: &str, version_id: Option<&str>) -> Result<()> {
         let heal_opts = HealOpts {
             recursive: false,
             dry_run: self.options.dry_run,
@@ -549,12 +551,22 @@ impl HealTask {
             set: self.options.set_index,
         };
         let mut expected = self.outcome_identity(bucket, object, version_id, self.options.pool_index, self.options.set_index);
-        let bucket_incarnation_id = self
+        let current_bucket_incarnation_id = self
             .outcome_bucket_incarnation_id(bucket, self.options.dry_run)
             .await?
             .ok_or_else(|| Error::TaskExecutionFailed {
                 message: format!("Missing bucket incarnation for durable MRF repair {bucket}/{object}"),
             })?;
+        let bucket_incarnation_id = self
+            .expected_mrf_bucket_incarnation_id
+            .ok_or_else(|| Error::TaskExecutionFailed {
+                message: format!("Missing source bucket incarnation for durable MRF repair {bucket}/{object}"),
+            })?;
+        if current_bucket_incarnation_id != bucket_incarnation_id {
+            return Err(Error::TaskExecutionFailed {
+                message: format!("Bucket incarnation changed before durable MRF repair {bucket}/{object}"),
+            });
+        }
         expected.bucket_incarnation_id = Some(bucket_incarnation_id);
 
         let storage_result = self
@@ -577,7 +589,30 @@ impl HealTask {
             storage_result.receipt.as_ref().map(|receipt| &receipt.disposition),
             Some(HealObjectDisposition::AuthoritativelyAbsent)
         );
-        if !self.record_verified_storage_receipt(expected, storage_result.receipt).await {
+        let receipt_missing = storage_result.receipt.is_none();
+        if !self
+            .record_verified_storage_receipt(expected.clone(), storage_result.receipt)
+            .await
+        {
+            let ok_drive_state = DriveState::Ok.to_string();
+            let healthy_legacy_without_repair = receipt_missing
+                && storage_result.item.detail == rustfs_heal_contracts::heal_channel::LEGACY_OBJECT_IDENTITY_UNVERIFIED_DETAIL
+                && storage_result.item.drives_healed() == Some(0)
+                && !storage_result.item.after.drives.is_empty()
+                && storage_result
+                    .item
+                    .after
+                    .drives
+                    .iter()
+                    .all(|drive| drive.state == ok_drive_state);
+            if healthy_legacy_without_repair {
+                self.outcome.write().await.record(HealObjectOutcome {
+                    identity: expected,
+                    disposition: HealObjectDisposition::Unknown,
+                    detail: Some(storage_result.item.detail.clone()),
+                });
+            }
+            self.record_result_item(storage_result.item).await;
             return Err(Error::TaskExecutionFailed {
                 message: format!("Missing exact storage proof for durable MRF repair {bucket}/{object}"),
             });

crates/heal/src/heal/task/tests.rs

@@ -2679,6 +2679,25 @@ async fn read_repair_object_heal_sets_read_repair_option() {
 }
 
 #[tokio::test]
+async fn durable_mrf_heal_rejects_a_recreated_bucket_before_storage_heal() {
+    let original_incarnation = Uuid::new_v4();
+    let recreated_incarnation = Uuid::new_v4();
+    let storage = Arc::new(MockStorage {
+        bucket_incarnation_id: Mutex::new(Some(recreated_incarnation)),
+        ..Default::default()
+    });
+    let mut request = HealRequest::object("bucket".to_string(), "object".to_string(), None);
+    request.source = HealRequestSource::Mrf;
+    request.expected_mrf_bucket_incarnation_id = Some(original_incarnation);
+    let task = HealTask::from_request(request, storage.clone());
+
+    let result = task.heal_object("bucket", "object", None).await;
+
+    assert!(matches!(result, Err(Error::TaskExecutionFailed { .. })));
+    assert!(storage.heal_object_calls.lock().unwrap().is_empty());
+}
+
+#[tokio::test(start_paused = true)]
 async fn read_repair_object_heal_is_not_failed_by_flat_task_timeout() {
     let storage = Arc::new(MockStorage {
         block_heal_object: Mutex::new(true),
@@ -2687,31 +2706,23 @@ async fn read_repair_object_heal_is_not_failed_by_flat_task_timeout() {
     let mut request = HealRequest::object("bucket".to_string(), "object".to_string(), None);
     request.source = HealRequestSource::ReadRepair;
     request.options.timeout = Some(Duration::from_millis(1));
-    let task = Arc::new(HealTask::from_request(request, storage.clone()));
-    let execution = tokio::spawn({
-        let task = task.clone();
-        async move { task.execute().await }
-    });
+    let task = HealTask::from_request(request, storage.clone());
+    // Isolate the storage boundary from preflight's separately enforced budget.
+    let execution = task.heal_object("bucket", "object", None);
+    tokio::pin!(execution);
+    assert!(futures::poll!(&mut execution).is_pending());
+    assert!(storage.object_heal_opts.lock().expect("heal options")[0].read_repair);
 
-    tokio::time::timeout(Duration::from_secs(1), async {
-        loop {
-            if !storage.object_heal_opts.lock().unwrap().is_empty() {
-                break;
-            }
-            tokio::task::yield_now().await;
-        }
-    })
-    .await
-    .expect("read-repair object heal should start");
-    tokio::time::sleep(Duration::from_millis(20)).await;
-    assert!(!execution.is_finished(), "read repair must not be failed by the flat task timeout");
-
-    execution.abort();
-    assert!(execution.await.is_err(), "aborted mock execution should not join successfully");
-    assert!(storage.object_heal_opts.lock().unwrap()[0].read_repair);
+    *task.task_start_instant.write().await = Some(Instant::now() - Duration::from_millis(2));
+    assert!(matches!(task.remaining_timeout().await, Err(Error::TaskTimeout)));
+    tokio::time::advance(Duration::from_millis(20)).await;
+    assert!(
+        futures::poll!(&mut execution).is_pending(),
+        "read repair must remain in storage after the flat task budget expires"
+    );
 }
 
-#[tokio::test]
+#[tokio::test(start_paused = true)]
 async fn non_read_repair_object_heal_still_uses_flat_timeout() {
     let storage = Arc::new(MockStorage {
         block_heal_object: Mutex::new(true),
@@ -2719,13 +2730,26 @@ async fn non_read_repair_object_heal_still_uses_flat_timeout() {
     });
     let mut request = HealRequest::object("bucket".to_string(), "object".to_string(), None);
     request.options.timeout = Some(Duration::from_millis(1));
-    let task = HealTask::from_request(request, storage);
+    let task = HealTask::from_request(request, storage.clone());
+    let execution = task.heal_object("bucket", "object", None);
+    tokio::pin!(execution);
+    assert!(futures::poll!(&mut execution).is_pending());
+    assert!(!storage.object_heal_opts.lock().expect("heal options")[0].read_repair);
+    tokio::time::advance(Duration::from_millis(20)).await;
+    assert!(matches!(futures::poll!(&mut execution), std::task::Poll::Ready(Err(Error::TaskTimeout))));
+}
 
-    let result = tokio::time::timeout(Duration::from_secs(1), task.execute())
-        .await
-        .expect("flat timeout should finish the task");
+#[tokio::test]
+async fn read_repair_preflight_still_rejects_an_expired_task_budget() {
+    let storage = Arc::new(MockStorage::default());
+    let mut request = HealRequest::object("bucket".to_string(), "object".to_string(), None);
+    request.source = HealRequestSource::ReadRepair;
+    request.options.timeout = Some(Duration::from_millis(1));
+    let task = HealTask::from_request(request, storage.clone());
+    *task.task_start_instant.write().await = Some(Instant::now() - Duration::from_millis(2));
 
-    assert!(matches!(result, Err(Error::TaskTimeout)));
+    assert!(matches!(task.heal_object("bucket", "object", None).await, Err(Error::TaskTimeout)));
+    assert!(storage.object_heal_opts.lock().expect("heal options").is_empty());
 }
 
 async fn make_resume_disk(temp: &TempDir) -> DiskStore {
@@ -3997,6 +4021,7 @@ async fn mrf_recreate_missing_object_records_exact_absence_receipt_with_scope()
         HealPriority::Normal,
     );
     request.source = HealRequestSource::Mrf;
+    request.expected_mrf_bucket_incarnation_id = Some(incarnation);
     let task = HealTask::from_request(request, storage.clone());
 
     task.execute()

crates/heal/tests/heal_concurrent_delete_test.rs

@@ -513,14 +513,24 @@ async fn delete_during_root(case: DeleteCase) {
 
 fn run_delete_case(case: DeleteCase) {
     // Incarnation-bound repairs use a spawned owner; retain the server's stack
-    // budget while exercising the real EC2+2 encode/decode futures in debug.
-    let runtime = tokio::runtime::Builder::new_multi_thread()
-        .worker_threads(4)
-        .thread_stack_size(8 * 1024 * 1024)
-        .enable_all()
-        .build()
-        .expect("concurrent delete runtime");
-    runtime.block_on(delete_during_root(case));
+    // budget for the whole scenario, including its foreground DELETE. block_on
+    // alone would poll that future on libtest's smaller calling-thread stack.
+    let stack_size = 8 * 1024 * 1024;
+    std::thread::Builder::new()
+        .name("heal-concurrent-delete".to_owned())
+        .stack_size(stack_size)
+        .spawn(move || {
+            let runtime = tokio::runtime::Builder::new_multi_thread()
+                .worker_threads(4)
+                .thread_stack_size(stack_size)
+                .enable_all()
+                .build()
+                .expect("concurrent delete runtime");
+            runtime.block_on(delete_during_root(case));
+        })
+        .expect("concurrent delete scenario thread starts")
+        .join()
+        .expect("concurrent delete scenario must not panic");
 }
 
 #[test]